ToutVirtual Enterprise Virtualization - A Proper Virtual World

This Article Will Be A Bit Different?

Virtualization Critical Evaluation - Chapter 01

This article will be a bit different.  How so?  Well, that is both easy to explain and hard to illustrate.  The year 2008 is effectively just past half over.  This is significant, because thoughts about the near future of virtualization come to mind, and how I will communicate to my clients what makes sense and what does not make sense over the next 18 months on paper?  Preparing them for the next 36 months?  Really strategic scope would be 60 months!  There will be long hours with spent with AIX (Dynamic LPARs), Xen (be it RHEL integrated or Citrix aligned), Solaris Zones, even Parallels, and the potential 800 pound gorilla, Hyper-V of Windows 2008 heritage with Microsoft SCVMM as well.  Why?  Because ESX 4.0 and its lean relative ESXi based on 4.0 core must be a winner.  Unfortunately for VMware, as Rome, there are many Huns at the gates of Rome.  I do hold out hope that VMware will have another winner, or I should say a better winner than ever before.  The history of ESX major versions for the most part has been one of best of breed, but the competition is prepared as never before.

VMware, with a few missteps, has achieved a notable hat-trick, 2.5.x, 3.x, and 3.5.x.  I do not quality 2.1.2, not because it was lacking significance, but it was just on the introduction of shared-storage as a serious feature or infrastructure component to virtualization, and was from my perspective the precursor to the serious modern hypervisor trend VMware established with 2.5.x and VirtualCenter 1.x.  This is subjective of course, but VMotion was the single most intriguing and significant feature that VMware has ever implemented, and it came into its own with ESX 2.5x.  As for those missteps?  Well, VirtualCenter comes to mind again, as does VCB, in fact, scaling seems to be a concept that VMware has and continues to struggle with. Including more recent solutions like Update Manager, and even in the core VMware API, which still seems slow for some reason at any significant scale, as I have noted in the past in verbose detail.  Unfortunately, this is where Microsoft can nail VMware to the wall.  The very same wall that has the writing on it, that says…Abandon All Hope, All Ye That Read This Here…if you see complete doom for VMware ahead?

In spite of what others may predict or even believe about my views, I do not yet, see failure for VMware.  VMware has potential, but so does every other virtualization platform.  As other solutions move or prepare to move into application instancing, VMware still holds firm that its future is ESXi and a pure hypervisor-ized vision.  That is dangerous.  Hypervisors will always exist, but will never dominate the industry over time.  VMware keeps its virtual-appliance model, only because it is a fake instance model?  That is a dead end, when any true application instancing solution out performs it hands down, this is where Citrix may be going?  There are just too many strategic reasons to reduce the complexity of virtualization, as only application instancing allows.  How do I know this to be true?  Why the emergence application instancing?  My wish list for 2009 and 2010 are not dependent on hypervisor technology, is why.  The wish list, in simple descriptors is as follows:

• Reduce the Total Number of Lines of Code Executed!  This is #1
• Avoid any solution that layers filter drivers upon filter drivers.  This is hard to do!
• Easy intra- and inter-site recovery, De-Duplication Models for Archival
• Thin Disking and Imaging, Greater alignment to Storage Arrays
• Reduced Support Staff (Yes, This Is Reality)
• Realign instancing back to its roots, Exchange, SQL, etc., all are instancing models that should never be in operating system isolation
• Utility Computing (Heterogeneous Application Hosting), Application Instancing
• Grid Computing?  To some degree a solution looking for a problem?
• Kill dependence on VMFS, Explore other file systems, NFS, iSCSI (Again)
• Become Even More Green, this is related to #1 (Energy Consumption Must Be Reduced)

How many of these goals is VMware presenting against?  Since LifeCycle solutions are nothing more than creative repackaging, not that many as I see it.  None of these goals or concepts is in any way new.  But achievement of these simple goals still is not consistent throughout the information technology industry or across hardware vendors.  We, all seem to be waiting for something, watching, believing that just around the corner, over the next slight hill; the ultimate solutions, in each scope is there just out of sight, the supreme utopia for those of us that are architects of infrastructure.  Why are we still chasing solutions to problems, not implementing?  I for one, can not continue to wait, VMware VI based on revision 4 must be a quantum leap again.

Remember I said, it is easy to explain, but hard to illustrate?  Well, we do well to itemize the problems, but do not do was well discovering the solutions?  So, we compromise against the options that are available, saying to ourselves, it is just the 80/20 rule, that I can not have everything, everything does not exist.  Food for thought no?  I want my cake, and will eat it too, and I want zero calories with that real sugar taste!  Well, for me, 2009 starts in the later half of 2008, and 2011 will start in 2009, if you understand my inference?  2015 is just not that far away, will you have the right solution, for 2015?  Global Datacenter will be the expectation for 2015.  Yes, I want everything just the way I want it, and to do that, I will have to implement a multiple vendor and tiered virtualization solution platform which may be, no will be, more complex than I want, but is the only way to achieve the efficiency I need.  The operations teams are not going to be happy?

Do I sound sad?  Or is the impression I present a bit dark?  That is reality, providing VMware ESX 4.x is a winner, it may not be as dark as it could be.  My hope is that VMware will spend major time in code validation and quality assurance certification with ESX 4.0, eliminating new introductions of features in 4.0, for inclusion in 4.0.1, 4.0.2, cough, or even 4.5 Update 1, in preference to solid stability.  I hope that VMware has a true instancing model that is a surprise?  Did I mention this all has to happen at a penance of a price?  Dang, knew I forgot something in my wish list…Adding…Everything Has To Be Done On The Cheap.

Add comment June 26th, 2008 Schorschi

Experimental? What Does That Really Mean?

Virtualization? What The Heck Is That? - Chapter 06

VMware does an interesting thing. It mixes alpha and beta code with general release code. Alpha? Yes, alpha because what VMware calls beta code at times is not what I would call beta code. However, the definitions come later, so bare with me. This inclusion of experimental code, cough beta, cough sometimes alpha code, has been around for years in VMware products, and from my perspective is both good and bad. Good in the sense that we get to see features and methods that will become core to the VMware products, one such feature that comes to mind the quickest is VirtualCenter not integrating support for experimental features that are on VMware ESX today, for example thin-disking use. You can create thin-disks today for virtual instances via the ESX service console CLI using vmkfstools, however, the VirtualCenter/Virtual Infrastructure client does not allow this within the virtual machine (instance) wizard, nor does VC/VI report the delta between what the guest OS understands and what is actually thin-disked allocation, so operational teams can thin-disk allocate themselves into a nasty situation that they can not see coming. Why does this difference exist? In short, thin-disking is experimental.

Experimental code in a general release is bad, because it violates a number of best practices rules. There are firms, corporations, out there that will decline a solution or product if it has any alpha or beta code that can end up in a production context. Even if the alpha/beta code is not leveraged, it very existence disqualifies the complete general release code base. This is fact not fiction. Check with our organization, ask your vendor product management team, or service vendor support team, I think you be surprised that in the small print this issue is well noted, and more to the point, enforced.

Now I am sure some of you are saying, experimental is not beta, well, it is. Per VMware documentation…VMware includes certain ‘experimental features’ in some of our product releases. These features are there for you to test and experiment with. We do not expect these features to be used in a production environment….I call that beta by definition, by intent, and by meaning, and since what VMware calls beta, I at times see as alpha? Well? I did tell you definitions would come later, here they are, but I digress. To be fair, VMware does offer some support for experimental features, but it is limited, and it is really us providing bug report feedback, more than it is VMware resolving issues realtime. If a known work-around exists, VMware publishes it, with the usual warnings and disclaimers.
So, in effect, VMware is releasing unfinished, rather let us say unpolished features, in general releases. How should this issue be addressed? To keep your vendor product management group from declaring VMware products illegal, I have a few suggestions:

• Experimental code must never be included in general release media. It should be on separate media, making it obvious that is an optional, additional component installation; it should not even be downloaded from the same basic web site location, to further isolate its intentional non-production use.
• Every single feature that is experimental should not be installed by default, from with in the general release installer process. Note, I said not installed. This does not mean, not enabled, disabled, or sitting there but not being used, it means the code is never there in physical sense, unless explicitly installed. No one can accidentally enable or use it.
• Never include experimental code in commands or features that can not be explicitly blocked, locked down, or otherwise disabled if it is installed. This does not side step the point above, it augments it. Meaning that if in a lab or test, non-production context, experimental code is installed it can be completely disabled, thus reverting test environment to a product like environment. Without this feature, duplicate environments would have to be maintained.
• Enable or disable of experimental features is done by global flag as well as feature specific flags. A master XML or configuration command. For example, esxcfg-exp -enable or -disable kills everything in one shot. Where as esxcfg-exp -enable thindisk, just enables thin-disking.

Open source code already scares the pants off of corporate vendor management groups. Weak certification methods by vendors, for example not testing on virtualization platforms, and continued rubber stamping of solutions from vendor to vendor, because strategic alliances are worthless, which results in garbage-in gospel-out scenarios freak everyone out, so why would VMware create this issue in the first place? The reason is straight-forward, VMware never wants to be seen as standing still, they are convinced that if they do not add more and more features, faster and faster that they will lose the race with Microsoft or Citrix or whomever is in the shadows of virtualization just waiting to pounce on the market leader. In some ways it is a surprise to me, why VMware, as strong as VMware is in virtualization, acts like the underdog even today? Winners act like winners, sometimes that is the difference between winning and losing.

I remember at the first VMworld session, in San Diego, what was it VMware 2004? Sitting at a table eating lunch - which was the best food any VMworld has ever had by the way - someone from VMware said, I am omitting the name since I consider the source a friend…“VMware ESX is a strong solution, it basically sells it self.” I remember watching the reaction of all sitting at the table… everyone agreed, as they were stuffing their faces. Maybe it was just the good food? No, the statement is true, today, as it was some four (4) years ago.

We do not need experimental code in general release media, nor in production environments, nor should VMware continue this policy in the future. After all, the general release code is strong…it should sell its-self, right?

Add comment June 5th, 2008 Schorschi

LifeCycle Wars? Gotta Be kidding… More Important Issues Remain To Be Resolved!

Virtualization, Fine, Well Sort of? - Chapter 14

Gartner commented that every thing will change in 2008 in reference to virtualization? Well duhe. Guess the old crystal ball needs new batteries? But to be honest and fail, Gartner was right, in concept, but not in subject. Everything is changing in virtualization, it has every single year for the last five (5) or so years running. Virtualization and the associated tools, methods and platforms have been, in my humble opinion, been the most dynamic aspect of the information technology sector, no? Fine, dual-cores and quad-cores were expected, come-on! Of course, what has changed and how it has changed is also a matter of debate and exhibited by various perspectives echoing across cyberspace. Including my views, of which, having been quite outspoken, well duhe, again, are no secret.

VMware for example, has constantly improved the enterprise hypervisor based platform, cough 3i, cough cough ESXi, product to a significant degree, but this has been completely predictable. Adding more capacity per virtual instance, making shared storage a core requirement a la VMotion if not Storage VMotion, failed to develop enterprise scalable archival options, and taken almost forever to improve its management application suite? Yes, VirtualCenter has gotten new features and add-ins, but VirtualCenter as a framework, well, still is horrible. If you don’t believe me? Try using it at its quoted scaling capability? Better yet, try using at half its quoted scaling recommendation? It does not do well. Even VMware acknowledges this. Not to mention inconsistent 64bit support for the client? Which has been fixed! But only after a lot of customer heated feedback. This is not to say that VirtualCenter has not been improved, it has, but it, like ESXi is still not enterprise class… yet. When it is a true enterprise class solution… Watch out! It could be competition killer, cough, Microsoft you listening? Did some one say (System Center Virtual Machine Manager (SCVMM)? Again, I have commented on this before, so why do I mention it now? Well because of VMware LifeCycle Manager of course! What? Did I lose anyone?

Yes, LifeCycle products are all the new rage, and if you widen the concept slightly, you get an entire range of related products, beyond VMware LifeCycle Manager. BladeLogic Operations Manager, IBM TPM, Opsware, and even VMware Lab Manager is/was a LifeCycle product for a focused niche. But the list goes on, ConfigureSoft Enterprise Configuration Manager, Encore, Vizioncore vCharter, and ToutVirtual has an entire framework devoted to LifeCycle concept for years, and I am sure I am forgetting a few others, but the list is sufficient for the point, all of these solutions support the basic features or concepts inclusive of total life cycle continuation of virtualization containers, in our biased view, virtual instances of course, right? Let’s itemize feature set at a high level a bit? Some key features for any life cycle product:

• Provisioning (Charge-Forward?)
• Retirement
• Configuration Management
• Security Management
• Patching
• Asset Management (Charge-Back?)
• Physical-To-Virtual, Well Ok

Provisioning and retirement are obvious no? The automated creation and destruction of instances makes sense. Once an instance exists it must be maintained, so functional configuration, security remediation, and patch management (fixing bugs, not just closing holes), all fall into place. Now, lets see, that is left? Oh, yes, asset management, which beyond inventory and capacity planning, tracking, and trending, that wonders of wonders, the charge forward and back model! Well, blah, blah, duhe, once more. There are quite of few CPAs I know that really do hate charge models, not because of the validity of the concept is not sound, but because establishing and maintaining such models is a pain, can you see blood? I even know one individual that equated capitalization and deprecated costing hang ups with virtual instances as… and I quote… spreading the peanut-butter. Of course I asked, smooth or chunky… the reply was… and I quote again… I don’t know, super chunky? Quite a few big scale clients of virtualization have even developed their own in-house LifeCycle solutions, why? Because enterprise class solutions took more than two (2) years to get to market in 2004 and/or 2005, and some products were/are incomplete, nothing but band-aiding VMware ESX as a Linux distribution into the same old gadgets, very un-cool. The one exception would physical-to-virtual (P2V) solutions, which have remained tight and focused on modest improvement objectives rather than chasing feature set expansion as the only valued goal.

The sad reality is that with all the rush to deliver LifeCycle solutions, VMware included, if debatable, late, the entire virtualization industry has not addressed some key issues. Many of these have been discussed in this same blog in the past, and quite a few other blogs. Including, but not limited to… we do not, and this true of just about every virtualization toting publisher, have enterprise class management tools from the market leaders, we do not have extensive options for archival, thin-disking, disk-instancing (read-only shared disk images) from many of the major storage venders, and the list goes on, heck, VMware has even removed features over time, rather than really improve them? The entire virtualization scope of the information technology industry is still developing new solutions, and not improving existing solutions. No one should be surprised; Gartner said they should make new products and add tons of new features, in order to survive, cough, Hypervisor Wars. So I guess everyone that develops Hypervisors reads Gartner? Can’t count the number of times I have heard customers say… improve what you have before you add new features, well for the last time, duhe. Seems like common sense, no? Not to sales and marketing gurus? Well, they read Gartner as well, right?

We have more clients, very day, going from 10s of virtual instances, to 100s, or from 100s to 1000s, and to be sure, more than a few going from 1000s to 10,000s in this calendar year, no? And what will they need? Oh lets see… very stable hypervisors, check. A fast and consistent management tool that work at significant scale, ah, not quite. Archival solutions that scale, well de-duplication is an initial step for that, so check, with a question mark. And for the 800 pound guerrilla, utility computing, leveraging application instancing, nope, this one is still not reality, well, except for Solaris Zones, or in the context of grid computing. Now, for crying out loud, is that not what we asked for in 2004, 2005, and 2006? And to think, Gartner told us in 2008, everything is going to change? Well has it? Duhe, oops sorry, said it again. Honestly, don’t see that virtualization has changed much at all; we still have the same basic problems we had years go.

Oh, Gartner, the new iBall (say Eye-Ball) still is not very accurate? But, remember, it has new features! To be fair, the newest iBall is an isolinear integrated, carbon based, globe, it glows in 16 million colors, at infinity minus one resolution! No more heavy lead-crystal from Baccarat? Going to miss that, to be sure. No realistically improved functionality, cough, accuracy? Why am I, not surprised, duhe, ops I said it again. It appears all the leading crystal ball makers read Gartner as well, and they have developed and retired three (3) generations of the popular predictive orbs, supported only for one (1) calendar year which is a shame. Purchase of sooth-saying devices has to be a significant capitalization for organizations like Gartner, right? So where do we ship the batteries, yes, batteries. Reading the datasheet, the latest iBall uses traditional chemical cells? Still? But, I just wonder about one more thing… Like the new smart-cars, does the iBall get even one (1) more additional hour of predictive premonitions, for the same size chemical cells? How environmentally green is the newest iBall anyway?

Add comment May 28th, 2008 Schorschi

Do Your Virtual Machines Eat Pizza?

Know What Virtualization Is, But What Is Next? - Chapter 13

Of course your virtual machines do not each pizza? Well mine do. I know, I am sure, some of you believe that I hang around the cold vaults, sniffing the ozone that comes off transformers, the power supply type, not the robotic type. There a few individuals at VMware, Microsoft, IBM, Dell, HP, etc., who are certain that I have an intellectual challenge due to secret consumption of Halon. In their collective opinions, I have limited brain capacity. It is a fact, I have been told to my face that I am crazy, that my views are embryonic, and that I just don’t know the facts about the state of virtualization in the industry today. All those that have believed me insane, the title of this article may be, again from their collective viewpoints, prove the point, how asinine my perception of virtualization is. Well, so be it. Maybe my professional frustration has, at least, damaged my logical diagnostic aptitude.

But, regardless of reality, perspective, or fact, my virtual machines eat pizza. Or to be explicit, my virtual instances eat resource pizza pie. Of course they do. Disk, Network, Memory IO, and CPU cycles, everyone in virtualization, be it operating system isolation based, application instance based, or something in the middle, we know this, pardon the cliché, these truths to be self-evident, and obvious. The entire rationale for virtualization is better consumption of resources? Over subscription, which I am not a fan of in most cases, if not accurate assumption of resources available. In the context of the title for today? Making sure every slice of pizza is eaten, very bite is enjoyed, that my clients are arguing about the last piece of pizza in the box, no pun indented. Rather than saying… go ahead, you can have the last slice, because if they are saying that? I did not do my homework right, because in virtualization there should never, never, ever, ever, be a virtual instance, cough, client which does not leave the table just a bit hungry. Yes, hungry, everyone has to be just a bit hungry. Why? When that is the case, I have achieved three nines utilization no?

Wait, hold up, and stop, what the heck does leaving the table hungry, pizza, and whatever you are saying have to do with virtualization. Well, it is obvious right? Predictive modeling! What? You did not see that coming? Odd, I would have thought everyone would have? Yes, predictive modeling. If the dirty secret of operating system isolation in virtualization is poor, or even worse, bad code, then the stupidity of virtualization, is the lack of extensive and exhaustive use of predictive modeling. This is not to say that this issue is not understood, it is, at least to some degree by the great minds of technology on some high plane of cognitive insight. Predictive Analytics (PA) is already well established in some areas of endeavor. Everyone of us has gained benefit from PA, every time we surf to a website, and said web site suggests a different but related product? Amazon and Netflix leverage the hell out the concept. The computing industry knows the power of PA, ComputerWorld commented on it in 2006, and CIO commented on the same concept in 2004. So I ask you, why has PA not turned virtualization on its head? We understand how to make pizza, cough, implement virtualization. But we don’t really know how to make virtualization resource utilization better, in a predictive manner? If you know just how many slices of pizza you need to make, you know just how much infrastructure you really need to pre-provision, right? Sorry, I just twisted the metaphoric linear relationship. But I believe you see the point no? True, there have been attempts to address the issue, BMC Software in 2005, and SearchDataCenter scratched the surface in 2007 referencing a few more player in the field of PA application to virtualization. However, I have yet to see PA change virtualization. Why is that? Microsoft System Center Virtual Machine Manager (SCVMM) and VirtualCenter (VC) to not have PA integrated? True, VC has HA and DRS, which on the QT (Quick Tip) makes recommendations on host-to-virtual-instance alignment could be done but that is historical trending based on existing infrastructure, not PA against pre-provisioning. PA should happen at initial P2V candidacy, as a what-if analysis before an entire virtualization infrastructure is evaluated, not just as virtual instances are introduced to established virtual clusters, hosts and/or sites.

So how to we kick start PA use? By way of example, I realize PA use in virtualization is not trivial. So here is a suggestion. Every hardware manufacturer, to receive inclusion to the any virtualization vendors respective HCL must execute and maintain baseline performance data. This is already done, no? Sales teams nee this or have this to some degree? Just needs to be standardized or normalized across all vendors. Every software publisher should do the same, should take the baseline configuration of the systems via VMmark, Sandrasoft, or whatever tool resource analysis tool was used, and run their application in the same context. No, not every variant, but say high, medium, and low scale host platforms as published by the hardware vendors. So every hardware vendor does three basic models and every software publisher does the three (3) largest hardware vendors, say HP, Dell, IBM. This is where the strategic alliances should make things happen, yes? I do not see nine (9) unit test scenarios for each software publisher as difficult or abusive. Share the data on the web, in a standard format that can then be imported and establish a baseline for PA for virtualization frameworks, like VC and SCVMM no? This datastore would grow, and develop like an open source initiative?

Every single PA integrated tool could mine this data to establish better modeling for virtualization! Aligned Strategy believes there is real money to be made in this scope. Does it not seem like common sense? PA needs a traditional hardware trending model to leverage for virtual instance what-if analysis, right? Is not the greenest thing we can do for ourselves; is figure out just how many slices of pizza our virtual instances need before we order the pizza? Dang it… while I have been trying convince everyone to change the world of virtualization… some rotten so and so snagged the last slice of pepperoni! I wish I could have predicated that… well, that is the point of the article, after all.

Add comment May 14th, 2008 Schorschi

Will De-Duplication Save Virtualization?

Know What Virtualization Is, But What Is Next? - Chapter 12

About 2 years ago, or a bit more, I was sitting at a small table in my favorite local Thai restaurant, eating my favorite Thai dish, green curry chicken, delivered hotter than heck, in fact, hot as Godzilla’s breath? But I digress. We were actually discussing troubles with virtualization in general. Several jokes were made about how crazy it was that virtualization was accelerating beyond any of our expectations, as I recall, we all agreed the rate of virtualization was going to get out of hand, to become a monster. Godzilla came to mind again.

Close your eyes, slip back in time, for about 2 years, remember the lack of mature management tools, issues with virtual instance scaling, environment scaling, emergence of dual-cores, and then coming of the quad-cores, relatively slow PCI bus/server backplane, which is still an issue, etc, etc. But one of the key issues, at least based on my recollection, was backup/restore of virtual instances. Regardless of the type of virtualization, regardless of the vendor discussed, backup and restore was expensive, if not, impossible at any significant scale. Lets be honest with ourselves, we all saw this, we all knew the scaling ceiling was coming, we all whined about it to anyone that would listen. Never mind the fact that disaster recovery models based on traditional models, at the time, could never be called efficacious. Ah, you can open your eyes now.

As I said, this was about two years ago, or maybe a bit more, and where we? Well, to be honest, not very far. Wait, wait… Hear me out. I say we are not far along, because the options marketed or published for the last two years have been functional, but not scalable to any reasonable degree. All the previous solutions required extensive infrastructure, based on traditional methods with a strong heritage based on cloning or dumping to tape in total, which was just about effective as tanks and rockets against Godzilla. Solutions that scaled 1-to-2 or even 3-to-1 virtualization hosts to a backup/restore proxy are not cost effective when hundreds of hosts and thousands of virtual instances are at risk. We need to leverage solutions that can leverage 5-to-1 sites, yes sites, avoiding extensive bandwidth consumption for disaster recovery preparation. Why? Because we continue to add sites or additional hosts at a significant rate, and bandwidth, be it SCSI based, fabric based, or even TCP/IP based, is still bandwidth. Given that many enterprise entities do not own their data storage or communication network infrastructure, extra bandwidth, can and does incur significant cost in a site to site context.

De-duplication is the anti-monster super weapon? What is de-duplication? In short, it is avoiding replication of identical data at a very low level integral to the respective storage solution in use. Although not limited to virtualization, de-duplication is wonderful for virtualization, since hypervisor based and even application instance virtualization often has a very high degree of static or common binary data per instance. Why would any one archive identical binary data over and over? But that is what is being done over and over. What a monster! Can we exhume Raymond Burr? Better yet, can we create a virtual Raymond Burr instance that reminds us all, as he did in 1955, that we are to blame for this monster?

Now, I am not going to make this easy for you, I am not going to recommend a specific de-duplication solution, but I will say this, NetApp, and EMC via Avamar, among others, know that de-duplication is a strategic direction, and are going at each other like Godzilla versus Radon, which means that all of us looking at de-duplication for virtualization are now citizens of Tokyo? That is a bit of stretch, true. But if you are not looking at de-duplication options for effective virtual instance backup/restore at significant scale, never mind, disaster recovery, do so. Because no one knows when a 60,000 ton monster is going to stomp, smash, and otherwise pancake flat, one or more datacenters like just a bunch of toy models sitting on green outdoor carpet. Not to mention what all that smoke from the cheesy firework based special effects will do to all the tape media, still being used.

Add comment April 22nd, 2008 Schorschi

Is Software Product Certification on Virtualization Today A Given?

Know What Virtualization Is, But What Is Next? - Chapter 11

By now just about everyone that keeps track of how things are going on in virtualization circles has heard about the Symantec Antivirus and VMware Virtual Machine issue, right? Even I, am not sure I understand all the implications. But I can think of one question that is significant. How the heck did this happen? Even if the issue turns out to be a VMware issue and not a Symantec issue, or visa-versus, or something else, something just does not add up, some where, some how, this issue should have been caught? Let me be clear, I am not saying who is at fault; I am simply asking how this issue happened. I sincerely hope it is not a comedy of errors scenario. Furthermore, since this issue did happen, I expect some real changes coming to how software publishers do quality assurance testing and what their respective clients will demand, yes, demand.

Do all software publishers test on virtualized platforms now? I can not answer this question, but absolutely every software publisher should. Does or has your favorite publishers of software test of virtualization platforms? Do you know the answer to this question? Ok, which ones? If not, do you have the right to expect it? Yes, of course you do. I don’t know what specific publishers do or do not do in reference to quality assurance testing, but I do know that with this recent issue with Symantec Antivirus and VMware, every vendor agreement in every major corporation is being reviewed and rewritten. It is something that corporations using enterprise solutions will demand, no? You bet they will. In fact, I will go an additional step beyond this basic requirement. I believe that corporations will demand testing schedules, explicit test plans, and even develop customized testing models that they will expect, no correction, will demand software publishers explicitly execute to enterprise class expectations, including virtualization platforms.

Now software publishers that refuse to do this, even hardware vendors that refuse to do this for BIOS and firmware updates, will see key enterprise clients walk. Yes, walk. There is no room to debate this, especially for financial, government, and other key industries. I can hear the inflexible vendors crying and moaning from all the way here, in front of my monitor as I am typing this text. Testing costs time and money, this is unreasonable, etc., etc. Bull. I can also see flexible vendors using their improved test models, that now explicitly include virtualization platforms, as part of their over all marketing efforts. Why not? As an enterprise customer, would you not want to know that a given software publisher is actually using the same virtualization platform that you are using today? Talk about common sense in product evaluation and selection. I can see any lack of testing on virtualization nailing a software publisher, in fact, we may even see a software publisher going bankrupt by ignoring leading virtualization platforms as a viable for overall product certification.

Why did this situation happen? Talking in a hypothetical context? Well, at some point someone in the publisher side of the industry decided myopic vision is a cost effective of course. This was justified? Sure, virtualization is so good, and so stable, and so consistent, that we, the trusted software publisher just never needed to worry about it testing on any virtualization platform. Oh, right, sure! How does that work for you? If you look at this issue, from the outside looking in, the lack of testing on at least the most popular virtualization platforms is more than just an honest oversight? Nuts. What publisher is not using virtualization just like the rest of us in the industry to save costs in lab or quality assurance infrastructure? How about the reverse scenario? Software publishers only tests on virtualization, never on hardware? Now that would be insane as well, but given the level of virtualization today, it would not be quite a complete surprise, not to me.

As I said before, I am not sure what happened or how, in reference to recent developments or should I say difficulties that Symantec and VMware encountered. But if this is not a very loud, obvious, and intense wake up call to all software publishers, that the significance of certification testing on virtualization platforms must not be overlooked, then what is? Having one or more enterprise customers smack the next software publisher, which fails to learn from recent history, in the preverbal head? No, not preverbal head, but profit margin., after all lose of revenue is the most painful impact, right?

Add comment March 25th, 2008 Schorschi

Why Do Hypervisor Based Platforms Dominate Virtualization?

Know What Virtualization Is, But What Is Next? - Chapter 10

Why are hypervisor based platforms for virtualization dominating the industry? There are a number of answers that come to mind:

• P2V (Physical To Virtual) Migration Ease
• Avoiding DLL or .NET hell (Yes, .NET Hell)
• Failure of Utility Computing?
• Dominate Operating System Support Gap

Hypervisors are easy for technical personnel to understand at a 1000 foot level. No they are not easy to manage, hard to understand at the 10 foot level, and require extensive skill to architect, design, and integrate as enterprise platforms, never mind support. Of course technical management hates to acknowledge this, but it is true. So, anything that makes some part of virtualization easy is focused on, and in the pursuit of pain avoidance, dominates thinking. This becomes quite obvious when you consider the list of issues that are noted above. Let us explore each a bit more!

Physical-to-Virtual is a given, moving from very old hardware to new virtualized agnostic platforms, is a pain to do unless you have 1GB dedicated pipes and almost painless if you actually have 10GB connectivity. All the variants, P2I (Physical-to-Image), I2V, V2P, I2P, etc., are applicable, no significant changes are done, but the operating system is kept in tact, more importantly the application installations are kept functional. When you have 100s or 1000s of migrations to get done, of course you want to keep is easy and straight-forward. I can not tell you the number of times I have heard business management say, just move my application, I don’t want to re-install or even update the operating system. Most of the time this because they refuse to let the technical teams have the time to do it right, or worse technical management has no clue how to do it consistently, often because they have laid off the knowledgeable staff that really knew the applications in question well.

Avoiding DLL hell, well this does not need much in explanation now does it? But .NET hell does, yes, .NET hell does exist. .NET, as much as Microsoft would like us to believe does almost nothing to fix the basic issues with DLLs. I best .NET hides the say old issues in the details. Worst, it makes things harder to resolve. Microsoft has never resolved the DLL memory loading issues, never changed the memory mapping model for DLLs to established true DLL integration between mismatched frameworks. I could go on, but anyone that has written serious code for the Windows platform understands these points. Microsoft slogan for .NET should be… The .NET Devil is in the DLL Details. Until mismatched DLL issues really disappear, the isolation benefits of hypervisor virtualization will help avoidance of pain.

Failure of Utility Computing? What the heck is this? To put is in simple terms, application instances, application framework components, etc. But it is a bigger issue, when heterogeneous applications are integrated, and way beyond DLL hell scenarios, resource management, process management, communication management, and configuration management are all aspects of utility computing models. Where you have many different applications running together, which may or may not be isolated by time or resource controls is a very difficult nut to crack. Time is often referenced as CPU loading, and resources are often referenced as Memory, Network, and/or Disk IO, now where do these elements come into play? Virtualization of course deals with the same basic issues, hiding such as part of its isolation. True utility computing is the long term future of virtualization, and it makes operating system publishers, like Microsoft freak. This is because the total number of operating system instances running is significantly reduced with true application instancing, and advance resource and time control. The maturing of management tools for utility computing on Windows is horrible, where as Solaris with its Zones implementation it is powerful. Apple Computer had the basic concept in its core Macintosh operating system about 1989. Never mind the fact that quad-cores beg for application instancing and heterogeneous application integration. Why pay for 4 or 8 package quad-cores, and then run 16, 32 or even 48, horrors 64, instances of various operating systems when you don’t have to do so?

Application frameworks, often integrate better controls, Microsoft SQL, IIS, Oracle, etc. and various other similar instancing models developed resource management within themselves, but the operating system was ignored. This is where the dominate operating system gap comes into focus. Any one that has tried to use Windows System Resource Manager knows exactly what I am saying, specifically soft caps, for resource control just do not work. It is quite a shame that Microsoft has not done more with proper application isolation at the operating system level. This is really a maturity issue, of course the methods and tools will improve, but never to the point that they impact operating system sales. Better solutions must come from publishers that do not publish operating systems. The same basic issue comes with storage vendors, in that as they improve thin-disk or imaging methods they actually impact the total number of storage frames they sell, it is just not going to happen the way we all would like to see it. I am sure some will disagree, fine, but until I see Microsoft really create a application virtualization model that does anything beyond the basics, I will keep my current opinion.
I am sure many others will think of additional rationales but the above noted are those most often encountered given my personal experience. Bottom-line, every short-cut coding to market drives, wastes real money in total cost of ownership, whereas coding for market saves total cost of ownership costs. Virtualization based on operating system isolation, which another way of saying hypervisor based virtualization, protects bad coding methods and techniques, shortened testing periods, promoting junk quality assurance efforts, etc., which are core to the entire coding to market model. The guy that invented single instance concept unit testing? Should be whipped with a bamboo cane, it was an excuse that code shop management was just itching for! So much code we have today is poor quality junk, and until this trend for dumping junk into the market is reversed, utility computing will never see the light of virtual day. Hypervisor based virtualization with its inherent operating system and application isolation is the only protection we have.

1 comment March 17th, 2008 Schorschi

Does Virtualization Reduce Staff?

Virtualization, Fine, Well Sort Of? - Chapter 06

In a word, yes. Ok, this article is done, right? Wrong. It may surprise you which staff you must maintain in order to maintain your virtualized infrastructure. However, I hope this discussion does not surprise you! There are a number of approaches to supporting an infrastructure; I am going to suggest one strategy that has tactical elements. This strategy is not cheap, cough, inexpensive. There are a few places where any organization just can not global source completely and expect to get quality results. You must have high quality associates or resources in a few key places:

• Engineering
• Operations
• Problem Management
• Project Management

Engineering is a key area, these resources must understand technology, emerging, architecture, and beyond the vendor, rah rah, roadmaps, they should also come from operations at least in part. Product evaluation by the vendors that publish said products is dumb, although there is a very strong trend to doing this, because it promotes reduction of engineering staff. Never make a decision on beta code or even release candidate code. Never implement general release code if you can avoid it. The trend is to be an early adopter, the concept of late adopting seems to be lost in the dust of the past. Funny, how no one wants to tabulate TCO in the context of late versus early adoption impact costing. The total time allowed to select, evaluate, develop and certify solutions, no entire architectures, continues to shrink, and yet I continue, as one observer, to hear about the bad results of this trend, when engineers are forced to make decisions on release candidate or worse beta code exposure. Some things can not be rushed, something that management never wants to acknowledge. What is the old joke? The manager asked the engineering team, “What is the number one thing I can do to improve the quality of your work?” An engineer speaks up and says, “Give us the time and resources to do the job right.” The manager replies after a short pause, “What did you say? Sorry. We need to rap this up so you and I can both get back to work; I have operations team coming in next to explain why they are having so many issues.”

Operations teams are virtualized in nature, smaller, non-site specific, lots of talent based on fewer brains. People world-wide, recently I called a vendor on a specific issue, and was transfer from the US, to Canada, and then to Costa Rica, yes, I went from low level customer support, to elite team support, to engineering for the vendor. No, this is not outsourcing, or global sourcing to use the newer term. Thus, these new operation teams you have will rely on external resources to be sure. But talent in-house, focused on the house, if you get my implication. No matter how good global sourcing, you must have good internal operational support. Quasi teams, more strike force in nature, interchangeable parts or cogs, experts in chronic issue resolution, root cause analysis, not to mention very good trouble shooters and problem solvers. Focusing on resolving the immediate impact, and then after the issue is resolved, worked-around, etc., figure out the long term solution. The focus is reduction of immediate impact first, not always the reduction of total impact. Real solutions often take real time. Odd, that operations management understands this, but engineering management does not? A solution that takes time to test and implement, opens the door for the problem management team.

Problem management might be the most misunderstood. Problem management is not the job of your operations team. Problem management is not the responsibility of the engineering team; however, many firms see this as the case. Let me explain, you are paying your operations team to support the environment, and the engineering team to create and improve the environment. The project management team to take what engineering team does and roll it into the environment in a way that the operations team can handle. Therefore, problem management team resolves problems by leveraging the vendors, to get issues addressed, and only then validated as viable by the engineering team, and ultimately adopted by the operations team. It makes no sense to have the engineering and operations teams debating with the vendors, developers, etc., until there is something concrete and solid for them to evaluate or support. To be direct, the project management team has to have balls. There is no other way to say it. If your project management team does not have the strength and intelligence to tell management where to go, when deadlines and deliverables are insane, then your engineering team can not get superior design work done. This in turn nails the operations team, since a bad design once implemented is horrible to support. And, your problem management team is buried with lots of rework or issues that never should have left the project evaluation/test effort. Unfortunately, I see this aspect of the total model is often an after thought. Weak project managers are the norm not the exception in many firms, and the results speak for themselves, customers are not happy, management is not happy, and most of all the technology teams are left holding the bag, from nose to tail, engineering gets blamed by the operations team, etc.

Of course, there is some overlap between all the teams at various points in the life cycle of a solution or even the development and recovery cycle of issue resolution, but that should be informal and synergistic in nature, not expected by default. Oh, did anyone notice, only now I have mentioned the word, virtualization, again? I guess the concepts are valid no matter how far they are applied from virtualization, no?

Add comment February 26th, 2008 Schorschi

The Risk of Virtualization

Virtualization, Fine, Well Sort Of? - Chapter 05

Let us consider the following situation…You walk into a conference room, before you are a group of your key clients, lines of businesses, what ever is applicable to your situation, and you ask the following questions:

• What percentage of your technology infrastructure is virtualized?
• Are you happy with the performance of virtualization?
• Do you consider virtualization a higher risk? If so, Why?
• Do you consider virtualization a lower risk? If so, Why?
• What is the true risk of this percentage of virtualization?
• Does the phrase “All Eggs in One Basket” meaning anything to you?
• Is the risk of virtualization worth the threat of virtualization?

What do you think the answers will be? Are you going to be happy with the answers? Are your clients going to be able to answer the questions? If they can not, is it your fault or theirs?

What is the percentage of your technology infrastructure is virtualized? This is a straight-forward question, or is it? Does your firm or infrastructure provider use virtualized storage? Does your solution use hypervisor or application virtualization?  Is your network virtualized? VLANs, QoS, etc., What? What the heck is that? Hypervisor and/or application virtualization, which is safer? Many clients have no real understanding that every information technology resource they contract for is virtualized in some way. It may be that what you think are LANs, are VLANs. It may be that what you think is physical infrastructure is, but not completely, what if you network or storage provider has their entire command-center tool set on virtualized instances and they are suffering from the same issues you are? You just don’t know what is virtualized below what you see as infrastructure.

Are you happy with the performance of virtualization? This is also a straight forward question, maybe the only straight-forward question in this entire discussion, or virtualization in general. From a quantitative perspective, providing you have rather extensive analysis methods, you can provide your clients with hard facts, from P2V, V2I, I2I, I2V, I2P, V2P, etc. But from a qualitative perspective the client is going to believe what they believe. Of course, from the client perspective, you get what you pay for, so if you can live with the performance virtualization provides, versus the cost of not using virtualization, this relatively straight forward. Did or does your process for virtualization candidacy force your project managers and design personnel to do proof-of-concept validation of virtualization? Or do you just trust the application vendors?

What is the true risk of the percentage of virtualization? Higher Risk? Lower Risk? We are not talking labs or non-critical environments but core production, the mission critical, the five (5) nines (9) realm of up time. No, not disaster recovery on virtualization, but true production, customer visible, if offline, you lose real money. Do your clients really understand this? Or did you just, ah, sort of, gloss over this point? This is not an argument for or against virtualization, because depending on your respective analysis, design, and implementation, this true risk could or should be less than traditional hardware, no? Transparent migration, i.e. VMotion for VMware, and/or stand-by hosts, or just available capacity on existing hosts? Not to mention network redundancy methods, 3DNS, Big-IP, etc. Storage redundancy infrastructure or methods? Argues the point that virtualization should have lower risk than traditional hardware. However, if you cut corners, or have non-shared storage, did not make the commitment to redundant network and storage fabric implementations? Then the risk of virtualization is higher than traditional hardware for sure, if you do not believe this, keep smoking what your are smoking. It is rather ironic that many clients do not really understand that they have many eggs in one basket on every virtual host server. The virtualization industry knows this, HA, DRS, SRM from VMware, and similar technologies or methods under development by every major virtualization solution provider, just screams this point no? Why was the biggest AH! At VMworld 2007 transparent host redundancy? Microsoft has learned this lesson a bit late, but to be sure Microsoft will never make that specific mistake again, of course I refer to the lack of a transparent virtual instance migration, or VMotion like feature set.

Taking all these questions and associated issues as potential impacts to your environment, is the risk of virtualization worth the thread of virtualization to your financial success? Returning to our conference room, let us up the stakes, what if it is really not a client conference, a board room meeting? Do you want to be the one that has to defend the use of virtualization when something significant goes wrong? Or worse, defend that you short-changed, or rushed to virtualization implementation? It does give pause to you does it not?

Now consider the concept of risk in virtualization, taking into account that hypervisor based operating systems are rather immature compared to traditional operating systems. Are we all just dodging the bullet? Will this risk get us in the end? Real issues for virtualization are and continue to be those of an immature platform, I will not exhaustively itemize these issues, but just think back to the early days of Windows or Linux, and you will recall the things that gave you pain, no? Rush to market code, inexperienced code development, hardware vendors struggling to understand a software operating system that is a moving target, etc., etc., I recall these and more in every operating system ever developed.

The bottom-line in avoiding risk is before your virtualization plan is invoked, why? Because regardless of how good the virtualization platform or your design around said platform is, it is people that designed it. If you have not given your people the time, resources, and had the patience to allow a superior design work to be done, your virtualization infrastructure is a house-of-cards. Once it is implemented, no matter how well it is supported, if the design is bad, the risk you are taking is bad. So I ask again, do your clients know this?

1 comment February 12th, 2008 Schorschi

Now That The Hype Over VMware 3i Has Finally Died Down!

Know What Virtualization Is, But What Is Next? - Chapter 09

VMware has released ESX 3.5, has released 3i installable, and many of us in the information technology industry are waiting for 3i on SAN disk, embedded USB, etc. But how long will we have to wait? I am not talking about 3i embedded its-self, no. I am talking about how long we will have to wait to be able to use 3i in any effective manner. The vaulted, the earth shattering next quantum step in virtualization, the oh-my-gosh, I can not believe how cool this is…are you sure it was not invented by those guys at Apple Computer, and just licensed to VMware? The great nirvana, no better yet, utopian, Zen state of virtualization, 3i is a complete and unabashed flop. Yes, a complete disappointment at general release.

Support for 3i out of the box, cough out of the solid-state memory, is not here. IBM, can not even get ESX 3.5 support documented right, HP is at least 60 to 90 days away from any significant support of 3i, and Dell, yes, Dell, which talked up VMware 3i 3.5 what? At least 3 or more months ago? Has yet to say much of anything. At VMworld 2007 everyone, yes, everyone in the exhibit booths was all, 3i, 3i, 3i! All we needed was hard-bodies on stage in cheerleader outfits, with the 3i logo stitched in interesting places, to have typical home team game rally. The hype, which was really hype, was so obvious it frankly should have embarrassed VMware then, as it is in true reality now doing. A home team rally has a bond fire right? Can we burn 3i on it as well? Sending 3i to the Halls of Valhalla!

Of course 3i has potential, of course it will take some of the major issues with deployment and configuration consistency of VMware ESX server out of the picture, it will be more secure, leaner, and meaner, and the ease of setup alone is going to be wonderful! At least compared to Anaconda/Kickstart, which is now 20+ years old no? But notice, please, that I said going to be…no one is there yet, in any realistic sense of the word. Monitoring for 3i is a joke, scaling of VirtualCenter is a joke, and now we have a virtualized appliance node, 3i, that depends on a single point of failure for monitoring and configuration, which is VirtualCenter. Talk about the cart before the horse, VirtualCenter enterprise view should have been release before 3i, hello, anyone thinking? Combine this weak infrastructure design issue, with the fact that you can not get any realistic information out of the hardware state of a 3i server, and 3i is, in not uncertain terms, dead on arrival. Enterprise use of 3i, which is the true long term cash cow for VMware, and the key threat that Microsoft is on target to nail, is left hanging in the wind.

Google has its yellow box appliance, so what color should 3i boxes be? How about pink? Since I don’t know what color signifies a colossal miscalculation? Until management, control, monitoring and system administrator level of access, be it scripting, or such, for 3i is as good as, if not better, than ESX console of service model, the enterprise client base will not adopt 3i. If Microsoft does not learn from this mission critical mistake, then Microsoft will never over take VMware by competitive advantage.

Who is on the hook for this release 3i screw up? Well, everyone, yes, everyone, IBM, Dell, and HP, but not ignoring Sun, and a few other smaller hardware platforms, contributed to VMware making this obviously stupid mistake.  We, the clients, are to blame as well, did we yell loud enough? Or more to the point, did we have the opportunity to do so? I am sure a small group of early demonstration observers or beta testers gave input on 3i issues, but did VMware listen? Right now we have nothing but a finger pointing game, hardware vendors are saying VMware API based on CIM is weak, or incomplete. VMware is saying the hardware vendors are moving slower than expected, what the heck is slower than not having a complete solution before anything was released? What good is any of these so called strategic alliances or partnerships between hardware vendors and VMware, if they can not coordinate something as dead-bang straight-forward as 3i as complete solution on initial release? An alliance or partnership is supposed to produce results, where are they, in reference to 3i? It is obvious that 3i design and development coordination between VMware and the hardware vendors was hype as well.

How do we explain to our enterprise customers that 3i is, in any reasonable fashion, at least one (1) year away from deployment? That 3i is nothing but hype, is still nothing but hype. I am sure just about everyone at the enterprise level sees 3i as hype. The final question is whether this silent majority have the guts to kick 3i back into the teeth of VMware and the lame hardware vendors that claimed support for 3i, but have absolutely nothing to show for it. Guess, not, we are all still trying to get closer to the stage, to see the cheerleaders, to get a better look at which interesting places, where the 3i logo is actually stitched. Hey, dude, have a care with that elbow will ya. Watch who you poke with your Blackberry too.

8 comments January 25th, 2008 Schorschi