Archive for September, 2008

VMworld This Week! What Will The Wizards Have For Us This Week?

Virtualization Critical Evaluation – Chapter 08

This week, 09/15/2008, is VMworld 2008! VMworld is always fun, sometimes more hype than fact, other times lots of facts, and minimal hype, you just never know which you will get. This is a good thing, it keeps everyone guessing, at least to some degree. I have been at every VMworld event, in the US, so far, and always enjoyed hearing the discussions in the hallways. For example, VMworld in Los Angeles, every one was talking iSCSI, the very first VMworld event, in San Diego, everyone was talking VMotion and ESX 2.5.0. VMworld 2008 should be no different, some thing will be buzzing. The question is what? ESXi? I do not think so. After the initial flop of ESXi, or ESX 3i, I should say, VMware needs a few winners, to freak out Microsoft, if for no other reason that it is fun to freak out Microsoft.

Microsoft has still missed the target on a number of issues, but they will address these. The question is can VMware exploit, oh, bad pun, these before Microsoft eliminates them? The issues I see with Hyper-V are as follows:

  • No VMotion function, yes, quick migration exists, and that is fine if you don’t need transparent migration, but most server virtualization does need this feature, as transparent as possible. VDI (Virtual Desktop Infrastructure), can survive without instance migration, no?
  • Hyper-V, due to its design, has a different performance model, the VMBus will never quite be as good as a dedicated Hypervisor, such as ESX. Microsoft strategy is, get it as good as we can; which is worth the effort, but close is not better.
  • Security is an issue with Microsoft, more so than VMware, not because there are so many instances of Windows out there, but because a generic operating system can never be as secure as a dedicated appliance or structured solution such as ESXi or ESX full. I don’t believe hyper-jacking is are threat yet, but it will be a threat for Microsoft sooner than VMware ESX, duhe.
  • Is Microsoft worried? Of course they are, why else did they certify VMware ESX 3.5 Update 2, finally? Microsoft is looking like for the 3rd time, in about 5 years, they are a dollar short and day late, more than that, really.

As I am traveling to VMworld 2008 I will be thinking about what my expectations and what my wishes for this VMworld will be. There are things I believe I will be expecting:

  • We need ESXi to be identical to full ESX installation, in reference to monitoring, and alert status reporting. Complete features set, such that ESXi installation with traditional agents must be identical to full ESX. Everyone at an enterprise level is struggling with this one, so ESXi will never grow to its potential until this is resolved.
  • We still need better archival/disaster recovery solutions. VCB is not living up to its potential, I liked the idea of VCB, but it still does not scale. Array based snap solutions like Avamar from EMC or the similar solution from NetApp, are still complex, hard to manage, and just a pain to implement. This is insane, and I believe the key issue for 2009, we have larger and larger VMs, but no realistic way to archival them.
  • We will need USB support in VMs on ESX. Security/License dongles of course. KVM dongles, yes KVMs, since new KVMs continue to add USB features, this is becoming an issue where key sites standardize on a type of KVM, the operational teams want the VMs interface to be identical, emulated in software but same look and feel. Maybe VMware should join forces with Avocent?
  • We still need IDE and SATA, emulation in VMs on ESX. How about SAS for VMs on ESX? Are any of these realistic? Maybe not, but to show clients a one-to-one emulation, is still requested over and over. VMware workstation does it, so ESX should too.
  • VMware Thin-Disk support in GUI for VirtualCenter? Better yet, Disk Imaging as well, were we can use a core OS volume, and VMs actually run delta off the core OS volume? Windows 2008 is going to drive this requirement; a full Windows 2008 install for server, Microsoft recommends 40GB just to start? Oh, never happen on VMs. More realistic for static VMDKs? Maybe 20GB.
  • VMware SRM (Site Recovery Manager)? We already know what this is, and how it works, but is anyone but every large enterprises going to implement this technology? Only time will tell. The real power of SRM is on a WAN scope, disasters in the same city are not the issue, it is mega events, storms, terrorists, etc., that will impact an entire city complex, lets get disaster recovery and load balancing of datacenters 1000s of miles away, not 10s of miles. Doing this is cross country, cross nation, that is the realistic need, but this is not cheap or easy. How will VMware solve this one?
  • What is VMware doing to block Xen or Solaris, as they emerge to run INTEL, x86, 64bit VT, etc., and run other operating systems other than their native OS?
  • What is VMware doing to improve its Application Instancing? How about a brand new product? VDM, and appliance VMs are not going to offset application streaming or Citrix next generation solutions.

I have one last question for VMware, before VMworld 2008 is official, and powered up…Where are the cheerleaders at? If you don’t understand the reference, look at my discussion in this blog of VMworld 2007, where I state that ESX 3i is nothing but hype, where I discuss some disappointments about ESX 3i, or, sorry, ESXi.

1 comment September 15th, 2008

Fear for the Sake Of Fear? Hyper-Jacking Myths?

Virtualization Critical Evaluation – Chapter 07

I ran straight into a brick wall this week. Not to worry, no injuries, much to the displeasure of some I am sure. However, I do this upon occasion, and some times I enjoy it, some times I don’t. Walls, of the virtual brick variety, in the computing industry are unique, because they often are built upon bad ideas, bad assumptions, bad conjectures, and mortared together with bad logic. Moreover, it has been my experience, that if you pull just the right brick, just the right way, then the entire structure is exposed as flawed, and comes down like, well, a ton of bricks onto the original builders. Hyper jacking myths are such brick walls. Hyper jacking is reality? Or is it? What are we afraid of? We know some day someone or some entity will hi-jack a hypervisor? Right? Or do we? Depends on who you asked, but to some, it is all the rage in the press right now, and has been since about the same time last year. Even the motives for discussing threats to hypervisors are suspect, based on my research. Although, not an article, but a thread, that illustrates the discussion of hyper-jacking as motive driven, is And of course, Google is just spilling over with topical discussions, about hyper-jacking, but seem to be all words and no real substance. There is so much misinformation about what can and can not be done in reference to hacking that security teams both public and private are having nervous breakdowns just trying to understand the risks and threats never mind formulating plans of action or as some like to call it, remediation of risk. One of my favorite articles that does nothing more than generate fog, or mild panic, is Unfortunately, this article is misleading. The key virtualization platforms that dominate the industry have been certified and vetted, against known methods and techniques, something this article, among others,never explains and thus never provides a balanced view of the issue. Of course, no one is secure against new techniques and methods, but this article does not explain that point well either, it raises questions, nothing more.

From my perspective, I have never liked the term, remediation, it smacks of re-active tasking. And mediation alone is still perceptive classification as post facto resultant state. But, then again, just what is a threat? When is a concept become a threat? Or even more than theory? We have so many threats to our existence, virtual reality is no different, but fear of threats that may never materialize? I am not saying you don’t plan for threats, but what I am saying, is that threats are just that, potentials, not impacts. What was the key concept in the Matrix? Neo had to believe he was the one, the reality, in all that was, is, and had been dominated by virtual reality? Or to explain it in a historical context, what is the quote…We have nothing to fear, but fear itself . So why you ask, did I say I hit a brick wall? Fear of what could happen was the wall this time. Fear has caused may a good idea to die on paper. Security teams are exceptionally susceptible to this scenario, the fear of what could happen, no matter how remote or indiscrete. Opinion can make what is rare or even not probable; appear to be a rather solid, a wall to success use of technology, no less. The specific wall that I ran straight into was built by a group of security experts, which had the best of intentions, but had a serious lack of foundation, just fears based on potential issues, causes with no realistic materialization that could someday be effects, nothing more.

For example, just because there maybe someday, a virus that can attack NICs directly, and just because someday someone may hyper-jack a hypervisor, therefore, virtualization is not as safe as traditional hardware. To repeat, because of the potential to be hacked at the physical NIC, or the virtual NIC, or the virtual switch, and subvert the hypervisor, by definition, virtualization is not as safe as traditional hardware. To which my response was…What? Do you know what circular logic is? These are potentials not realities. So you decide what reality is, based on what casual fantasies you see as potentials? That is like saying you never will use a horse and cart to deliver carrots to market because the horse might die from pulling the cart, because the cart may break down, because the left rear wheel may fall off, because you loaded 27 carrots not 270? Therefore, horses are dangerous?

Trying to get to the bottom of this line of thinking as urban legend or not, I found a number of articles that all discuss what the new threats to corporate entities must be, yes I say must be, because the articles all promote their authoritative position, with little or not objective explanation. One that stood out as such, hyperjacking the latest threat to servers. To me this type of article is less than useful, it hints at a possibility and nothing more, talk about true hype, just to get a hit on a web page? Looking for threats is fine, but we don’t see a flood of articles discussing real results, real attacks, now do we? Hackers tend to brag about what other hackers have done, Even if some professional hacking group, from China or Russia, has done, or some defense contractor has done it in a lab, or worse in the real world, the word would get out, in short order, that is fact, the web is horrible for hiding the truth, just as it is horrible at only reporting the truth. At the risk of being connected to the X-Files franchise…The Truth Is Out There…if it exists, the internet always gives up its secrets, even if, the secrets are buried under a ton of garbage.

What is the goal of some of these articles? I am not sure, other than to generate headaches, and as I said before, generating web page hits. The results however, are real, when this type of hyped vague popinjaying is believed; inaction due to threat results, and is a classic psychological warfare technique, as well. Bully for the conspiracy theorist. A less militaristic context for the same basic situation is called analysis paralysis. The lack of action because of fear of the consequences of taking said action. This is never rational or logical? Good point, when is a potential issue deemed a true threat that needs to be acted upon? What an idea, action to offset threats, rather than inaction. This is a good thing, because when action is taken; the technology is actually used, leveraged, what have you. So how do we put this into the context of a security strategy and still use the technology? This is not trivial to be sure, but a large dose of common sense is a key to rational success.

First, threats will continue to emerge and develop, just as methods to eliminate threats will be soon to follow, if not offset by changes in theory, design, and implementation. The only constant here is that change always happens. Second, being offensive is not always possible, and being defensive is reasonable, as long as the technology is used, at some point. Early adoption of technology is a risk. However, never adapting a technology for us, is just giving into the fear of potential threats. Third, trust that ideas, solutions, methods, etc. to combat real attacks will come, or can be implemented before impact occurs. Fourth, no matter how good, how strong, how extensive the strategy implemented to protect the environment, there will be some event or situation that will compromise the environment at some time. To believe otherwise is foolish, or worse to believe that it is impossible, is irrational.

Keeping the above concepts in mind, and looking at the hypervisors of today, which are safer than others? That is a rational question, a proactive one, not a reactive one. Saying that all are unsafe is not rational. True, all have some level of risk, but so does using every operating system, and we still use operating systems in everything, from servers to cell phones. Never using an operating system, because it may someday be hacked, is defensive, and irrational. Focusing on the proactive aspect, and agreeing that hypervisors should be used, there are a few basic design features that offset risk, not quasi threats.

  • An embedded solution is safer than any generic operating system based solution. This is straight-forward. Operating systems, in the traditional sense, have a large surface area of attack because they are designed to be flexible. Flexibility is difficult to manage. Embedded models are focused functional elements. Easier to manage.
  • Hypervisors should be designed to never allow themselves to be executed by themselves in abstracted context. This is obvious no? It takes only a few lines of code to validate that a hypervisor is hosting a hypervisor. This is true of multiple vendor or different vendor stacking, say Hyper-V refuses to execute ESX, and ESX refuses to execute Hyper-V. No hypervisor should ever host another hypervisor, therefore, nesting is forbidden.
  • Never violate the context of function versus access. What does this mean? It means that virtual instances should never have access to the hypervisor, nor know they are hosted, and the hypervisor should never inform a virtual instance that it is hosted. There should never be any inter-process communication between virtual worlds and the framework that hosts the worlds. This is a pain at times, because we sometimes want to cheap as developers, but don’t do it.
  • Never ask a hypervisor to be a firewall. This is similar to the point above, but is an external design issue to the hypervisor. Never connect the hypervisor management functions to a less secure environment, than the virtual instances. This import in the case of a DMZ environment, but should be true for any environment. Is it a real risk? Today maybe not, but if someone, some day actually does figure out how to effectively hyper-jack a hypervisor? The environment design should be strong as possible and still be useful.

There is one guaranteed way to address fear, specifically the fear of hyper jacking as security teams today feel they must. However, these same security teams need to understand, they can not deny or disqualify solutions because of fear of the unknown or the future. They need to understand a concept that I coined in a meeting about 4 years ago, Engineering-By-Fact. This is based on a concept a former boss, of my boss at the time, coined, Management-By-Fact. In explanation of Management-By-Fact and Engineering-By-Fact, as well, this boss of my boss stated that he never wanted anyone in his organization say…I Think, I Believe, It Should, Most Likely, Maybe, or Most of the Time, or any variation of the same expression of opinion, in the same sentence as voicing a solution to an issue. Solutions, like problems, are black and white, if you don’t know for a fact, that you have the right, correct, and factual solution, or even worse you really do not understand the problem, don’t venture opinion over facts. Now, if we could only get the people which express opinions as facts to honor this concept, the poor security teams out there, which seem to be afraid of their own shadows, at times, could spend less time building brick-walls, and more time configuring fire-walls, right?

3 comments September 10th, 2008

VMware Really Hurting? Or Just Really Bad Timing For A Simple Mistake?

Virtualization Critical Evaluation – Chapter 06

Is VMware really hurting, meaning are they coming a part of the seams? Or was the latest licensing bug issue just bad timing? It was a minor code mistake, but a major perception mistake. For something that happens all the time in a code development shop, more often than a non-coder would care to understand. Just imagine you are the one that make the mistake? Just imagine you are the one that missed catching the mistake? There will be some careers that will end or at least be derailed for a while, at best, at worst, change lives of some with dramatic impact. Is this fair? No. Is it really reasonable, maybe it is given what VMware must now do to recover from this perception of running a lose development shop. No one is perfect, but the world expects perfection, and more than the perception of perfect. VMware is not perfect. No software publisher is. However, the real issue here is not the mistake that was made. But why the mistake was made. Perception has real significant impact. Why do I say this?

Some time ago, just short of a year ago, VMware promised a specific group of enterprise customers that the licensing model would be, and I believe I quote, “we are discussing options for making the licensing model more informational, rather than enforcement oriented.” Those in the room, there were some 50 or 100 of us that heard this asked almost in one voice…When? Where? How will this passive licensing model be implemented? VMware at that point became very vague. In fact the topic never came up again. This is the real issue, because if VMware had owned up to this promise, at least I saw it as a promise as a time, as good as a promise, and then the impact of the August 12th bug would not have been the fire drill it was. VMware seems to be doing damage control as a matter or routine throughout 2008. Was the exit of Greene not a type of damage control?

Even the evaluation version of VMware ESX OS has a 60 day try it window before features are disabled. Now what an interesting idea! The commercial version of License Server would generate events and warnings but not actually disable functions for 60 days; this would have avoided the issue no? Never mind the fact that I think that the Flex License Manager solution is horrible, I am just not a fan of restrictive licensing, and I have never been impressed with Flex, it has a very long and negative history depending on who or whom you ask. And, yes I know all the issues and debatable points that surround software piracy and theft. So the following questions come to mind? First, just how many Lawyers will get sacked at VMware for the August 12 issue? Lawyers, yes lawyers. This is terrible, sad even, because I am sure some heads will roll across the floor and down the stairs, out the door of VMware. I hope the heads migration, includes the entire brain trust that thought proactive enforcement of licensing was a good idea, I bet it was a lawyer that initiated the idea! Am I wrong? VMware say something if I am.

Just how many of senior management will get nailed? After all, there are serious issues with VMware quality if this is a trend. Blaming Greene is a cheap shot at this point. Lets be honest, there is a growing trend in the entire information technology (IT) industry to release solutions to the customer that are flat out incomplete, broken, or worse pushed out the door because of a fixed deadline. The quality assurance process, I believe, is seen by the marketing, sales, and even top management, as an evil thing, that holds back solutions from being released. After all, most customers pay for support, so we, the customers pay twice? Once for the product as concept that is incomplete, and again for getting issues fixed that never should have gotten out to us in the first place? You bet, your sweet posterior, you do. Just how long should a list of known issues be, to be deemed reasonable? My eyes almost bugged out of my head when I read the release notes for ESX 3.5 Update 2, very long, does not install a sense of confidence? All I could think of was… If the known issues list is this long, how long was the issues list that they actually fixed? And what did they miss?

Guess the issue was not big enough for the CEO of VMware to make a public statement? Maybe at VMworld 2008, in the key note address, someone at VMware will do the right thing, and state that VMware will and has improved quality assurance methods and processes, so customers are not impacted in a similar manner in the future? How about a known issues list that is only 5 items long in total, or less than 10 items in total? That would answer a lot of critics, including me. And go a significant way in the positive direction to answering the question…Is this the end of a bad series of missteps for VMware? Or I sincerely hope this is not the case…Is this just the next incremental step in a longer trend, before VMware goes down in flames?

1 comment September 2nd, 2008


September 2008
« Aug   Oct »

Posts by Month

Posts by Category