ToutVirtual Enterprise Virtualization - A Proper Virtual World

Never Fall A Sleep At Work After A Big Office Holiday Party!

Christmas Eve is upon us yet again, and I leaned back in my chair, put up my feet, balancing somewhat poorly, in my standard corporate chair, which is not designed for leisure repose. The boss via instance messaging, and all my other co-workers, few as they are now, that come into the office routinely, are long gone for the day, at last. I have a mug of tea heated to perfection, now at my elbow on my desk. I am content; another year of successful virtualization engineering and implementation is done. VMworld was a blast; to be sure, all things considered it has been a good year. So relaxed am I, proud of the achievements thus far this year, I feel the creep of sleep approach my consciousness… I start to think this is not the place to dream… but this thought is never finished as I slip into unconsciousness…

I sat in my favorite chair, in fact, the only chair in my bed chamber, the last bite of my warmed gruel, still favoring my tongue. The cold of the house barely avoided by the weak flames in the fireplace, I bemoaned the high price of cool. I hear the wind out side, move across the house structure, but I am warm, comfortable in my miserly surroundings. Unexpectedly, I felt a sudden chill, a phantasm like wave of cold. I fear for my life, but no rational evidence to do so is apparent, or so I thought. Faintly and then louder, I heard the clanking and jingling of metal objects linked by almost silent, fiber optic cable, the scratch or scrape of heavy objects across the hard oak floor. The objects, which appeared to be servers of various sizes and models, all dragged by a ghostly image of the contractor technical support person we, lost to the severe cold of the server vault all those years ago. As this person, of whom I fail to recall their given name, perished while struggling to rack hundreds of servers in just a couple of days. A cruel end for a poor soul that was just doing as instructed. Never mind the task was impossible for any mere mortal to do alone.

This poor unfortunate soul, this poltergeist condemned to travel the spiritual waste lands of the nether world in server support hell, approached me as though solid walls had no substance, having come through my bedroom door as if it was nothing more than smoke or mist. Again the noise of the looped together servers and other miscellaneous fiber fabric infrastructure, I now recognize, assaults my ears. I am speechless, the enjoyed gruel after taste in my mouth, now soured on my tongue. As the apparition spoke with haunting, hollow tones, words drawn out like they were painful to speak, and slurred in duress, as though each syllable was an extreme effort of will to be formed.

It, the thing unreal shadow, said “You will be visited by three wraiths of the virtual realm. They will come after the midnight hour but before dawn, each in its own time. Beware, beware, these wraiths will part the veil of reality. Be warned, be warned, that you heed their words of wisdom and advice, so you have been warned, so you should be aware. Fore ignoring their respective councils is doom assured, and thus doom earned.”

I squealed in fright, I jumped from my comfortable, if suspected quality of chair, and made flight to my bed, drawing the curtains of the bed frame around me, hiding my sight from the specter that voiced such dire prophetic whispers. No, I thought, no, this can not be real, it is not real, I am a man, a man of significant years. I can not and will not be boxed into a corner like timid child. I thought to myself, be gone, be gone you unholy spirit, I have no need of your contempt or compassion. But I stayed my tongue, voicing no sound, for I was, I admit, afraid. And as I thought these thoughts to my self, I heard again the noises I heard before, the rattle, clanking, and merciless scratching, of the technical resource ghost that was once human make its exit as it had its entrance. I thought anew, had this same technical position been not out-sourced under the circumstances then done, this poor entity might yet live still?

Some time passed, and I don’t recall the old clock on the fireplace mantel chiming at all, but when the chime toned thrice, I realized it was now three(3) in the morning or there about? For the old clock was wont to keep imperfect time. As poor in resources I felt I was, I had no desire to acquire a more accurate time piece. After all, approximate time, is good enough, only a fool desires the real time, all the time. A penny saved, is a penny earned, no? But my self congratulation for being frugal was interrupted, and I do not reference such as a computer processor would be. For a physical presence was near me, just out side the curtains of my bed I believed, nay, I felt there. In trembling movements I opened the curtains and there before me, was a wraith, as promised before. I yelled in spite of myself, and hid my direct view with a nearby pillow, saying “Go away oh wraith, leave me in peace. I deserve not this visitation.”

The wraith, in response did nothing more than motion with an appendage that appeared to be a very old and weathered hand, that I should come forth. No word or sound was made, nothing but an android or robotic like stiff jester with the hand. I could not resist the beckoning, though I hated being intrigued by this event. As I left the false security of my bed, I donned my robe and slippers, leaving my night hat a top my head, I also attired myself in my shabby old robe that was at hand at the end of the bed, where I had left it before sleep. Once I seemed prepared, the wraith placed that same aged hand against my forehead, and I lost all since of time and space, as I closed my eyes, for this was an uncomfortable happening. Movement of some type I knew was done, but the means of transportation and location selection were beyond my observation or ability to describe. It felt like I was falling, losing balance, but I never seemed to fall. After some brief time of this sensation, I sensed it was time to open my eyes. I did so. But what I say was both irrational and yet believable. Something I could some how identify as true.

Before me aligned in straight rows, some 100s, no 1000s of server racks, in long rows, stretched across a room longer and wider than I could determine with unassisted vision. Every make and model of server were represented, all manner of distributed server by vendors I recognized were in this unworldly presentation. I walked up to the nearest row of these racks, looking at the enclosed devices, and touched the keyboard nearest me. The corresponding flat-panel, be it small, and quite old in appearance did flicker into refractive light, the backlighting of screen still functional, a miracle given its apparent age. Before me was the simple console of a VMware Virtual Infrastructure (VI) ESX or it appeared to be such? It was hard to tell the specific version, since no reference to the version of ESX was obvious on the screen, this struck me odd, that I could see it was a VMware VI host before me, executing, but I could not focus on the specific version. All I gathered from the screen that this was old infrastructure; this entire location was in fact very old, the distant past, in a word, obsolete? The wraith, still not saying a single word, nor making any sound, motioned to all the racks in a wide sweeping movement, that imparted to my understanding that the entire room, as such, was all the same, VMware VI hosts in all its forms and variances was shown before me.

As I worked the keys on the keyboard before me, I realized in horror, that even though the host I connected to at random, was functional, no virtual instances were in existence. This realization allowed a cold chill to walk down my spine. “Was in fact, this entire expanse empty of all virtual reality? Were there not virtual instances in existence here?” I did not realize I had thought this idea aloud, but I had. Again, I asked, “Were there not virtual instances in existence here?” The wraith moved not the slightest, nor gave any indication that my question was acknowledged or understood. Still, I asked, pointing to the rack both to the left and right of where I stood saying, “Pray, tell me wraith, is this the future? Is VMware not but doomed to be abandoned?” Again, as before, the wraith gave no response. I yelled in frustration, “Wraith before me, speak!” Still this elicited no response.

After a few moments more, the wraith raised that same aged hand, a move I was now familiar with, and motioned for me to follow, I did this, taking one step then another with honest dread as to what I would see next. I was led to a cubical via that same falling sensation of transportation. This cube was long layered in dust and neglect, for it appeared, had not been used for some significant length of time. The trash bins were over flowing with papers and empty cans of Jolt cola. The desk was covered in corporate snail mail and other correspondence from various vendors, unsolicited marketing information to be sure, it all. An empty coffee cup placed at the edge of the desk, which read simple script, “We have done so much, for so long, we can now do anything with nothing, and management could still, care less!” It was clear this was a dead-end cube, which some unfortunate life form once used but was now long forgotten, long abandoned to a fate unknown. The wraith pointed to the name plate on the cube wall just at the entrance to same. Reading the name there, I fell to my knees, the strength of my legs abandoning me. Disregarding the pain of this maneuver, crying in a weak voice I said, “No, no. Please spirit, no.” I was in shock, and despaired at what I had just read. Saying with difficulty around my tears, “Spirit tell me this is not true, that this is only what could be?” But as before, the phantasm before me said nothing, making no jester or response in any way as recognition of my query. I felt confused, I felt lost.

The next instant or so it seemed to me, I awoke. I was sitting in my chair, in my cube. I found it odd that I recalled nothing about two (2) of the three (3) wraiths in my dream. Was I visited by all three or just one? No sooner did I think this thought, than a cold chill walked up my spine, where before it walked only down. The same chill? It changed directions? But that was in a dream, I told myself. Nothing but a dream, I told my self again. With practiced ease, I pulled my legs down to their proper business position, if only the boss knew how often I have my feet above knees with my keyboard placed across my hips as I sit in a relaxed pose, while at work. But the boss is 1000s of miles away, never yet to visit my physical location. So be it. With little thought to other needs of work, I started to disconnect my laptop, planning to stuff same into my portable bag. Thinking I never know when I might need to access the company network over the holidays. When my eye just catches one specific iconic flash in the bottom right corner… an icon that warns that I have new mail, yet unread. I freeze, I think, then say to no one but myself, “Ah, dang it, ignore it, it is Christmas day!” But truth be told, my personified view of life is closer to that of a cat, than of some other animal, whereas my cursorily often gets the better of me.

I click the icon, thinking, with luck it is something of no consequence, something that will be banished to the delete folder or little better. However, it is addressed to all my peers, my manager and other managers in the department, as well as me. It is from the boss of my boss, a communiqué from the top of my world, which makes me nervous. For as everyone knows this type of note, this time of year, is never good news, or never has been in the past. Regardless of my fears, and because I clicked the icon, the note opens to a larger window, which displays the message…

“Everyone, sorry for the late notice, but I wanted to get this to you all before the end of year. It has been decided, that starting the first of next month, we will begin the process of implementing Xen, and Hyper-V as our core virtualization platform, variable class of service, for virtual instances that leverage the Windows operating system. Furthermore, all Solaris virtual instances now run on VMware VI will be retired, and functionality migrated to Solaris LDOMs. All Linux virtual instances on VMware VI will be migrated or otherwise recreated as RHEL KVM based instances, in parallel, to the Solaris move.” As I read this, my knees start to shake; my hands take on a noticeable tremor. What? Has VMware VI not served us well? I read on, having trouble focusing on the meaning behind the words, “I am sure everyone saw this coming, so this should not be a surprise to anyone. Please understand the timeline for this is very aggressive, we need to get this done sooner than later. Right after the holidays, we will have a few meetings to work out the tactical objectives now that the strategic direction has been established. Remember, we have to get more done this coming year than ever before, faster than ever before, or else. Oh, I want to also say, of course, to everyone, Happy Holidays!” Is it a dream, did the wraith have it right? What is real? What is virtual? I am again scared…

Abruptly, I scream in pain, this hurts. No this really hurts… I am now, really, awake! I realize I have fallen out of my chair, spilling my almost empty cup of tea on myself in the process. What a horrible dream? Yes, a dream about a dream? As I set to rights my chair, and clean up the last of the spilt tea, I think to myself… Never fall a sleep at work after a big office holiday party… It is then, and only then, I notice that infernal inhuman soulless iconic flashing at bottom right side of the flat panel monitor on my desk. The unread electronic mail indicator is demanding immediate attention.

2 comments December 24th, 2008 Schorschi

Cloud Computing, Utility Computing, Bah Humbug!

Virtualization Critical Comparison - Chapter 03

Well, boys and girls, of which there should be more girls in the information technology industry right? Christmas is upon us, and the New Year will be here faster than you can say Jack Frost. Thanksgiving is gone, Halloween is long gone, so where are we? We are in that dead space between the old and new, when all planning is up in the air, resource planning, time planning, project planning are all but done, but could change over night this time of year, here to day, gone tomorrow, is a real situation this time of year. Wonderful, but what have we done to change the world, in the information technology industry as a whole? What will we do in 2009 that will shake the foundation of a strategic planning? No, I am not consulting with the ghosts of Christmas past or Christmas present they do not understand binary computing, or even virtualization.

Sure, virtualization has changed server side computing, sure virtualization is changing desktop computing, heck, virtualization is changing cell phone computing, yes, virtual machines on cell phones, of course. But in the rush to improve computing utilization, have we missed something? We have, and we have missed it in a big, big way. Clouding or utility computing, depending on how your organization defines it, is wonderful, the SETI project proved that distributed computing, or floating processing is viable, and functional, so of course the idea of floating resources between datacenters is an easy goal to promote, no? And virtualization makes it, at least from a tactical perspective, possible, and from a strategic perspective, logical. But what did we miss? Still guessing?

No it is not application instancing per se, although I have long said that application instancing is going to be a big deal, a least a few times in this blog as well as elsewhere. No, the next big deal is operating system reduction. Gartner do you have your ears open? What? What the heck are you smoking? I am sure some are saying…You stuff a bunch of mistletoe in a pipe and smoke it? Virtualization does not reduce operating system instances, but application instancing should or does if done right. But that is not what I am saying, I am saying, operating system reduction according to type. Yes, type. How many organizations are struggling with multiple operating systems while trying to craft a cloud or utility infrastructure? Why do multiple operating systems exist? Well that is a complex question and a straight forward question? The answers are complex and straightforward as well.

Let us take the simple question first, why do multiple operating systems exist? In simple prose, if you wanted a specific feature or variant of performance, you needed the right operating system. Well, those days are gone, at least in distributing computing, there is no reason for any organization to have more than two operating systems, if that, one for servers and one for desktops, done deal. Why? Simple, it is not worth the resources and expense to maintain a multiple operating system scope and scale engineering and operational support models for each operating system. You can do everything on every operating system within common reason today, the performance differences are disappearing for must applications. Virtualization has encouraged this, by further abstraction of the hardware. After reducing the total number of servers, and reducing the total number of operating systems, increasing the utilization curve, what is a big cost issue? Personnel, yes, even in these hard economic times…I heard the word depression echoing in the vast, empty, cube spaces of corporate America, no? It does not make sense to have a UNIX team, a Linux team, and a Windows team, now does it? Never mind that many large enterprise firms have several different variants of UNIX or Linux and even Windows running of very long periods of time? Why does it take so long to certify solutions? Why do cross operating system solutions fail to work well? You avoid these issues, when you eliminate operating system types. Imagine if you had one and only one engineering team focused the one operating system? That would save personnel expense, because yes fewer persons, but more important, less time to get to solutions to your clients, since only one platform is focused on, be it Solaris, Red Hat, or Windows 2008, to be specific via example.

Let us now tackle the complex question, how many organizations are struggling with cloud or utility computing when multiple operating systems are available or viable in the enterprise of the organization? Cross operating system solutions never do well consistently across all operating systems, in fact, they often lose focus and support all operating systems according to lowest common denominator. Why? Oh, it would be easy to say political turf, right? And that is actually true to some degree. There is absolutely no synergistic effect when you are trying to be all things to all platforms. At some point someone at the top has to make a hard decision and stick to it. This is both true for the developer of a solution and the users of said solution. But the greater cause is avoidance of pain, it is much easier to do what has always been done, because UNIX or Linux or Windows has always been there, it is easier not to force change.

Even virtualization is nothing new, just a slight improvement on the original problem, supportability and utilization. Virtualization has improved utilization, but made supportability worse, or should I say more complex? Absolutely everything in the information technology scope is complex, more complex than it needs to be. Why? Because marketing loves new features, and suckers, I mean upper management types find it every easy to fall in love with a specific concept or feature that makes a quick bonus, then standing up for a long term, far more superior objective, which should be more profitable years down the road, but then again, no high end of year bonus? Tell me I am not right? Has the number of vendors demanding to present new toys to your engineering group increased or decreased over the last year? I for one, have never seen so much junk, yes, junk, solutions flood into by electronic mail, snail mail, or even the door to my office, because there is a hard blitz on to chase the latest new toy. When did early adoption become strategic planning? And of course absolutely every new toy must be implemented in less than 12 months, or else.

We need to de-complex the computing infrastructure in order to create a new cloud or utility computing model that will give us long term, or more than 5 years is what I call long term, and significant strategic benefit. The first step to that goal, is don’t implement every operating system in the cloud. And if not implemented in the cloud, let those orphaned operating systems decline, and disappear. Until this is done, cloud or utility computing will have a flawed foundation, a weak point, that will drive up cost and complexity, which is not acceptable. Cloud or utility computing must be easier than ever before, must be leaner than ever before, it can not survive the silo effect of conflicting operating system goals, objectives and needs. Cloud or utility computing should be monolithic or it will just fail to live up to even modest expectations. So, until the number of operating systems up for consideration in your cloud or utility model is reduced, I say, bah humbug. Oh, and the ghost of Christmas future, it, agrees with me…Ha! So stick that in your pipe and smoke it.

Personal note…To all that read this blog, including those that have agreed, disagreed, and stated such in comment or reply to this blog; I wish you all happy holidays for 2008 and best wishes for 2009. I hope that you have enjoyed this blog, and will continue to enjoy it next year.

Add comment December 17th, 2008 Schorschi

Virtual Instance Performance Revisited

Virtualization, Fine, Well Sort Of? - Chapter 08

This is a revisited article, not because of a correction or change of view, but advance a topic that I have always indented to revisit, but never seemed to have time to do so, until now. A loyal reader of this blog reminded me of this fact recently, so I am honor bound to resolve the gap or lack of continued discussion on this topic. Oh, I am speaking of virtual instance performance of course, as the title notes. Unfortunately, virtual instance performance is a complex topic that gets into the swampy weeds full of tangles and hidden snags, faster than water and dirt make mud.

In part I of this topic, I discussed the context of performance, to which, Peer-to-Peer or Inter Virtual Instance Performance, which is what the host infrastructure reports. I will not rehash this topic here, but it is important to note that only the host infrastructure can accurately report virtual instance performance. Also in part I of this topic I referenced, Host to Instance Performance, Host to Host Performance, and Host to Cluster Performance. Respectively I will summarize each concept, but for more detail, refer to part I of this topic. In brief, Host to Instance Performance, is overhead of the host or host impact to performance, what does your hardware and hypervisor cost in reference to performance. Whereas, Host to Host Performance is which host executes which virtual instances the best, all things beyond individual instance deltas being equal. Moreover, Host to Cluster Performance is one step short of cloud or grid computing modeling, focusing on which host in a given cluster is the most efficient given a known set of virtual instances. This is important, when you consider data center globalization, to which hosts should be consistent, and so should clusters of hosts, across different datacenters, for example.

Now if you are tracking all these This-Versus-That models above, then you will realize that one model is missing, which it is? Give up? Cluster to Cluster Performance! There is a good reason for this; I neglected it in part I, my bad. As life cycle and management tools have approved over the last year or so, this is a viable and significant performance model, especially when you have heterogeneous hyper-visor based environments. Consider VMware versus Hyper-V, or Xen versus VMware, or Xen versus Hyper-V? Obviously if you offer a class of service aspect to your virtualization, you need to be able to compare different virtualization infrastructures in real time, with little or no explicit normalization. I for one, hate normalization, it is often abused and biased to a specific or narrow criteria set, so normalization devalues the analysis and results. But I digress, Cluster to Cluster Performance is beyond the scope of this specific article, but will be discussed in the future, did someone say Virtual Instance Performance Part III?

But the title is Virtual Instance Performance Revisited, and so the key to all performance evaluation starts and ends with the virtual instance, this is the corner stone of virtualization, be it application instance, virtualization container, or operating system isolation based. The vast majority of tools available for virtualization performance evaluation focus on the virtual instance of course, since the goal is, to always have the fastest instances possible, given the constraints of the associated infrastructure. The last comment begs the question, what constraints? Well, these are discussed extensively by virtualization gurus over and over, including processor context switching or processor cycle loading, memory IO, disk IO and network IO. It is quite common for various hardware vendors to focus only on one or two of these constraints and publish misleading or flat-out inaccurate statistics declaring they have the best or fastest virtual instances in the known universe, only on their respective hardware of course. Bah Humbug! They even normalize their results in comparison to their competitors to prove the point that they have the best hardware. Bah Humbug, Again!

Unless you evaluate virtual instances only under severe load for all four (4) constraints, and inclusive of these constraints, you are not doing your clients or yourself right. A classic garbage-in gospel-out (GIGO) scenario if there ever was one. Virtualization abuses hardware, it is and always does this, this is by design, and after all, virtualization is attempting to fully utilize resources that are often unused or wasted, no? So the single most import issue with virtualization instance performance evaluation is the selection of the tools, not I stated, tools, to do the evaluation, Ah Ha! Bet you did not see, or read, that one coming now did you? VMware VMark, vCompute, IOMeter, etc., all have their weak points, you must understand these limits or issues before you design your evaluation criteria and methodology. Consider this, if your specific testing for virtual instance performance testing is only looking at processor loading and memory IO, are your clients not going to be unhappy when their network IO and disk IO results are horrible? Did you analyze your environment right? Did you evaluate your proposed environment right? If the virtual instance evaluation for performance is skewed, then your entire environment performance evaluation for Host or even Cluster scope performance will be horrible.

Now it is time to get into the weeds, and get mud in between the toes. Now that we know that we must test all constraints explicitly and inclusively, and we must test at the virtual instance scope before all else, what do we do? The virtualization gurus will argue over this, but below is what works for me.

1. Establish a control. Establish a performance history baseline. If you are testing virtual instances on a new hyper-visor, or new hardware vendor, do the exact same test on an environment you already understand. If you have HP, and test on Dell, don’t normalize your results, just make sure you understand that HP and Dell are different, and make sound inferences based on the raw results. If you can test on 3 or more hardware vendors at the same time, or have historical data using the same tools and methods, you don’t need to normalize the data. Normalization is for management and others that do not know how to analyze resultant data.
2. Processor and memory differences, including changes in caching speed and size of buffers, are often a shifted scale comparison, so normalization is not needed. This is also true of power consumption curves. This is rational and logical, since network and disk sub-systems should remain consistent for a longer period, so by definition the number of factors to be compared can be reduced if the sub systems remain consistent, including, of all things, the PCI bus architecture per host. Production performance data always trumps lab data. So if you have HP and Dell in production, and are evaluating IBM, use the production data as the baseline or control, then test HP and Dell with the newer tools or methods, or processors and memory, etc., then and only then, test IBM. Bingo! No normalization is required. I can just hear the slick stylized marketing types for all the various vendors crying over their iced-mocha-lattes, when they find out I always reject normalization based evaluations by default.
3. Always run the same test, in the same environment, at the same time, with the same characteristics. This is just basic common sense. However, don’t be surprised when you see something that does not make sense. Iterations are key to the entire evaluation effort. Remember that basic statistical analysis requires that a sample size of 30 or more is needed to get to any standardization and variance deviations accuracy. Every time a change is done, changing the experiment is done, and performance evaluation is an experiment. Think scientific method all the time when doing any performance evaluation, be it in the lab or otherwise.
4. Make sure you understand where and when you can introduce error into the results. The only way to do this is through peer review, getting more eyes on the proposed test plan, is the significant objective. Everyone sees the same process with or from a different perspective, whereas tunnel vision is evaluative death. Sometimes eating crow at the beginning is better than getting heart-burn while coughing up feathers at the end of an evaluation effort.
5. Control expectations. Data often goes around the world faster then the executive summary. Expect that someone, somewhere, will take the evaluation tools and methods, as well as the results out of context. Results will be challenged, be prepared for it. Don’t defend results, only explain how results are generated and analyzed. Vendors hate this, and often forget this point, when they sponsor or quote so called independent analysis, focusing the resultant explanations as the authoritative final qualitative statement, when the raw data objectively discounts or obviously points to other conclusions. Normalization often hides the true results.
6. The developer of the given virtualization environment is the start of the process not the end. Do not rely on the developer tool set, nor what a given vendor demands as the only acceptable tool for analysis. Of course the vendor has tuned the given tool or methodology to illustrate the strengths of the platform in question. Would it not be a wonderful world if HP performance tools worked on Dell and IBM, and Dell tools for same, worked on IBM and HP, etc., etc. Would make for some interesting evaluations no? Or Fabric tools worked on FCoE infrastructure, and iSCSI tools worked on FC infrastructure? Sounds insane? Not so. Generic tools sets exist, independent tools exist, use them. Even if every vendor in the world has used VMmark, VMmark means nothing to Hyper-V.
7. Repeat, repeat and repeat, change only one thing at a time, for example, only change loading of one constraint at a time, be it processor loading versus memory IO, versus disk IO or versus network IO. Use the same dataset or streamed sequence for each test. Never change the dataset or streamed sequence between iterative testing for a given factor. Complete an entire set of tests before mucking with the variables beyond the planned test set. Could be considered a repeat of the point above, about running the evaluation in a consistent manner, but it is so important, it if it is a repeat, so be it.

Well, at this point, I am sure someone is yelling…But he has not told us anything useful yet? What the Heck?! Not true. It is true, I have not spelled out an explicit methodology for evaluation, as a do this, do that, then do this scenario. To do that, cough, would be to create a bias that should, no, must be avoided. But to be fair, I will summarize things a bit, and recommend a best practices approach.

• Analyze the virtualization environment, focus on the virtual instances first, look at processor loading, memory, network, and disk IO loading, create and execute tests that stress all constraints as applicable to your expected needs, and well beyond your expected needs. If the majority of your virtual instances are encoding unique video data creating results, expect lots of disk IO, if your virtual instances are web servers, expect lots of network IO, etc. Be smart in your evaluation design for performance.
• Remember, virtual instances are the corner stone of all evaluation, Hosts and Clusters have their own performance characteristics, but they are impacted or resultant based on the virtual instances. Dynamic resource sharing, high-availability, etc., are wonderful features, but mean nothing if individual and grouped virtual instancing performance is not understood. The goal is to have most of the virtual instances perform well, most of the time, nothing more. The number of instances, the number of hosts, the number of clusters, even the number of virtualized datacenters, if it comes to that scale or scope of evaluation, will be obvious and straight-forward, if the methodology and tools used are sound according to the virtual instance modeling.
• Performance evaluation is a living breathing animal, and should be viewed as dynamic and experienced based, no pun intended. Nothing in virtualization is static, so allow and expect the methods and tools to be flexible and adaptive to the effort at hand. This is not to say that change is good for the sake of change. Only change tools and methods when it makes sense to do so. Never change technique in the middle of an evaluation effort. To do so is statistical resultant evaluation suicide.

Add comment December 10th, 2008 Schorschi

It is Veterans Day

Virtualization Critical Comparison - Chapter 10

As I write this, it happens to be Veterans Day. No, I never served, but have friends and family that have and did. I don’t have any family or friends in current service, but I wanted to say to those that maybe Veterans, that happen to read this blog… Thank You. If I had been able to serve, I would have. I believe all citizens of the United States should serve, as all citizens do, in Germany of today for example do. I take a very deep sense of pride in America, when non-citizens, became citizens, in serving in the Armed Forces. And to be clear, I thank you for every type or era of service from every armed service, from the Revolutionary War, to Civil War, to WWI, WWII, Korea, Vietnam, Iraq I and II, etc. I do not make delineations between peace keepings, police actions, or world wars, combat is combat, this is one lesson I learned at a young age listing to family and friends that participated in the various conflicts throughout recent history.

Moreover, I had close relatives on both sides of WWII, German and American. So I feel I have unique perspective from mein Grossvater who served in the German Technical Corps in Poland, in active combat with the Russians and Americans, as well as a Great Uncle, a pig-boat man and medic, who lost the back of his knee on a beach on Iwo Jima, because when they lost all the medics at one point, he was volunteered. There is nothing honorable or romantic, or even glorious, about combat. Saying thanks, and meaning it, I know has special meaning to all veterans that only they can appreciate.

As my best friend in high-school said, a former Marine, once said…Semper Fidelis, or Semper Fi, meaning Always Faithful, is a state of mind, not a motto. Thinking of this fact, I realized VMware is at a moment of truth, since it is facing an intense battle, a Battle Royale, really, against all comers. I discussed this at length with a good friend of mine at VMware recently, just after VMworld 2008. No, I will not name this person here, since to do so would not only violate their respective privacy, but since my blog is at times critical of VMware solutions, it would not be appropriate to create a perception of association, beyond our friendship of course, here, or with my, at times, intense views of VMware products. The basic discussion was an honest debate, in policy context, about how VMware should combat the threats, VMware sees in the virtualization sub culture of the Information Technology (IT) industry.

A standard SWOT (Strength, Weaknesses, Opportunities, and Threats) analysis model is illustrative of the debated items within our discussion. MBA students hate these; I did when I got my MBA as well, professors love them, or so it seemed at the time. The itemization here is not exhaustive, but illustrative. I am sure many readers of this blog will find more items then are listed here, feel free to comment on such in reply to this blog as you would. All feedback is good, even that which we don’t enjoy, VMware are you listening? Sorry, was just too easy to say or type rather.

Strengths

• Market Share, VMware owns officially, last I looked at the various reports something close to 60% of the market? But with Datacenter adoption rates of only 10% or so, is this significant enough? The competition is getting stronger; well, Microsoft not withstanding, maybe VMware should acquire Citrix, Ironworks, Parallels, etc., just to eliminate confusing options to the potential customer base? VMware should focus on virtualization, even developing its own application instancing solution, which, is not based on VI? Something new and radical?
• Product Feature Set, no doubt VMware VirtualCenter (VC) is still key to the success of VMware, even with its scaling limits and top-heavy implementation of today, single point of failure (SPOF), it is, for now the best Ease-of-Use (EOU) solution for virtualization. HA, DRS, and UM have all added to this model, for EOU, while detracting from VC scaling and SPOF issues. Not to forget (Storage) VMotion, and VMware SRM. True, VMware IV 4 promises to address these issues, improve integration, but only time will tell how successful this is, or how well received.

Weaknesses

• Lack of focus on Enterprise client base, this is changing in VMware, but it took a long time for it to become reality, years in fact. As the economic situation worsens, it is only the Enterprise and strong growth segments in Healthcare, Financial (cough, those that survive), etc., which for the most part are very big globalized entities, which will have the resources to continue with VMware.
• Cost, VMware in comparison is expensive. Unlike Apple Computer, Inc. VMware can not survive on only the top 10 or 20% of the market share that will always need or be able to justify VMware cost over other solutions.
• Loss of true innovation? Has VMware lost its ability to innovate? Yes and No. Yes, in that we are not seeing the great insightful leaps from VMware as a leader in virtualization, but what new big advances are still to be made? Hypervisor, operating system isolation virtualization, is quite mature. Application instancing, streaming, and state-less solution delivery appear to be eclipsing virtualization. Cloud computing, utility computing, even Grid computing are all gaining acceptance, if slowly as concepts that Enterprise entities are struggling to define, delineate, and implement, of which virtualization as we know it today, is only part of the overall strategy that is typical.

Opportunities

• Innovation? Well, VMware is still acquiring or should I say is now almost completely, acquiring solutions, not really creating new solutions? Is that a fair statement? I think so, now. VMware has learned from Microsoft, that it is easier to purchase rather than innovate? This is an opportunity to be sure, but also a weakness.
• VMware cost model is its greatest weakness but also an obvious opportunity. But does VMware have the strategic will to convert a weakness to an opportunity? Can this opportunity offset the impact of the significant threat, Microsoft? Will VMware continue to be a high-priced option, and elite solution? Or will VMware become the Walmart of virtualization? Which strategic direction will force Microsoft to re-think its strategy?
• VMware must differentiate its self from the competition, showing values of its solution as superior thus, worth the investment. This is an opportunity that VMware only has as long as others continue to goof up, who are goofees, or is it goofies? Microsoft, with its rather weak, compared to VMware, Hyper-V solution, has established yet another opportunity for VMware to get crazy and surprise all of us again?

Threats

• Microsoft, is not the Borg, they are not that efficient or effective yet…I fear we have awoken a sleeping dragon, is just not accurate yet, and VMware knows this. Not to be confused with the famous phrase…I fear we have done nothing more than awaken a sleeping giant. I leave it to the readers to discover the famous context of the later phrase; the hints are WWII and Pear Harbor. Microsoft has goofed twice, well three times, first the failed to acquire VMware, then they never got Microsoft Virtual Server right, then Hyper-V was released, as was ESX 3i, before, in my view it was ready. This does not mean that Microsoft is not a threat, only that it is more potential than reality at this point. Hyper-V is weak, but System Center Virtual Machine Manager (SCVMM) for all its potential, is a disappointment to me. I really expect more from the Borg, I expect Microsoft to adapt much faster than they have so far. Still an emerging threat.
• Xen (Citrix or Red Hat), Virtual Iron, Parallels, Solaris Zones, etc., the various virtualization container models and smaller organizations all are threats that will become more significant over time, as long as VMware cost model is significantly higher than the others, but will they survive when VMware and Microsoft own virtualization? I would say they are a declining threat.

Well, Gartner is not as outspoken on this topic as they could be, which strikes me as odd. Maybe no one wants to predict what will happen? As for me, I am going back and look at VI 4 again, now that I just completed the deep dive on Hyper-V, and look at VMware VI 4, including ESXi, yes, I said ESXi, with a fresh perspective, being a good critic means revisiting conclusions and assumptions, no solution is static so no critic should be static. I also plan to look at the latest version of Xen, again, in light of pending VI 4 and recent Hyper-V analysis, and see if my original SWOT analysis as noted above holds true or not.

Oh, as for my friend at VMware, what were the applicable conclusions from their perspective? In brief, agreed with the strengths, and to some extent saw reasonable merits in the opportunities, especially the idea of differentiating value of VMware quality and depth of offering over cost, but disagreed to some extent with most of the weaknesses and threats, noting that Microsoft was an obvious future threat.

Since this person is my friend, I would not wish to offend, therefore, I did not ask…Did you enjoy the cool-aid?  Instead, I thought to myself…What flavor was selected for next month, hoping the flavor selected would be agreeable. Cherry is just horrible, if I remember right. I am hoping for Grape, after all, I will be drinking it as well, because friends should never drink alone? Or I should say… I will be drinking it, at least until a strong and significant threat to VMware materializes. It is a question of when, not if, of course. All suggestions for future flavors of the month for cool-aid should be posted to this blog, the flavors the next 18 months or so, thinking of Microsoft for some reason since it took about that to get Hyper-V off the drawing board, are still open for selection. Don’t even suggest Cherry, yuck.

Add comment November 11th, 2008 Schorschi

Hyper-V, No Warp Drive, Struggling With Impulse Power Still

Virtualization Critical Comparison - Chapter 09

Well, I did it, I did the deep dive into Hyper-V, it was demanded, it was not an option, and when I came up for air, I felt like I was covered in dirt, well to be more specific, not dirt. The stuff was brown, but it had an odd smell, and it did not come off easy, too greasy for my liking, more like something unexpected that oozed out of the cracks of a weak impulse propulsion engine design prototype. Which I am not sure was what Microsoft was hoping for? Or was it? Microsoft released something that was better left stuck to the bottom of my pressure suit magnetic boots, rather than introduced into my virtualization lab. But in reflection this is what Microsoft intended, yes, intended. How so? Microsoft wanted to avoid looking like they were standing still, that they had not completely lost the Hypervisor market for at least another 18 or so months?

Look at Microsoft goals and objectives, from the past, they have done this before. They almost never have a winner out the door, but something that is functional, or looks to be anyways. Something that can be built up and refined, and polished into a mirror finish of Brass. Not Gold or Silver, at least not yet. Hyper-V gets you from Earth to Saturn and back again. But if you need to leave the Alpha quadrant, such as VMware has already done? For get it. Of course, along the way, Microsoft gets just enough of a following to make the solution look passable, if not viable at some scale or degree. Look at WordPerfect and Lotus, they dominated their respective markets, and Microsoft took them down hard, over time.

The gloves are off, that is obvious, and Microsoft has VMware in the phasers targeting array, and is locked on. The interesting thing is that VMware is a moving target, at warp speed, so Microsoft is having trouble getting a kill. Microsoft is still not strong on the feature set, no matter how many their sales teams decry the faults of VMware or Microsoft otherwise discounts the gaps in Hyper-V to cool-aid sipping CIOs of firms world wide, VMware is still the best of breed, still avoiding assimilation or destruction. And I am not just referring to VMware VMotion, or what Microsoft has tried over and over to rename as transparent migration. Lets be real, quick-migration is not quick, if any reboot is required, and in Hyper-V, that is what is required, be it planned or as a recovery on a cluster, it is not HA (High Availability) but AHA (Almost High Availability).

Let us dig a bit deeper. If some of that brown stuff gets into your suit vends, sorry about that, but when you decided to read about Hyper-V, you should have expected the brown stuff would get all over the place. The incomplete features that hurt Hyper-V are as follows:

• HA (High Availability), well sort of, Microsoft Cluster running VMs (Virtual Machines) as a resource is not HA, it is really a band-aid, no matter how you look at it.
• QM (Quick Migration), a reboot is a reboot is a reboot. This was discussed above, so no need to itemize again here.
• One VM (Virtual Machine) per LUN? Are You Kidding? What bozo thought this up? Did I say band-aid again? Microsoft does not have a shared IO model that is really a shared IO model. VMware VMFS is not perfect, but it did set the expectation of standard high. Even in the Microsoft Clustering model, Microsoft says that disks should be duplicated not shared, the old X-Cluster versus Y-Cluster argument. Well, even if shared in a Y-Cluster, there are scaling problems, for anyone with 1000s of VMs. Like 1000s of LUNs?
• Networking? It is horrible in Hyper-V, and Microsoft says, well, we depend on vendors to develop new or better drivers. Bull. Microsoft has all the leverage, the design is weak. For example, HP is still working on getting teaming to work right with Hyper-V, to which HP PSP 8.2 will support teaming in Hyper-V. Dell and IBM are silent so far on the issue, stating that they rely on the NIC drivers as provided. But simple teaming to avoid Single Point of Failure (SPOF) is not the issue I am yelling about here, it is the fact that Hyper-V does not do load-balancing or even an active/active pool of NICs, similar to VMware Bond, for me, that just yells weak design out of the space dock.
• You have to have VMs with IDE for boot? But can have SCSI for other VM disks? Dumb. I think Microsoft did this to make transportation between VMware ESX and Hyper-V and back again, which will be a key trick for enterprise scale organization, as painful as possible.
• Microsoft Clustering is horrible. Yes, horrible. When it works it is great, when it is ill or sick, you often will find it easier to take a node offline and shoot it with a phaser, because recovering a node that is whacked is one step short of a digital miracle. How do I know this? I have been supporting and designing solutions around Microsoft Clustering since 1998!
• The entire Hyper-V solution is dependent on MOM, SCVMM and other Microsoft tools, so Hyper-V is not free, in fact, if you need to scale Hyper-V, it is not inexpensive. Anyone look at the cost of SCVMM and MOM? They are expensive and going up in price, with each new version, are they not?

The single most disappointing issue, well more of an architectural concept to be fair, with Hyper-V, for me, is the SPOF (Single Point of Failure) issues. With VMware ESX, to eliminate SPOF, I just double my components, and of course when needed, add an additional VMware ESX servers, maybe one or two or three, etc., and I am done, yes, done. No SPOF, unless I don’t know why physical switches and/or storage processors are. I can have as many physical NICs as the given hardware can handle, and I can map the physical NICs as I see fit. The same for HBA channels to the fabric, as long as I have shared-storage, I am good to go; a Virtual Cluster is just a few clicks away… Make it so, Number One. Microsoft, I need so many other components, applications, and layers of integration, it takes a Xenomorphic degree to keep things straight.

VMware VirtualCenter (VC) server? Sure if you need it, but not required, to achieve about 80% of the benefits of the virtualization architecture we need to avoid SPOF. VC components and features are integrated into one interface, where as with Hyper-V, you really need to use three (3) different tools just to match VC for configuration tasks, lets see, 1) the Cluster Administrator, 2) SCVMM, 3) Hyper-V MMC plug-in, because? Why Cluster Administrator and SCVMM don’t talk or play nice, you end up in Hyper-V MMC, to figure out what the heck has gone sideways. I will not itemize the other features in VC, which would only kick more of the brown stuff into the Microsoft fans suit cooling system, while they are reading this. Now I ask you? Does Hyper-V look that appealing, or is this a case of the old adage…You get what you pay for?

True, I am not the biggest fan of VMware, and at times I hammer VMware in this blog, to get things right, to improve, to be a better steward of the virtualization cutting edge, to boldly go where no, man, cough, virtualization company has gone before, sure. But compared to Hyper-V? This time I am going to praise VMware, a bit. VMware has a better solution; it may not be perfect, from my perspective, but compared to Hyper-V? VMware has achieved warp drive, where Hyper-V is still stuck on impulse drive. Maybe Microsoft should hire a few Vulcan technical experts? Hey, I wonder, VMware, Vulcan? Is it just me? Or is there a connection here? How many years did it take Star Fleet to get out from under the guidance of Vulcan? Vulcan developed 2nd and 3rd generation warp systems under the support of the Vulcan Science Directorate, I mean VMware developed VI 3 and 4, while Microsoft is still trying to get out of the space dock without training wheels. It was 3, maybe 4 years at least, right? Well Microsoft has already had 3 years already, and the training wheels are still welded to the side of the hull of Hyper-V, at best it is embarrassing for Microsoft, at worst, again for Microsoft, Vulcan, cough, I mean VMware has escaped the Borg, yet again!

1 comment November 4th, 2008 Schorschi

ToutVirtual Announces Support for Microsoft Windows Server 2008 Hyper-V Virtualization

Carlsbad, Calif. - October 27, 2008 - ToutVirtual, an emerging leader in optimization and performance-management software for virtual computing infrastructures, today announced that its VirtualIQ suite of products now also supports Microsoft Windows Server 2008 Hyper-V virtualization.

The VirtualIQ suite of products is designed to support virtual server room operations through three stages of virtualization - design, deploy, and deliver stages - helping users make correct decisions for virtualization optimization along the way. The suite of products allows users to compare how various virtualization platforms, such as Hyper-V, perform running different applications and then provides visibility and policy-based control in managing the Hyper-V based environment. VirtualIQ suite of products supports multiple virtualization platforms for an apple-to-apple comparison and provides all essential decision-making data in a single, integrated web console that is simple to install and use.

ToutVirtual is partnering with Microsoft as an Independent Software Vendor (ISV). Users of Microsoft Windows Server 2008 Hyper-V can get more information about ToutVirtual VirtualIQ at: http://www.microsoft.com/virtualization/partner-profile.mspx?id=81

“We are pleased that ToutVirtual is delivering support for Windows Server 2008 Hyper-V,” said Jim Schwartz, Director for Virtualization Solutions at Microsoft Corp. “ToutVirtual’s VirtualIQ products help our customers running Hyper-V assess and optimize applications for their environments, assisting them in making the right planning decisions.”

“ToutVirtual is excited to strengthen its relationship with Microsoft with the support of Windows Server 2008 Hyper-V,” said Vipul Pabari, chief technology officer for ToutVirtual. “Hyper-V users can use our products to compare the performance of applications on multiple virtualization platforms. Whether the user is just getting started in the design phase or further along in deployment or delivering advanced services, VirtualIQ suite of products is simple and cost effective.”

Pricing, Availability and Platforms Supported
More information about the VirtualIQ suite of products is available at:
http://www.toutvirtual.com/downloads/downloads.php

About ToutVirtual
ToutVirtual, Inc. is an emerging leader in virtualization system optimization software to manage and automate virtual computing processes and ease the transition from design to deployment. VirtualIQ, the company’s flagship product suite, allows organizations to obtain a holistic view and control their virtual infrastructure including servers, applications, storage, and clients independently of the underlying virtual computing platform. Unlike other companies whose products are vendor specific, platform specific, or network tier specific, ToutVirtual software operates across multiple platforms and is multi-tier to prolong product life, protect IT investments, and maximize ROI. Additional information about the company and its products is available at http://www.toutvirtual.com

ToutVirtual and VirtualIQ are registered trademarks or trademarks of ToutVirtual, Inc. All other marks and names mentioned herein may be trademarks of their respective companies.
###

Add comment October 27th, 2008 Administrator

VMworld This Week! What Will The Wizards Have For Us This Week?

Virtualization Critical Evaluation - Chapter 08

This week, 09/15/2008, is VMworld 2008! VMworld is always fun, sometimes more hype than fact, other times lots of facts, and minimal hype, you just never know which you will get. This is a good thing, it keeps everyone guessing, at least to some degree. I have been at every VMworld event, in the US, so far, and always enjoyed hearing the discussions in the hallways. For example, VMworld in Los Angeles, every one was talking iSCSI, the very first VMworld event, in San Diego, everyone was talking VMotion and ESX 2.5.0. VMworld 2008 should be no different, some thing will be buzzing. The question is what? ESXi? I do not think so. After the initial flop of ESXi, or ESX 3i, I should say, VMware needs a few winners, to freak out Microsoft, if for no other reason that it is fun to freak out Microsoft.

Microsoft has still missed the target on a number of issues, but they will address these. The question is can VMware exploit, oh, bad pun, these before Microsoft eliminates them? The issues I see with Hyper-V are as follows:

• No VMotion function, yes, quick migration exists, and that is fine if you don’t need transparent migration, but most server virtualization does need this feature, as transparent as possible. VDI (Virtual Desktop Infrastructure), can survive without instance migration, no?
• Hyper-V, due to its design, has a different performance model, the VMBus will never quite be as good as a dedicated Hypervisor, such as ESX. Microsoft strategy is, get it as good as we can; which is worth the effort, but close is not better.
• Security is an issue with Microsoft, more so than VMware, not because there are so many instances of Windows out there, but because a generic operating system can never be as secure as a dedicated appliance or structured solution such as ESXi or ESX full. I don’t believe hyper-jacking is are threat yet, but it will be a threat for Microsoft sooner than VMware ESX, duhe.
• Is Microsoft worried? Of course they are, why else did they certify VMware ESX 3.5 Update 2, finally? Microsoft is looking like for the 3rd time, in about 5 years, they are a dollar short and day late, more than that, really.

As I am traveling to VMworld 2008 I will be thinking about what my expectations and what my wishes for this VMworld will be. There are things I believe I will be expecting:

• We need ESXi to be identical to full ESX installation, in reference to monitoring, and alert status reporting. Complete features set, such that ESXi installation with traditional agents must be identical to full ESX. Everyone at an enterprise level is struggling with this one, so ESXi will never grow to its potential until this is resolved.
• We still need better archival/disaster recovery solutions. VCB is not living up to its potential, I liked the idea of VCB, but it still does not scale. Array based snap solutions like Avamar from EMC or the similar solution from NetApp, are still complex, hard to manage, and just a pain to implement. This is insane, and I believe the key issue for 2009, we have larger and larger VMs, but no realistic way to archival them.
• We will need USB support in VMs on ESX. Security/License dongles of course. KVM dongles, yes KVMs, since new KVMs continue to add USB features, this is becoming an issue where key sites standardize on a type of KVM, the operational teams want the VMs interface to be identical, emulated in software but same look and feel. Maybe VMware should join forces with Avocent?
• We still need IDE and SATA, emulation in VMs on ESX. How about SAS for VMs on ESX? Are any of these realistic? Maybe not, but to show clients a one-to-one emulation, is still requested over and over. VMware workstation does it, so ESX should too.
• VMware Thin-Disk support in GUI for VirtualCenter? Better yet, Disk Imaging as well, were we can use a core OS volume, and VMs actually run delta off the core OS volume? Windows 2008 is going to drive this requirement; a full Windows 2008 install for server, Microsoft recommends 40GB just to start? Oh, never happen on VMs. More realistic for static VMDKs? Maybe 20GB.
• VMware SRM (Site Recovery Manager)? We already know what this is, and how it works, but is anyone but every large enterprises going to implement this technology? Only time will tell. The real power of SRM is on a WAN scope, disasters in the same city are not the issue, it is mega events, storms, terrorists, etc., that will impact an entire city complex, lets get disaster recovery and load balancing of datacenters 1000s of miles away, not 10s of miles. Doing this is cross country, cross nation, that is the realistic need, but this is not cheap or easy. How will VMware solve this one?
• What is VMware doing to block Xen or Solaris, as they emerge to run INTEL, x86, 64bit VT, etc., and run other operating systems other than their native OS?
• What is VMware doing to improve its Application Instancing? How about a brand new product? VDM, and appliance VMs are not going to offset application streaming or Citrix next generation solutions.

I have one last question for VMware, before VMworld 2008 is official, and powered up…Where are the cheerleaders at? If you don’t understand the reference, look at my discussion in this blog of VMworld 2007, where I state that ESX 3i is nothing but hype, where I discuss some disappointments about ESX 3i, or, sorry, ESXi.

1 comment September 15th, 2008 Schorschi

Fear for the Sake Of Fear? Hyper-Jacking Myths?

Virtualization Critical Evaluation - Chapter 07

I ran straight into a brick wall this week. Not to worry, no injuries, much to the displeasure of some I am sure. However, I do this upon occasion, and some times I enjoy it, some times I don’t. Walls, of the virtual brick variety, in the computing industry are unique, because they often are built upon bad ideas, bad assumptions, bad conjectures, and mortared together with bad logic. Moreover, it has been my experience, that if you pull just the right brick, just the right way, then the entire structure is exposed as flawed, and comes down like, well, a ton of bricks onto the original builders. Hyper jacking myths are such brick walls. Hyper jacking is reality? Or is it? What are we afraid of? We know some day someone or some entity will hi-jack a hypervisor? Right? Or do we? Depends on who you asked, but to some, it is all the rage in the press right now, and has been since about the same time last year. Even the motives for discussing threats to hypervisors are suspect, based on my research. Although, not an article, but a thread, that illustrates the discussion of hyper-jacking as motive driven, is http://www.wilderssecurity.com/showthread.php?t=179419. And of course, Google is just spilling over with topical discussions, about hyper-jacking, but seem to be all words and no real substance. There is so much misinformation about what can and can not be done in reference to hacking that security teams both public and private are having nervous breakdowns just trying to understand the risks and threats never mind formulating plans of action or as some like to call it, remediation of risk. One of my favorite articles that does nothing more than generate fog, or mild panic, is http://rationalsecurity.typepad.com/blog/vm_hyperjacking/index.html. Unfortunately, this article is misleading. The key virtualization platforms that dominate the industry have been certified and vetted, against known methods and techniques, something this article, among others,never explains and thus never provides a balanced view of the issue. Of course, no one is secure against new techniques and methods, but this article does not explain that point well either, it raises questions, nothing more.

From my perspective, I have never liked the term, remediation, it smacks of re-active tasking. And mediation alone is still perceptive classification as post facto resultant state. But, then again, just what is a threat? When is a concept become a threat? Or even more than theory? We have so many threats to our existence, virtual reality is no different, but fear of threats that may never materialize? I am not saying you don’t plan for threats, but what I am saying, is that threats are just that, potentials, not impacts. What was the key concept in the Matrix? Neo had to believe he was the one, the reality, in all that was, is, and had been dominated by virtual reality? Or to explain it in a historical context, what is the quote…We have nothing to fear, but fear itself . So why you ask, did I say I hit a brick wall? Fear of what could happen was the wall this time. Fear has caused may a good idea to die on paper. Security teams are exceptionally susceptible to this scenario, the fear of what could happen, no matter how remote or indiscrete. Opinion can make what is rare or even not probable; appear to be a rather solid, a wall to success use of technology, no less. The specific wall that I ran straight into was built by a group of security experts, which had the best of intentions, but had a serious lack of foundation, just fears based on potential issues, causes with no realistic materialization that could someday be effects, nothing more.

For example, just because there maybe someday, a virus that can attack NICs directly, and just because someday someone may hyper-jack a hypervisor, therefore, virtualization is not as safe as traditional hardware. To repeat, because of the potential to be hacked at the physical NIC, or the virtual NIC, or the virtual switch, and subvert the hypervisor, by definition, virtualization is not as safe as traditional hardware. To which my response was…What? Do you know what circular logic is? These are potentials not realities. So you decide what reality is, based on what casual fantasies you see as potentials? That is like saying you never will use a horse and cart to deliver carrots to market because the horse might die from pulling the cart, because the cart may break down, because the left rear wheel may fall off, because you loaded 27 carrots not 270? Therefore, horses are dangerous?

Trying to get to the bottom of this line of thinking as urban legend or not, I found a number of articles that all discuss what the new threats to corporate entities must be, yes I say must be, because the articles all promote their authoritative position, with little or not objective explanation. One that stood out as such, hyperjacking the latest threat to servers. To me this type of article is less than useful, it hints at a possibility and nothing more, talk about true hype, just to get a hit on a web page? Looking for threats is fine, but we don’t see a flood of articles discussing real results, real attacks, now do we? Hackers tend to brag about what other hackers have done, Even if some professional hacking group, from China or Russia, has done, or some defense contractor has done it in a lab, or worse in the real world, the word would get out, in short order, that is fact, the web is horrible for hiding the truth, just as it is horrible at only reporting the truth. At the risk of being connected to the X-Files franchise…The Truth Is Out There…if it exists, the internet always gives up its secrets, even if, the secrets are buried under a ton of garbage.

What is the goal of some of these articles? I am not sure, other than to generate headaches, and as I said before, generating web page hits. The results however, are real, when this type of hyped vague popinjaying is believed; inaction due to threat results, and is a classic psychological warfare technique, as well. Bully for the conspiracy theorist. A less militaristic context for the same basic situation is called analysis paralysis. The lack of action because of fear of the consequences of taking said action. This is never rational or logical? Good point, when is a potential issue deemed a true threat that needs to be acted upon? What an idea, action to offset threats, rather than inaction. This is a good thing, because when action is taken; the technology is actually used, leveraged, what have you. So how do we put this into the context of a security strategy and still use the technology? This is not trivial to be sure, but a large dose of common sense is a key to rational success.

First, threats will continue to emerge and develop, just as methods to eliminate threats will be soon to follow, if not offset by changes in theory, design, and implementation. The only constant here is that change always happens. Second, being offensive is not always possible, and being defensive is reasonable, as long as the technology is used, at some point. Early adoption of technology is a risk. However, never adapting a technology for us, is just giving into the fear of potential threats. Third, trust that ideas, solutions, methods, etc. to combat real attacks will come, or can be implemented before impact occurs. Fourth, no matter how good, how strong, how extensive the strategy implemented to protect the environment, there will be some event or situation that will compromise the environment at some time. To believe otherwise is foolish, or worse to believe that it is impossible, is irrational.

Keeping the above concepts in mind, and looking at the hypervisors of today, which are safer than others? That is a rational question, a proactive one, not a reactive one. Saying that all are unsafe is not rational. True, all have some level of risk, but so does using every operating system, and we still use operating systems in everything, from servers to cell phones. Never using an operating system, because it may someday be hacked, is defensive, and irrational. Focusing on the proactive aspect, and agreeing that hypervisors should be used, there are a few basic design features that offset risk, not quasi threats.

• An embedded solution is safer than any generic operating system based solution. This is straight-forward. Operating systems, in the traditional sense, have a large surface area of attack because they are designed to be flexible. Flexibility is difficult to manage. Embedded models are focused functional elements. Easier to manage.
• Hypervisors should be designed to never allow themselves to be executed by themselves in abstracted context. This is obvious no? It takes only a few lines of code to validate that a hypervisor is hosting a hypervisor. This is true of multiple vendor or different vendor stacking, say Hyper-V refuses to execute ESX, and ESX refuses to execute Hyper-V. No hypervisor should ever host another hypervisor, therefore, nesting is forbidden.
• Never violate the context of function versus access. What does this mean? It means that virtual instances should never have access to the hypervisor, nor know they are hosted, and the hypervisor should never inform a virtual instance that it is hosted. There should never be any inter-process communication between virtual worlds and the framework that hosts the worlds. This is a pain at times, because we sometimes want to cheap as developers, but don’t do it.
• Never ask a hypervisor to be a firewall. This is similar to the point above, but is an external design issue to the hypervisor. Never connect the hypervisor management functions to a less secure environment, than the virtual instances. This import in the case of a DMZ environment, but should be true for any environment. Is it a real risk? Today maybe not, but if someone, some day actually does figure out how to effectively hyper-jack a hypervisor? The environment design should be strong as possible and still be useful.

There is one guaranteed way to address fear, specifically the fear of hyper jacking as security teams today feel they must. However, these same security teams need to understand, they can not deny or disqualify solutions because of fear of the unknown or the future. They need to understand a concept that I coined in a meeting about 4 years ago, Engineering-By-Fact. This is based on a concept a former boss, of my boss at the time, coined, Management-By-Fact. In explanation of Management-By-Fact and Engineering-By-Fact, as well, this boss of my boss stated that he never wanted anyone in his organization say…I Think, I Believe, It Should, Most Likely, Maybe, or Most of the Time, or any variation of the same expression of opinion, in the same sentence as voicing a solution to an issue. Solutions, like problems, are black and white, if you don’t know for a fact, that you have the right, correct, and factual solution, or even worse you really do not understand the problem, don’t venture opinion over facts. Now, if we could only get the people which express opinions as facts to honor this concept, the poor security teams out there, which seem to be afraid of their own shadows, at times, could spend less time building brick-walls, and more time configuring fire-walls, right?

2 comments September 10th, 2008 Schorschi

VMware Really Hurting? Or Just Really Bad Timing For A Simple Mistake?

Virtualization Critical Evaluation - Chapter 06

Is VMware really hurting, meaning are they coming a part of the seams? Or was the latest licensing bug issue just bad timing? It was a minor code mistake, but a major perception mistake. For something that happens all the time in a code development shop, more often than a non-coder would care to understand. Just imagine you are the one that make the mistake? Just imagine you are the one that missed catching the mistake? There will be some careers that will end or at least be derailed for a while, at best, at worst, change lives of some with dramatic impact. Is this fair? No. Is it really reasonable, maybe it is given what VMware must now do to recover from this perception of running a lose development shop. No one is perfect, but the world expects perfection, and more than the perception of perfect. VMware is not perfect. No software publisher is. However, the real issue here is not the mistake that was made. But why the mistake was made. Perception has real significant impact. Why do I say this?

Some time ago, just short of a year ago, VMware promised a specific group of enterprise customers that the licensing model would be, and I believe I quote, “we are discussing options for making the licensing model more informational, rather than enforcement oriented.” Those in the room, there were some 50 or 100 of us that heard this asked almost in one voice…When? Where? How will this passive licensing model be implemented? VMware at that point became very vague. In fact the topic never came up again. This is the real issue, because if VMware had owned up to this promise, at least I saw it as a promise as a time, as good as a promise, and then the impact of the August 12th bug would not have been the fire drill it was. VMware seems to be doing damage control as a matter or routine throughout 2008. Was the exit of Greene not a type of damage control?

Even the evaluation version of VMware ESX OS has a 60 day try it window before features are disabled. Now what an interesting idea! The commercial version of License Server would generate events and warnings but not actually disable functions for 60 days; this would have avoided the issue no? Never mind the fact that I think that the Flex License Manager solution is horrible, I am just not a fan of restrictive licensing, and I have never been impressed with Flex, it has a very long and negative history depending on who or whom you ask. And, yes I know all the issues and debatable points that surround software piracy and theft. So the following questions come to mind? First, just how many Lawyers will get sacked at VMware for the August 12 issue? Lawyers, yes lawyers. This is terrible, sad even, because I am sure some heads will roll across the floor and down the stairs, out the door of VMware. I hope the heads migration, includes the entire brain trust that thought proactive enforcement of licensing was a good idea, I bet it was a lawyer that initiated the idea! Am I wrong? VMware say something if I am.

Just how many of senior management will get nailed? After all, there are serious issues with VMware quality if this is a trend. Blaming Greene is a cheap shot at this point. Lets be honest, there is a growing trend in the entire information technology (IT) industry to release solutions to the customer that are flat out incomplete, broken, or worse pushed out the door because of a fixed deadline. The quality assurance process, I believe, is seen by the marketing, sales, and even top management, as an evil thing, that holds back solutions from being released. After all, most customers pay for support, so we, the customers pay twice? Once for the product as concept that is incomplete, and again for getting issues fixed that never should have gotten out to us in the first place? You bet, your sweet posterior, you do. Just how long should a list of known issues be, to be deemed reasonable? My eyes almost bugged out of my head when I read the release notes for ESX 3.5 Update 2, very long, does not install a sense of confidence? All I could think of was… If the known issues list is this long, how long was the issues list that they actually fixed? And what did they miss?

Guess the issue was not big enough for the CEO of VMware to make a public statement? Maybe at VMworld 2008, in the key note address, someone at VMware will do the right thing, and state that VMware will and has improved quality assurance methods and processes, so customers are not impacted in a similar manner in the future? How about a known issues list that is only 5 items long in total, or less than 10 items in total? That would answer a lot of critics, including me. And go a significant way in the positive direction to answering the question…Is this the end of a bad series of missteps for VMware? Or I sincerely hope this is not the case…Is this just the next incremental step in a longer trend, before VMware goes down in flames?

1 comment September 2nd, 2008 Schorschi

Emerging Technology Is More Painful Than Management Wants To Believe

Virtualization Critical Evaluation - Chapter 05

There is a trend in the Information Technology (IT) industry that is well known, well understood, and the truth about it, well hidden at the same time. Not because it is held under non-disclosure-agreement, not because it is patent or copyright owned, but because no one wants to acknowledge it. The fact is that the concept of faster, better, cheaper is foolishness in the IT industry. Moreover, it is an excuse, not a strategy. Real work, real quality in design and development takes effort, and exactly what management does not want to here, real time. Good work always takes good time to do.

Want an example? VoIP (Voice Over IP), it is horrible, to be honest. The voice quality is one step short of ridiculous, the quality and consistency is nothing compared to good old analog. I am now on my third vendor for VoIP, and considering looking for a fourth, if I could or even can. And I bet you thought I was going to illustrate that virtualization would be the example a technology before its time? Well, give me a few minutes, maybe I will.

Compromise is the dominate theory in the development in the IT industry today. I dare anyone reading this blog to say otherwise. I dare anyone to debate this issue in an honest, rationale, and objective manner. Change context for a minute, where are the 5, 10, 15 or even 20 year strategic plans for product development? How is the IT industry really going to go green, and not just pay lip service to the concept of green for decades? The IT goals of 20 years ago have not been very effective now have they? The paperless office never really got off the ground did it? For years, laser printers generated more volume of print on page than the total book publishing houses, world wide, ever did or do? Want an example? Purchase a new automobile, there are still somewhere between 25 and 40 pages printed for getting out of a dealership. Where did we go wrong? Such great ideas, such poor implementation? Why, it all comes down to compromise of ideals, goals, and objectives. Why is it that all the vendors only have roadmaps that are 12 or 18 months long? Because no one is thinking long-term, no one wants to make real commitment in a specific direction, Edison labs did what, some 6000 attempts to get a better light bulb? I doubt the typical technology firm that develops a new PCI device or USB device does more than 100 code builds once alpha code is locked, before market release.

The IT industry, reference to product and solution development is a mob mentality. Baby steps all the time, not leaps and bounds. This is not to say that the IT industry as not done some wonderful things. It has. Computing technology has revolutionized many fields of science and technology, in ways, that even 40 or just 20 years ago would have been closer to a Star Trek series, pick your favorite and insert here episode, than what even I could have dreamed up at the time, and believe me, I have some crazy ideas. Just ask my friends! But am I making my point? How many great ideas have been scrapped because someone was unwilling to wait for the solution to be realized? Or never wanted to commit resources, financial and temporal, to the solution when it was nothing but an idea? How many ideas never made it to a napkin in a coffee shop because time and resources were impossible to get? Or, and this is frightful, how many solutions have been strangled in the validation and certification effort phases, because some one, some where, in the chain of command, was unwilling to wait? Because someone refused to believe in potential because someone could not see short-term profit? Imagine if Edison, Tesla, or even von Braun, von Neumann, Babbage, and sorry, almost forgot, Einstein, had a project time line that strangled their explorations of thought, never mind, prototyping in a lab.

I experienced this first hand as a very young support technical resource person. I was assigned to an evaluation team, for a new software application, this was about 15 years ago. We worked day and night to improve the application in question. Feedback to the developers, feedback to us from the end-user population that was doing alpha testing, then beta testing, and even, finally, release candidate testing for the application, it was wonderful to feel progress, that a quality solution was near completion. However, there was not enough time, we just did not move fast enough it seemed. For whatever reason, even though real progress was made, we were not making everyone happy. The timeline for release was predetermined by management more than a year before; the first line of code was typed. This was a new experience for me. I was taught, both at home and in school, you do the job right, you do the right thing. Meaning, in my innocent, rose-colored glass view of the world, in the IT industry world, that if the application was not quite done, you delayed release, you got it right, then, and only then, you released it. Quality was the key to success. The key to true massive profitability, I was quite wrong, or so it seemed at the time.

The lead project manager walked into the end-user test lab, early one morning with no warning and began red-lining the project test plan. Days and weeks of regression, system integration, and component validation testing just disappeared from the master timeline. We would miss real bugs and real issues by doing this. I was about half way done with my breakfast, when I got the news from others on the evaluation tea. Why was I eating breakfast? Well I had been up all night chasing a nasty bug in the code, trying to isolate the issue, so the developers could move forward sooner rather than later. When I happened to see the project manager, after he had done the nasty to the master time line. I asked…What is this? Are we closing the project? The reply was…No. We are selectively shifting features to the planned version 2.0, rather than version 1.0 release. Of course, being young, and lacking tact in political scenarios, I asked…What about those features that were agreed upon with the end-users, our customers, our clients, before we started? How will you explain that significant features are really there, but not enabled because they are incomplete, when we are so close to being done? The answer, and I am being explicit, was…The application must be released on 15 days; we will never train the end-users on the additional features that were not validated, we will never acknowledge that some features have been dropped from 1.0 release. I was in shock, I was confused, I felt betrayed. This decision just did not make sense. Why? We still had some time, but they shortened the total timeline, some 5 days ahead of the original planned release. What the heck!

Now, you may be asking, why I explained this story from so long ago? Because I now realize, 15 years later, that this theory of product release is so ingrained into the IT industry, at all levels, that it is killing the industry. Management in the IT industry is under so much pressure to make things happen, on a strict mathematical schedule, with no exceptions, no flexibility. Thus all the true creative effort and the artistic aspect of idea development and design are dying out. The ugly aspect of this is that quality is something you get with version 3.0, which actually costs the customer or client even more. Look at Windows? Was it not 3.1 that really was functional to any reasonable degree? What did early adopters do? Spend a ton of money on Windows 1.0 or 2.0?

When was the last time we had any true, knock your socks off, quantum leap in the IT industry? 15 years, or 25 years, or more? Is it not true? Nothing new under the sun should be slogan for the IT industry. Tell me I am wrong? I am not bitter about this; I am not even surprised by it…any more. I was at a technical conference recently, sponsored for the most part, by one of the big three hardware vendors. Which one it was is not significant to my discussion here, but what every technical session screamed at me, was, yes, you guessed it, compromise. In ideas, in design, in implementation, and the attempts to compare these just average products to the competition only reinforced how all three vendors are in lock-step with each other, with solutions that are so close in capacity and function, that picking one over the other is almost insignificant. Of course, we all know the one that offers the cheapest cost, will be declared the best, quasi faster and better. Great, faster, better, cheaper is back!

A number of things have contributed to this, out-sourcing, why own when you can leverage? Lowering of educational standards, hey, expecting results above and beyond the average is not fair; you might damage some below average student esteem, rather than encourage improvement and achievement? The lack of large firms willing to develop talent, create careers, versus steal talent, only to let it go when out sourced? Why invent, when you can purchase? The Japanese still work according to 20 year or longer timelines, they expect achievement, but they also commit to technologies that seemed logical in reference to maturity in future years, not quarters. Ask General Motors? They have had more than 30 years to get something on the table, to really change the world, and they have failed, and a 100 year firm is all but dust. Look at Toyota? They are only just now peaking on plans established more than 30 years ago. Just imagine what the computing industry would be like if that type of effort was done? Don’t like automobiles as an example? What about fuel? Brazil has done better than most Countries along the same idea, did someone yell sugarcane?

And how does this have anything to do with virtualization? It is simple and easy to see, if you take the time to look. We have so much computing power compared to the past, cores upon cores, that we over purchased, over scaled, and under use it to the point, that an entire new segment of the IT infrastructure was created and now dominates said IT industry, and it is called virtualization. What is really stupid is that it is hypervisor virtualization, not application instance virtualization that dominates now. Why, because we want to achieve, faster, better, cheaper of course! Hypervisors are a result of faster, better, cheaper, mind set in virtualization. Virtualization should have only resulted in flexible environments, not utilization redirection. Iif all of those project managers, developers, designers, etc., years ago, took just that extra bit of time and effort, to do something right before hand, then there would be no afterwards, no emergence of virtualization, as we know it today. No outrageous cost avoidance because the environment would have been lean and mean. No zealous endorsement of…faster, better, cheaper. Well, at least not in the IT industry.

I am putting my rose-colored-glasses on e-bay, I have just enough faith in the future, that there must be someone out there that needs them…I hope.

Add comment August 22nd, 2008 Schorschi