Archive for February, 2008

Does Virtualization Reduce Staff?

Virtualization, Fine, Well Sort Of? – Chapter 06

In a word, yes. Ok, this article is done, right? Wrong. It may surprise you which staff you must maintain in order to maintain your virtualized infrastructure. However, I hope this discussion does not surprise you! There are a number of approaches to supporting an infrastructure; I am going to suggest one strategy that has tactical elements. This strategy is not cheap, cough, inexpensive. There are a few places where any organization just can not global source completely and expect to get quality results. You must have high quality associates or resources in a few key places:

  • Engineering
  • Operations
  • Problem Management
  • Project Management

Engineering is a key area, these resources must understand technology, emerging, architecture, and beyond the vendor, rah rah, roadmaps, they should also come from operations at least in part. Product evaluation by the vendors that publish said products is dumb, although there is a very strong trend to doing this, because it promotes reduction of engineering staff. Never make a decision on beta code or even release candidate code. Never implement general release code if you can avoid it. The trend is to be an early adopter, the concept of late adopting seems to be lost in the dust of the past. Funny, how no one wants to tabulate TCO in the context of late versus early adoption impact costing. The total time allowed to select, evaluate, develop and certify solutions, no entire architectures, continues to shrink, and yet I continue, as one observer, to hear about the bad results of this trend, when engineers are forced to make decisions on release candidate or worse beta code exposure. Some things can not be rushed, something that management never wants to acknowledge. What is the old joke? The manager asked the engineering team, “What is the number one thing I can do to improve the quality of your work?” An engineer speaks up and says, “Give us the time and resources to do the job right.” The manager replies after a short pause, “What did you say? Sorry. We need to rap this up so you and I can both get back to work; I have operations team coming in next to explain why they are having so many issues.”

Operations teams are virtualized in nature, smaller, non-site specific, lots of talent based on fewer brains. People world-wide, recently I called a vendor on a specific issue, and was transfer from the US, to Canada, and then to Costa Rica, yes, I went from low level customer support, to elite team support, to engineering for the vendor. No, this is not outsourcing, or global sourcing to use the newer term. Thus, these new operation teams you have will rely on external resources to be sure. But talent in-house, focused on the house, if you get my implication. No matter how good global sourcing, you must have good internal operational support. Quasi teams, more strike force in nature, interchangeable parts or cogs, experts in chronic issue resolution, root cause analysis, not to mention very good trouble shooters and problem solvers. Focusing on resolving the immediate impact, and then after the issue is resolved, worked-around, etc., figure out the long term solution. The focus is reduction of immediate impact first, not always the reduction of total impact. Real solutions often take real time. Odd, that operations management understands this, but engineering management does not? A solution that takes time to test and implement, opens the door for the problem management team.

Problem management might be the most misunderstood. Problem management is not the job of your operations team. Problem management is not the responsibility of the engineering team; however, many firms see this as the case. Let me explain, you are paying your operations team to support the environment, and the engineering team to create and improve the environment. The project management team to take what engineering team does and roll it into the environment in a way that the operations team can handle. Therefore, problem management team resolves problems by leveraging the vendors, to get issues addressed, and only then validated as viable by the engineering team, and ultimately adopted by the operations team. It makes no sense to have the engineering and operations teams debating with the vendors, developers, etc., until there is something concrete and solid for them to evaluate or support. To be direct, the project management team has to have balls. There is no other way to say it. If your project management team does not have the strength and intelligence to tell management where to go, when deadlines and deliverables are insane, then your engineering team can not get superior design work done. This in turn nails the operations team, since a bad design once implemented is horrible to support. And, your problem management team is buried with lots of rework or issues that never should have left the project evaluation/test effort. Unfortunately, I see this aspect of the total model is often an after thought. Weak project managers are the norm not the exception in many firms, and the results speak for themselves, customers are not happy, management is not happy, and most of all the technology teams are left holding the bag, from nose to tail, engineering gets blamed by the operations team, etc.

Of course, there is some overlap between all the teams at various points in the life cycle of a solution or even the development and recovery cycle of issue resolution, but that should be informal and synergistic in nature, not expected by default. Oh, did anyone notice, only now I have mentioned the word, virtualization, again? I guess the concepts are valid no matter how far they are applied from virtualization, no?

Add comment February 26th, 2008

The Risk of Virtualization

Virtualization, Fine, Well Sort Of? – Chapter 05

Let us consider the following situation…You walk into a conference room, before you are a group of your key clients, lines of businesses, what ever is applicable to your situation, and you ask the following questions:

  • What percentage of your technology infrastructure is virtualized?
  • Are you happy with the performance of virtualization?
  • Do you consider virtualization a higher risk? If so, Why?
  • Do you consider virtualization a lower risk? If so, Why?
  • What is the true risk of this percentage of virtualization?
  • Does the phrase “All Eggs in One Basket” meaning anything to you?
  • Is the risk of virtualization worth the threat of virtualization?

What do you think the answers will be? Are you going to be happy with the answers? Are your clients going to be able to answer the questions? If they can not, is it your fault or theirs?

What is the percentage of your technology infrastructure is virtualized? This is a straight-forward question, or is it? Does your firm or infrastructure provider use virtualized storage? Does your solution use hypervisor or application virtualization?  Is your network virtualized? VLANs, QoS, etc., What? What the heck is that? Hypervisor and/or application virtualization, which is safer? Many clients have no real understanding that every information technology resource they contract for is virtualized in some way. It may be that what you think are LANs, are VLANs. It may be that what you think is physical infrastructure is, but not completely, what if you network or storage provider has their entire command-center tool set on virtualized instances and they are suffering from the same issues you are? You just don’t know what is virtualized below what you see as infrastructure.

Are you happy with the performance of virtualization? This is also a straight forward question, maybe the only straight-forward question in this entire discussion, or virtualization in general. From a quantitative perspective, providing you have rather extensive analysis methods, you can provide your clients with hard facts, from P2V, V2I, I2I, I2V, I2P, V2P, etc. But from a qualitative perspective the client is going to believe what they believe. Of course, from the client perspective, you get what you pay for, so if you can live with the performance virtualization provides, versus the cost of not using virtualization, this relatively straight forward. Did or does your process for virtualization candidacy force your project managers and design personnel to do proof-of-concept validation of virtualization? Or do you just trust the application vendors?

What is the true risk of the percentage of virtualization? Higher Risk? Lower Risk? We are not talking labs or non-critical environments but core production, the mission critical, the five (5) nines (9) realm of up time. No, not disaster recovery on virtualization, but true production, customer visible, if offline, you lose real money. Do your clients really understand this? Or did you just, ah, sort of, gloss over this point? This is not an argument for or against virtualization, because depending on your respective analysis, design, and implementation, this true risk could or should be less than traditional hardware, no? Transparent migration, i.e. VMotion for VMware, and/or stand-by hosts, or just available capacity on existing hosts? Not to mention network redundancy methods, 3DNS, Big-IP, etc. Storage redundancy infrastructure or methods? Argues the point that virtualization should have lower risk than traditional hardware. However, if you cut corners, or have non-shared storage, did not make the commitment to redundant network and storage fabric implementations? Then the risk of virtualization is higher than traditional hardware for sure, if you do not believe this, keep smoking what your are smoking. It is rather ironic that many clients do not really understand that they have many eggs in one basket on every virtual host server. The virtualization industry knows this, HA, DRS, SRM from VMware, and similar technologies or methods under development by every major virtualization solution provider, just screams this point no? Why was the biggest AH! At VMworld 2007 transparent host redundancy? Microsoft has learned this lesson a bit late, but to be sure Microsoft will never make that specific mistake again, of course I refer to the lack of a transparent virtual instance migration, or VMotion like feature set.

Taking all these questions and associated issues as potential impacts to your environment, is the risk of virtualization worth the thread of virtualization to your financial success? Returning to our conference room, let us up the stakes, what if it is really not a client conference, a board room meeting? Do you want to be the one that has to defend the use of virtualization when something significant goes wrong? Or worse, defend that you short-changed, or rushed to virtualization implementation? It does give pause to you does it not?

Now consider the concept of risk in virtualization, taking into account that hypervisor based operating systems are rather immature compared to traditional operating systems. Are we all just dodging the bullet? Will this risk get us in the end? Real issues for virtualization are and continue to be those of an immature platform, I will not exhaustively itemize these issues, but just think back to the early days of Windows or Linux, and you will recall the things that gave you pain, no? Rush to market code, inexperienced code development, hardware vendors struggling to understand a software operating system that is a moving target, etc., etc., I recall these and more in every operating system ever developed.

The bottom-line in avoiding risk is before your virtualization plan is invoked, why? Because regardless of how good the virtualization platform or your design around said platform is, it is people that designed it. If you have not given your people the time, resources, and had the patience to allow a superior design work to be done, your virtualization infrastructure is a house-of-cards. Once it is implemented, no matter how well it is supported, if the design is bad, the risk you are taking is bad. So I ask again, do your clients know this?

Add comment February 12th, 2008


February 2008
« Jan   Mar »

Posts by Month

Posts by Category