By Dave Welch (@OraVBCA)

Tech Target’s Beth Pariseau asked for my opinion on Oracle’s recent Nimbula purchase announcement. She then posted an article titled “Oracle's Nimbula IaaS buy sparks open source speculation” on March 14th.

I always appreciate the opportunity to contribute to Beth’s pieces. On this occasion, I opted to give her my input in writing only. As I’ve thought about it, I think the OVM concerns that I shared with Beth are important enough that I’ve chosen to post my entire response to her. I’m posting it as-is, resisting the temptation to expound on how it relates to Oracle’s March 20 announcement of its hardware/cloud sales decline.

March 13, 2012

Beth,

It’s always great to hear from you. Quite frankly, I’m surprised I didn’t hear from you on the occasion of IBM’s announcement last week that they were committing to OpenStack. I’m inclined to believe that’s a much bigger deal than Oracle’s Nimbula announcement.

Oracle is floundering with virtualization. Oracle’s been on its Xen project for a full six years now (assuming it put at least a few months of prep into its OVM OOW 2007 announcement) and still hasn’t produced anything we consider remotely usable.

I’ve been verbal from the start about how VMware needed a technical and financial competitor on the Linux side. When Oracle announced OVM at OOW 2007, I was dearly hoping that Oracle would be that competitor. Over five years later, I’m still waiting for Oracle to step up.

Oracle announced OVM 3.0 at OOW 2011 with a lot of fanfare. Within two days, VMware blasted its partners a 15-point technical competitive analysis based solely on downloadable OVM 3.0 documentation. I thought VMware’s analysis totally missed the mark. I have maintained for years that OVM chose the wrong open source platform in Xen, and that would continue to be the case until Xen got rid of the parent partition, otherwise known as Dom0. The three Xen issues—any one of which was a bigger deal for me than anything on the OVM 3.0/vSphere partner compete sheet—were: 

• Tier 1 workload I/O & network scalability: difficult to accomplish when the same device driver code has to live and function in both the base native Linux install image as well as the parent partition virtual layer.
• The parent partition is inherently anti-HA as there are lots of required Linux patches that require a full down of the environment yet have nothing to do with virtualization.
• Security analysts agree that the ultimate way to reduce a security threat at the end of the day is to reduce the size of the attack surface. The installation of the parent partition mini-Linux image on top of the base native hardware image bloats rather than reduces the attack surface.

Oracle’s long-standing Senior VP of OVM/OEL Wim Coekaerts announced in his May 2011 blog post that the parent partition (Dom0) was finally going away as of the 2.6.39+ kernel. My hopes soared. At the time, I said that we were probably still a hardware depreciation lifecycle away from getting any respectable OVM GA adoption on the street. Meanwhile, Red Hat would have to muck with it and release. Then Oracle in turn would muck with it and release.

A year later in May 2012, Wim posted that OVM 3.1 with Dom0 was finally GA. Just a month ago, House of Brick followed Oracle Consulting Services into a shop that was trying to take a scaled workload live within days on Oracle E-Business Suite with RAC on OVM 3.1. OCS had done its own OVM configuration per its best practices for the customer. What we found was anything other than stable or even remotely usable. We retrofitted vSphere 5.1 per the customer’s plea, and they went live with the corrected stack days later. Takeaway: a year after Wim’s May 2011 Dom0 post, I’m regretfully predicting that we’re still at least a hardware depreciation life-cycle away from OVM becoming anything usable under Tier 1 workloads.

So what’s going on at Oracle? Wim Coekaerts won my respect a decade ago with his team’s engineering and support of Oracle Cluster File System. I wonder if this has anything to do with Wim and his team’s capabilities and rather wonder if his team is the victim of resource constraints. To me, it sure looks like another indicator that Oracle’s leadership’s primary objective with OVM may have been trade rag competition. Oracle’s leadership can’t be blind to the accelerating hemorrhage of accounts optimizing their Oracle red stacks on VMware, with the associated crippling reduction on Oracle processor-based licensing revenue.

Given the choices that Oracle appears to have made to-date and its current in-the-mire status with OVM, I think that for Oracle to bag Xen and go with OpenStack would have to be an improvement for two reasons: 

• Anything would be better than current state OVM.
• Any other corporate entity overseeing design and development of the hypervisor and surrounding tooling would be an improvement.

I wouldn’t be at all surprised if Oracle went there.

Regards,
Dave

By Dave Welch (@OraVBCA)

I advocate that IT management at all levels and administrators of peer technical disciplines read the Oracle Database Concepts Guide’s Introduction chapter. Yes, that includes the C-level. That read will go a long way in preparing you to participate in database-related discussions at any level. Although this is beneficial if your organization deals with Oracle databases at all, it will prepare you to understand and accept a qualified architect’s suggestion that running Oracle business-critical workloads on the vSphere platform can provide remarkable benefits with minimal if any risks.

As I post this, it’s interesting for me to introspect on how little my mentoring activities have shifted since I authored my first version of this document. This is the centerpiece of guidance I developed and field-tested over seventeen elapsed years of Oracle DBA team building and mentoring.  If you intend to become a DBA, study my selected subset of the Introduction chapter as if you were preparing to take a test on it. In 1995, one of my protégés started with the ingredient goo that evolved into this guidance table. He went on inside a year to become a successful consultant working for Oracle Corporation.

Why am I recommending a document and release version that ostensibly became obsolete years ago when Database release 10g went into mainstream adoption? Certainly not as an excuse to save time by repurposing existing collateral. I just rewrote this approach table from scratch. I find the Database Concepts Guide’s Introduction chapter got cumbered as of release 10g, and the nice self-contained introductory chapter is no longer what it used to be. I’m not saying I could have done a better job authoring the newer Concepts Guides. You can return later to more recent versions’ concepts guides to spot look up newer features as needed. But I believe the benefit of a coherent, consolidated read exceeds the risk of wasting time in older release documentation. It can be difficult to find quality vendor-provided documentation. Oracle’s documentation, at least up through 9i, is a notable exception.

The 9i Concepts Guide’s Introduction chapter is 67 pages. In the approach guidance table that follows, I’m suggesting that you read only about two thirds of that chapter. Your first read needs to be slow and careful to understand the concepts. Then return to review the concepts sufficient to suit your purposes.

The following approach table follows all the way through for DBA candidates’ needs. Non-DBA candidates will know where short of that to stop. DBA candidates should follow this overview with a study of the 2-Day DBA guide first published with Database 10g. I have my own selective approach to that document as well, which is beyond the scope of this post.

Here is your syllabus:

Oracle9i Database Concepts
Release 2 (9.2)
March 2002
http://docs.oracle.com/cd/B10501_01/server.920/a96524.pdf

By Dave Welch (@OraVBCA)

The most impressive session I have taken in from the last two VMworlds was 2011 US session BCO2874 “vSphere High Availability 5.0 and SMP Fault Tolerance – Technical Overview and Roadmap”. This session was reprised at VMworld 2012 as session INF-BCO2655 “VMware vSphere Fault Tolerance for Multiprocessor Virtual Machines—Technical Preview and Best Practices”. Don’t miss these sessions’ live demo of a 4 vCPU workload failover.

I’ve been watching Fault Tolerance (FT) eagerly since Mendel Rosenblum did a live demo of FT alpha code at VMworld 2007. Alas, FT’s 2009 GA release has been little more than bait for future capability to House of Brick customers as none of those customers’ Tier-1 workloads fit inside FT’s current single vCPU limit.

I wonder if VMware’s executive leadership has any idea what it is sitting on with SMP Fault Tolerance (SMP FT). I’m wondering if SMP FT could turn out to be the most disruptive technology anyone has seen in years. SMP FT certainly threatens a massive disruption to the clustered relational database market. I make that prediction due to two key SMP FT features that Oracle RAC can’t touch: approximately single-second failover, and no client disconnect.

The VMworld 2012 US session included engineering’s confession that the alpha code still has very substantial performance latency (they use the word “overhead”, which I am replacing with “latency” for clarity). I got the impression that they’re quite worried about the latency, and wondered if they even considered their current performance numbers fatal to a GA release. Here’s a screen shot of their latency numbers (at video minute 35):

The presenters then refer participants to the vSphere 4 FT Performance Whitepaper (“Section 3. Fault Tolerance Performance” beginning on p. 7) for further elaboration on their performance measurement methodology.

I’m betting there are plenty of extremely HA SLA-sensitive workload owners that will be more than happy to tolerate that much latency. For example, look no further than the stock exchanges where extremely distributed transactions must commit all transaction phases within a few seconds to avoid substantial revenue loss let alone regulatory penalties.

A corollary to Moore’s law is Intel executive David House’s prediction that chip performance would double every 18 months. I’ve read clarifications by other parties arguing that chip performance actually doubles every 20 to 24 months. Fine. Let’s call it 24 months. In his 2006 blog post, John Sloan extends that math to a 30-fold performance improvement each decade. (Related to that, my non-scientific mental survey tells me that less than 10% of House of Brick’s performance diagnostic engagements find CPU-constrained workloads.) In light of the fact that we are swimming in a world of CPU performance and it just keeps getting better, how obsessed do I imagine I’d be with SMP FT’s latency if I were the CTO of NYSE or NASDAQ? Not very. I could stand up my SMP FT-protected workload in a hardware refresh and still run circles around any x86 chip vendor's offering of just two hardware depreciation lifecycles ago.

Accordingly, I suggest the SMP FT product and engineering teams gate the product’s dot-zero release based on code stability only. Subsequent to that, I suggest the engineering/product teams prioritize their engineering efforts as follows:

1. Stability
2. vCPU horizontal scalability
3. Latency (and a distant third at that)

During the 2012 VMworld SMP FT session Q&A, Jim Chow answered a question by saying that SMP FT engineers were still considering the subject of the question in their architectural discussions. That answer gave me the impression that SMP FT’s GA release was probably at least a year away at that time. Despite the wait, I believe enterprises would do well to evaluate SMP FT’s business continuity promise now and get project planning underway accordingly.

As I find my social media chops, no doubt I’ll post more frequently with less content per post. But meanwhile…

I’m told that Oracle’s Larry Ellison also began contributing to the Twitterverse this summer.

House of Brick’s Oracle VBCA Boot Camp Sunday 9/30 came off better than I expected, and I had high expectations. I think the attendees feel good about it, too, based on their end-of-session group chorus calling for more. I’m inclined to put extra weight on this particular group’s feedback. They were more interactive than most groups and therefore challenged me in helpful ways. I’ll be channeling the feedback into House of Brick’s SQL Server boot camp Tuesday November 6^th in Seattle in conjunction with the SQL Pass conference.

I’m declaring Colin Bieberstein of Husky Energy – Calgary as my boot camp guest of honor. I got the impression that Colin may have made significant personal sacrifices to attend on just two weeks’ notice.

As for Database 12c, I’m pleased that the announcement made before Larry Ellison took the stage aligned with what I’ve been already telling everyone. A few months ago Larry predicted the 12c GA release this December or January. That was never going to happen, minimally due to the increasing complexity of the release. Add to that the coding and QA challenges Oracle faces with what has always given the appearance of involving an Exa code branch.

The announcer said sometime in calendar year 2013. I’ve been telling everyone not a minute before June but more probable toward December 2013. I’ve also been saying don’t look for release stability worthy of production systems anytime before the middle of 2014. That’s not a hit on Oracle. It is just the nature of the beast with code this complex underpinning business-critical systems. I continue to be in love with their red stack software, and sincerely hope that the love affair continues.

As for Database 12c’s multitenancy (pluggable databases on top of a consolidated instance), I firmly believe it is tooling the wrong way at the wrong stack layer. That was my immediate reaction on December 19^th, 2011 when I first got wind of the release’s architectural direction. On the contrary, I have been preaching for three years an architectural direction that I have dubbed “atomicity.” That is, in vSphere environments, move toward an alignment of one database instance, middle tier component, or utility per guest OS.

Atomicity accomplishes two major objectives. It makes architecture with tinker toy components much easier for those building initial stack prototypes and with less technical administrator involvement. Think vCloud Director. That accelerates time to market. Smaller memory/CPU workload alignments also dramatically facilitate live migration. That facilitates dramatically better utilization of an Oracle processor-based license, which in many cases can be a stack’s most expensive component.

I am constantly challenged in my travels by hardware-centric DBAs and System Administrators justly concerned about the prospect of managing four to ten times as many Oracle executable and OS instances. I tell them that 100% of their vSphere-experienced peers that I interact with in shops that know what they’re doing with vSphere say they would never go back. The model is to allocate a fraction of the savings that vSphere provides to patch automation tooling.

Database 12c is attempting virtualization not one but two layers higher in the stack than the vSphere platform that accomplishes virtualization in spades. When architecting system stacks, always assume to begin the discussion by tooling at the lowest layer of the stack possible, unless there are very compelling business and/or technical justifications to do otherwise.

Oracle VBCA is predictable when you know what you’re doing. I tell people that the one-time replatform from big iron to x86 is the hard part, minimally because, unlike us, enterprises don’t do it every day. I’m convinced the RISC UNIX->x86 hop is the single most important thing organizations can plan for in their move toward the world’s premier platform vSphere. Accordingly, Jeff Browning’s replatforming preso will get my nod for OOW best of show. He sleuthed out the fact that RMAN CONVERT’s sys.dbms_backup_restore endian translation used to work just fine into any x86 platform. That is until three years in October 2008, when patch 13340675 “fixed” it to only work with Exadata. You can also pick up Jeff’s preso recording at VMworld 2012.

Jeff invited me to lunch yesterday. The more time I spend with Jeff, the more humbled I am by how big his heart is. Thanks, Jeff. I continue to benefit from our professional and personal friendship.

The VMware 2012 Pavilion is appropriately themed with vFabric at every turn. Charles Fan is a humble, self-effacing man who would never bring up the incredible things he’s done for Joe Tucci. Charles’ vision for what is now called vFabric is really finding its voice, and that’s clear as you stroll around the VMware pavilion.

Yesterday vFabric’s Bill Bonin shared perspective with me--a month after VMware’s Chief Performance Officer Richard McDougall did the same thing--on how big the in-memory Hadoop market is projected to be in just a few short years. Guys, thanks to both of you for your one-on-one attention to make sure I keep Hadoop in my periscope.

VMware, I need you to align vFabric Data Director with vCloud Director strategically if not integrate them technically. Yet they are being marketed independently and have segregated internal organizations. I’m at a loss to explain the lack of product alignment as I now add the vFabric Data Director message to what I have always felt was VMware’s best tooling: was Lab Manager is vCloud Director. There’s an apparent massive operational deployment intersection here. This is no different than my incessant internal harping to you years ago that Lab Manager and Stage Manager were the same thing, despite the fact that they were separate code bases. You eventually merged Stage Manager into Lab Manager. C’mon, vFabric Data Director was announced a year ago. As one who provided architectural guidance into what was code-named “Aurora” and as a lightning rod evangelist for VBCA, I’m struggling to articulate the cogent unified vision for these separate products. It’s got to be even harder for your prospects to capture that vision. It’s going to get even worse now with your phenomenally shrewd acquisition of DynamicOps.

I ran into a former Oracle RAC employee yesterday who confirmed what we always knew. The statement that got deleted from the published My Oracle Support note (off the top of my head) “There are technical restrictions that prohibit the certification of RAC in a VMware environment” had to do with clock drift. (Clock drift went away with vSphere 4 in 2009).

Ron Zellars from the world’s largest ice cream factory - Wells Dairy - stopped by the VMware pavilion to say hi. We are proud to have helped them over a year ago with their EBS R12 upgrade and replatform of RAC to vSphere.

Brian and Richard from the U.S.’s largest appliance manufacturer - G.E. Appliances and Lighting - also stopped by to say hi. Brian’s business card now sports “Chief Evangelist” because CTO Lance Weaver thought it was cool on my card when GE and HoB got introduced here last year. I may have to declare Brian’s business card title as my biggest professional accomplishment for 2011!

Having said that, my most enjoyable encounter of the day was with Wize Commerce’s DBA Selina Lin out of San Mateo, CA. Selina took in my “Oracle RAC and VMware HA Tooling - A Decision Tree” VMware theater preso but couldn’t return Tuesday for the replay of my “Business Critical Applications Performance – VM vs. Native” preso. So we found a couple chairs just outside exhibit hall doors and took our time with that preso one-on-one. Selina, our ½ hour working session made my day. All the best to you.

My esteemed colleague Cisco’s Tushar Patel invited me to lunch today. Tushar was the VMware-side engineering force behind HoB’s groundbreaking VMware Oracle Solutions Lab at OOW 2007.

My only VMware 2012 pavilion complaint: inadequate white board space. Thank goodness for the iPad Paper app!

And speaking of apps, Uber is the coolest most practical iPhone/iPad app I’ve seen lately! Map all the available drivers in your geo and summon up the nearest one with a click without placing a call to a cab dispatcher.

Run Uber on the iPad 3 you’re going to win when you physically present Tuesday and Wednesday in the VMware Pavilion at 5:30. Your odds are pretty good as I’m thinking there just weren’t that many bodies present given the size of the daily prize.

To close out this mega-post: I was asked yesterday for feedback on a partner strategy event earlier this year. I answered with a concept that I’ve introduced into my professional activities from my volunteer teaching experience: “Never let planned content interfere with a productive discussion.”

There is some discord and significant misinformation in the Oracle community as to what the contractual obligations are associated with licensing only a subset of a vSphere cluster’s hosts for Oracle. Furthermore, there is all manner of opinion out there regarding the risks of asserting one’s contractual rights. This definitive opinion piece waters down relevant Oracle License and Services Agreement language to a summary that you can get your hands around.  It includes observations on what we are seeing and, more importantly, what we are not seeing in terms of legal actions with regard to the OLSA.

I just offered a fairly comprehensive Oracle-on-vSphere licensing opinion piece to my colleague Jeff Browning who is EMC’s chief spokesman for all things Oracle storage-related. My post appears in Jeff’s Oracle sub-cluster licensing thread at April 2nd.

Jeff then took the initiative to replicate my post to his own blog here and the EMC Community Network here.

Storage Performance Metrics

This blog entry is the continuation of the blog entry dated August 5, 2011

Average Read Time

• 20ms
• Sustained peaks of >20ms for no more than about five minutes

Attributes:

• Un-cached random reads
• Assuming 90% read, 10% write, with writes colliding

The Average Read Time performance attribute is important because there are schema/application designs that can pierce the cache, regardless of how big that cache is.

Average read time can be pulled from Oracle Statspack and AWR reports at the tablespace and datafile level.

Read time can also be pulled from the vSphere Client:

Select a host or a VM in the navigation pane > Performance Tab > [Advanced] > select Data Store > select Real Time > click the Read Latency box.

Spindle Busy Average

< 50%

To run R/esxtop to display device, kernel, guest and queue latency start an r/esxtop session:

$ esxtop > v > [Enter]

Look at the %USD column.

LUN percent busy

$ nmon > d

SCSI Queue Depth

<= teens (after having reconfigured all storage path devices to 128, up from default of 32)

The default SCSI queue depth for ESX is 32.

$ esxtop > d > f > [Enter].  Verify that QSTATS is selected with an asterisk to the left as pictured below.

Storage Latency

For day-to-day monitoring of disk throughput, latency is the ideal metric. It is more accurate in determining if your database is suffering from I/O throughput issues. The performance redlines for disk latency should be 20ms or lower. With latency it is acceptable to have burst periods of higher than 20ms latency, but not consistently.

R/esxtop monitors disk latency at three distinct layers, the device or HBA, the kernel or ESX hypervisor and the guest or virtual machine. Use the free IOmeter storage benchmarking tool in a Windows guest (only the Windows version can do asynchronous writes, not the Linux version).  Configure the tool for the Oracle database default block size and a 32GB streaming write (large enough to pierce the write cache of many SANs).

To run R/esxtop to display device, kernel, guest and queue latency start an r/esxtop session:

$ esxtop v > f > h > i > j > enter

• The GAVG or guest latency is actually a combination of the device latency + hypervisor + any additional guest OS overhead.
• The KAVG is the kernel latency or hypervisor latency. High latency reported at the kernel can be due high SCSI queues or device drivers.
• DAVG is the device latency or HBA latency. We use the term HBA generically, including an actual fiber channel HBA or iSCSI NIC. Device latency usually indicates a bottleneck at the storage or SAN layer.

By monitoring the four distinct layers we can clearly see the HBA is an issue with the SAN configuration. If just GAVG was high then we would have had looked at the Linux guest. This customer had very serve device latency (DAVG). This related back to a SAN issue that was later fixed by the storage vendor with a firmware upgrade.

Storage Throughput

Given a 4Gb Fibre Channel fabric, you should be looking at >100MB/sec/storage path sustained. If you are getting less than that, consider splitting out storage paths.

CPU Ready Time

The VMware definition of CPU Ready is, "the time a guest waits for CPU from the host while in a ready-to-run state" (VMware ESX Server 3: Ready Time Observations - http://www.vmware.com/pdf/esx3_ready_time.pdf)  We also refer to CPU Ready Time as the ”guest heartbeat.”

We generally monitor CPU Ready Time through VI client.  ESXTOP can also be used.   However, we prefer the VI client because it measures CPU Ready in milliseconds.

Select the virtual machine > Performance tab > click Advanced > click Chart Options... > CPU > Real Time > and select Ready

It is normal for a guest to average between 0 - 50ms of CPU ready time.  Anything over 300ms and you will experience performance problems. We’re comfortable with up to 300ms CPU Ready Time on average, with a high water mark of 500ms.

ESX Memory Oversubscription

Do not oversubscribe memory shared by Tier-1 Oracle workloads.  VCS should report no ballooning.

Oracle is very aggressive at using all of the resources available to it. When an Oracle instance starts up, it allocates memory for the Oracle SGA. This allows Oracle to use a contiguous space of memory for caching of data and SQL statements. As long as the instance is running, it will not give the memory back even if blocks are unused. Oracle memory is less dynamic then CPU. It tends to level out after the Oracle SGA has been allocated. You may see brief periods of spikes when the PGA is being used for a RMAN backup, but other than slight anomalies it tends to stay static. As with CPU, measure memory during peak workloads.

We recommend memory sizing by using the following 3 tools:

• Oracle SGA Advisor.
• Oracle PGA Advisor.
• OS metric collection (for example, nmon)

Start with the Oracle advisors first to determine if the SGA and PGA are accurately sized. We consider the SGA critical to virtual machine sizing. If, for example, the Oracle SGA is undersized this will affect your buffer cache hit and shared pool, negatively impacting performance. It will also skew your OS metric collection. After the SGA and PGA are sized accurately then use the OS collected metrics to determine the appropriate memory size of the virtual machine.

Allocate enough ESX physical memory to at least cover the Oracle SGA, and preferably as large as the sum of the SGA, the PGA high water mark, and memory used by the shadow processes.  If you use reservations, make sure that there is enough extra physical memory in the cluster that the failed over VM will not refuse to start.

In vSphere Client, select the VM > Performance tab > Advance tab > Chart Options… > Memory > Real Time > Active

Network Performance

We confess that we rarely run into network configuration issues that impede VMware Infrastructure performance.  Just the same, we’ll offer a standard.  There is always the possibility that a workload changes its virtual machine memory blocks at a high rate.  A workload so hot that it generates dirty memory blocks faster than the proprietary vMotion interconnect can move in thirty seconds is justification for beefing up the interconnect hardware capacity through faster NICs (or converged technology), teaming NICs, or both.

iperf Network Load Test

Conceptually the network load test should be used to force the max throughput of the network fabric and interfaces. Similar to the memory it flushes out any potential drive issues, hardware issues or network topology issues. This test is to validate performance and push the max throughput of the interfaces to determine if there are driver issues or stability problems.

Iperf is a network testing tool. It is able to create a TCP or UPD data stream between two nodes (or virtual machines). Iperf is open source software available for both Linux and Windows at http://sourceforge.net/projects/iperf/. Below is a simple test that runs for 500 seconds testing the maximum throughput of the network interfaces.

Server Node

$ iperf -s -i 5

Client Node

$ iperf -c -t 500 -i 5 –m

Dropped Packets

For network packets on a LAN, you ideally should not see dropped packets. Dropped packets typically indicate congestion in the network or failing hardware. One percent dropped packets in either direction can throttle throughput by as much as 15 percent.

In vSphere, dropped packets can be monitored by selecting the ESX host > Performance tab > Advanced > Network > Real Time > select None in Counters > then check Receive packets dropped and Transmit packets dropped

As we move forward, there are more layers in the system stack. Virtualization is being introduced for a whole bunch of reasons, each of which is justification alone in my mind for doing so.

From my viewpoint, virtualization blends the lines somewhat between the various IT administrators’ functions. I see that as a good thing for the admins, for IT, and for the business overall. This could be a huge paradigm shift in particular for some shops we’ve worked in where the various IT teams barely communicate between themselves.

My first-steps formula for moving in this direction:

1. Select up to ten performance attributes from these hardware resource groups that I list in what is usually descending order of importance: IO, memory, CPU, and network.

2. Adopt metrics for each attribute.

3. Adopt a handful of performance tools freely available or bundled with native Linux or Windows.

4. Author and promote elevator-pitch summary conceptual overviews of the performance attributes, the needs they fulfill, and your associated SLAs.

5. Regularly publish your performance metrics to the organization including the C-level and business users.

I believe there is significant benefit to IT proactively educating superiors and business peers on these performance attributes, unilaterally suggesting associated SLAs/red lines, and freely making the information available. Rather than self-inducing pressure, such information liberates. It naturally removes organizational barriers. It goes a long way to removing fear, uncertainty, and doubt. When appropriate, it brings more focus and leverage to IT CAPEX discussions including budget exceptions when necessary. Last but not least, it can demonstrate that VMware Infrastructure is for all intents and purposes transparent.

I’ll publish some key performance attributes, straw-man metrics, and suggest a handful of tools in a follow-up blog post.

A few weeks ago an industry colleague asked me that question. My answer was immediate: enhancements around VMDK.

This blog entry is not about any new specific features of the code name MN release.

And no, I’m not talking about enhancements to the VMFS file system. It wouldn’t be much of an exaggeration to say that about the only evidence of VMFS's aging is how many calendar pages have flipped since its last revision under ESX 3.5. No doubt VMFS yet has room for improvement, just as the ESX kernel scheduler had room for improvement when it was completely re-written more granularly for vSphere 4. As the GA VMFS release stands, it’s already capable of a scant 100 microseconds of latency per IO in the hypervisor, and that with linear scalability.

What I’m talking about is the absolute certainty that the list of features that require VMDK storage will continue to expand. That’s going to put all the more pressure on organizations that are worried about keeping a rapid V2P rip cord handy next to their virtual tier-1 production Oracle databases.

Direct-mounted storage and RDM both hold out the possibility of a very rapid replatform to native. Or better yet use either storage paradigm to leave production on vSphere where it belongs and hot-fork a branch to native.

As of 4.1, here’s my list of key features that are only enabled by VMDK, listed in what I would say is their descending order of importance:

• Lab Manager/vCloud Director

• Storage vMotion

• Storage IO Control

• Fault Tolerance

The odds of getting into a Sev-1 yelling match with Oracle Support over vSphere’s presence in a production stack are largely mitigated by proper system stack architecture. Our CEO Nathan Biggs wonders if it is a coincidence that in all the years we’ve been doing this, no House of Brick customer has ever needed to V2P their production Oracle system for support purposes.

VMware is hardware to Oracle. Our team's day-to-day experience has me thinking that  a major purpose of the replatforming language in Oracle’s VMware support statement may be to serve as a legal hedge in the unlikely event that anybody were to ever encounter a VMware hypervisor problem that actually induced an Oracle bug. It doesn’t surprise me that we’ve been seeing a significant up-tick in the number of enterprises determined to virtualize the entire stack.

From my view, there is no significant body of street experience to justify pre-emptively avoiding VMDK out of fear for Oracle Support. To that subset of shops that say “We’re not virtualizing production without a rapid V2P insurance policy,” I’d still say being virtual with restricted storage tooling beats the heck out of being stuck on native. Better yet, if you stick with VMDK, you’re in an expanding crowd of good company.

Side notes on the VMDK-enabled features mentioned above:

Fault Tolerance: I yawn in the Tier-1 space today anytime FT is mentioned. Precious few of our customers’ Tier-1 workloads' can accommodate their high water marks with a single vCPU. But in some future moment when someone crosses my palm with a SMP FT announcement, in that same moment FT will hold out the prospect of high availability fail-over SLAs of just seconds. As such, SMP FT will demand to be re-evaluated against much more expensive, complex, incumbent RDBMS clustering technologies.

Lab Manager: yes, the vCloud Director source code branch allows multiple data stores under a single configuration of VMs. But I’ll keep promoting Lab Manager until its vCloud Director cousin is enhanced with linked clone capability which I consider critical for the pre-production product development lifecycle. And I’ll continue to advocate that shops pull together single highly-performant data stores underneath Lab Manager until then.

As for Storage vMotion, Storage IO Control or even SAN vendor FC/Solid State dynamic reallocation algorithms, I see great value in each of them. I’m also certain they are not a wise substitute for poor or missing storage architecture. Rather, they should be viewed as nice insurance policies on top of properly-architected storage.

It's official.  Yesterday Oracle announced support for RAC 11.2.0.2 or newer on VMware.  No exclusions or conditions.

I laud Oracle Corp. for their decision.  They are going to sell more RAC as a result of removing the RAC-on-VMware support restriction.

House of Brick has been relentless in the promotion of RAC-on-VMware.  If RAC is appropriate for a workload, then every aspect of the stack's product lifecycle can be optimized when virtualized on what I consider to be the world's premier platform.

House of Brick did our first production-grade RAC-on-VMware implementation in May 2007 and I spoke on it at VMworld later that year.  I dedicated an entire session in April at IOUG 2010 to RAC-on-VMware and the results of our Break/Fix lab.  This is probably as good a time as any for me to come clean and admit Break/Fix Lab was a misnomer because there is inherently nothing broken about RAC-on-vSphere.

With this, I'm going to find the time to get my six minute video summary posted of our RAC-on-VMware Break/Fix Lab.  I promised the attendees of our EA8700 session at VMworld 2010 that I'd get it out there, but it kept getting back-burnered.

So, if Oracle Corp. now acknowledges the advisability of removing the VMware support restriction for the world's premier High Availability RDBMS and their most complex database offering, what does that tell you about the viability of single instance Oracle on VMware?

I’m happy to report that House of Brick’s most recent event was back in Omaha—our Solid Foundations seminar December 4th.  My presentation was “Oracle DB 11g R2 New Features”.  My slides are available here.  This presentation is not a comprehensive list of 11g R2 DB new features.  Rather, it is an overview of those features that have caught my eye from our perspective of involvement with our customer base.  There is at least one feature in here that isn’t in the New Features Guide: the new Result Cache, that Oracle placed in the Shared Pool, of all places.  I intentionally held off in reviewing Tom Kyte’s must-see Oracle Open World ’09 presentation “11 things about Oracle Database 11g Release 2” until I was done with my own new features compilation.

Our Omaha customer/prospect base may have noticed we’ve been a little quiet lately.  Lately we’ve been guilty of distraction, and that’s a welcome problem.  On the one hand our local customer base continues to expand.  On the other hand, our local customer base is getting to be a smaller piece of our total business.  Moving forward, we anticipate we will supplement our Omaha-based Solid Foundations live events with webinars to connect with our expanding customer base and prospects out of our local region.

Years ago we faced another welcome problem: House of Brick live events were drawing large crowds while the Nebraska Oracle User Group had as few as 15 at some events.  The NOUG board proposed a merger of the events.  We were humbled by NOUG’s overture and we agreed.  Since then, our Oracle business has grown but at the same time become a smaller piece of our overall activities.  Because of this, we suggested that the NOUG re-evaluate if it makes sense to continue to fold into our Omaha live events as Oracle is getting proportionately less of our presentation bandwidth and indeed is no longer even our primary topic.  NOUG leadership agreed and have announced that they will combine with the Heartland Oracle Applications User Group (OAUG)’s Omaha events.

When I asked Friday’s Solid Foundations crowd how many shops were running Oracle RAC, easily a third of the hands went up.