By David Klee (@kleegeek)

In our world, monitoring an IT infrastructure is defined as having accurate, up-to-date knowledge of the current state and health of critical servers. Sending alerts based on monitoring metrics means that administrators should be notified when something is unhealthy and that admins can (and should) take corrective measures. However, people sometimes blur the lines and alert on items that should not be alerted on. Alternatively, overlooking critical issues and not alerting on them can destroy a business.

When we set up monitoring on business critical systems (specifically VMware, Windows, and SQL Server), many key metrics are monitored. Occasionally we store them for historical purposes. These items include host and guest CPU, memory and disk activity, disk used and capacity, and event and error tracking. Sometimes we will run in-depth SQL Server performance data collection and collect items such as Page Life Expectancy, Buffer Cache Hit Ratio, recompilations, data or log file autogrows, tempdb usage, and other important items to profile and trend.

Most of these items are initially thought to be important enough to set an alert However, is there really anything you should do about your CPU running higher than average for five minutes? What if my Page Life Expectancy falls below some predetermined threshold for ten minutes during a backup? What if I miss an alert that I care about because other alerts were bulk deleted and this alert was in the middle of the list?

At House of Brick, we have two levels of alerting: warning and critical. A warning causes a notification to go out, but sometimes it is more informational. Other times the notification is something to be investigated during the next business day. Either way, these items are not critical and administrators should not be alerted in the middle of the night. Critical alerts, on the other hand, are very important. These alerts are for items that simply cannot wait. Even more important is that the alerts are delivered in a timely manner. If a business critical server crashes, the business demands that it get resolved as soon as possible. The following items jump to mind as examples for critical alerting:

• SQL Server error states 17-25
• Core services are stopped
• Database integrity problems
• Scheduled job failures
• Disks approaching full in the operating system or datastores approaching full in VMware
• Host memory ballooning that is not resolving itself
• Active Directory domain controllers failing to replicate to other DCs
• And the obvious server or device failure

Warning alerts are less critical. Examples of these conditions are:

• A sample benchmark query is running a bit slower than average, but still returns a recordset
• SAN to SAN replication is backed up but still transmitting
• VMware host memory utilization is over 80% but below 90%
• A virtual disk is approaching 80% full

Other alerting is subtler: What if a scheduled job runs long? What if you have ten times over the average inbound database connections? Even more subtle: What if my primary data loading scheduled job is taking twice as long today as it did six months ago? What if my database growth has an organization that will completely run out of available space in four months? How do you detect these sorts of trends? How do you classify these? Monitoring and alerting should be well thought out before going live. A full list of devices, performance metrics, error states, and scenarios for your specific environment should be developed and then analyzed. They should be configured by resource type so that the critical alerts are appropriate for the underlying cause. Critical alerts should all be items that you welcome being woken up in the middle of the night or interrupted on weekends to handle so that they cause the least disruption to business. They should be something you can actively do something about to fix. Do not send out critical alerts on items that the administrators cannot take action on immediately. Monitoring your environment also means that you proactively review the environment for baseline changes. Are things running slower today than they were six months ago? If so, can you quantify specifically how much of a different exists? Have you automated the analysis and have the reports automatically delivered so you do not skip the routine checks?

You should be monitoring your environment and capturing performance statistics. You should be alerting on items that are of high importance. You should be proactively protecting your business.

This week while going through some old VMware documentation, I discovered a reference to performance gains while using Windows Large-Page allocations. It is located on page 11 of the Performance and Scalability of Microsoft SQL Server on VMware document (http://www.vmware.com/files/pdf/perf_vsphere_sql_scalability.pdf). However, while it claims performance of SQL Server improves when Large Pages are enabled, the document is from the ESX 3.5 era. What happens to the performance on a modern system like vSphere 5.0 and SQL Server 2008R2? If there are benefits, is it worth the drawbacks? Talk to any seasoned SQL Server DBA and most will tell you to avoid Large Pages unless Microsoft explicitly tells you to enable it, but most do not have solid reasons or experience backing their opinions. Why the FUD?

What are Large Pages?

First, let’s talk about Windows memory. Windows uses the Virtual Address Space to manage its memory map. These Virtual Addresses are stored and managed by Windows in a structure called Page Tables. Each Virtual Address has a corresponding Page Table Entry in the Page Table. This table has to be searched when the OS references memory pages. To increase performance of this lookup process, the processor maintains a cache called the Translation Look-Aside Buffer (TLB). Memory is stored in the Virtual Address Space in two block sizes – Large and Small. For this conversation, I’m ignoring the Itanium platform, which has different block sizes. Small Pages are 4KB in size. Large Pages are 2MB in size. When Large Pages are enabled, the number of entries in the TLB is dramatically reduced. More translation is required to reference Small Pages than Large Pages, and a penalty in performance is felt.

Large Pages and VMware

VMware ESX and ESXi hosts can accommodate large page requests of a guest, and this feature has been supported and enabled by default since ESX 3.5. The host CPU must support hardware-assisted memory virtualization (i.e. Intel EPT Hardware Assist and AMD RVI Hardware Assist). VMware has published studies describing Large Page performance improvements of guest workloads at the following URLs.

• www.vmware.com/files/pdf/large_pg_performance.pdf
• http://www.vmware.com/files/pdf/techpaper/vsp_41_perf_memory_mgmt.pdf
• http://communities.vmware.com/docs/DOC-6912

VMware claims up to a 19% performance gain in SQL Server 2008 by enabling Large Pages in SQL Server and enabling hardware-assisted memory virtualization in the CPUs. Reference http://www.vmware.com/files/pdf/perf_vsphere_sql_scalability.pdf for more details.

Large Pages and SQL Server

Trace flag 834 allows SQL Server to use Large Pages for the Buffer Pool. A few restrictions apply.

• Windows Server and SQL Server must both be 64-bit.
• The service account that SQL Server is running under must have the “Lock Pages in Memory” privilege set.

Previously, Enterprise edition was required to support Large Pages natively. However, trace flag 845 on appropriately patched servers (reference MS KB 970070 for more details) allows Standard Edition SQL Servers to join the party.
To add the trace flag into the SQL Server startup parameters list, start the SQL Server Configuration Manager and add the following entry into the SQL Server instance Advanced Startup Parameters property: ;-T834.

When the trace flag is set and the SQL Server service restarted, you will notice that the ERRORLOG now contains entries for ‘Using large pages for buffer pool’ and the amount of memory used

Make sure to set the maximum buffer pool memory by configuring the ‘Maximum server memory’ setting to prevent this problem from ever occurring.

Drawbacks

Enabling Large Pages might give a boost to performance but does not come without downsides. First, VMware’s Transparent Page Sharing (TPS) is rendered ineffectual for this virtual machine. A 2MB memory block is almost guaranteed to be unique, rather than a 4KB block, so do not count on TPS to save any memory on the host. If a host is overcommitted, this could reduce performance of other guest VMs because the balloon driver could be more aggressive in ballooning for memory reclamation. However, best practices dictate that a host not be overcommitted for business critical systems so this should be a nonissue.
Second, verify the boot time of your SQL Server when Large Pages are enabled. SQL Server must claim and zero out the memory it will be using before the service can completely start up. This process takes time. It can potentially take quite a while. This configuration can take a SQL Server startup time from a minute or two to over 30 minutes, depending on your environment. Be very careful that by enabling Large Pages you do not inadvertently violate your SLA during an unplanned outage. Remember, VMware HA requires a restart of a server on the new host if a host fails. This reboot time could be substantial if large amounts of memory are allocated.
Another problem can come if the service needs to restart and the memory it now requires to be contiguous is no longer that. If so, SQL Server will fail to start and you will see the following error in the ERRORLOG file.

*snip*

Performance Benefits

Check back soon for our performance summary of SQL Server 2008 R2 on vSphere 5.0 Large Pages performance!

Summary

So, in summary:

Requirements

• Minimum guest memory allocation of 8GB
• ‘Lock Pages in Memory’ permissions granted to service account
• 64-bit Windows and SQL Servers only
• Max SQL Server buffer pool memory setting must be configured

Pros

• Potential measurable performance improvement (YMMV)

Cons

• Potential significant delay in service startup time
• Eliminates transparent page sharing for this VM on VMware host
• SQL Server service restarts success rate could be an issue

Keep in mind that Large Pages is not a VMware feature – it’s a Windows feature. VMware simply extends it. Large Pages can be enabled whether the workload is physical or virtual, but all workloads should be virtualized, right?

Storage Performance Metrics

This blog entry is the continuation of the blog entry dated August 5, 2011

Average Read Time

• 20ms
• Sustained peaks of >20ms for no more than about five minutes

Attributes:

• Un-cached random reads
• Assuming 90% read, 10% write, with writes colliding

The Average Read Time performance attribute is important because there are schema/application designs that can pierce the cache, regardless of how big that cache is.

Average read time can be pulled from Oracle Statspack and AWR reports at the tablespace and datafile level.

Read time can also be pulled from the vSphere Client:

Select a host or a VM in the navigation pane > Performance Tab > [Advanced] > select Data Store > select Real Time > click the Read Latency box.

Spindle Busy Average

< 50%

To run R/esxtop to display device, kernel, guest and queue latency start an r/esxtop session:

$ esxtop > v > [Enter]

Look at the %USD column.

LUN percent busy

$ nmon > d

SCSI Queue Depth

<= teens (after having reconfigured all storage path devices to 128, up from default of 32)

The default SCSI queue depth for ESX is 32.

$ esxtop > d > f > [Enter].  Verify that QSTATS is selected with an asterisk to the left as pictured below.

Storage Latency

For day-to-day monitoring of disk throughput, latency is the ideal metric. It is more accurate in determining if your database is suffering from I/O throughput issues. The performance redlines for disk latency should be 20ms or lower. With latency it is acceptable to have burst periods of higher than 20ms latency, but not consistently.

R/esxtop monitors disk latency at three distinct layers, the device or HBA, the kernel or ESX hypervisor and the guest or virtual machine. Use the free IOmeter storage benchmarking tool in a Windows guest (only the Windows version can do asynchronous writes, not the Linux version).  Configure the tool for the Oracle database default block size and a 32GB streaming write (large enough to pierce the write cache of many SANs).

To run R/esxtop to display device, kernel, guest and queue latency start an r/esxtop session:

$ esxtop v > f > h > i > j > enter

• The GAVG or guest latency is actually a combination of the device latency + hypervisor + any additional guest OS overhead.
• The KAVG is the kernel latency or hypervisor latency. High latency reported at the kernel can be due high SCSI queues or device drivers.
• DAVG is the device latency or HBA latency. We use the term HBA generically, including an actual fiber channel HBA or iSCSI NIC. Device latency usually indicates a bottleneck at the storage or SAN layer.

By monitoring the four distinct layers we can clearly see the HBA is an issue with the SAN configuration. If just GAVG was high then we would have had looked at the Linux guest. This customer had very serve device latency (DAVG). This related back to a SAN issue that was later fixed by the storage vendor with a firmware upgrade.

Storage Throughput

Given a 4Gb Fibre Channel fabric, you should be looking at >100MB/sec/storage path sustained. If you are getting less than that, consider splitting out storage paths.

CPU Ready Time

The VMware definition of CPU Ready is, "the time a guest waits for CPU from the host while in a ready-to-run state" (VMware ESX Server 3: Ready Time Observations - http://www.vmware.com/pdf/esx3_ready_time.pdf)  We also refer to CPU Ready Time as the ”guest heartbeat.”

We generally monitor CPU Ready Time through VI client.  ESXTOP can also be used.   However, we prefer the VI client because it measures CPU Ready in milliseconds.

Select the virtual machine > Performance tab > click Advanced > click Chart Options... > CPU > Real Time > and select Ready

It is normal for a guest to average between 0 - 50ms of CPU ready time.  Anything over 300ms and you will experience performance problems. We’re comfortable with up to 300ms CPU Ready Time on average, with a high water mark of 500ms.

ESX Memory Oversubscription

Do not oversubscribe memory shared by Tier-1 Oracle workloads.  VCS should report no ballooning.

Oracle is very aggressive at using all of the resources available to it. When an Oracle instance starts up, it allocates memory for the Oracle SGA. This allows Oracle to use a contiguous space of memory for caching of data and SQL statements. As long as the instance is running, it will not give the memory back even if blocks are unused. Oracle memory is less dynamic then CPU. It tends to level out after the Oracle SGA has been allocated. You may see brief periods of spikes when the PGA is being used for a RMAN backup, but other than slight anomalies it tends to stay static. As with CPU, measure memory during peak workloads.

We recommend memory sizing by using the following 3 tools:

• Oracle SGA Advisor.
• Oracle PGA Advisor.
• OS metric collection (for example, nmon)

Start with the Oracle advisors first to determine if the SGA and PGA are accurately sized. We consider the SGA critical to virtual machine sizing. If, for example, the Oracle SGA is undersized this will affect your buffer cache hit and shared pool, negatively impacting performance. It will also skew your OS metric collection. After the SGA and PGA are sized accurately then use the OS collected metrics to determine the appropriate memory size of the virtual machine.

Allocate enough ESX physical memory to at least cover the Oracle SGA, and preferably as large as the sum of the SGA, the PGA high water mark, and memory used by the shadow processes.  If you use reservations, make sure that there is enough extra physical memory in the cluster that the failed over VM will not refuse to start.

In vSphere Client, select the VM > Performance tab > Advance tab > Chart Options… > Memory > Real Time > Active

Network Performance

We confess that we rarely run into network configuration issues that impede VMware Infrastructure performance.  Just the same, we’ll offer a standard.  There is always the possibility that a workload changes its virtual machine memory blocks at a high rate.  A workload so hot that it generates dirty memory blocks faster than the proprietary vMotion interconnect can move in thirty seconds is justification for beefing up the interconnect hardware capacity through faster NICs (or converged technology), teaming NICs, or both.

iperf Network Load Test

Conceptually the network load test should be used to force the max throughput of the network fabric and interfaces. Similar to the memory it flushes out any potential drive issues, hardware issues or network topology issues. This test is to validate performance and push the max throughput of the interfaces to determine if there are driver issues or stability problems.

Iperf is a network testing tool. It is able to create a TCP or UPD data stream between two nodes (or virtual machines). Iperf is open source software available for both Linux and Windows at http://sourceforge.net/projects/iperf/. Below is a simple test that runs for 500 seconds testing the maximum throughput of the network interfaces.

Server Node

$ iperf -s -i 5

Client Node

$ iperf -c -t 500 -i 5 –m

Dropped Packets

For network packets on a LAN, you ideally should not see dropped packets. Dropped packets typically indicate congestion in the network or failing hardware. One percent dropped packets in either direction can throttle throughput by as much as 15 percent.

In vSphere, dropped packets can be monitored by selecting the ESX host > Performance tab > Advanced > Network > Real Time > select None in Counters > then check Receive packets dropped and Transmit packets dropped

During Sunday’s Partner Event keynote, Larry Ellison said,”I think it’s enormously exciting that we can take this great company Sun and combine it with another great company Oracle and then merge those technologies and maybe do things neither company would be able to do by themselves.  That’s our goal, to tackle problems even bigger, to integrate hardware and software, and deliver revolutionary systems.”

Monday Oracle President Safra Catz said: “That it turns out that us and all the other software vendors, were all sending you little pieces of technology all these years, and it was at your site that you had to make it all work together.  And what we thought was that this really didn’t make sense.  That long-term, companies like us had to take more and more of the responsibility of bringing you systems that work together.”  She continued, “But many of you know there’s a reason we call this Oracle Open World, because we are just slavishly devoted to open standards.  And if you want some of the pieces and some from others, that’s ok.  We’re going to make that possible.  But our hope is that little by little, you take many of the pieces from us.”

Yet it appears to me that Oracle has already started stepping away from openness, in the very moment when its hardware partners, OS partners, and customers rather need to be reassured as to Oracle’s commitment to a somewhat level playing field moving forward.  I submit as evidence the “Database Smart Flash Cache” feature new in 11g R2.

11.2 New Features E10881-03 October 2009, p. 1-27: “New in Oracle Database 11g Release 2 (11.2), the Database Smart Flash Cache feature is a transparent extension of the database buffer cache using solid state device (SSD) technology. The SSD acts as a Level 2 cache to the (Level 1) SGA. Database Smart Flash Cache can greatly improve the performance of Oracle databases by reducing the amount of disk I/O at a much lower cost than adding an equivalent amount of RAM."

Sounds pretty good.  But get this restriction in the 11.2 Concepts Guide part # E10713-04 October 2009 p. 14-9) “Note: Database Smart Flash Cache is available only in Solaris and Oracle Enterprise Linux.”  The 11.2 Admin Guide (part # E10595-06 October 2009 p. 6-21) says Database Smart Flash Cache is only supported on the Solaris or Oracle Enterprise Linux operating systems.  This is confirmed by 11.2 Release Notes for Linux (part # E10838-03 October 2009 p. 2) and 11.2 Database Release Notes -03 p. 9.  11.2 Database Licensing Information (p. 1-4) says Database Smart Flash Cache is Enterprise Edition only and Solaris and Oracle Enterprise Linux only.  The 11.2 Reference (E10820-03 October 2009 p. 1-49, 1-50) says nothing about OS restrictions on either of the new db_flash_cache_file and db_flash_cache_size parameters.

What technical justification could Oracle have to not make this feature available or not support it in the HP-UX, AIX, or Windows ports for example, let alone the Red Hat and SuSe Linux distributions?  I’m feeling the need to reevaluate my enthusiasm for Oracle Enterprise Linux, for the first time since its announcement in 2006.

Mr. Ellison said, “There are some advantages of the single organization having control of the engineering of the hardware also engineering the software and engineering all the pieces to fit together well.”

It would appear that Oracle is already taking initiatives with respect to the Sun/Oracle announcement that give me pause.  If we’re already seeing what may be a competition-motivated lock-out of a key feature like database smart flash cache today, what kinds of additional lock-outs may be coming once the hardware and DB software are subjected to co-engineering?  I am not in favor of Oracle’s Sun acquisition.
 

Mr. Ellison continued in his Sunday keynote, “We are not selling the hardware business, no part of the hardware business are we selling.  And we think Sparc is a fantastic technology, and with a little more investment it could be even better.  You know the CMT line is fantastically innovate, and we think again by adding to the already spectacularly good engineering team we have around Sparc, we can increase our rate of innovation.  Make the chip even more reliable while consuming less power.  Make the chip even faster while consuming less power.  So we intend to invest in Sparc.”

I’ve been dazzled by how elegantly Oracle has integrated opened new business lines beginning with the Oracle Application Server curve back at the turn of the decade.  I had my doubts Oracle was going to make it with an app server, but they proved it to me, technically as well as with actual market share.  Oracle has consistently impressed me with their integration of acquired companies both organizationally and technically.  I’ve wondered since the Oracle/Sun announcement if Oracle was finally in over their head by attempting to get into hardware.  This week I’m swayed over to believe that Oracle can probably pull off the integration and management of Sun’s hardware lines.  Larry Ellison’s in charge and he’s a remarkably impressive visionary, strategist and tactician.

Mr. Ellison continued, “Solaris is unquestionably the number one enterprise operating system in the world in terms of linear scalability, in terms of reliability, in terms of features and functions.  It is the leading operating system running the Oracle database.  But it’s just the leading enterprise operating system period.”

Mr. Ellison continued, “And we’re very proud to be working with Sun to make sure that all the Oracle software runs better on Solaris, more reliably on Solaris, and faster on Solaris than it’s ever run before, and it runs faster there than it runs any place.”

So what are Oracle’s intentions with respect to Linux?  I think Chris Josephes sheds useful light on the issue, especially as it relates to Oracle’s 2005 declaration of Solaris 10 as the preferred development platform for these 64 bit platforms: UltraSparc, AMD-64, and Intel-64. 

Mr. Ellison continued, “We’re going to spend more, not less on MySQL.  We think it’s a fantastic piece of technology, it’s extremely popular, it’s an open source product, and we’re going to increase our rate of contribution to that product.”

On Wednesday, Scott McNealy said: “The one I hear a lot about is MySQL.  Open Source.  Well, what’s going to happen?  This is the volume, open source, low end database out there in the marketplace.  Is it going to get clobbered?  Well, I don’t think so.  And again Larry has said, “It doesn’t compete with the Oracle database,” which is true, this competes with Microsoft.”  Mr. McNealy put up a slide with this text: “’MySQL and Oracle don’t compete at all,’ said Mr. Ellison adding that he would not sell MySQL.  Financial Times, 22 September 2009”.

So what about the whole engineering effort behind Oracle 10g with one of the prominent objectives being that it would compete with SQL Server’s relative ease of installation and operations?  At the time we were led to understand that the single CD 20 minute 10g Database install and the improved manageability of 10g Database, was in part a response to Oracle’s concern about the hemorrhage of customers jumping ship for SQL Server.  What about Oracle’s open admission in the fall of 2005 that Oracle Express was a direct answer to SQL Server 2005 Express?  If MySQL and Oracle don’t compete at all, then one would have to talk themselves into the new concept that SQL Server and Oracle don’t compete at all.  A MySQL customer today doesn’t have the obvious technical upgrade path to paid Oracle versions compared to an Oracle Express customer.  What then could Oracle’s financial/business interest in continuing to promote MySQL?  I’d like to hear an answer to that question, rather than Oracle continuing to attempt to convince us that they’re going to support and contribute to MySQL simply because they say they will.  The existence of the Oracle employee assigned to MySQL documentation, that I met at Tuesday evening’s bloggers’ meeting, did more to convince me of the prospects of Oracle’s on-going support for MySQL, than all of these executive statements combined.

On Wednesday during his Exadata II comments, Mr. Ellison said, “The important thing here I think is our belief that eventually the virtual machine, the VM, and the operating system have to work exceptionally well together.  That means they have to be engineered together.”

With respect to the last sentence, I’m afraid I’m not similarly disposed.   VMware and the guest operating systems weren’t engineered together, although current versions were most definitely engineered with awareness of each other.  On the other hand, this seems like a huge forward looking statement on Oracle’s part, and one that I believe sets up a straw man to fill market demand that doesn’t today exist.  On Thursday while leaving the conference, I had an epiphany about my full-time experience in VMware’s OOW pavilion: not one person asked me about Oracle Virtual Machine or even mentioned it.

I’m finally getting around to blogging on Oracle RAC One Node, coincidentally announced on the second day of VMworld.

The day Oracle announced this (September 1st), I took a quick look at the Oracle-provided collateral.  It said, “…you can migrate the instance to another node in the cluster using the Omotion online migration utility with no downtime for application users”.  I immediately had a negative reaction to that:

• “Omotion” – clever name.  Any attempt to piggy-back through implication on VMware’s successful “VMotion” capability?
• In a RAC environment, there is no ability to hot-move a client connection.  That's unchanged through 11g.  So I guessed that this involved installing the RAC views in the database, temporarily firing up a second RAC instance, connecting to the new instance, then disconnecting from the old.

I told colleagues at VMworld that I was betting this would have some tool that gave you say 30 minutes to get all your connections migrated, prior to killing off any straggler sessions on the original instance.

The Oracle RAC SIG webinar on September 24^th pretty much confirmed my hunches.  The Omotion utility gives you a default/maximum migration window of (drum roll…) 30 minutes to get your connections moved.  And oh by the way, if you should happen to have a long running query that can’t clear out in 30 minutes, tough luck; there’s gonna be “downtime for application users” after all.  No big deal?  I'd say 90% of our customers' OLTP systems are hybrid, involving canned or ad-hoc decision support, or both.

Other than looking up the Oracle RAC SIG webinar abstract paragraph, I’m writing this weeks later without reviewing the webinar ppt.  The speaker made the point that clients/middle tiers should pick up the instance migration alert from Oracle Notification Server, and do their automatic disconnect/reconnect thing.  Funny, as a former Oracle University RAC instructor, I’ve been promoting the ONS framework for years and encouraging RAC shops to code to it.  The problem is, in all of our travels, it appears to be extremely rare for RAC shops to code to the ONS framework.  That I may think they’re missing a huge opportunity, doesn’t change the fact that ONS is seriously under-utilized.

On the one hand, Oracle has declined to support RAC on VMware for years, hiding behind this statement in the published RAC FAQ: “...there are technical restrictions that prevent the certification of RAC in a VMware environment.”  On the other hand, I’ve never seen Oracle attempt to go more aggressively after VMware line for line than in the “RAC One Node” webinar.  I find it incongruous that they are willing to go into unprecedented detail in their RAC One Node/VMware product comparison, yet have been unwilling to provide any detail whatsoever as to what the alledged RAC-on-VMware problems are.  This is all the more frustrating given the two-year-old announcement that RAC will be not only supported but certified on a (still) future Oracle VM release.  What is the probability that any vendor's comparatively challenged Xen platform (judgment call mine—another blog entry for another day), is going to come up with a solution for some as yet unspecified RAC-on-VMware technical restriction, that VMware and/or the Linux kernel development team haven't long since made moot?  Since our team did our first customer production-grade RAC-on-VMware install in the spring of 2007, I’d put those odds at close to nil.

So what about the detailed RAC One Node/VMware comparison?  I remember feeling some of the line items were spurious, and thought others were given undue emphasis in the talk.  Here’s one off the top of my head.  They were saying that encapsulating the OS in a VM provides savings for the server hardware guys, but not for OS system administration.  I have a lead SA in mind who I’d like to get on the phone, tell him I’m about to hit the record button, and see if he can think of any benefits to system administration on VMware at the OS level only.  I'll start recording first, then give him the question because I want to capture the laughter.  I can think of a couple off top of my head:

• Reduced if any need to re-install the OS on new machines due to VM templates
• Shops that get away with running very old unsupported Windows versions on VMs, that would never dream of continuing to do it on fully-depreciated unsupported hardware

Not mentioned in the webinar, is the isolation benefit that VMware provides, that's difficult to attain in a native environment short of following the long-standing, expensive recommendation of running production RAC instances on dedicated nodes.  I’ve always agreed with the recommendation's technical merits.  But finding shops that practice it seems just about as difficult as finding shops that code to ONS.  RAC One Node doesn’t help me there.  On the other hand, RAC on VMware gives me all the isolation benefits while letting me load up the node with as many other workloads as I can get away with.  That blends substantial technical and financial benefits, which is why we’ve been promoting the stack’s virtues for years.

I wonder if the Oracle RAC SIG VMware comparisons were misplaced in terms of audience.  I suspect the Oracle RAC SIG audience is largely technicians and technical leads.  I'd be surprised if they were swayed by the competitive analysis.  Two years ago in our VMware Oracle Solutions Lab at OOW 2007, a steady stream of DBAs told us how wide spread VMware Infrastructure was in their shops, including Oracle test and dev instances, except for (at the time) Oracle production.  It’s getting harder to find DBAs that aren’t familiar with VMware’s proven operational capabilities.  So I wonder if Oracle did more harm than good with this presentation.  If I were Oracle and had any hope of staving off the increasing rush of interest in Oracle on VMware by announcing Oracle RAC One Node, I would have attempted to suggest those comparisons in a C-level event, if I put them forth at all.

By the way, the tooling necessary to run Oracle RAC One Node wasn’t in the 11g R2 GA release.  In the webinar, they said that they needed more time in QA with the tools.  Hmmm.

What about pricing?  $10K per processor, except the temporary 2^nd RAC One Node migration instance is free subject to the 10-day rule.  So where’s the flexibility in that, short of calling it $20K per processor?  Imagine VMotion or DRS restricted to 10 cumulative migration days per year.  With vSphere 4’s ability to increase a VM's core count on the fly, I’d rather take a fraction of that $10K per processor, and buy VI 4 Advanced at $2,245 per processor.  Run single instance Oracle on that, and I’d have true uninterrupted workload migration for all transaction types, let alone the other features in VI 4 Advanced.  At risk of sounding too demanding, at least I'd get a control GUI out of that deal.

I’ll finish by commenting on the webinar’s abstract:

“Server virtualization in the data center promises to reduce server footprint while providing flexibility, load balancing and high availability for any application running in a virtual machine. However, adoption for database as been slowed due to limitations in the technology. Oracle now provides Oracle Real Application Clusters-One as an alternative virtualization technology. Oracle Real Application Clusters-One provides the benefits of server virtualization, and more, with easier consolidation, more flexible load balancing, and better high availability features. This session will introduce Oracle Real Application Clusters-One, and explain how you can use it to standardize your environment while increasing flexibility and agility”

• “…adoption for databases as (sic) been slowed due to limitations in the technology”?  It's our observation that adoption has been slowed not because of technology, but primarily due to mixed messages from Oracle.  Oracle Sales has an adversarial stance to customers running VMware, no doubt due to the license revenue threat.  Yet Oracle support consistently provides quite a different experience.  And, since this is a RAC SIG webinar, Oracle’s refusal to detail the RAC on VMware support issue could make this "limitations" statement sound like just more hiding.
• I don’t see “…the benefits of server virtualization, and more…” for the reasons I’ve discussed.
• I don’t see “…easier consolidation…”.  For starters, has Oracle pulled back on the one-RAC-instance-per-node recommendation?
• I don’t see “…more flexible load balancing…”.  I don't even see equivalent load balancing.  Can RAC One Node automatically move active workloads let alone long running workloads without restrictions, and with no connection awareness on the part of the clients?
• As for better HA features, yes, we've always told everybody that RAC has Cadillac HA features that are superior to VMware HA for shops with explicit SLAs tight enough to justify the significant expenditure, and scale so large that they're knocked out of the Oracle 10g SE RAC bundle.  But that only applied to what I'm going to have to start calling "Real RAC"--the EE RAC line item, or 10g SE RAC bundle, both of which have multiple RAC instances in normal operations.  "Real RAC" provides the superior HA solution.  RAC One Node can't provide better HA than VMware HA or IP stack fail-over solutions, because there's no second hot RAC instance waiting to catch the fail-over football in the event of a production emergency.
• The final sentence suggests using RAC One Node to “…standardize your environment while increasing flexibility and agility”.  Well...

...the moment I ask any of our single instance Oracle-on-VMware customer shops to scrap that stack and go native with Oracle RAC One Node for purposes of "...increasing flexibility and agility", I’ll probably be subjected to an impromptu comparative features analysis that you can take to the bank.