It’s been hopping since Oracle Open World.  I’ve been doing a lot of events and Fortune 500 customer site visits—all of them on the topic of tier-1 workloads on VMware.

I think it inevitable that the enterprise trend to evaluate options to VMware, will collide with enterprises’ need to leverage virtualization’s benefits into tier-1 workloads, let alone horizontal scaling.  I look forward to the word-on-the-street revelations that will come from that.  Forget vendor posturing.  The passage of time makes it more difficult for people that over-represent themselves to maintain control of the message.  I find it interesting that the majority of the scaled shops we are talking to already have VMware Enterprise License Agreements in place but haven’t yet moved into tier-1 virtualization.  I also find it interesting that configuration defects in many shops go unnoticed but become sudden roadblocks when the shops attempt to virtualize a tier-1 workload.

I continue to slam into a consistent, significant uptick in RAC interest wherever I go.  On that topic, shops interested in RAC will want to make sure they look at Barb Lundhild’s OOW ’09 presentation “Understanding Oracle Real Application Clusters Internals”. There are significant RAC enhancements and component realignments as of DB 11g R2.

During Sunday’s Partner Event keynote, Larry Ellison said,”I think it’s enormously exciting that we can take this great company Sun and combine it with another great company Oracle and then merge those technologies and maybe do things neither company would be able to do by themselves.  That’s our goal, to tackle problems even bigger, to integrate hardware and software, and deliver revolutionary systems.”

Monday Oracle President Safra Catz said: “That it turns out that us and all the other software vendors, were all sending you little pieces of technology all these years, and it was at your site that you had to make it all work together.  And what we thought was that this really didn’t make sense.  That long-term, companies like us had to take more and more of the responsibility of bringing you systems that work together.”  She continued, “But many of you know there’s a reason we call this Oracle Open World, because we are just slavishly devoted to open standards.  And if you want some of the pieces and some from others, that’s ok.  We’re going to make that possible.  But our hope is that little by little, you take many of the pieces from us.”

Yet it appears to me that Oracle has already started stepping away from openness, in the very moment when its hardware partners, OS partners, and customers rather need to be reassured as to Oracle’s commitment to a somewhat level playing field moving forward.  I submit as evidence the “Database Smart Flash Cache” feature new in 11g R2.

11.2 New Features E10881-03 October 2009, p. 1-27: “New in Oracle Database 11g Release 2 (11.2), the Database Smart Flash Cache feature is a transparent extension of the database buffer cache using solid state device (SSD) technology. The SSD acts as a Level 2 cache to the (Level 1) SGA. Database Smart Flash Cache can greatly improve the performance of Oracle databases by reducing the amount of disk I/O at a much lower cost than adding an equivalent amount of RAM."

Sounds pretty good.  But get this restriction in the 11.2 Concepts Guide part # E10713-04 October 2009 p. 14-9) “Note: Database Smart Flash Cache is available only in Solaris and Oracle Enterprise Linux.”  The 11.2 Admin Guide (part # E10595-06 October 2009 p. 6-21) says Database Smart Flash Cache is only supported on the Solaris or Oracle Enterprise Linux operating systems.  This is confirmed by 11.2 Release Notes for Linux (part # E10838-03 October 2009 p. 2) and 11.2 Database Release Notes -03 p. 9.  11.2 Database Licensing Information (p. 1-4) says Database Smart Flash Cache is Enterprise Edition only and Solaris and Oracle Enterprise Linux only.  The 11.2 Reference (E10820-03 October 2009 p. 1-49, 1-50) says nothing about OS restrictions on either of the new db_flash_cache_file and db_flash_cache_size parameters.

What technical justification could Oracle have to not make this feature available or not support it in the HP-UX, AIX, or Windows ports for example, let alone the Red Hat and SuSe Linux distributions?  I’m feeling the need to reevaluate my enthusiasm for Oracle Enterprise Linux, for the first time since its announcement in 2006.

Mr. Ellison said, “There are some advantages of the single organization having control of the engineering of the hardware also engineering the software and engineering all the pieces to fit together well.”

It would appear that Oracle is already taking initiatives with respect to the Sun/Oracle announcement that give me pause.  If we’re already seeing what may be a competition-motivated lock-out of a key feature like database smart flash cache today, what kinds of additional lock-outs may be coming once the hardware and DB software are subjected to co-engineering?  I am not in favor of Oracle’s Sun acquisition.
 

Mr. Ellison continued in his Sunday keynote, “We are not selling the hardware business, no part of the hardware business are we selling.  And we think Sparc is a fantastic technology, and with a little more investment it could be even better.  You know the CMT line is fantastically innovate, and we think again by adding to the already spectacularly good engineering team we have around Sparc, we can increase our rate of innovation.  Make the chip even more reliable while consuming less power.  Make the chip even faster while consuming less power.  So we intend to invest in Sparc.”

I’ve been dazzled by how elegantly Oracle has integrated opened new business lines beginning with the Oracle Application Server curve back at the turn of the decade.  I had my doubts Oracle was going to make it with an app server, but they proved it to me, technically as well as with actual market share.  Oracle has consistently impressed me with their integration of acquired companies both organizationally and technically.  I’ve wondered since the Oracle/Sun announcement if Oracle was finally in over their head by attempting to get into hardware.  This week I’m swayed over to believe that Oracle can probably pull off the integration and management of Sun’s hardware lines.  Larry Ellison’s in charge and he’s a remarkably impressive visionary, strategist and tactician.

Mr. Ellison continued, “Solaris is unquestionably the number one enterprise operating system in the world in terms of linear scalability, in terms of reliability, in terms of features and functions.  It is the leading operating system running the Oracle database.  But it’s just the leading enterprise operating system period.”

Mr. Ellison continued, “And we’re very proud to be working with Sun to make sure that all the Oracle software runs better on Solaris, more reliably on Solaris, and faster on Solaris than it’s ever run before, and it runs faster there than it runs any place.”

So what are Oracle’s intentions with respect to Linux?  I think Chris Josephes sheds useful light on the issue, especially as it relates to Oracle’s 2005 declaration of Solaris 10 as the preferred development platform for these 64 bit platforms: UltraSparc, AMD-64, and Intel-64. 

Mr. Ellison continued, “We’re going to spend more, not less on MySQL.  We think it’s a fantastic piece of technology, it’s extremely popular, it’s an open source product, and we’re going to increase our rate of contribution to that product.”

On Wednesday, Scott McNealy said: “The one I hear a lot about is MySQL.  Open Source.  Well, what’s going to happen?  This is the volume, open source, low end database out there in the marketplace.  Is it going to get clobbered?  Well, I don’t think so.  And again Larry has said, “It doesn’t compete with the Oracle database,” which is true, this competes with Microsoft.”  Mr. McNealy put up a slide with this text: “’MySQL and Oracle don’t compete at all,’ said Mr. Ellison adding that he would not sell MySQL.  Financial Times, 22 September 2009”.

So what about the whole engineering effort behind Oracle 10g with one of the prominent objectives being that it would compete with SQL Server’s relative ease of installation and operations?  At the time we were led to understand that the single CD 20 minute 10g Database install and the improved manageability of 10g Database, was in part a response to Oracle’s concern about the hemorrhage of customers jumping ship for SQL Server.  What about Oracle’s open admission in the fall of 2005 that Oracle Express was a direct answer to SQL Server 2005 Express?  If MySQL and Oracle don’t compete at all, then one would have to talk themselves into the new concept that SQL Server and Oracle don’t compete at all.  A MySQL customer today doesn’t have the obvious technical upgrade path to paid Oracle versions compared to an Oracle Express customer.  What then could Oracle’s financial/business interest in continuing to promote MySQL?  I’d like to hear an answer to that question, rather than Oracle continuing to attempt to convince us that they’re going to support and contribute to MySQL simply because they say they will.  The existence of the Oracle employee assigned to MySQL documentation, that I met at Tuesday evening’s bloggers’ meeting, did more to convince me of the prospects of Oracle’s on-going support for MySQL, than all of these executive statements combined.

On Wednesday during his Exadata II comments, Mr. Ellison said, “The important thing here I think is our belief that eventually the virtual machine, the VM, and the operating system have to work exceptionally well together.  That means they have to be engineered together.”

With respect to the last sentence, I’m afraid I’m not similarly disposed.   VMware and the guest operating systems weren’t engineered together, although current versions were most definitely engineered with awareness of each other.  On the other hand, this seems like a huge forward looking statement on Oracle’s part, and one that I believe sets up a straw man to fill market demand that doesn’t today exist.  On Thursday while leaving the conference, I had an epiphany about my full-time experience in VMware’s OOW pavilion: not one person asked me about Oracle Virtual Machine or even mentioned it.

Oracle has somehow managed to stay out of USA Today and the Wall Street Journal so far this week, according to what I have noticed.  Expect that to change today with Larry Ellison’s keynote, assisted on-stage by California Gov Schwarzenegger.

I’m intrigued by Oracle’s Golden Gate acquisition announcement.  Golden Gate specializes in real time decision support data integration.  Look for Golden Gate to play well with Oracle’s strategic business intelligence acquisitions of Siebel and Hyperion business intelligence acquisitions.  These compliment Oracle’s tenured Data Warehouse Builder and OLAP—already tightly integrated with the database.

Major changes in the Oracle Partner program announced on Sunday.  Oracle’s acquisitions have outgrown the existing partner framework.  The partner program now allows and encourages specialization.  Among the changes is the separation (at least for partners) of decoupling certification from Oracle University training.  The Exadata Storage servers and Sun Oracle Database hardware are available for re-sale by Oracle Partners as of December 1.  I thank Ray Wang for his excellent recap of the Sunday announcements.

After hanging with VMware’s Chris Rimer (owns the Oracle relationship world-wide) and even speaking with him around the country, I finally get it on the Oracle-on-VMware “certification” issue.  Oracle doesn’t certify below the OS.  VMware is hardware to the Oracle red stack, because it’s below the OS.  With respect to Oracle’s VMware support clause that Oracle may ask a customer to assist replicating the bug on physical hardware, that makes Oracle’s VMware support policy no different than Oracle’s policy with HP or other hardware vendors.  This is consistent with Wim Coekaert’s response in Sunday’s partner Oracle Virtual Machine meeting to a guy asking a one-off question about VMware support.  Wim is reported to have said that Oracle’s no different than most other ISV’s with respect to that policy.  This is an appropriate time for me to remind that to-date Oracle support has never asked any HoB Oracle VMware customer to replicate any problem on native hardware.

I ran into Tim Gorman at last night’s Bloggers’ meeting.  We flew Tim into Omaha a few years ago to keynote at Solid Foundations.  Tim is on HoB’s very short list of Oracle business intelligence specialists to call.  He’s also an Oak Table member.  I also enjoyed chatting with John, a ten-year Oracle employee who blogs independently of Oracle’s blog site, and who works on Oracle’s MySQL documentation project.
 

I’ve enjoyed the steady stream of HoB customers and associates who have been stopping by the VMware pavilion.  It feels like a high school reunion.  Nebraska Workers’ Compensation Court’s Randy Cecrle observes that the exhibit floor is substantially shifted from previous years.  He’s not finding the third party content/document management, IDE or BPM vendors out there.  Much fewer development tool vendors.  He says he checked the program guide to make sure he wasn’t overlooking them.

I’ll be taking a look at what’s new in OVM 2.2, announced this week.

The best break-out session recommendation that I’ve heard yet from Oppenheimer Fund’s Roger Rose.  Monday Steve Shaw’s session S312645: “Oracle Database Performance on Linux: Tips, Tools, and Tuning for Intel Platforms.”  This included the option to disable the Linux background process that can slow down the clock for power consumption.  It also included changing the NUMA memory setting/interleave from 6K to 2M.

Look for my comprehensive review of the show’s keynotes tomorrow. 

This show’s down to about 35K attendees, compared to 45K two years ago.  Oracle’s cutting back; no third year of the mega LCD screens with Oracle ads on either end of Howard St.

Surprising trend: In day one of my VMware Pavilion volunteer work, I got asked as many questions about the mechanics of running RAC on VMware, as I did any other Oracle-on-VMware topic.

From Sunday’s Oracle/Sun talk: I noticed Larry Ellison’s defense of TPC fining Oracle a whopping $10K for making a comparative IBM performance claim based on numbers that Oracle hadn’t yet published.  In my opinion, Mr. Ellison seems to be saying in so many words that the end justified the means and they made the decision fully anticipating the $10K fine.

I’ll no doubt say more about Oracle’s Sun acquisition later.  Out of all of the Sun issues, I probably the most curious about what Oracle will do with Sun’s MySQL business line.  Six years ago, we used to laugh at this little database that had no stored procedures and didn’t even have commit/rollback.  We stopped laughing at MySQL a few years ago.  Now it’s one of our business lines.  I keep reminding myself that if Oracle were to attempt to scuttle or ignore (same thing) MySQL with intent to reduce competition for the Oracle database, that someone out there would no doubt take the open source remnants and keep it alive.  I just scanned two sheets that I picked up today at the Sun pavilion.  The first sheet describes each of their 23 demo stations.  The second lists the ~50 Sun-related sessions at the show.  Not a word in either about MySQL.  That's a disappointment.

 
News today that caught my eye since we do a lot with E-Business Suite shops: datango claims reduction of Oracle application rollout and support costs by over 40%.

HP targeted Oracle Apps shops today, announcing a series of new provisioning templates for BladeSystem Matrix users.  I'd like to see a vendor step forward with Oracle Apps VMware virtual machine templates.

I’m finally getting around to blogging on Oracle RAC One Node, coincidentally announced on the second day of VMworld.

The day Oracle announced this (September 1st), I took a quick look at the Oracle-provided collateral.  It said, “…you can migrate the instance to another node in the cluster using the Omotion online migration utility with no downtime for application users”.  I immediately had a negative reaction to that:

• “Omotion” – clever name.  Any attempt to piggy-back through implication on VMware’s successful “VMotion” capability?
• In a RAC environment, there is no ability to hot-move a client connection.  That's unchanged through 11g.  So I guessed that this involved installing the RAC views in the database, temporarily firing up a second RAC instance, connecting to the new instance, then disconnecting from the old.

I told colleagues at VMworld that I was betting this would have some tool that gave you say 30 minutes to get all your connections migrated, prior to killing off any straggler sessions on the original instance.

The Oracle RAC SIG webinar on September 24^th pretty much confirmed my hunches.  The Omotion utility gives you a default/maximum migration window of (drum roll…) 30 minutes to get your connections moved.  And oh by the way, if you should happen to have a long running query that can’t clear out in 30 minutes, tough luck; there’s gonna be “downtime for application users” after all.  No big deal?  I'd say 90% of our customers' OLTP systems are hybrid, involving canned or ad-hoc decision support, or both.

Other than looking up the Oracle RAC SIG webinar abstract paragraph, I’m writing this weeks later without reviewing the webinar ppt.  The speaker made the point that clients/middle tiers should pick up the instance migration alert from Oracle Notification Server, and do their automatic disconnect/reconnect thing.  Funny, as a former Oracle University RAC instructor, I’ve been promoting the ONS framework for years and encouraging RAC shops to code to it.  The problem is, in all of our travels, it appears to be extremely rare for RAC shops to code to the ONS framework.  That I may think they’re missing a huge opportunity, doesn’t change the fact that ONS is seriously under-utilized.

On the one hand, Oracle has declined to support RAC on VMware for years, hiding behind this statement in the published RAC FAQ: “...there are technical restrictions that prevent the certification of RAC in a VMware environment.”  On the other hand, I’ve never seen Oracle attempt to go more aggressively after VMware line for line than in the “RAC One Node” webinar.  I find it incongruous that they are willing to go into unprecedented detail in their RAC One Node/VMware product comparison, yet have been unwilling to provide any detail whatsoever as to what the alledged RAC-on-VMware problems are.  This is all the more frustrating given the two-year-old announcement that RAC will be not only supported but certified on a (still) future Oracle VM release.  What is the probability that any vendor's comparatively challenged Xen platform (judgment call mine—another blog entry for another day), is going to come up with a solution for some as yet unspecified RAC-on-VMware technical restriction, that VMware and/or the Linux kernel development team haven't long since made moot?  Since our team did our first customer production-grade RAC-on-VMware install in the spring of 2007, I’d put those odds at close to nil.

So what about the detailed RAC One Node/VMware comparison?  I remember feeling some of the line items were spurious, and thought others were given undue emphasis in the talk.  Here’s one off the top of my head.  They were saying that encapsulating the OS in a VM provides savings for the server hardware guys, but not for OS system administration.  I have a lead SA in mind who I’d like to get on the phone, tell him I’m about to hit the record button, and see if he can think of any benefits to system administration on VMware at the OS level only.  I'll start recording first, then give him the question because I want to capture the laughter.  I can think of a couple off top of my head:

• Reduced if any need to re-install the OS on new machines due to VM templates
• Shops that get away with running very old unsupported Windows versions on VMs, that would never dream of continuing to do it on fully-depreciated unsupported hardware

Not mentioned in the webinar, is the isolation benefit that VMware provides, that's difficult to attain in a native environment short of following the long-standing, expensive recommendation of running production RAC instances on dedicated nodes.  I’ve always agreed with the recommendation's technical merits.  But finding shops that practice it seems just about as difficult as finding shops that code to ONS.  RAC One Node doesn’t help me there.  On the other hand, RAC on VMware gives me all the isolation benefits while letting me load up the node with as many other workloads as I can get away with.  That blends substantial technical and financial benefits, which is why we’ve been promoting the stack’s virtues for years.

I wonder if the Oracle RAC SIG VMware comparisons were misplaced in terms of audience.  I suspect the Oracle RAC SIG audience is largely technicians and technical leads.  I'd be surprised if they were swayed by the competitive analysis.  Two years ago in our VMware Oracle Solutions Lab at OOW 2007, a steady stream of DBAs told us how wide spread VMware Infrastructure was in their shops, including Oracle test and dev instances, except for (at the time) Oracle production.  It’s getting harder to find DBAs that aren’t familiar with VMware’s proven operational capabilities.  So I wonder if Oracle did more harm than good with this presentation.  If I were Oracle and had any hope of staving off the increasing rush of interest in Oracle on VMware by announcing Oracle RAC One Node, I would have attempted to suggest those comparisons in a C-level event, if I put them forth at all.

By the way, the tooling necessary to run Oracle RAC One Node wasn’t in the 11g R2 GA release.  In the webinar, they said that they needed more time in QA with the tools.  Hmmm.

What about pricing?  $10K per processor, except the temporary 2^nd RAC One Node migration instance is free subject to the 10-day rule.  So where’s the flexibility in that, short of calling it $20K per processor?  Imagine VMotion or DRS restricted to 10 cumulative migration days per year.  With vSphere 4’s ability to increase a VM's core count on the fly, I’d rather take a fraction of that $10K per processor, and buy VI 4 Advanced at $2,245 per processor.  Run single instance Oracle on that, and I’d have true uninterrupted workload migration for all transaction types, let alone the other features in VI 4 Advanced.  At risk of sounding too demanding, at least I'd get a control GUI out of that deal.

I’ll finish by commenting on the webinar’s abstract:

“Server virtualization in the data center promises to reduce server footprint while providing flexibility, load balancing and high availability for any application running in a virtual machine. However, adoption for database as been slowed due to limitations in the technology. Oracle now provides Oracle Real Application Clusters-One as an alternative virtualization technology. Oracle Real Application Clusters-One provides the benefits of server virtualization, and more, with easier consolidation, more flexible load balancing, and better high availability features. This session will introduce Oracle Real Application Clusters-One, and explain how you can use it to standardize your environment while increasing flexibility and agility”

• “…adoption for databases as (sic) been slowed due to limitations in the technology”?  It's our observation that adoption has been slowed not because of technology, but primarily due to mixed messages from Oracle.  Oracle Sales has an adversarial stance to customers running VMware, no doubt due to the license revenue threat.  Yet Oracle support consistently provides quite a different experience.  And, since this is a RAC SIG webinar, Oracle’s refusal to detail the RAC on VMware support issue could make this "limitations" statement sound like just more hiding.
• I don’t see “…the benefits of server virtualization, and more…” for the reasons I’ve discussed.
• I don’t see “…easier consolidation…”.  For starters, has Oracle pulled back on the one-RAC-instance-per-node recommendation?
• I don’t see “…more flexible load balancing…”.  I don't even see equivalent load balancing.  Can RAC One Node automatically move active workloads let alone long running workloads without restrictions, and with no connection awareness on the part of the clients?
• As for better HA features, yes, we've always told everybody that RAC has Cadillac HA features that are superior to VMware HA for shops with explicit SLAs tight enough to justify the significant expenditure, and scale so large that they're knocked out of the Oracle 10g SE RAC bundle.  But that only applied to what I'm going to have to start calling "Real RAC"--the EE RAC line item, or 10g SE RAC bundle, both of which have multiple RAC instances in normal operations.  "Real RAC" provides the superior HA solution.  RAC One Node can't provide better HA than VMware HA or IP stack fail-over solutions, because there's no second hot RAC instance waiting to catch the fail-over football in the event of a production emergency.
• The final sentence suggests using RAC One Node to “…standardize your environment while increasing flexibility and agility”.  Well...

...the moment I ask any of our single instance Oracle-on-VMware customer shops to scrap that stack and go native with Oracle RAC One Node for purposes of "...increasing flexibility and agility", I’ll probably be subjected to an impromptu comparative features analysis that you can take to the bank.