By Jim Hannan (@HoBHannan)
I strongly believe one of the most beneficial exercises an IT staff can undergo is benchmarking. There are truly some substantial benefits to benchmarking:

• Building of baselines
• Establishing key metrics
• Communicating the key metrics across the organization
• Identifying scalability
• Testing for Tier 1 readiness
• Validating the infrastructure (network, storage, CPU, software, drivers, and configurations)

I think many organizations agree with this concept, but benchmarking can be resource intensive and inaccurate if done improperly. This tends to deter organizations from doing this kind of work.

Because I am a consultant, I have the benefit of working with many different organizations. I get to see what works and what does not. In fact, I spend quite of bit of time with customers doing benchmarking and Tier 1 readiness testing. In this blog I will discuss the various methods and present what works and what does not.

Why benchmark? Benchmarking provides a measurement of your hardware and scalability. Often during benchmarking, customers expose weaknesses or misconfiguration. Additionally, benchmarking establishes baselines for future benchmarking and performance troubleshooting. But most importantly, benchmarking tests the platform for Tier 1 readiness.

I think it is important to clarify a few things before continuing on. What does it mean to baseline? A baseline establishes a performance profile when things are running well. The baseline can record things like TPS (transactions per second), I/O peaks, CPU usage, maximum I/O throughput, and many other performance indicators. The baselines can come from tools like AWR reports, NMON statistics (OS metrics), storage tools, and user response time.

What does Tier 1 readiness mean? Tier 1 readiness is simply a key set of benchmarks designed to validate and record the performance of a platform (hardware and software) in its ability to service the workloads.

There are several different approaches to benchmarking. In the table below we categorized the different approaches and identified the level-of-effort for each category. A more detailed description of each test is below the table.

Load Generation

Working from the top down, lets look at Load Generation techniques. Load Generation tools like Swingbench and even Spec 2006 (CPU/hardware analysis) have a preset of tests. They may not behave anything like your database or application. They have the benefit of being typically the easiest to setup. They fall short in accuracy of results.  That said, people who are very familiar with Swingbench can accurately measure performance by running tests across many different hardware configurations. The results have a point of reference – meaning that they can compare the previous results to the new results.

Tier 1 Readiness

I should tell you that some of my favorite tools live in this category. These tests tend to be very focused on particular elements like, for example, storage, network, or CPU. I like to tell customers these test expose the tipping points. What I mean by that is, these tests give you the absolute best case for throughput. Another reason that I am a really big proponent of these tests is because they are the most approachable. They do not take a lot of time to setup. They don't need an Oracle installation. In fact, our SQL Server team will use a lot of the same tools.

Right-Sizing

It could be argued that right-sizing does not fit into the benchmarking category, but I would disagree with that argument. It’s a process, and probably the most important of the four. After several edits of this paragraph, I realized something. I was originally going to say that right-sizing is a virtualization only step. That is simply not true. Right sizing is just as important for physical hardware as it is for virtual.

What data do you need for right-sizing? For Oracle, customers are most successful when they have collected OS metrics and AWR reports. The metrics should be collected every day. And metrics should exist as far back as a month ago – or even a year ago.

Why so much data? For your business critical applications, understanding their peaks is important. For example: what does month end look like? Does the application get very busy around Christmas or the beginning of the school year?

Capture and Replay

This is the most accurate of the testing options available. A capture and replay benchmark takes the application code and data and tests it on the new hardware. I have seen all kinds of various sets of tests. Some customers only replay critical or most prevalent SQL – others test all SQL over a defined period.

Summary

I mentioned in the beginning of the blog that I had the luxury of working with many different customers. I get to see what works and, in some cases, what does not work as well. For most customers a combination of the 4 different approaches in the right mix. Understanding when to use the appropriate tool is important. Additionally, the customers that are most successful with benchmarking have adopted their own methodology and processes. This allows the organization to be clear about what the key metrics are and how to run the test consistently each time. This is a deep topic with host of considerations. If you have questions or comments, I can be reached via Twitter @HoBHannan.

By Dave Welch (@OraVBCA)

Tech Target’s Beth Pariseau asked for my opinion on Oracle’s recent Nimbula purchase announcement. She then posted an article titled “Oracle's Nimbula IaaS buy sparks open source speculation” on March 14th.

I always appreciate the opportunity to contribute to Beth’s pieces. On this occasion, I opted to give her my input in writing only. As I’ve thought about it, I think the OVM concerns that I shared with Beth are important enough that I’ve chosen to post my entire response to her. I’m posting it as-is, resisting the temptation to expound on how it relates to Oracle’s March 20 announcement of its hardware/cloud sales decline.

March 13, 2012

Beth,

It’s always great to hear from you. Quite frankly, I’m surprised I didn’t hear from you on the occasion of IBM’s announcement last week that they were committing to OpenStack. I’m inclined to believe that’s a much bigger deal than Oracle’s Nimbula announcement.

Oracle is floundering with virtualization. Oracle’s been on its Xen project for a full six years now (assuming it put at least a few months of prep into its OVM OOW 2007 announcement) and still hasn’t produced anything we consider remotely usable.

I’ve been verbal from the start about how VMware needed a technical and financial competitor on the Linux side. When Oracle announced OVM at OOW 2007, I was dearly hoping that Oracle would be that competitor. Over five years later, I’m still waiting for Oracle to step up.

Oracle announced OVM 3.0 at OOW 2011 with a lot of fanfare. Within two days, VMware blasted its partners a 15-point technical competitive analysis based solely on downloadable OVM 3.0 documentation. I thought VMware’s analysis totally missed the mark. I have maintained for years that OVM chose the wrong open source platform in Xen, and that would continue to be the case until Xen got rid of the parent partition, otherwise known as Dom0. The three Xen issues—any one of which was a bigger deal for me than anything on the OVM 3.0/vSphere partner compete sheet—were: 

• Tier 1 workload I/O & network scalability: difficult to accomplish when the same device driver code has to live and function in both the base native Linux install image as well as the parent partition virtual layer.
• The parent partition is inherently anti-HA as there are lots of required Linux patches that require a full down of the environment yet have nothing to do with virtualization.
• Security analysts agree that the ultimate way to reduce a security threat at the end of the day is to reduce the size of the attack surface. The installation of the parent partition mini-Linux image on top of the base native hardware image bloats rather than reduces the attack surface.

Oracle’s long-standing Senior VP of OVM/OEL Wim Coekaerts announced in his May 2011 blog post that the parent partition (Dom0) was finally going away as of the 2.6.39+ kernel. My hopes soared. At the time, I said that we were probably still a hardware depreciation lifecycle away from getting any respectable OVM GA adoption on the street. Meanwhile, Red Hat would have to muck with it and release. Then Oracle in turn would muck with it and release.

A year later in May 2012, Wim posted that OVM 3.1 with Dom0 was finally GA. Just a month ago, House of Brick followed Oracle Consulting Services into a shop that was trying to take a scaled workload live within days on Oracle E-Business Suite with RAC on OVM 3.1. OCS had done its own OVM configuration per its best practices for the customer. What we found was anything other than stable or even remotely usable. We retrofitted vSphere 5.1 per the customer’s plea, and they went live with the corrected stack days later. Takeaway: a year after Wim’s May 2011 Dom0 post, I’m regretfully predicting that we’re still at least a hardware depreciation life-cycle away from OVM becoming anything usable under Tier 1 workloads.

So what’s going on at Oracle? Wim Coekaerts won my respect a decade ago with his team’s engineering and support of Oracle Cluster File System. I wonder if this has anything to do with Wim and his team’s capabilities and rather wonder if his team is the victim of resource constraints. To me, it sure looks like another indicator that Oracle’s leadership’s primary objective with OVM may have been trade rag competition. Oracle’s leadership can’t be blind to the accelerating hemorrhage of accounts optimizing their Oracle red stacks on VMware, with the associated crippling reduction on Oracle processor-based licensing revenue.

Given the choices that Oracle appears to have made to-date and its current in-the-mire status with OVM, I think that for Oracle to bag Xen and go with OpenStack would have to be an improvement for two reasons: 

• Anything would be better than current state OVM.
• Any other corporate entity overseeing design and development of the hypervisor and surrounding tooling would be an improvement.

I wouldn’t be at all surprised if Oracle went there.

Regards,
Dave

by Jim Hannan (@HoBHannan)

In my previous post I discussed tools to automate deployments for virtualized Oracle workloads. I outlined some of the current industry methods and their tools. In this blog post I will take a deeper look at one of those tools, Oracle’s DBCA (Database Configuration Assistant). I cannot advocate for DBCA being the perfect solution for every company, but I think majority of IT teams will find it very well suited for their deployments. Before jumping into DBCA, we should first quickly revisit the table from the last blog post. The table below outlines the opportunities for automation. Each step has various methods available to it. In this blog we are targeting the final step, automated deployment of the Oracle database after the virtual machine and software has been deployed.

Table 1 - Oracle Automated Deployment and Tools

Most DBAs are familiar with DBCA and the GUI wizard. Probably less familiar is DBCA's ability to run silently from the command line. The command line options really lend themselves to automation. You can create a database template or a catalog of templates. In the following example I have created the template SAND-forblog.dbc. This is stored with the other default templates. After creating and selecting the configuration you can than override the setting by using DBCA command line options. An example of overriding a parameter is changing the character set or datafile location. From the command line you can type “dbca –help” to get a full list of options. I have listed some of the key options in the table below. Remember this command line options can override settings defined in the template, very flexible.
Creating the database:

Output of the database creation:

You can delete a database with the command:

And output of the delete command:

I thought it would be worth mentioning that customers that have successfully implemented automation typically have matured their processes first by building standards. This reminds me of software development. First, build the flow chart (standard process), and then look for automation opportunities.

If you have any questions about DBCA automation or previous topics I can be reached on twitter @HoBHannan.

House of Brick is pleased to announce the hiring of Bob Lindquist, industry veteran and leader in the virtualization of business critical applications (VBCA), as the business development director for the west region.

“Bob was one of the early visionaries in the industry to grasp the importance of virtualizing business critical applications such as Oracle and SQL Server-based systems,” said Jim Ogborn, VP of Client Services at House of Brick. “I am excited that he will bring his vision of virtualization and cloud computing coupled with House of Brick’s industry-leading VBCA services to our customers and partners in the west region.”

House of Brick is leading the industry in virtualizing business critical systems, having provided services to dozens of the Fortune 500 companies, as well as hundreds of other enterprises of all sizes. Service bundles such as the Oracle or SQL Server on VMware Enablement, and the Oracle License Review and Architecture Optimization have helped House of Brick customers save countless millions of dollars while improving high availability, disaster recoverability, and new feature time to market.

“Virtualizing business critical systems, such as those built on top of Oracle DB or MS SQL Server is sometimes an intimidating thing for our customers,” says Nathan Biggs, House of Brick CEO. “This fear can come from not having done this type of virtualization successfully before, and can also come from misunderstandings about performance, licensing, and support from their software vendors. House of Brick knocks down the technical and emotional barriers that our customers face in taking on projects like this, and provides key services for successful implementations.”

In commenting on the hiring of Bob Lindquist, Biggs said “Bob will bring tremendous energy to the growth we are experiencing at House of Brick. With the increased VBCA activity we are seeing in the west, Bob is a welcome and critical addition to our team.”

"In my 25 years as a customer, solution provider, and consultant, I have seen the transformation of IT in all organizations due to enabling technologies like virtualization,” said Bob Lindquist. “VBCA is the next enabler for providing IT-as-a-Service to lines of business. House of Brick is the industry specialist in this space, and I feel privileged to join such a strong team of IT visionaries."

Bob Lindquist immediately assumes business development responsibility for House of Brick’s customers and partners in the western states region, including Arizona, California, Colorado, Nevada, New Mexico, and Utah. Come meet Bob at House of Brick’s Suite at Mandalay Bay during VMware’s Partner Exchange, February 23rd through the 28th in Las Vegas.

By Dave Welch (@OraVBCA)

I advocate that IT management at all levels and administrators of peer technical disciplines read the Oracle Database Concepts Guide’s Introduction chapter. Yes, that includes the C-level. That read will go a long way in preparing you to participate in database-related discussions at any level. Although this is beneficial if your organization deals with Oracle databases at all, it will prepare you to understand and accept a qualified architect’s suggestion that running Oracle business-critical workloads on the vSphere platform can provide remarkable benefits with minimal if any risks.

As I post this, it’s interesting for me to introspect on how little my mentoring activities have shifted since I authored my first version of this document. This is the centerpiece of guidance I developed and field-tested over seventeen elapsed years of Oracle DBA team building and mentoring.  If you intend to become a DBA, study my selected subset of the Introduction chapter as if you were preparing to take a test on it. In 1995, one of my protégés started with the ingredient goo that evolved into this guidance table. He went on inside a year to become a successful consultant working for Oracle Corporation.

Why am I recommending a document and release version that ostensibly became obsolete years ago when Database release 10g went into mainstream adoption? Certainly not as an excuse to save time by repurposing existing collateral. I just rewrote this approach table from scratch. I find the Database Concepts Guide’s Introduction chapter got cumbered as of release 10g, and the nice self-contained introductory chapter is no longer what it used to be. I’m not saying I could have done a better job authoring the newer Concepts Guides. You can return later to more recent versions’ concepts guides to spot look up newer features as needed. But I believe the benefit of a coherent, consolidated read exceeds the risk of wasting time in older release documentation. It can be difficult to find quality vendor-provided documentation. Oracle’s documentation, at least up through 9i, is a notable exception.

The 9i Concepts Guide’s Introduction chapter is 67 pages. In the approach guidance table that follows, I’m suggesting that you read only about two thirds of that chapter. Your first read needs to be slow and careful to understand the concepts. Then return to review the concepts sufficient to suit your purposes.

The following approach table follows all the way through for DBA candidates’ needs. Non-DBA candidates will know where short of that to stop. DBA candidates should follow this overview with a study of the 2-Day DBA guide first published with Database 10g. I have my own selective approach to that document as well, which is beyond the scope of this post.

Here is your syllabus:

Oracle9i Database Concepts
Release 2 (9.2)
March 2002
http://docs.oracle.com/cd/B10501_01/server.920/a96524.pdf

By Jim Hannan (@HoBHannan)

As virtualization adoption increases for Tier 1 workloads, many organizations are looking for ways to automate deployments. Oracle VM has touted this as one of its premier features (Users Can Deploy Oracle Real Application Clusters Up to 10 Times Faster with Oracle VM Templates). In my opinion Oracle has done a really good job with this. As a virtualization software provider, Oracle recognized early on the importance of automating the deployment of Oracle workloads. Oracle primarily does this with Oracle VM templates, which I believe have some shortcomings (these shortcomings are the same for VMware templates). Templates for both Oracle and VMware have the same concept. A virtual machine is created and the copied to a template for future deployments. The problem with templates is that are they are out-of-date within the next release of a security patch or organizational standard change. Managing and updating the template itself becomes a burden with dimensioning returns the older the template gets.

Many organizations have recognized the shortcomings of templates and have moved to automated deployments. HoB has had the opportunity to assist and help customers with automation. Each organization we have worked with has had a slightly different approach to the solution, but fundamentally they all used a similar approach.

Common Approach:

1. Create a virtual machine (vCPU, memory, network)
2. Boot the virtual machine (typically using a boot image)
3. Begin automated OS install (most common for Linux is kickstart)
4. Use scripts to further customize the build
5. Install Oracle software (process varies)
6. Create the Oracle database (process varies)

The table below contains the different steps as outlined above with the various tool options. Take note that in the tool column, I have added numbers, e.g. [1] or [2]. This represents different tools for the particular step.

In my next blog, I will do a walk through of some of the tools mentioned above.

By Jim Hannan (@HoBHannan)

Oracle Ksplice (also referred to as uptrack or Ksplice uptrack) http://www.ksplice.com is a kernel patching tool for Linux that does not require a reboot after applying a patch.

"On July 21, 2011, Oracle announced that it has acquired Ksplice, a privately-held company based in Cambridge, MA. Ksplice is the creator of innovative zero downtime update technology for Linux."

To date, this tool is the first of its kind for Linux. Ksplice allows for Linux administrators to patch the Linux kernel without taking an outage or shutting down any applications — like Oracle RDBMS!

For business critical applications, like Oracle databases, this is a really compelling feature.

Oracle Ksplice offers the following:

• No downtime
• No disruption (application like Oracle are unaffected)
• No rebooting
• And potentially better security (In today’s industry, security patching is a must. Often the Linux kernel has security updates — Ksplice allows you apply the updates without an outage.)

Now, to be fair, I have not been a big proponent of Oracle Enterprise Linux (OEL) over Red Hat and Suse Linux. There are primarily two reasons for this, one is because of hardware support — most hardware vendors do not officially support Oracle Enterprise Linux. This does not mean that most hardware vendors will not help you if you call with an issue. However, it does mean that if you have driver issues running in OEL, the vendor will most likely not help you. The other reason I have not advocated for OEL in the past is because of VMware support. Until September 2012, VMware did not offer support for OEL and UEK and a supported guest OS. As the announcement below shows, UEK on OEL is now supported in vSphere 5.1.

How to get Ksplice

Ksplice is offered free of charge with the purchase of Oracle Premier Support for OEL. Ksplice can also be purchased for Red Hat Linux, but you will need Oracle Premier Support. You will also need to use Oracle’s software repositories instead of Red Hat’s.

How Ksplice works

Ksplice is a kernel-patching tool only. This means you need to continue using yum to update your RPMS.

Oracle releases a Ksplice patch by using the following workflow:

Kernel update released -> Ksplice technology (Oracle (using Ksplice technology) takes the patch and creates a Ksplice patch) -> Customers can download and apply Ksplice kernel patch.

As I mentioned before, the tool is often referred to as uptrack. That is because of the command line wrapper script names (see below). The tool also has web interface, which I am only familiar with by demo. The interface offers a portal of all the Linux nodes checked in under your Ksplice key.

¹Oracle recommends running this commnd over uname -a because the running Linux  OS by become unaware is a hot patch has been applied.

There you have it, a great feature for customers with tight SLAs. I also applaud VMware for supporting OEL. This to me re-demonstrates VMware’s commitment to their Oracle customers running on VMware. And in my opinion, the best platform to run OEL is VMware vSphere.

As I find my social media chops, no doubt I’ll post more frequently with less content per post. But meanwhile…

I’m told that Oracle’s Larry Ellison also began contributing to the Twitterverse this summer.

House of Brick’s Oracle VBCA Boot Camp Sunday 9/30 came off better than I expected, and I had high expectations. I think the attendees feel good about it, too, based on their end-of-session group chorus calling for more. I’m inclined to put extra weight on this particular group’s feedback. They were more interactive than most groups and therefore challenged me in helpful ways. I’ll be channeling the feedback into House of Brick’s SQL Server boot camp Tuesday November 6^th in Seattle in conjunction with the SQL Pass conference.

I’m declaring Colin Bieberstein of Husky Energy – Calgary as my boot camp guest of honor. I got the impression that Colin may have made significant personal sacrifices to attend on just two weeks’ notice.

As for Database 12c, I’m pleased that the announcement made before Larry Ellison took the stage aligned with what I’ve been already telling everyone. A few months ago Larry predicted the 12c GA release this December or January. That was never going to happen, minimally due to the increasing complexity of the release. Add to that the coding and QA challenges Oracle faces with what has always given the appearance of involving an Exa code branch.

The announcer said sometime in calendar year 2013. I’ve been telling everyone not a minute before June but more probable toward December 2013. I’ve also been saying don’t look for release stability worthy of production systems anytime before the middle of 2014. That’s not a hit on Oracle. It is just the nature of the beast with code this complex underpinning business-critical systems. I continue to be in love with their red stack software, and sincerely hope that the love affair continues.

As for Database 12c’s multitenancy (pluggable databases on top of a consolidated instance), I firmly believe it is tooling the wrong way at the wrong stack layer. That was my immediate reaction on December 19^th, 2011 when I first got wind of the release’s architectural direction. On the contrary, I have been preaching for three years an architectural direction that I have dubbed “atomicity.” That is, in vSphere environments, move toward an alignment of one database instance, middle tier component, or utility per guest OS.

Atomicity accomplishes two major objectives. It makes architecture with tinker toy components much easier for those building initial stack prototypes and with less technical administrator involvement. Think vCloud Director. That accelerates time to market. Smaller memory/CPU workload alignments also dramatically facilitate live migration. That facilitates dramatically better utilization of an Oracle processor-based license, which in many cases can be a stack’s most expensive component.

I am constantly challenged in my travels by hardware-centric DBAs and System Administrators justly concerned about the prospect of managing four to ten times as many Oracle executable and OS instances. I tell them that 100% of their vSphere-experienced peers that I interact with in shops that know what they’re doing with vSphere say they would never go back. The model is to allocate a fraction of the savings that vSphere provides to patch automation tooling.

Database 12c is attempting virtualization not one but two layers higher in the stack than the vSphere platform that accomplishes virtualization in spades. When architecting system stacks, always assume to begin the discussion by tooling at the lowest layer of the stack possible, unless there are very compelling business and/or technical justifications to do otherwise.

Oracle VBCA is predictable when you know what you’re doing. I tell people that the one-time replatform from big iron to x86 is the hard part, minimally because, unlike us, enterprises don’t do it every day. I’m convinced the RISC UNIX->x86 hop is the single most important thing organizations can plan for in their move toward the world’s premier platform vSphere. Accordingly, Jeff Browning’s replatforming preso will get my nod for OOW best of show. He sleuthed out the fact that RMAN CONVERT’s sys.dbms_backup_restore endian translation used to work just fine into any x86 platform. That is until three years in October 2008, when patch 13340675 “fixed” it to only work with Exadata. You can also pick up Jeff’s preso recording at VMworld 2012.

Jeff invited me to lunch yesterday. The more time I spend with Jeff, the more humbled I am by how big his heart is. Thanks, Jeff. I continue to benefit from our professional and personal friendship.

The VMware 2012 Pavilion is appropriately themed with vFabric at every turn. Charles Fan is a humble, self-effacing man who would never bring up the incredible things he’s done for Joe Tucci. Charles’ vision for what is now called vFabric is really finding its voice, and that’s clear as you stroll around the VMware pavilion.

Yesterday vFabric’s Bill Bonin shared perspective with me--a month after VMware’s Chief Performance Officer Richard McDougall did the same thing--on how big the in-memory Hadoop market is projected to be in just a few short years. Guys, thanks to both of you for your one-on-one attention to make sure I keep Hadoop in my periscope.

VMware, I need you to align vFabric Data Director with vCloud Director strategically if not integrate them technically. Yet they are being marketed independently and have segregated internal organizations. I’m at a loss to explain the lack of product alignment as I now add the vFabric Data Director message to what I have always felt was VMware’s best tooling: was Lab Manager is vCloud Director. There’s an apparent massive operational deployment intersection here. This is no different than my incessant internal harping to you years ago that Lab Manager and Stage Manager were the same thing, despite the fact that they were separate code bases. You eventually merged Stage Manager into Lab Manager. C’mon, vFabric Data Director was announced a year ago. As one who provided architectural guidance into what was code-named “Aurora” and as a lightning rod evangelist for VBCA, I’m struggling to articulate the cogent unified vision for these separate products. It’s got to be even harder for your prospects to capture that vision. It’s going to get even worse now with your phenomenally shrewd acquisition of DynamicOps.

I ran into a former Oracle RAC employee yesterday who confirmed what we always knew. The statement that got deleted from the published My Oracle Support note (off the top of my head) “There are technical restrictions that prohibit the certification of RAC in a VMware environment” had to do with clock drift. (Clock drift went away with vSphere 4 in 2009).

Ron Zellars from the world’s largest ice cream factory - Wells Dairy - stopped by the VMware pavilion to say hi. We are proud to have helped them over a year ago with their EBS R12 upgrade and replatform of RAC to vSphere.

Brian and Richard from the U.S.’s largest appliance manufacturer - G.E. Appliances and Lighting - also stopped by to say hi. Brian’s business card now sports “Chief Evangelist” because CTO Lance Weaver thought it was cool on my card when GE and HoB got introduced here last year. I may have to declare Brian’s business card title as my biggest professional accomplishment for 2011!

Having said that, my most enjoyable encounter of the day was with Wize Commerce’s DBA Selina Lin out of San Mateo, CA. Selina took in my “Oracle RAC and VMware HA Tooling - A Decision Tree” VMware theater preso but couldn’t return Tuesday for the replay of my “Business Critical Applications Performance – VM vs. Native” preso. So we found a couple chairs just outside exhibit hall doors and took our time with that preso one-on-one. Selina, our ½ hour working session made my day. All the best to you.

My esteemed colleague Cisco’s Tushar Patel invited me to lunch today. Tushar was the VMware-side engineering force behind HoB’s groundbreaking VMware Oracle Solutions Lab at OOW 2007.

My only VMware 2012 pavilion complaint: inadequate white board space. Thank goodness for the iPad Paper app!

And speaking of apps, Uber is the coolest most practical iPhone/iPad app I’ve seen lately! Map all the available drivers in your geo and summon up the nearest one with a click without placing a call to a cab dispatcher.

Run Uber on the iPad 3 you’re going to win when you physically present Tuesday and Wednesday in the VMware Pavilion at 5:30. Your odds are pretty good as I’m thinking there just weren’t that many bodies present given the size of the daily prize.

To close out this mega-post: I was asked yesterday for feedback on a partner strategy event earlier this year. I answered with a concept that I’ve introduced into my professional activities from my volunteer teaching experience: “Never let planned content interfere with a productive discussion.”

By Jim Hannan

I recently discovered a white paper published by VMware on tuning latency-sensitive workloads:

Best Practices for Performance Tuning of Latency-Sensitive Workloads in vSphere VMs

Being part of a performance team that virtualizes business critical applications, we are always looking for better methodologies. So obviously a white paper written by VMware on how to improvement performance for latency sensitive workloads would be of interest.

This blog discusses some the tweaks and settings introduced in the white paper. I have also provided recommendations on whether we would suggest using each performance tweak. I think it is important to stress two things before considering using any of the settings.

• VMware very clearly states that for most workloads these tweaks are unnecessary. The category of application that would benefit from these settings are truly for latency sensitive applications that have SLAs in the sub-second range.
• In some cases we have not had the opportunity to benchmark or examine the results of settings. So care should be taken. At HoB, we believe the best approach is to first benchmark in a test environment before implementing in production.

Turn off BIOS Power Management

This is something we hope all customers are doing. The Intel Nehalem has two power management options:

• C-state
• Intel Turbo Boost

C-state can increase memory latency according to VMware and is not recommended. Intel Turbo Boost should be left on. According to VMware it will increase the frequency of the processor should the workload need more power.

Tickless Kernel in RHEL 6

We have watched this from afar for 5 plus years. In the early versions of ESX, the guest would suffer from clock drift. Clock drift is when the clock of the OS falls behind. This was common for SMP would loads that were busy doing work or on an ESX host with constrained resources.

Moving off the RHEL 5 to RHEL 6 and the tickless kernels can reduce application reduce latency. The tickless kernel is a better time keeping mechanism. Additionally, VMware is claiming is can offer a better performance for latency sensitive applications.

NUMA Node Affinity

NUMA affinity basically assigns a VM to a NUMA node. This can be monitored with ESXTOP. I would not recommended this until it has been determined that NUMA latency is an issue. We say this because each application handles NUMA differently. Oracle for example chooses to not use NUMA as of 11g.

To monitor NUMA latency with ESXTOP

esxtop > f > g > enter > (capital) V

N%L < 80 for any VM than you the workload may have NUMA latency issues.

Interrupt Moderation

Most NICs support a feature called interrupt moderation or interrupt throttling. This basically buffers (or queues) interrupts for the network card so the host CPU does not get overwhelmed. This can be disabled on most NICs to give latency sensitive application better network throughput. VMware recommends this in only the most extreme cases. We agree that this should be used carefully. We would consider this with Oracle RAC workloads that are suffering latency issues on the RAC interconnect.

VMXNET3 Interrupt Coalescing

As of vSphere 5, VMXNET3 supports interrupt coalescing. Interrupt coalescing is similar to a physical NICs interrupt moderation. The philosophy behind interrupt coalescing is to benefit the entire cluster by reducing the CPU overhead of TCP traffic. However for latency sensitive applications--like Oracle RAC--it is best to disable the interrupt coalescing.

Go to VM Settings →Options tab →Advanced General →Configuration Parameters and add an entry for ethernetX.coalescingScheme with the value of disabled.

If you have a cluster or host that runs latency sensitive applications you can also disable it for the entire cluster with the setting below.

Click the host go to the configuration tab → Advance Settings → networking performance option CoalesceDefaultOn to 0 (disabled).

VMXNET3 Large Receive Offload (LRO)

Similar to the feature above, the VMXNET3 feature LRO aggregates multiple received TCP segments into a large segment before delivery to the guest TCP stack.

We recommend that you disable LRO all Oracle virtual machines.

# modprobe -r vmxnet3

Add the following line: /etc/modprobe.conf

(Linux version dependent): options vmxnet3 disable_lro=1

Next, reload the driver: # modprobe vmxnet3

Prevent De-scheduling of vCPUs

I have a hard time imagining that heavy workloads need to use this setting. You can ensure that a virtual machine is never de-scheduled from a pCPU (physical CPU) by configuring the advance setting monitor_control.halt_desched. This setting is similar to a tight loop an application might have when a process never sleeps. It just continually executes a loop. VMware describes this setting as the VM becoming the owner of the pCPU (or pCPUs). Which indicates to me that the pCPU is not eligible to schedule other workloads, it in effect becomes dedicated to the “pinned” workload.

Go to VM Settings →Options tab →Advanced General →Configuration Parameters and add monitor_control.halt_desched with the value of false.

Reference Chart

By Jim Hannan

Huge Pages (also known as Large Pages) is something we at House of Brick are often asked about. There is significant information on the subject, however, the information is scattered throughout MetaLink and various other resources. I have attempted to consolidate some of this information, and have added some of our experiences and best practices for implementing Huge Pages.

Linux Memory Kernel Parameters

Before implementing Huge Pages, a check of kernel memory settings should be performed. Two key settings for successfully implementation are shmall and shmax.

Oracle makes use of one of the three memory management models to create the SGA during database startup, and it does this in the following sequence: First, Oracle attempts to use the one-segment model (most optimal). If this fails, it proceeds with the next one, which is the contiguous multi-segment model (contiguous smaller chunks). If that fails as well, it goes with the last option, which is the non-contiguous multi-segment model. This model can potentially cause memory fragmentation.

Reference: http://docs.oracle.com/cd/B12037_01/server.101/b10755/initparams166.htm

Oracle memory kernel parameters adjusted:

kernel.shmall = 1073741824
kernel.shmax = 4398046511104

Operating System Huge Pages

Using Huge Pages is a technique that allows the operating system to allocate and manage memory using a larger-than-standard page size. This term is synonymous with “Large Memory Pages.” As of ESX 3.5 the hypervisor is enhanced to support large pages. Before 3.5, ESX Server used small pages to emulate large pages. The default page size for most systems is 4KB (4096 bytes). Huge Pages are typically 2MB or 4MB. Managing memory using a larger page size reduces the required number system calls by a factor of 500 or 1,000.

Due to dramatic improvements in hardware and memory performance, the smaller page size has not traditionally been a significant problem. However, as memory capacity has the potential to exceed 1TB, and databases can effectively manage hundreds of GBs of memory, it becomes significant. Huge Pages were introduced to address this issue. The performance impact associated with the system calls relating to memory management is particularly noticeable when looking at very large memory allocations under a hypervisor, because memory operations must be protected. Protected system calls require a degree of serialization and can experience a measurable performance penalty under a hypervisor. Therefore reducing the number of calls has a high return.

A 14% performance improvement has been shown under load for an Oracle database with a 24GB SGA (shared memory pool) when using huge pages as compared with standard pages. A 40% improvement was measured for a system with a 96GB SGA with no other changes.

Under The Hood of Huge Pages

The CPU's Memory Management Unit (MMU) stores a cache of recently used memory address. This is called Translation Lookaside Buffer (TLB). TLB is a memory address table stored in the near CPU cache. Without TLB, a modern day OS does a "Page Walk" walking the physical memory to find the physical memory address, this process causes higher latency. With TLB the OS can check the TLB cache for the memory address for the required memory location, if successful this is called a "page hit". A TLB miss is called "page fault", which results in a Page Walk. When your application uses Huge Pages, a larger memory range is addressed, allowing for more page hits. This reduces the time it takes to find data in memory and the lowers the CPU cost.

Linux handles huge pages gracefully by reserving a pool of memory at system boot time. This amount of memory can be increased or decreased on the fly using sysctl, but an increase requires that the memory be available to be successful without a reboot. Using Huge Pages in Linux implies memory locking. User limits must accommodate locking of the entire memory segment for Huge Pages to be successful.

Database 11g Huge Pages

Enabling expanded Automatic Memory Management (AMM) in Database 11g prohibits use of Linux huge pages. This is true of all current Database 11g R1 and R2 releases and patches, including the latest (Database 11.2.0.2). Database 11g R1 expanded AMM to include the Program Global Area (PGA). AMM is not active by default in Database 11g; it is enabled via the parameters MEMORY_TARGET and MEMORY_MAX_TARGET. However, the Database 10g SGA_TARGET and SGA_MAX_SIZE parameters continue to work as they did previously.

AMM does not use standard System V-style IPC shared memory. Instead, AMM uses memory mapped files in /dev/shm. The drawback is that memory mapped files do not support Huge Pages. This is a fundamental problem with AMM if Huge Pages are desired.

Of the two features, it is strongly recommended that Huge Pages take priority, regardless of the workload characteristics and database instance memory size. In general, Huge Pages is a performance feature, whereas AMM is a configuration feature.

Database 11.2.0.2 introduced the initialization parameter USE_LARGE_PAGES to provide more Huge Pages-related instance control. Its values are TRUE, FALSE, or ONLY. The default is TRUE and causes Oracle to behave as it always has. Set the parameter to ONLY to enforce the use of Huge Pages and prevent instance startup if sufficient huge pages are not available.

Finally, although all versions of Oracle since version 7 have supported Huge Pages, Database 11.2.0.2 is the first release that provides any meaningful output in the alert log relating to Oracle’s use of them.

Metalink References:
HugePages on Oracle Linux 64-bit [ID 361468.1]
HugePages on Linux: What It Is... and What It Is Not... [ID 361323.1]
Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration [ID 401749.1]
HugePages and Oracle Database 11g Automatic Memory Management (AMM) on Linux [ID 749851.1]

Huge Pages Implementation Example

Note: As stated previously, Huge Pages are not compatible with Oracle AMM.

alter system set sga_max_size = 44000M scope=spfile;
alter system set sga_target = 44000M scope=spfile;
alter system set pga_aggregate_target = 4000M scope=spfile;
alter system set pre_page_sga = TRUE scope=spfile;

Check to see if hugepages are currently configured:

grep Huge /proc/meminfo

If hugepages_total is > zero, then this is the number of Huge Pages currently configured. Huge Pages are allocated in 2MB chunks. Next, calculate the number of Huge Pages needed for a 44GB SGA, and add ten pages for overhead.

(44000/2) + 10 = 22010

The setting for hugepages is configured in /etc/sysctl.conf. To check the current setting, run the following command:

grep nr_hugepages /etc/sysctl.conf

Check /etc/security/limits.conf to make sure that Oracle can lock shared memory. This number is expressed in KB, set this to a number higher than 44000000 (44 GB expressed in KB)

grep oracle /etc/security/limits.conf
oracle    soft    memlock 50000000
oracle    hard    memlock    50000000

Note: This section must be done as root.

Add or modify the following line to /etc/sysctl.conf:

vm.nr_hugepages=22010

Add or modify the following two lines to /etc/security/limits.conf:

oracle    soft    memlock 50000000
oracle    hard    memlock    50000000

Add or modify the following two lines in /etc/sysctl.conf:

kernel.shmmax = 4398046511104
kernel.shmall = 1073741824
kernel.shmmni = 4096

kernel.shmax = 4398046511104

The system must be rebooted to have the hugepages setting take effect. After the reboot check the Huge Pages setting.

# grep Huge /proc/meminfo
HugePages_Total:  22010
HugePages_Free:   22010
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

If the settings are correct then start the database. Confirm Huge Pages are allocated with the command below.

grep Huge /proc/meminfo
HugePages_Total:  22010
HugePages_Free:      9
HugePages_Rsvd:      0
Hugepagesize:     2048 kB