Oracle and the deceptive art of benchmarking

Last week, Oracle came out with a new TPC-C benchmark that delivered a record 30 million tpmC, breaking the prior record of 10 million tpmC delivered by IBM DB2 in August. You might think that Oracle has regained their crown as the transactional database leader. That is, unless you actually looked at how they came up with those results, as Conor O’Mahony eloquently points out

  1. Oracle’s December TPC-C results came on a configuration of 27 SPARC servers with a total of 1,728 processor cores, while IBM’s August results came on a configuration of only 3 IBM Power 780 Servers with 192 cores. Oracle used 9 times the hardware, yet only got 3 times the performance.
  2. This result is very consistent with the SAP two-tier SD standard application benchmark run in August, where DB2 delivered three times the performance of Oracle when both were running on 256 core configurations.
  3. In order to avoid performance degradation, Oracle turned off page integrity checking (which few customers would actually do in production). IBM’s test were conducted with page integrity checking on.
  4. Oracle’s TPC-C price/performance results were calculated using lease licenses rather than standard perpetual licenses, and reflected special support terms that don’t include unlimited problem resolution, 24×7 unlimited mission critical direct-to-engineer support, or upgrade protection. If Oracle had gone with their standard licenses and 24×7 support agreement, as IBM did, they would have been quite a bit more expensive than IBM.

While you might say that Oracle outfoxed IBM by exploiting loopholes in the benchmark requirements, is this really what you want from your vendor?

What this shows is that benchmarks can’t be viewed as absolute comparisons. You have to dig under the covers to see what they truly include. And some vendors will do deceptive things to make their numbers look better.

— Results on Transaction Processing Performance Council web site at  Results are as of 12/2/10.
— Oracle SPARC SuperCluster with T3-4 Servers (27 x 64 core) (108 chips, 1728 cores, 13824 threads); 30,249,688 tpmC; $1.01/tpmC; available 6/1/11.
— IBM Power 780 cluster (3 x 64 core) (24 chips, 192 cores, 768 threads); 10,366,254 tpmC; $1.38/tpmC; available 10/13/10.

Tagged ,

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: