Last week, Oracle came out with a new TPC-C benchmark that delivered a record 30 million tpmC, breaking the prior record of 10 million tpmC delivered by IBM DB2 in August. You might think that Oracle has regained their crown as the transactional database leader. That is, unless you actually looked at how they came up with those results, as Conor O’Mahony eloquently points out…
- Oracle’s December TPC-C results came on a configuration of 27 SPARC servers with a total of 1,728 processor cores, while IBM’s August results came on a configuration of only 3 IBM Power 780 Servers with 192 cores. Oracle used 9 times the hardware, yet only got 3 times the performance.
- This result is very consistent with the SAP two-tier SD standard application benchmark run in August, where DB2 delivered three times the performance of Oracle when both were running on 256 core configurations.
- In order to avoid performance degradation, Oracle turned off page integrity checking (which few customers would actually do in production). IBM’s test were conducted with page integrity checking on.
- Oracle’s TPC-C price/performance results were calculated using lease licenses rather than standard perpetual licenses, and reflected special support terms that don’t include unlimited problem resolution, 24×7 unlimited mission critical direct-to-engineer support, or upgrade protection. If Oracle had gone with their standard licenses and 24×7 support agreement, as IBM did, they would have been quite a bit more expensive than IBM.
While you might say that Oracle outfoxed IBM by exploiting loopholes in the benchmark requirements, is this really what you want from your vendor?
What this shows is that benchmarks can’t be viewed as absolute comparisons. You have to dig under the covers to see what they truly include. And some vendors will do deceptive things to make their numbers look better.
— Results on Transaction Processing Performance Council web site at http://www.tpc.org Results are as of 12/2/10.
— Oracle SPARC SuperCluster with T3-4 Servers (27 x 64 core) (108 chips, 1728 cores, 13824 threads); 30,249,688 tpmC; $1.01/tpmC; available 6/1/11.
— IBM Power 780 cluster (3 x 64 core) (24 chips, 192 cores, 768 threads); 10,366,254 tpmC; $1.38/tpmC; available 10/13/10.