Have you looked at YCSB? It's a pathologically poorly designed benchmark. It essentially tests how good a developer is at implementing a Java wrapper around a toy object model and interfacing that with a database.
A proper benchmark would hold constant things which are reasonably expected to be constant: the use case scenario, the data, the warmup requirements, the concurrency, and the count. If feeling frisky, the hardware, operating system and networking environment. And then step back.
For example, "On an EC2 Extra Large instance running Ubuntu 11.10 with whatever tuning the package recommends, what is the transactions per second when 50 vendor-provided clients on the same LAN are attempting to write and read random records out of a pool of 10mm 5k records?"
Comments
Have you looked at YCSB? It's a pathologically poorly designed benchmark. It essentially tests how good a developer is at implementing a Java wrapper around a toy object model and interfacing that with a database.
A proper benchmark would hold constant things which are reasonably expected to be constant: the use case scenario, the data, the warmup requirements, the concurrency, and the count. If feeling frisky, the hardware, operating system and networking environment. And then step back.
For example, "On an EC2 Extra Large instance running Ubuntu 11.10 with whatever tuning the package recommends, what is the transactions per second when 50 vendor-provided clients on the same LAN are attempting to write and read random records out of a pool of 10mm 5k records?"