After finishing the implementation of mapping and conversion from Ecore to ProtoBuf I wanted to benchmark the performance in comparison to the existing BinaryResourceImpl and XMIResourceImpl. So, I started looking for articles and frameworks for Java performance benchmarks. I found the most interesting article series with the title “Robust Java benchmarking” at IBM developerWorks. It even includes a benchmarking framework, implementing all hints given in the articles. Running the benchmarks with this framework worked and returned sensible results, but two things bothered me: the long execution time of 2-3 minutes for a benchmark run and that it required 3 JARs including a large amount of unused code. Therefor, I looked for an alternative. Besides, I don’t need nanosecond accurate measurements and all the statistics stuff provided by the mentioned framework.
My main requirements for an alternative framework were:
- support for code warmup, so the JIT compiler has a chance to optimize the code
- time measurement with System.nanoTime() instead of System.currentTimeMillis()
- compatibility with Eclipse’ JUnit Plug-in Tests
Here is the list of candidates I assessed and which requirements they met:
I tried to implement the benchmarks with JUnitBenchmarks and Caliper. Although Caliper seemed to be quite promising, it turned out to not work well on Windows and with JUnit Plug-in Tests, because it spawns new JVM processes for benchmarking. I didn’t have these problems with JUnitBenchmarks. The downside of JUnitBenchmarks is that it uses System.currentTimeMillis() and displays results only with a resolution of 0.01 seconds. Actually, there is a discussion on whether to use System.nanoTime() or not. Finally, I decided to use JUnitBenchmarks and could verify the results I got with the “Robust Java benchmarking” framework, but with a much lower overall execution time.
In the next post I will present the benchmark results.