Comments on: Java vs. C benchmark #2: JET, harmony and GCJ

By: Another Question on What Language to use!

Another Question on What Language to use! — Sun, 14 Mar 2010 20:07:05 +0000

[...] hence they are not directly equivalent. Really? Where’d you get these statistics? Stefan Krause.blog() Blog Archive Java vs. C benchmark #2: JET, harmony and GCJ Well excuse me if I don’t own my own server farm. It is not needed. Server Java executes [...]

By: Ivan Maidanski

Ivan Maidanski — Fri, 12 Feb 2010 09:59:49 +0000

Stefan, I could suggest another tool for the benchmark – JCGO (http://www.ivmaisoft.com/jcgo/), a Java-to-C ahead-of-time translator, which, in conjunction with a good C/C++ compiler, could produce a highly optimized native code for the targeted platform.

By: Stefan Krause.blog() » Blog Archive » Update For Java Benchmark

Stefan Krause.blog() » Blog Archive » Update For Java Benchmark — Wed, 02 Jul 2008 18:32:56 +0000

[...] Stefan Krause.blog() { throw new NoFunnyProverbFoundException(); } « Java vs. C benchmark #2: JET, harmony and GCJ [...]

By: Brent

Brent — Thu, 01 May 2008 08:46:51 +0000

Stefan, I just wrote a Java benchmarking framework that may be of interest to you to use in benchmarking Java code. In fact, this framework will be discussed as part of an article that I am writing for IBM developerWorks on measuring Java performance. If interested, send me an email, and I can send you an article preprint.

By: Deniz Oguz

Deniz Oguz — Tue, 11 Mar 2008 06:01:51 +0000

I am afraid you are right. There is only x64 versions. Currently only 6u4-p download is available and said to be %10 behind of 6u5-p. I hadn’t check it before
I do not have too much information between performance releases and normal releases. From the blog I expect their JIT compiler produce more effective code.

By: admin

admin — Mon, 10 Mar 2008 22:05:40 +0000

Wow – it’s the first time I heard of “-p” releases. The link and download page don’t have much information about them. For linux there’s just a x64 version, is that right? What are the differences to the normal releases? Is 6u5-p already available as a download?

By: Deniz Oguz

Deniz Oguz — Mon, 10 Mar 2008 20:09:13 +0000

Hi stefan,

Could you try lates performance release of JDK 6.0? JDK 6.0-u5-p is a result of collaboration between intel and sun and as stated in the following blog it has world record on specjbb2005. http://blogs.sun.com/dagastine/entry/slam_specjbb2005_world_record_on1
You could download it at http://java.sun.com/javase/technologies/performance.jsp
Thanks for your efforts.

By: Vitaly Mikheev

Vitaly Mikheev — Tue, 06 Nov 2007 08:32:21 +0000

Hi Stefan,

If people start reading the “Java vs. C performance” series from the November issue, they may be unaware of that you “warm-up” JVMs with dynamic compilers before measuring the time so the peak performance is actually reported.

I think it would be useful to clearly mention it each time and give a link to the description of your testing methodology.

—————

['bout GCJ:]
“These settings prevent array store and bounds checking so I’ve even allowed aggressive optimizations that violate the JLS. This optimization might be considered unfair because the other VMs have to perform those checks”

It’s ok for benchmarking. I beleive JVMs should do a better job to remove index checks and show good comparative results even if checks are turned off for GCJ or no checks are generated by C compilers.

Thanks,
–Vitaly

By: Isaac Gouy

Isaac Gouy — Mon, 05 Nov 2007 21:40:11 +0000

“aren’t perfectly scientific”

My guess is that with so few samples the t-test is pretty meaningless.

I’m won’t quarrel about whether you have enough time or not, obviously it’s different from my perspective. You have programs running 10 20 50 seconds and the benchmarks game has programs running 9,000 13,000 seconds – so it just seemed possible for you to do more samples.

By: admin

admin — Mon, 05 Nov 2007 20:52:54 +0000

Hi Issac, the sample size hasn’t changed (i.e. the sample size is only 4), I understand and accept your criticism that these benchmarks aren’t perfectly scientific (with one reason that the sample size is too small).
However I’m not going to change the sample size simply because I don’t have enough time for it and that’s not the purpose of those benchmarks (you could do that on the language shootout, but since JRockit has a much larger warmup phase than Hotspot the tests should be adapted to run multiple times in one process like here or at least take much longer).
These benchmarks here should show the peek performance of JVMs compared to each other and C++. I think the results presented here are exact enough for deciding whether the factor 2-3 is true or not (And depending on the benchmark it is or is not…). Harmony wouldn’t beat C++ if sample size went up and the hotspot compilers wouldn’t perform any better for fannkuch either.

Still to justify publishing a ranking even when results were very close like for nbody: I agree that small differences could affect the results, but the samples had very small variation.Tests that showed a large variation were rerun to eliminate influence by other OS processes as much as I could. You can take a look at a “scatterplot” (or at least what can be done easily with openoffice) for NBody on http://www.stefankrause.net/wp/imgs/scatterplot_nbody.jpeg (The first sample is the warmup run that has been left out. You can see that JRockit indeed has a much slower warmup phase)