> Using stats –trim-outliers, which throws away best and worse
people make this mistake over and over again. there's no point in taking anything but the best time when talking about performance in a preemptive multiprocessing environment. the best time isn't an outlier, every time other than it is!
The best time is irrelevant in practice if it only happens very infrequently, under a very specific set of circumstances.
Benchmarks are only useful if they're indicative of what the likely outcome in a real-world scenario will be.
What you propose is much akin to those completely unrealistic "The Computer Language Benchmarks Game"-style benchmarks that show JavaScript (or Java, or PHP, or whichever slower language you prefer to use instead) as being as fast as C or C++. Perhaps that's true, but only when you discard reality almost completely. JavaScript only ends up performing decently because the benchmark is trivial, and the implementation has been highly tuned, often into a form that we'd never actually see in any real-world product. And predictably, JavaScript's performance ends up being much worse than C's or C++'s when it comes to real software, contrary to what the flawed benchmark suggests.
> The best time is irrelevant in practice if it only happens very infrequently, under a very specific set of circumstances.
This isn't a practical benchmark, though. This is an experimental performance comparison. So to ensure that your results aren't influenced by other variables, it's a good idea to attempt to produce the ideal environment for best performance across both tests. So binding cores(to eliminate as much variance from Linux's thread scheduler) and running nothing else on the machine are actually good decisions to make here.
Comparisons to the Computer Language Benchmarks Game is somewhat irrelevant here, because the comparison being run is actually extremely simple: Using a GCC executable compiled as C versus a GCC executable compiled as C++. The traditional failings of the game aren't here(implementation issues, language-specific tuning issues, etc.) because the code that GCC is compiled from is identical, the compiler options should be identical(except for C vs C++ compile setting), and the test setup should be run on the same machine, with the same compiler options and tuning choices.
>> that show JavaScript (or Java, or PHP, or whichever slower language you prefer to use instead) as being as fast as C or C++ <<
I've learned that it's unrealistic to expect someone to have actually looked at "The Computer Language Benchmarks Game" before pointing to it as an example for whatever point they wish to make ;-)
The only mystery here is why you think we wouldn't notice that "the implementation has been highly tuned" etc applies just as much to the C and C++ programs as to the JavaScript programs.
real-world is not preditcable. i can make any program run arbitrarily slow by making it swap or nice it into oblivion. if i'm measuring microbenchmark performance, i want to know how fast can it possibly go. measuring real-world averages makes sense only in a real-world deployment setting - this obviously isn't the case here.
No, there are other things which affect timing besides switching to other processes. For example, virt:physical mapping of the process memory affects cache performance unpredictably.
Comments
> Using stats –trim-outliers, which throws away best and worse
people make this mistake over and over again. there's no point in taking anything but the best time when talking about performance in a preemptive multiprocessing environment. the best time isn't an outlier, every time other than it is!
The best time is irrelevant in practice if it only happens very infrequently, under a very specific set of circumstances.
Benchmarks are only useful if they're indicative of what the likely outcome in a real-world scenario will be.
What you propose is much akin to those completely unrealistic "The Computer Language Benchmarks Game"-style benchmarks that show JavaScript (or Java, or PHP, or whichever slower language you prefer to use instead) as being as fast as C or C++. Perhaps that's true, but only when you discard reality almost completely. JavaScript only ends up performing decently because the benchmark is trivial, and the implementation has been highly tuned, often into a form that we'd never actually see in any real-world product. And predictably, JavaScript's performance ends up being much worse than C's or C++'s when it comes to real software, contrary to what the flawed benchmark suggests.
> The best time is irrelevant in practice if it only happens very infrequently, under a very specific set of circumstances.
This isn't a practical benchmark, though. This is an experimental performance comparison. So to ensure that your results aren't influenced by other variables, it's a good idea to attempt to produce the ideal environment for best performance across both tests. So binding cores(to eliminate as much variance from Linux's thread scheduler) and running nothing else on the machine are actually good decisions to make here.
Comparisons to the Computer Language Benchmarks Game is somewhat irrelevant here, because the comparison being run is actually extremely simple: Using a GCC executable compiled as C versus a GCC executable compiled as C++. The traditional failings of the game aren't here(implementation issues, language-specific tuning issues, etc.) because the code that GCC is compiled from is identical, the compiler options should be identical(except for C vs C++ compile setting), and the test setup should be run on the same machine, with the same compiler options and tuning choices.
>> that show JavaScript (or Java, or PHP, or whichever slower language you prefer to use instead) as being as fast as C or C++ <<
I've learned that it's unrealistic to expect someone to have actually looked at "The Computer Language Benchmarks Game" before pointing to it as an example for whatever point they wish to make ;-)
The only mystery here is why you think we wouldn't notice that "the implementation has been highly tuned" etc applies just as much to the C and C++ programs as to the JavaScript programs.
real-world is not preditcable. i can make any program run arbitrarily slow by making it swap or nice it into oblivion. if i'm measuring microbenchmark performance, i want to know how fast can it possibly go. measuring real-world averages makes sense only in a real-world deployment setting - this obviously isn't the case here.
Honestly, more people should use stabilizer http://plasma.cs.umass.edu/emery/stabilizer . The best can still be an outlier.
haven't heard of it - thanks for the link. looks great.
What is the reasoning?
The best time measured is the time of execution with least switching to other processes. So, it's actually the closest to the real time.
No, there are other things which affect timing besides switching to other processes. For example, virt:physical mapping of the process memory affects cache performance unpredictably, see http://d3s.mff.cuni.cz/publications/download/hocko-sipew10.p...
No, there are other things which affect timing besides switching to other processes. For example, virt:physical mapping of the process memory affects cache performance unpredictably.