Cython is actually what is faster than Julia in Wes' comparison, not Python. Cython looks kinda, sorta like Python, but it is actually a static language with C-like types (but quite different syntax for those types), no polymorphism, and, afaict, ill-defined semantics. The best answer I seem to get about Cython's semantics is that Cython's semantics are whatever it does. I'm not alone in this complaint – Travis Oliphant expressed a similar concern at this year's SciPy (in this panel [http://www.youtube.com/watch?v=7i2vhoQY-K4], if I recall correctly), which is part of his motivation to work on Numba [https://github.com/numba/numba].
If you look at the comments on Wes' post, when I used the dot(x,y) function, which ships with Julia and uses a BLAS to compute the inner product just like the fastest "Python" version does, Julia is equally fast. That stands to reason – they're both just calling a BLAS.
Finally, that blog post is months old – since then Julia passed the milestone of being no slower than 2x C++ on its microbenchmarks suite [http://julialang.org/]. That's not a guarantee that all code is that fast, but most things we see can be pretty easily tweaked to get there (counterintuitively for those coming from Matlab, Python or R, usually by devectorizing the code rather than vectorizing it). And of course, there's a lot of room for improving Julia's performance, the compiler is still quite young and there are many optimizations that we haven't implemented. Basically, there's nothing but work standing in the way of reaching C or Fortran's speed across the board.
I just ran Wes's benchmarks (not the BLAS call versions) on my machine with a Julia I built on 10/13 (17c3c13), and the timings have indeed improved. For the details, see this gist: https://gist.github.com/3901139 (including the comment I posted on it).
The highlight is
numpy: (x * y).sum() => 41.1 ms
julia: inner(x,y) => 37.4 ms
julia: x*y => 19.5 ms
cython: inner(x,y) => 13.8 ms
The numpy and Julia versions are much easier to write and run.
Disclaimers: I've never written or built cython code before just now, and I think Julia is the coolest.
EDIT: whoops, missed the most important one (inner() written in pure Julia). Added it. Any thoughts on why inner() in Julia isn't faster?
Nice. Thanks for running those. The reason inner isn't faster is probably that we do bounds checks on every array access. This is surprisingly inexpensive on modern hardware but it still takes some time. We're working on a couple of things to address this: generating code so that llvm can more easily hoist bounds checks out of loops, and allowing turning bounds checking off entirely for blocks of Julia code.
Do y'all have a performance roadmap (or issue tracker tag?) that lists some optimizations that you foresee as julia matures? If not, I hope you and/or Jeff will do a blog post on this at some point! One key point of interest would be the split (obviously hand-wavy) between:
- llvm hotness that you don't use yet
- well-known techniques from HotSpot, Smalltalk, V8, etc.
- researchy optimizations that julia is particularly suited for?
This would be a nice resource for newcomers with a language bent, and also as a building block for your Google Summer of Code applications ;)
That's a good idea. Jeff is the real compilers genius so I'll have to see if I can convince him to write a blog post along those lines. The bounds check stuff I mentioned above is one of the important planned optimizations. Another important move is making composite types immutable by default, which is surprisingly unstiffling, yet allows a large number of clever optimizations (stack allocation, memory layout optimizations). More playing around with llvm optimization passes could help and is pretty easy to do, but we haven't spent much time on that. Also lots of gc improvements (approximate ref counting, escape analysis).
From a user's point of view it doesn't matter if it is python or cython. Slow things get implemented carefully and are packed away in a 3rd party libraries.
If the performance comes down to BLAS, then language speed benchmarks are moot. Saying one language is faster than another becomes disingenuous.
> If the performance comes down to BLAS, then language speed benchmarks are moot. Saying one language is faster than another becomes disingenuous.
Fully agreed. That's why comparing NumPy calling BLAS to Julia calling BLAS is a silly exercise. Comparing NumPy calling BLAS to a summation loop written in pure Julia goes beyond silly to just unreasonable, but that's what the blog post does.
Comments
A couple of nits...
Cython is actually what is faster than Julia in Wes' comparison, not Python. Cython looks kinda, sorta like Python, but it is actually a static language with C-like types (but quite different syntax for those types), no polymorphism, and, afaict, ill-defined semantics. The best answer I seem to get about Cython's semantics is that Cython's semantics are whatever it does. I'm not alone in this complaint – Travis Oliphant expressed a similar concern at this year's SciPy (in this panel [http://www.youtube.com/watch?v=7i2vhoQY-K4], if I recall correctly), which is part of his motivation to work on Numba [https://github.com/numba/numba].
If you look at the comments on Wes' post, when I used the dot(x,y) function, which ships with Julia and uses a BLAS to compute the inner product just like the fastest "Python" version does, Julia is equally fast. That stands to reason – they're both just calling a BLAS.
Finally, that blog post is months old – since then Julia passed the milestone of being no slower than 2x C++ on its microbenchmarks suite [http://julialang.org/]. That's not a guarantee that all code is that fast, but most things we see can be pretty easily tweaked to get there (counterintuitively for those coming from Matlab, Python or R, usually by devectorizing the code rather than vectorizing it). And of course, there's a lot of room for improving Julia's performance, the compiler is still quite young and there are many optimizations that we haven't implemented. Basically, there's nothing but work standing in the way of reaching C or Fortran's speed across the board.
I just ran Wes's benchmarks (not the BLAS call versions) on my machine with a Julia I built on 10/13 (17c3c13), and the timings have indeed improved. For the details, see this gist: https://gist.github.com/3901139 (including the comment I posted on it).
The highlight is
numpy: (x * y).sum() => 41.1 ms
julia: inner(x,y) => 37.4 ms
julia: x*y => 19.5 ms
cython: inner(x,y) => 13.8 ms
The numpy and Julia versions are much easier to write and run.
Disclaimers: I've never written or built cython code before just now, and I think Julia is the coolest.
EDIT: whoops, missed the most important one (inner() written in pure Julia). Added it. Any thoughts on why inner() in Julia isn't faster?
Nice. Thanks for running those. The reason inner isn't faster is probably that we do bounds checks on every array access. This is surprisingly inexpensive on modern hardware but it still takes some time. We're working on a couple of things to address this: generating code so that llvm can more easily hoist bounds checks out of loops, and allowing turning bounds checking off entirely for blocks of Julia code.
Do y'all have a performance roadmap (or issue tracker tag?) that lists some optimizations that you foresee as julia matures? If not, I hope you and/or Jeff will do a blog post on this at some point! One key point of interest would be the split (obviously hand-wavy) between: - llvm hotness that you don't use yet - well-known techniques from HotSpot, Smalltalk, V8, etc. - researchy optimizations that julia is particularly suited for?
This would be a nice resource for newcomers with a language bent, and also as a building block for your Google Summer of Code applications ;)
That's a good idea. Jeff is the real compilers genius so I'll have to see if I can convince him to write a blog post along those lines. The bounds check stuff I mentioned above is one of the important planned optimizations. Another important move is making composite types immutable by default, which is surprisingly unstiffling, yet allows a large number of clever optimizations (stack allocation, memory layout optimizations). More playing around with llvm optimization passes could help and is pretty easy to do, but we haven't spent much time on that. Also lots of gc improvements (approximate ref counting, escape analysis).
From a user's point of view it doesn't matter if it is python or cython. Slow things get implemented carefully and are packed away in a 3rd party libraries.
If the performance comes down to BLAS, then language speed benchmarks are moot. Saying one language is faster than another becomes disingenuous.
> If the performance comes down to BLAS, then language speed benchmarks are moot. Saying one language is faster than another becomes disingenuous.
Fully agreed. That's why comparing NumPy calling BLAS to Julia calling BLAS is a silly exercise. Comparing NumPy calling BLAS to a summation loop written in pure Julia goes beyond silly to just unreasonable, but that's what the blog post does.