It's interesting, but it looks like it has a loooong way to go before reaching feature parity with PyPy, which has itself a number of challenges to overcome before becoming mainstream. I would be interested to read the rationale for starting work on Pyston vs using/improving PyPy.
There are already a number of Python implementations using JIT techniques, often in sophisticated ways. PyPy has achieved impressive performance with its tracing JIT; Jython and IronPython are both built on top of mature VMs with extensive JIT support. So why do we think it’s worth starting a new implementation?
In short, it’s because we think the most promising techniques are incompatible with existing implementations. For instance, the JavaScript world has switched from tracing JITs to method-at-a-time JITs, due to the compelling performance benefits. Whether or not the same performance advantage holds for Python is an open question, but since the two approaches are fundamentally incompatible, the only way to start answering the question is to build a new method-at-a-time JIT.
Another point of differentiation is the planned use of a conservative garbage collector to support extension modules efficiently. Again, we won’t know until later whether this is a better approach or not, but it’s a decision that’s integral enough to a JIT that it is difficult to test in an existing implementation.
The downside of starting from scratch is, unsurprisingly, that creating a new language implementation is an enormous task. Luckily, tools are starting to come out that can help with this process; in particular, Pyston is built on top of LLVM, which lets us achieve top-tier code generation quality without having to deal with the details ourselves. Nonetheless, a new Python implementation is a huge undertaking, and Pyston will not be ready for use soon.
Their technique (by-method JIT, LLVM toolchain) is more likely to produce a mature implementation in the short term, possibly with a modest (but welcome) speedup, without major incompatibilities with extensions. If it happens, it's better than "That Perfect JIT VM" that is never finished.
The major issue with PyPy is that it has a radically different C API than CPython, leading to incompatibility with NumPy, Pandas, etc. to mention just a few prominent performance-critical Python packages that are written in C. PyPy works if you want to use pure Python code, but that's rarely the case in real-world high-performance Python projects.
Compatibly with CPython's C API will always be a problem for any other implementation. The only practical solution is to switch to a better FFI, like cffi.
Comments
It's interesting, but it looks like it has a loooong way to go before reaching feature parity with PyPy, which has itself a number of challenges to overcome before becoming mainstream. I would be interested to read the rationale for starting work on Pyston vs using/improving PyPy.
From the site: https://tech.dropbox.com/2014/04/introducing-pyston-an-upcom...
Why a new implementation
There are already a number of Python implementations using JIT techniques, often in sophisticated ways. PyPy has achieved impressive performance with its tracing JIT; Jython and IronPython are both built on top of mature VMs with extensive JIT support. So why do we think it’s worth starting a new implementation?
In short, it’s because we think the most promising techniques are incompatible with existing implementations. For instance, the JavaScript world has switched from tracing JITs to method-at-a-time JITs, due to the compelling performance benefits. Whether or not the same performance advantage holds for Python is an open question, but since the two approaches are fundamentally incompatible, the only way to start answering the question is to build a new method-at-a-time JIT.
Another point of differentiation is the planned use of a conservative garbage collector to support extension modules efficiently. Again, we won’t know until later whether this is a better approach or not, but it’s a decision that’s integral enough to a JIT that it is difficult to test in an existing implementation.
The downside of starting from scratch is, unsurprisingly, that creating a new language implementation is an enormous task. Luckily, tools are starting to come out that can help with this process; in particular, Pyston is built on top of LLVM, which lets us achieve top-tier code generation quality without having to deal with the details ourselves. Nonetheless, a new Python implementation is a huge undertaking, and Pyston will not be ready for use soon.
Thanks for that. Well, I won't say no to open-source R&D :)
Their technique (by-method JIT, LLVM toolchain) is more likely to produce a mature implementation in the short term, possibly with a modest (but welcome) speedup, without major incompatibilities with extensions. If it happens, it's better than "That Perfect JIT VM" that is never finished.
Except PyPy is here now and you can use it. Not necessarily perfect, but the design is sound.
The major issue with PyPy is that it has a radically different C API than CPython, leading to incompatibility with NumPy, Pandas, etc. to mention just a few prominent performance-critical Python packages that are written in C. PyPy works if you want to use pure Python code, but that's rarely the case in real-world high-performance Python projects.
Compatibly with CPython's C API will always be a problem for any other implementation. The only practical solution is to switch to a better FFI, like cffi.