As one of the main drivers of Python at Netflix, this one is near and dear to my heart. Let me know if you have any questions, or come visit our booth!
You know how Google claims that by saving fractions of a second for every search that billions of people do, they've saved many man-lives of productivity? As a leading engineer at both Reddit and Netflix, you've undone all of that. Well done.
Posts like these are great on a tech level, but they're also good recruitment - Seeing Python, along with hackers that I respect (like you) at Netflix make me at least the littlest bit tempted to go there ;)
But in all seriousness, we like to share what we do because sharing is cool and it attracts other people who like to share cool technology, which are the kind of people we like to work with.
> It's tough as everyone is drinking the ruby kool-aid.
Ruby and Python are similar in terms of execution speed and developer productivity. If your work requires something that Python has and Ruby doesn't(say numpy), then it might make sense; otherwise what would you accomplish by switching from Ruby to Python?
We were actually talking about your company in our sprint planning meeting a few minutes ago, since we're doing in-house Python analytics. You might be getting a call.
Awesomeness. We are making great headway on Blaze and the new persistence format to supplant HDF5/PyTables (Although, Blaze will always work great with carray, NumPy, and Pytables), and the Numba compiler project is working out great.
Wakari (http://wakari.io) is nearing 1.0 and we're about to turn on paid monthly plans for beefier nodes and GPU nodes, and we have also been in talks with several companies about in-house private deployments.
Give us a shout or shoot me an email at pwang (at) continuum.io! :-)
There was many interesting things listed at the blog post. I have been planning a system similar in idea to Sting, your OLAP server (?).
I'm planning a light RPC style system on top of Pandas dataframe objects, transferring HTTP requests to Pandas method calls. Memory usage of such a system with our data is too expensive if all data is in use. So, I'm investigating possible caching and database solutions.
Could you perhaps elaborate on the implementation of Sting's in memory database?
I just posted this above: "I don't have any info about Sting, but we're working on some blog posts that go deeper into each of these topics. I'll make sure that the interest in Sting is accounted for in the prioritization."
I don't have any info about Sting, but we're working on some blog posts that go deeper into each of these topics. I'll make sure that the interest in Sting is accounted for in the prioritization.
Kurt Brown had some slides about data & Python at the NYC Python meetup last October. He did a Sting demo there and some of the slides talk about it (I think... it might be a different thing, though. October was a long time ago..)
I have to say that after doing a python project I totally don't get why people love it so much. The syntax is slightly cleaner, but the lack of threading, first-class functions and dependency management tools make it waste at least as much time as it saves. I honestly don't believe in the concept of a programming language being able to make code more or less readable. A Java project with a POM file can be up and running on any OS in about 5 minutes.
Sorry, I should have said "lack of true threading in CPython" by which I mean the GIL limitation. And lambda expressions are lame compared to what you can do in JavaScript/Perl/Scala. And the build tools suck compared to Maven which is saying a lot. Maven is cross-platform and install-free. It takes a long time to makes head and tails of pip, easy_install and virtualenv and it still does less than Maven.
>And lambda expressions are lame compared to what you can do in JavaScript/Perl/Scala.
Python lets you pass functions other ways than using lambdas, you know. If you've defined a function with the standard "def foo" style, you can pass "foo" as an argument.
Even the standard "1-line lambdas LOL" limitation can be circumvented by defining the function elsewhere and making the lambda a call to it.
Lambda expressions are not necessarily first-class functions (I mean, they are in Python, but so is any other old function).
First class functions simply mean that a language treats functions as it would nearly any other data type, namely that they may be passed as parameters, bound to variables, and returned from other functions.
Python's poor handling of lambda expressions do not invalidate that functions are first-class citizens.
What do you use "true" threading for? Are you multiplying matrices? If so use Numpy. Are you downloading data and accessing database? If so, you threading, threading in Python works great for that. If you hit memory size issues, switch to gevent or eventlet for green threads.
I have been building large projects in Python for the last 10 yeas and never really thought "Hmm I wish I had more powerful lambda functions". You can always just have regular function and pass that around. Why do you need lambdas so much?
I often get the idea that people confuse anonymous and first class functions (apparently because they discover the latter expressed in terms of the former).
It has its share of warts, but doesn't lack any of those things you mention.
Java shines at the huge end, but for most everything else python is a more productive choice as it is so much more concise. Perhaps you are not yet using it optimally?
a) python-pip combined with virtualenv gives very good requirement support.
b) python does have first class functions?
c) threading still happens, you just can't take advantage of multiple cores. This isn't as big of a problem as you'd think though, you can still write very fast code.
Comments
As one of the main drivers of Python at Netflix, this one is near and dear to my heart. Let me know if you have any questions, or come visit our booth!
You know how Google claims that by saving fractions of a second for every search that billions of people do, they've saved many man-lives of productivity? As a leading engineer at both Reddit and Netflix, you've undone all of that. Well done.
I always tell people that I've spent the last six years of my life as a digital crack dealer.
Unfair tone. I hope it was in jest. My life is not all about productivity. There is a leisure aspect as well:
http://en.chessbase.com/home/TabId/211/PostId/4008728
FWIW, I knew he was kidding. He's a reddit engineer too. :)
That's the other bsimpson. ;-)
I hoped the "of productivity" would be a cue that I was being playful.
ah, did not catch that - sorry :)
Keep it up!
Posts like these are great on a tech level, but they're also good recruitment - Seeing Python, along with hackers that I respect (like you) at Netflix make me at least the littlest bit tempted to go there ;)
You've caught on to our ploy!
But in all seriousness, we like to share what we do because sharing is cool and it attracts other people who like to share cool technology, which are the kind of people we like to work with.
I strongly suspect preventing me from visiting PyCon is one of the reasons my CTO scheduled so many product launches for the next two weeks...
I'm pushing for more python use at my company as well. It's tough as everyone is drinking the ruby kool-aid.
P.S. I really enjoyed your talk at QCon SF last year!
> It's tough as everyone is drinking the ruby kool-aid.
Ruby and Python are similar in terms of execution speed and developer productivity. If your work requires something that Python has and Ruby doesn't(say numpy), then it might make sense; otherwise what would you accomplish by switching from Ruby to Python?
what would you accomplish by switching from Ruby to Python?
Simpler syntax?
A larger and more varied developer community, resulting in applicability to a wider range of problems.
You should also come by our booth (Continuum Analytics) and we can talk shop!
We were actually talking about your company in our sprint planning meeting a few minutes ago, since we're doing in-house Python analytics. You might be getting a call.
Awesomeness. We are making great headway on Blaze and the new persistence format to supplant HDF5/PyTables (Although, Blaze will always work great with carray, NumPy, and Pytables), and the Numba compiler project is working out great.
Wakari (http://wakari.io) is nearing 1.0 and we're about to turn on paid monthly plans for beefier nodes and GPU nodes, and we have also been in talks with several companies about in-house private deployments.
Give us a shout or shoot me an email at pwang (at) continuum.io! :-)
Hi.
There was many interesting things listed at the blog post. I have been planning a system similar in idea to Sting, your OLAP server (?).
I'm planning a light RPC style system on top of Pandas dataframe objects, transferring HTTP requests to Pandas method calls. Memory usage of such a system with our data is too expensive if all data is in use. So, I'm investigating possible caching and database solutions.
Could you perhaps elaborate on the implementation of Sting's in memory database?
[posted from yet another HN account]
I just posted this above: "I don't have any info about Sting, but we're working on some blog posts that go deeper into each of these topics. I'll make sure that the interest in Sting is accounted for in the prioritization."
So now that's +2. :)
+3 please :-)
The idea of marrying "slow" Python and OLAP is disturbing my sleep!
Have you looked at the Cubes project?
http://databrewery.org/cubes.html
would you be interested to talk about these tools on the http://foodfightshow.org devops podcast? we would love to discuss them with you!
I would, but I could probably find you people who know more about the tools, like the folks who wrote them. :)
Hit me up on email (it's in my profile).
Where can I find more info about Sting? I didn't see a link in the article and some quick Google searches haven't proved fruitful.
I don't have any info about Sting, but we're working on some blog posts that go deeper into each of these topics. I'll make sure that the interest in Sting is accounted for in the prioritization.
Kurt Brown had some slides about data & Python at the NYC Python meetup last October. He did a Sting demo there and some of the slides talk about it (I think... it might be a different thing, though. October was a long time ago..)
http://www.meetup.com/nycpython/files/
I have to say that after doing a python project I totally don't get why people love it so much. The syntax is slightly cleaner, but the lack of threading, first-class functions and dependency management tools make it waste at least as much time as it saves. I honestly don't believe in the concept of a programming language being able to make code more or less readable. A Java project with a POM file can be up and running on any OS in about 5 minutes.
Ummm, not sure where you're getting your information but Python has threads (http://docs.python.org/2/library/threading.html), first-class functions (http://en.wikipedia.org/wiki/First-class_function), and dependency management tools (google for pip, easy_install, etc.)
Sorry, I should have said "lack of true threading in CPython" by which I mean the GIL limitation. And lambda expressions are lame compared to what you can do in JavaScript/Perl/Scala. And the build tools suck compared to Maven which is saying a lot. Maven is cross-platform and install-free. It takes a long time to makes head and tails of pip, easy_install and virtualenv and it still does less than Maven.
>And lambda expressions are lame compared to what you can do in JavaScript/Perl/Scala.
Python lets you pass functions other ways than using lambdas, you know. If you've defined a function with the standard "def foo" style, you can pass "foo" as an argument.
Even the standard "1-line lambdas LOL" limitation can be circumvented by defining the function elsewhere and making the lambda a call to it.
Lambda expressions are not necessarily first-class functions (I mean, they are in Python, but so is any other old function).
First class functions simply mean that a language treats functions as it would nearly any other data type, namely that they may be passed as parameters, bound to variables, and returned from other functions.
Python's poor handling of lambda expressions do not invalidate that functions are first-class citizens.
What do you use "true" threading for? Are you multiplying matrices? If so use Numpy. Are you downloading data and accessing database? If so, you threading, threading in Python works great for that. If you hit memory size issues, switch to gevent or eventlet for green threads.
I have been building large projects in Python for the last 10 yeas and never really thought "Hmm I wish I had more powerful lambda functions". You can always just have regular function and pass that around. Why do you need lambdas so much?
I think he was talking about threads that run on multiple processors.
Also, packaging is a whole different can of worms (but hopefully will be sorted by distutils2!).
Python doesn't have first class functions?
I often get the idea that people confuse anonymous and first class functions (apparently because they discover the latter expressed in terms of the former).
It has its share of warts, but doesn't lack any of those things you mention.
Java shines at the huge end, but for most everything else python is a more productive choice as it is so much more concise. Perhaps you are not yet using it optimally?
a) python-pip combined with virtualenv gives very good requirement support.
b) python does have first class functions?
c) threading still happens, you just can't take advantage of multiple cores. This isn't as big of a problem as you'd think though, you can still write very fast code.
Hey jedberg - any plans/timeline for open sourcing aminator? It sounds pretty cool...