I switched from mostly using R to Python about a year ago for gluing together my data pipeline (from data source all the way to production models and frontends/visualizations). It hasn't really impacted what I'm capable of doing or my productivity, except the standard extra googling that comes in the first couple years I use any language.
The main reason I went for Python is purely practical: it's a language people outside my team will respect and deal with. It makes it easier for me to collaborate in many different ways: share tools with other teams, transfer ownership of my code, get help when I need it, etc. Data science at some companies has the reputation of "hack something together and throw it over the wall for someone else to deal with". In my experience R only furthers this reputation. Which is too bad, it's really great at what it does.
Computation has grown more complicated. They need real computer scientists and a real language that supports real development, not some scientists which learned just enough Python to automate running some 20 year old Fortran code.
The part about sharing makes a lot of sense since Python use is so wide spread. The throwing-over-the-wall effect isn't a language specific issue, more of a work culture issue. Seems to me if you practice "literate programming" with R markdown you can greatly improve the sharing aspect and reduce the throw-it-over-the-wall issue.
Totally agree about it being a cultural thing, have first hand experience at some of the usual suspects. I have a far more cynical label for it: deliver your turd (typically formed in MATLAB) for someone else to polish. It is surprising how common this is in some places and groups. I mention groups because when the group moves from one place to another it brings that turd polishing culture along.
I went through the very same process :). I really like your comment, you highlight something that I forgot to mention in this clarity "It makes it easier for me to collaborate in many different ways: share tools with other teams, transfer ownership of my code, get help when I need". Would you mind if I add it as to a "other people's experiences" section at the bottom of the article?
Yep. This is exactly why I gravitated toward python and scikit-learn. So much of data science is just getting the data into the right format. Grab it from this file, and this database, and this web service, then get it formatted into this table structure, and then clean it with this filter, and then plug these holes this way and those other holes that way, and now you're finally ready for a random forest baseline.
Python is a really good language for merging and parsing data from lots of different sources. For many problems, a general purpose language with very good data science library may actually be the better choice than a dedicated data science language/environment.
I've been interviewing candidates recently and 'I just prototype in matlab and throw it over the wall to the implementation team' is a big negative in my evaluations (we don't have a wall in this company and I will fight to keep it that way).
Comments
I switched from mostly using R to Python about a year ago for gluing together my data pipeline (from data source all the way to production models and frontends/visualizations). It hasn't really impacted what I'm capable of doing or my productivity, except the standard extra googling that comes in the first couple years I use any language.
The main reason I went for Python is purely practical: it's a language people outside my team will respect and deal with. It makes it easier for me to collaborate in many different ways: share tools with other teams, transfer ownership of my code, get help when I need it, etc. Data science at some companies has the reputation of "hack something together and throw it over the wall for someone else to deal with". In my experience R only furthers this reputation. Which is too bad, it's really great at what it does.
Yeah, there is a large community of Python users in scientific computing. It's great.
I like well-established languages with a large user base.
So, I was dismayed by Big Data Genomics' ADAM Project's choice of Scala, which has almost no uptake in the genomics/bioinformatics community.
They do it because they run over Spark. But Spark has an excellent Python binding.
Python's days have come and gone.
Computation has grown more complicated. They need real computer scientists and a real language that supports real development, not some scientists which learned just enough Python to automate running some 20 year old Fortran code.
So Julia then :)
The part about sharing makes a lot of sense since Python use is so wide spread. The throwing-over-the-wall effect isn't a language specific issue, more of a work culture issue. Seems to me if you practice "literate programming" with R markdown you can greatly improve the sharing aspect and reduce the throw-it-over-the-wall issue.
Totally agree about it being a cultural thing, have first hand experience at some of the usual suspects. I have a far more cynical label for it: deliver your turd (typically formed in MATLAB) for someone else to polish. It is surprising how common this is in some places and groups. I mention groups because when the group moves from one place to another it brings that turd polishing culture along.
I went through the very same process :). I really like your comment, you highlight something that I forgot to mention in this clarity "It makes it easier for me to collaborate in many different ways: share tools with other teams, transfer ownership of my code, get help when I need". Would you mind if I add it as to a "other people's experiences" section at the bottom of the article?
Definitely feel free to add that!
"for gluing together my data pipeline"
Yep. This is exactly why I gravitated toward python and scikit-learn. So much of data science is just getting the data into the right format. Grab it from this file, and this database, and this web service, then get it formatted into this table structure, and then clean it with this filter, and then plug these holes this way and those other holes that way, and now you're finally ready for a random forest baseline.
Python is a really good language for merging and parsing data from lots of different sources. For many problems, a general purpose language with very good data science library may actually be the better choice than a dedicated data science language/environment.
I've been interviewing candidates recently and 'I just prototype in matlab and throw it over the wall to the implementation team' is a big negative in my evaluations (we don't have a wall in this company and I will fight to keep it that way).