I think what's going on here is that tech leadership folks know that the models the scientists develop eventually need to feed into their live product (so need to be "production ready"), but there isn't enough work to have two teams; one to develop the models, and one to run them in production. Thus, the ideal employee is an expert in everything! That's valuable, but not likely to be something you find when both data science and SRE are deep fields where people are very successful only knowing one of them ;)
I work on something called Pachyderm, which is a Kubernetes-based data storage and job execution system that tries to bridge this gap. We have a managed solution (https://hub.pachyderm.com) where we provision your Kubernetes cluster and do all the management (keeping the software up to date, authentication and authorization, etc.) and in fact don't even expose kubectl to you. You'll never see any of the Kubernetes stuff (though you might recognize certain error messages, I suppose). You just supply your code and a specification for how data flows around your pipelines, add your data, and we do the rest. Data scientists can interact with the versioned inputs and outputs through notebooks, but you're getting the full suite of production features behind the scenes -- a history of exactly which data inputs went into which data outputs, incremental processing, seamless autoscaling (set cpus: 8, gpus: 1 in your pipeline specification, and we find you a machine that meets that spec, add it to your cluster in less than a minute, schedule your work there, and remove the machine when the job finishes), etc.
Sorry for the sales pitch. I pretty much never use HN to shill my paid work, but it seems especially relevant to this sort of problem. Maybe you don't need the unicorn employee that is an expert in multiple fields -- focus on the data science and let us actually deal with the ugliness of computers ;)
(And if you do like Kubernetes but don't want to write your own orchestration system, Pachyderm itself is open source.)
Two teams causes an issue where scientists chuck models over the wall for the engineers to somehow rebuild into a semi-workable approach. The end result isn't great because you can't build good production models without taking production deployment into account. You also can't convert non-production models into production models without understanding the modeling assumptions that happened.
The general result is that the engineers and leadership finds the results underwhelming to horrible. The scientists often don't care because what happens on the other side of the wall isn't their problem.
That doesn't mean everyone has to know everything but separating people into teams is not the answer. Have a single team with people of different focuses and areas of expertise.
There may not be enough work for two teams 100% of the time, but there sure is when TSHTF. Manufacturers understood the need for some slack, but software companies still haven’t figured this out.
Comments
I think what's going on here is that tech leadership folks know that the models the scientists develop eventually need to feed into their live product (so need to be "production ready"), but there isn't enough work to have two teams; one to develop the models, and one to run them in production. Thus, the ideal employee is an expert in everything! That's valuable, but not likely to be something you find when both data science and SRE are deep fields where people are very successful only knowing one of them ;)
I work on something called Pachyderm, which is a Kubernetes-based data storage and job execution system that tries to bridge this gap. We have a managed solution (https://hub.pachyderm.com) where we provision your Kubernetes cluster and do all the management (keeping the software up to date, authentication and authorization, etc.) and in fact don't even expose kubectl to you. You'll never see any of the Kubernetes stuff (though you might recognize certain error messages, I suppose). You just supply your code and a specification for how data flows around your pipelines, add your data, and we do the rest. Data scientists can interact with the versioned inputs and outputs through notebooks, but you're getting the full suite of production features behind the scenes -- a history of exactly which data inputs went into which data outputs, incremental processing, seamless autoscaling (set cpus: 8, gpus: 1 in your pipeline specification, and we find you a machine that meets that spec, add it to your cluster in less than a minute, schedule your work there, and remove the machine when the job finishes), etc.
Sorry for the sales pitch. I pretty much never use HN to shill my paid work, but it seems especially relevant to this sort of problem. Maybe you don't need the unicorn employee that is an expert in multiple fields -- focus on the data science and let us actually deal with the ugliness of computers ;)
(And if you do like Kubernetes but don't want to write your own orchestration system, Pachyderm itself is open source.)
Two teams causes an issue where scientists chuck models over the wall for the engineers to somehow rebuild into a semi-workable approach. The end result isn't great because you can't build good production models without taking production deployment into account. You also can't convert non-production models into production models without understanding the modeling assumptions that happened.
The general result is that the engineers and leadership finds the results underwhelming to horrible. The scientists often don't care because what happens on the other side of the wall isn't their problem.
That doesn't mean everyone has to know everything but separating people into teams is not the answer. Have a single team with people of different focuses and areas of expertise.
Forgive my ignorance, but wasn't (isn't?) Pachyderm a Hadoop data version control tool? Did the product pivot?
The (very) short version is that it is a much better alternative to Hadoop, built for the container era.
There may not be enough work for two teams 100% of the time, but there sure is when TSHTF. Manufacturers understood the need for some slack, but software companies still haven’t figured this out.