I don't think it's a particularly new feature of software development that a few highly paid employees who've got the entire stack in their brains are vastly more productive than a vast cross functional team.
There is some fantastic tooling for machine learning.
Databricks, GCP, everyone knows it.
The issue is that the data industry was raised from birth in complete fear of the boogeyman.
The boogeyman is Oracle. And the frankly ridiculous things Oracle did in the bad old days.
Hence most places have a constant internal conflict between "look here are all these brilliant data science tools" and "ah shit, GCP costs a ton of money when some idiot runs a select * query on a join across 5 TB of data."
Comments
I don't think it's a particularly new feature of software development that a few highly paid employees who've got the entire stack in their brains are vastly more productive than a vast cross functional team.
Also I will say.
There is some fantastic tooling for machine learning.
Databricks, GCP, everyone knows it.
The issue is that the data industry was raised from birth in complete fear of the boogeyman.
The boogeyman is Oracle. And the frankly ridiculous things Oracle did in the bad old days.
Hence most places have a constant internal conflict between "look here are all these brilliant data science tools" and "ah shit, GCP costs a ton of money when some idiot runs a select * query on a join across 5 TB of data."
But there are plenty of great tools.
Can you speak a bit more to this? I dislike Oracle with a passion, but I am not sure how the GCP comment connects.
It's just $$$.
You can save a LOT of money in GCP by specifying the columns you actually need in your queries, and various very simple SQL optimization techniques.
Everyone is scared of the cost of these vendor tools.