Might not be your target audience, but high energy physics has been operating an infrastructure like that for years:
https://www.etp.physik.uni-muenchen.de/research/grid-computi...
You can basically use these frameworks to run your analysis jobs on any of the connected HPC centers and interactively move workloads around. The data never has to touch your hard drive either but gets moved to the compute on demand. This is how thousands of physicists do statistical analysis on petabytes of data.
Comments
If it is not ML Ops, how would you describe it?
it's honestly just a different approach. ML ops is adjusting your code to work with the cloud and managing all that.
For us is basically integrating clouds directly into your code so the barrier disappears and the cloud providers become an extension of your laptop.
Might not be your target audience, but high energy physics has been operating an infrastructure like that for years: https://www.etp.physik.uni-muenchen.de/research/grid-computi... You can basically use these frameworks to run your analysis jobs on any of the connected HPC centers and interactively move workloads around. The data never has to touch your hard drive either but gets moved to the compute on demand. This is how thousands of physicists do statistical analysis on petabytes of data.
super cool!
One of the professors at my lab at NYU CILVR (Kyle Cranmer) i believed was super involved with this. Will definitely sync up with him!
Thanks for the heads up!