A previous project I worked on was persisting more than 10 TB of data per week, data that had to be analyzed, the requirement being to extract meaningful user profiles from it. We also went back to analyze older data with new queries / algorithms quite a lot and playing with a petabyte is fun. It was a startup, it was dissolved at some point, so you've never heard of it.
I don't like the 90% argument. 90% of apps are just dumb frontends to a MySQL database, so you don't need Rust, or Spark. As you can see, that doesn't tell you anything meaningful about the other 10%.
Comments
A previous project I worked on was persisting more than 10 TB of data per week, data that had to be analyzed, the requirement being to extract meaningful user profiles from it. We also went back to analyze older data with new queries / algorithms quite a lot and playing with a petabyte is fun. It was a startup, it was dissolved at some point, so you've never heard of it.
I don't like the 90% argument. 90% of apps are just dumb frontends to a MySQL database, so you don't need Rust, or Spark. As you can see, that doesn't tell you anything meaningful about the other 10%.