Mine is that neural networks as we understand them now get replaced by much more solid methods, based on the principles from scientific machine learning, where sophisticated differentiable models that are designed to mimic the problem space get tuned.
Maybe so. But I suspect that scientific machine learning would be difficult to grasp by most working data scientists and ML users who don't have graduate-level training in a quantitative science or engineering, which implies an uphill battle for adoption. You can do a lot with ML and deep learning without to construct a sophisticated mathematical model of your problem space.
(I took one class in differential equations in 2004, touched up on that in a boot camp before starting a PhD in econ, and have probably forgotten everything I ever learned about the topic ever since. Why yes, I feel mathematically inadequate sometimes.)
would be difficult to grasp by most working data scientists
Most of what those same people do now was difficult to grasp when they started. The next group will adopt whatever is the best tools for the problems they face.
which implies an uphill battle for adoption
The same was true for current ML and data science. It was true for OOP. It was true for structured programming. This is always true, but tools that solve problems get adopted.
You can do a lot with ML and deep learning without to construct a sophisticated mathematical model of your problem space.
True, but as such simple tasks are mined out, everyone doing much work will move past that. For example, pick any task on say paperswithcode.com and look at the top performing models. It's hard to find any that are simple old school networks. All of them I just sampled do much more sophisticated modeling of the precise problem space, and the trend is that all current edge and future work will involve modeling the problem much more carefully.
I expect the naive deep learning approach will be a tiny blip in the progress of using learning to solve problems.
Future progress looks like it relies more and more on making a good model of the problem with some untuned parameters, then using gradient descent or related to tune those parameters. The better the original model, the better it performs, and the less data one needs to train it since it has relevant structure already.
For example, learning basic physics in a net takes significantly more training, parameters, and cost than simply building in basic physics and tuning the small unknown parts. The net version also doesn't scale well since it is built from pieces (linear chunks) that simply don't match functional relations in the underlying problem.
Comments
Maybe so. But I suspect that scientific machine learning would be difficult to grasp by most working data scientists and ML users who don't have graduate-level training in a quantitative science or engineering, which implies an uphill battle for adoption. You can do a lot with ML and deep learning without to construct a sophisticated mathematical model of your problem space.
(I took one class in differential equations in 2004, touched up on that in a boot camp before starting a PhD in econ, and have probably forgotten everything I ever learned about the topic ever since. Why yes, I feel mathematically inadequate sometimes.)
Most of what those same people do now was difficult to grasp when they started. The next group will adopt whatever is the best tools for the problems they face.
The same was true for current ML and data science. It was true for OOP. It was true for structured programming. This is always true, but tools that solve problems get adopted.
True, but as such simple tasks are mined out, everyone doing much work will move past that. For example, pick any task on say paperswithcode.com and look at the top performing models. It's hard to find any that are simple old school networks. All of them I just sampled do much more sophisticated modeling of the precise problem space, and the trend is that all current edge and future work will involve modeling the problem much more carefully.
I expect the naive deep learning approach will be a tiny blip in the progress of using learning to solve problems.
Future progress looks like it relies more and more on making a good model of the problem with some untuned parameters, then using gradient descent or related to tune those parameters. The better the original model, the better it performs, and the less data one needs to train it since it has relevant structure already.
For example, learning basic physics in a net takes significantly more training, parameters, and cost than simply building in basic physics and tuning the small unknown parts. The net version also doesn't scale well since it is built from pieces (linear chunks) that simply don't match functional relations in the underlying problem.