Having done a small amount of machine learning, I can see how the advice here is "true". And by "true", I mean appropriate for the way that machine learning exists and operates in present-day space. Algorithms are difficult, temperamental and requires expert "tuning".
The sequence seems to be:
- First you learn the formal theory, the math and statistics.
- Then you learn the "squinting", the ad-hoc rules for how to apply which algorithm.
- Then implement the thing
This works better than just starting your editor and piecing code. However, I would claim that this doesn't actually work well in the sense that this is kind of where AI/ML have bogged down. I mean, there are only 5 main approaches, 20 main algorithms and whatever subsidiaries and random stuff. They don't work great and the only progress is incremental (though there is progress and throwing more computer power around at the same time enhances - while masking the low amount of conceptual progress).
What's lacking is any modularity in combining algorithms. The power of ordinary programming is, essentially, using function calls to put together what you want. ML doesn't do that and for all the magic, that makes it weak and fragile - when one magic algorithm doesn't work well, rather than improving it, it really is better, at present, in the interest of getting stuff done, to start with a different magic algorithm. This is true, I'm a realistic in the sense of accepting the present but an idealist in the sense of saying "that kind of sucks, we should be able to fix problems, not surrender and regroup".
Yes, I'm happy to denigrate the good and proper in my question for the best. But I'm an idealist, I suppose it's a matter of taste.
I would seriously look in to deep learning. http://deeplearning.net/ I am doing everything with it now.
This includes principal component analysis/compression, face detection,hand writing recognition, named entity recognition, clustering, topic modeling, semantic role labeling, among other things.
There is a very common structure to this. Despite neural nets having their own baggage, they're worth understanding.
The structure you're wanting is definitely in there. Edit: Yes a bit of self promotion here. Just making a point with patterns I've found as I've built this out.
Half the battle is understanding the linear algebra going on here. Beyond that you can pretty much do everything with one set of algorithms and terminology.
For those who go WTF java are you insane? The core idea I'm linking to here is the fact that deep nets are composed of singular neural networks with slight variations having a very common structure for both the singular layer as well as the deep nets themselves.
I wouldn't claim the Neural nets don't work. We've seen many use cases where they do work (And I have looked at deep learning just a little, it may superior in ways but it doesn't seem in any way fundamentally different from the other stuff).
I would add that Support Vector Machines also work and they are similar and have a much clearer math to them [1]. But SVM and neural nets are ultimately just linear matchers on a nonlinear pattern space, they ultimately involve adhoc choices that experts learn over time.
As I said above, once you learn the maths (linear algebra, statistics, functional-analysis or whatnot), it become less basic understand and more "understanding how", a series of tweaks that experts "with a feel for this stuff" do. But this "feel" level understanding seems exactly what stands in the way of serious, rational progress on the topic.
Binary compatible yes. Star the repo and watch it in the next few days. Example apps are on the way. I plan on implementing a full "easy to use" machine learning lib around this.
I have a lot more example usage in each of the tests. Test coverage was a higher priority above the documentation, but example usage is there. I'm more than happy to answer emails around the usage of the library as well. I also take feature requests.
I plan on implementing convolutional nets, recursive neural nets and some other ones based around that same structure; That includes the scale out versions with akka for easy multi threading or clustering ( I have built in service discovery with zookeeper among other things in there.
Advice question - how far into the linear algebra should I go? I'm currently working my way through a book - and it's not too bad, but I really don't want to invest more than I really need. Any suggestions?
One other thing might be understanding different ways you can manipulate data. In this case, numerical representation here is an example per row when I toss in one matrix for training. This is applicable to many machine learning problems.
You could look at the appendixes in texts by Bishop, MacKay, Barber and Rasmussen/Williams to see what they expect (and they expect a pretty thorough understanding. The last 3 are freely available content
This is essentially the goal of probabilistic programming[1]: let the programmer specify and tweak the model, and have the analog of a compiler handle inference. Finding a good model is then analogous to debugging.
You are mostly stuck with Bayesian models, since that's what we have general-purpose inference algorithms for. And in practice an understanding of Bayesian statistics is a prerequisite for writing useful probabilistic programs.
Well, you can kind of use decision forests, and similar approaches, to combine algorithms. The lack of human-readability makes it pretty tough unless you're in a domain where you can effectively apply weighted heuristics (which I found to work pretty well in NLP, albeit with a lot of experimental tweaking to arrive at the right model for my domain).
Comments
Interesting,
Having done a small amount of machine learning, I can see how the advice here is "true". And by "true", I mean appropriate for the way that machine learning exists and operates in present-day space. Algorithms are difficult, temperamental and requires expert "tuning".
The sequence seems to be:
- First you learn the formal theory, the math and statistics.
- Then you learn the "squinting", the ad-hoc rules for how to apply which algorithm.
- Then implement the thing
This works better than just starting your editor and piecing code. However, I would claim that this doesn't actually work well in the sense that this is kind of where AI/ML have bogged down. I mean, there are only 5 main approaches, 20 main algorithms and whatever subsidiaries and random stuff. They don't work great and the only progress is incremental (though there is progress and throwing more computer power around at the same time enhances - while masking the low amount of conceptual progress).
What's lacking is any modularity in combining algorithms. The power of ordinary programming is, essentially, using function calls to put together what you want. ML doesn't do that and for all the magic, that makes it weak and fragile - when one magic algorithm doesn't work well, rather than improving it, it really is better, at present, in the interest of getting stuff done, to start with a different magic algorithm. This is true, I'm a realistic in the sense of accepting the present but an idealist in the sense of saying "that kind of sucks, we should be able to fix problems, not surrender and regroup".
Yes, I'm happy to denigrate the good and proper in my question for the best. But I'm an idealist, I suppose it's a matter of taste.
I would seriously look in to deep learning. http://deeplearning.net/ I am doing everything with it now. This includes principal component analysis/compression, face detection,hand writing recognition, named entity recognition, clustering, topic modeling, semantic role labeling, among other things.
There is a very common structure to this. Despite neural nets having their own baggage, they're worth understanding.
The structure you're wanting is definitely in there. Edit: Yes a bit of self promotion here. Just making a point with patterns I've found as I've built this out.
See:
https://github.com/agibsonccc/java-deeplearning/blob/master/...
https://github.com/agibsonccc/java-deeplearning/blob/master/...
Half the battle is understanding the linear algebra going on here. Beyond that you can pretty much do everything with one set of algorithms and terminology.
For those who go WTF java are you insane? The core idea I'm linking to here is the fact that deep nets are composed of singular neural networks with slight variations having a very common structure for both the singular layer as well as the deep nets themselves.
I wouldn't claim the Neural nets don't work. We've seen many use cases where they do work (And I have looked at deep learning just a little, it may superior in ways but it doesn't seem in any way fundamentally different from the other stuff).
I would add that Support Vector Machines also work and they are similar and have a much clearer math to them [1]. But SVM and neural nets are ultimately just linear matchers on a nonlinear pattern space, they ultimately involve adhoc choices that experts learn over time.
As I said above, once you learn the maths (linear algebra, statistics, functional-analysis or whatnot), it become less basic understand and more "understanding how", a series of tweaks that experts "with a feel for this stuff" do. But this "feel" level understanding seems exactly what stands in the way of serious, rational progress on the topic.
[1]http://en.wikipedia.org/wiki/Support_vector_machine
Just today, I finished a deep-learning example in C# with Accord .NET. It solves a simple XOR function, as well as classifying ascii digits.
If you're looking for a basic example in .NET, I think the XOR one is as simple as it gets.
Deep-Learning XOR: https://github.com/primaryobjects/deep-learning/blob/XOR/Dee...
Deep-Learning Digits: https://github.com/primaryobjects/deep-learning/blob/master/...
Very cool stuff. A .NET implementation is a great project. If you want any advice, I'm working with a friend on a hadoop version as well.
https://github.com/jpatanooga/Metronome
We have a wiki and some resources put up.
Very interesting!
Is that Word2Vec implementation you have roughly equivalent of the Google version[1]?
Any examples of how to use deeplearning4j generally?
[1] https://code.google.com/p/word2vec/
Binary compatible yes. Star the repo and watch it in the next few days. Example apps are on the way. I plan on implementing a full "easy to use" machine learning lib around this.
Edit: Poke around in the tests, Here's an example of it learning a compressed version of MNIST: https://github.com/agibsonccc/java-deeplearning/blob/master/...
I have a lot more example usage in each of the tests. Test coverage was a higher priority above the documentation, but example usage is there. I'm more than happy to answer emails around the usage of the library as well. I also take feature requests.
I plan on implementing convolutional nets, recursive neural nets and some other ones based around that same structure; That includes the scale out versions with akka for easy multi threading or clustering ( I have built in service discovery with zookeeper among other things in there.
Advice question - how far into the linear algebra should I go? I'm currently working my way through a book - and it's not too bad, but I really don't want to invest more than I really need. Any suggestions?
The main thing to understand is the basics. For example element wise multiplication vs full blown matrix operations. Think something like this: http://en.wikipedia.org/wiki/Matrix_multiplication
Matlab/Octave is a great way to practice this due to the native data types. If python is your thing numpy's arrays are also pretty easy to digest.
Subtle little tricks like this: https://www.youtube.com/watch?v=evF-3ykjRU0
And understanding the dynamics of scalar operations vs matrix - vector operations.
The machine learning class has some good fundamentals if you need a refresher on how something works.
There will be more complex things like some optimization algorithms have different uses for eigen values: http://see.stanford.edu/materials/lsocoee364b/11-conj_grad_s...
See: http://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors
One other thing might be understanding different ways you can manipulate data. In this case, numerical representation here is an example per row when I toss in one matrix for training. This is applicable to many machine learning problems.
Very much appreciated. That's really helpful and feels very doable. :)
You could look at the appendixes in texts by Bishop, MacKay, Barber and Rasmussen/Williams to see what they expect (and they expect a pretty thorough understanding. The last 3 are freely available content
http://www.inference.phy.cam.ac.uk/itila/book.html
http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf
http://www.gaussianprocess.org/gpml/chapters/RW.pdf
Also, an excellent list of ML resources: http://www.reddit.com/r/MachineLearning/comments/1jeawf/mach...
This is essentially the goal of probabilistic programming[1]: let the programmer specify and tweak the model, and have the analog of a compiler handle inference. Finding a good model is then analogous to debugging.
You are mostly stuck with Bayesian models, since that's what we have general-purpose inference algorithms for. And in practice an understanding of Bayesian statistics is a prerequisite for writing useful probabilistic programs.
[1] e.g. http://probabilistic-programming.org/
Well, you can kind of use decision forests, and similar approaches, to combine algorithms. The lack of human-readability makes it pretty tough unless you're in a domain where you can effectively apply weighted heuristics (which I found to work pretty well in NLP, albeit with a lot of experimental tweaking to arrive at the right model for my domain).