I also dismissed LLMs given their "accuracy." But I think that's the wrong thing to compare. The sheer fact that a transformer architecture model gets better at scale and that we can feed it at scale is insane. It's what makes it AI and not just ML.
The fact that you can get billions of parameters to do anything useful from a relatively simple algorithm on a relatively small amount (high GBs / low TBs) of text means the algorithm is insane. That's what people miss - they think GPT is trained on "the whole internet" and is similar to some of low-variate regression model that is "approximating things." It is absolutely approximating things - so does all intelligence -- but it is truly sifting / "attending to" what is important over a relatively small corpus and organizing into billions of parameters the way a brain would organize data.
Will it hallucinate details? statistics? Etc.? Yes, and it should not be used in its current form for "truth." But it's very different from a low-variate model that is synthesizing in a low-dimensional space (which is how we gradually learn about the world) and an extremely-high dimensional model that is starting to see "what is important" in ways that are far, far above human intelligence. Similar to a human brain (due to the underlying neural architecture and any type of hierarchical compression of knowledge) but with far more input data, and a simplicity that maybe the brain has maybe it doesn't -- but is far more scalable and capable of hierarchies of information that out-scale us by so many orders of magnitude, and more every 6 months.
3blue1brown's https://www.youtube.com/watch?v=wjZofJX0v4M and upcoming videos I think will show the beauty and simplicity of the algorithm more. To put it another way -- the fact that you get a remotely true outcome with a model that just improves with scale, a remotely true outcome by the algorithm sifting what is important -- means that with time it will know what is more important in ways that far surpass humans.
If you approach interacting with LLM chatbots that way it is absolutely mind-blowing how "on point" the answers are. If you ask ChatGPT why the internet is important? Or why AI/ML models are important? Or why the "Attention is All You Need" paper is important? (yes with some RLHF but that's just to improve a few more percentage points). It will create an incredibly well-sifted, highly compressed answer* all from an algorithm that outputs matrix numbers from fairly limited, fairly shitty internet text, compressed into what is useful in a very eloquent way. That's the excitement of LLMs. Super-human intelligence from an algorithm and low-quality information.
Comments
I also dismissed LLMs given their "accuracy." But I think that's the wrong thing to compare. The sheer fact that a transformer architecture model gets better at scale and that we can feed it at scale is insane. It's what makes it AI and not just ML.
The fact that you can get billions of parameters to do anything useful from a relatively simple algorithm on a relatively small amount (high GBs / low TBs) of text means the algorithm is insane. That's what people miss - they think GPT is trained on "the whole internet" and is similar to some of low-variate regression model that is "approximating things." It is absolutely approximating things - so does all intelligence -- but it is truly sifting / "attending to" what is important over a relatively small corpus and organizing into billions of parameters the way a brain would organize data.
Will it hallucinate details? statistics? Etc.? Yes, and it should not be used in its current form for "truth." But it's very different from a low-variate model that is synthesizing in a low-dimensional space (which is how we gradually learn about the world) and an extremely-high dimensional model that is starting to see "what is important" in ways that are far, far above human intelligence. Similar to a human brain (due to the underlying neural architecture and any type of hierarchical compression of knowledge) but with far more input data, and a simplicity that maybe the brain has maybe it doesn't -- but is far more scalable and capable of hierarchies of information that out-scale us by so many orders of magnitude, and more every 6 months.
3blue1brown's https://www.youtube.com/watch?v=wjZofJX0v4M and upcoming videos I think will show the beauty and simplicity of the algorithm more. To put it another way -- the fact that you get a remotely true outcome with a model that just improves with scale, a remotely true outcome by the algorithm sifting what is important -- means that with time it will know what is more important in ways that far surpass humans.
If you approach interacting with LLM chatbots that way it is absolutely mind-blowing how "on point" the answers are. If you ask ChatGPT why the internet is important? Or why AI/ML models are important? Or why the "Attention is All You Need" paper is important? (yes with some RLHF but that's just to improve a few more percentage points). It will create an incredibly well-sifted, highly compressed answer* all from an algorithm that outputs matrix numbers from fairly limited, fairly shitty internet text, compressed into what is useful in a very eloquent way. That's the excitement of LLMs. Super-human intelligence from an algorithm and low-quality information.
* https://chat.openai.com/share/00a5f9b7-7ee1-4641-92bf-999185...