I have a feeling the hardware architecture for LLMs are completely wrong. There's no way hundreds of kilowatts is required for intelligence.. just in terms of the physics. Is there someone out there in the analog/neuromorphic computing world that could make these power-hungry monsters completely redundant?
It's more the economics of what sells that lets the power hungry monsters thrive. If you can tape out one chip design, and one gives you 1 TOPS and the other is 5 TOPS, the sales will go to the 5 TOPS because power is a downstream cost. That being said, it's only a matter of time before both chips are maximum TOPS and the only differentiator is power. My latest simulations are showing a competitive design around 70W, but I'm still confirming it and it would really be beneficial to do some data gathering tapeouts at larger nodes to feel like it's worth the risk of more expensive nodes.
Yes smaller nodes give lower power operation with faster transistor switching speeds. You can also underpower a node to intentionally cause slower transistor switching speeds.
You're completely wrong about hardware being the bottleneck.
The software architecture LLMs run on was chosen precisely because hardware can brute force it and brute force has become the method of choice ever since, because it is easier to scale.
And here you're saying "what if everyone was wrong and what we need is just more brute force?", without looking at the reason that makes brute force necessary to begin with.
This is also the problem, we are continuing with LLMs because they fit hardware well, instead of experimenting with trying to model knowledge (how do I know what I know) and metacognition (what do I know).
Current architecture can't tell the difference between fact and fiction, so to speak. (Which I think is a better way to say it rather than truth and falsehood; facts have provenance and context, not just "true" or "false".)
I mean, we ground that in external verification (or, use math, which in theory is accessible to everyone, but also has an "empirical" loop now via lean).
I'm not saying AI has remotely solved this problem, not even close. But plenty of philosophy, especially in the 1950s+, from analytic philosophers that these kinds of statements aren't coherent.
Symbolic and formal AI has failed as a general path to intelligence, heuristic bash with deep learning won. There needs to be some kind of actual breakthrough on the actual structure of how we formalize things - we can either describe processes and just let them run (physics, CS), or verify well (math), but if you look at any messy, real-life domain, how "knowledge" is encoded is a bajillion heuristics that would not really be the "rules" of the domain. The heuristics being linear algebra rather than SAT equations doesn't change how bad the "knowledge" is compared to what humans idealize as "knowledge" or even empirically, the kinds of things humans cut at for "knowledge".
Comments
I have a feeling the hardware architecture for LLMs are completely wrong. There's no way hundreds of kilowatts is required for intelligence.. just in terms of the physics. Is there someone out there in the analog/neuromorphic computing world that could make these power-hungry monsters completely redundant?
It's more the economics of what sells that lets the power hungry monsters thrive. If you can tape out one chip design, and one gives you 1 TOPS and the other is 5 TOPS, the sales will go to the 5 TOPS because power is a downstream cost. That being said, it's only a matter of time before both chips are maximum TOPS and the only differentiator is power. My latest simulations are showing a competitive design around 70W, but I'm still confirming it and it would really be beneficial to do some data gathering tapeouts at larger nodes to feel like it's worth the risk of more expensive nodes.
Systolic hardware?
Race logic with pulses, it's a descent of systolic arrays. There is less of an array shape and more about time.
Dumb question, wouldn't node size impact the logic latency itself on some level?
Yes smaller nodes give lower power operation with faster transistor switching speeds. You can also underpower a node to intentionally cause slower transistor switching speeds.
Where can I find the whitepaper?
You're completely wrong about hardware being the bottleneck.
The software architecture LLMs run on was chosen precisely because hardware can brute force it and brute force has become the method of choice ever since, because it is easier to scale.
And here you're saying "what if everyone was wrong and what we need is just more brute force?", without looking at the reason that makes brute force necessary to begin with.
This is also the problem, we are continuing with LLMs because they fit hardware well, instead of experimenting with trying to model knowledge (how do I know what I know) and metacognition (what do I know).
Current architecture can't tell the difference between fact and fiction, so to speak. (Which I think is a better way to say it rather than truth and falsehood; facts have provenance and context, not just "true" or "false".)
I mean, we ground that in external verification (or, use math, which in theory is accessible to everyone, but also has an "empirical" loop now via lean).
I'm not saying AI has remotely solved this problem, not even close. But plenty of philosophy, especially in the 1950s+, from analytic philosophers that these kinds of statements aren't coherent.
Symbolic and formal AI has failed as a general path to intelligence, heuristic bash with deep learning won. There needs to be some kind of actual breakthrough on the actual structure of how we formalize things - we can either describe processes and just let them run (physics, CS), or verify well (math), but if you look at any messy, real-life domain, how "knowledge" is encoded is a bajillion heuristics that would not really be the "rules" of the domain. The heuristics being linear algebra rather than SAT equations doesn't change how bad the "knowledge" is compared to what humans idealize as "knowledge" or even empirically, the kinds of things humans cut at for "knowledge".
human brain uses 20 watts.
I have a gut feeling the particular stochastic processes we’re currently using are the wrong ones to go beyond LLMs into actual intelligence.