Building LLM’s. In my mind those engineers are the ones that have more intimate knowledge of the data and input, and can create the LLM’s for their specific tasks. Everyone else is a customer to them.
I can tell you how a house is built, that doesn’t make me a builder that makes me informed and opinionated.
I can decorate my house however I like but im not a painter/decorator or a tradesman. I can assemble some ikea furniture, but I’m not a carpenter. I’m a consumer and I can tweak something to my liking but I can’t do anything significant.
I find this obsession with building strange. It's a very SaaS Silicon Valley mindset. There are whole swathes of very talented engineers who spend most of their time debugging, characterizing systems, doing performance analysis and resolving bottlenecks. Some of it might require writing significant code but mostly it's writing small test cases. The key skill is to treat a computing system as the object of study and to be a good empirical scientist (which requires understanding theory pretty well). These are people with deep expertise in networking, GPUs, CPUs, memory etc. One only has to look at national labs that do large-scale HPC (high-performance computing) to see examples.
One can argue that a lot of "building with AI" is commoditized by fine-tuning and RAG libraries or even reduced to prompt engineering. A lot of it is also tricks that might work on one dataset but not others. Putting together libraries fueled by pizza and coke gives an illusion of skill and speed.
Are there grifters who are jumping onto the AI bandwagon? Of course! In spades. Are there also engineers who want to build up their skills and are failing to do so or in the process of doing so? Of course, this happens too! But there are also people who are trying to understand, debug and improve models who are not necessarily "building". After all, the scaling laws paper (the original one) was a result of pure analysis of empirical data.
No, it is being able to evaluate models. 5 builders without eval produce zero value. 1 eval person can pick and choose the best model out of a bunch that are open source and commercial and maybe one of them is good enough. Put an eval person together with N builders and you have a chance of making a good enough model.
If you want to know why Hacker News is full of people disappointed or skeptical with AI ask yourself why they put 99.9% of their effort into “zero-shot” when it is clear as day that if you get a few thousand examples and train in that you wipe the floor with “zero-shot”
Comments
The real talent is building this stuff, everyone else is just part of the marketing effort.
What do you mean with "building this stuff"? As in building LLMs, or building applications on-top of them.
Building LLM’s. In my mind those engineers are the ones that have more intimate knowledge of the data and input, and can create the LLM’s for their specific tasks. Everyone else is a customer to them.
I can tell you how a house is built, that doesn’t make me a builder that makes me informed and opinionated. I can decorate my house however I like but im not a painter/decorator or a tradesman. I can assemble some ikea furniture, but I’m not a carpenter. I’m a consumer and I can tweak something to my liking but I can’t do anything significant.
I find this obsession with building strange. It's a very SaaS Silicon Valley mindset. There are whole swathes of very talented engineers who spend most of their time debugging, characterizing systems, doing performance analysis and resolving bottlenecks. Some of it might require writing significant code but mostly it's writing small test cases. The key skill is to treat a computing system as the object of study and to be a good empirical scientist (which requires understanding theory pretty well). These are people with deep expertise in networking, GPUs, CPUs, memory etc. One only has to look at national labs that do large-scale HPC (high-performance computing) to see examples.
One can argue that a lot of "building with AI" is commoditized by fine-tuning and RAG libraries or even reduced to prompt engineering. A lot of it is also tricks that might work on one dataset but not others. Putting together libraries fueled by pizza and coke gives an illusion of skill and speed.
Are there grifters who are jumping onto the AI bandwagon? Of course! In spades. Are there also engineers who want to build up their skills and are failing to do so or in the process of doing so? Of course, this happens too! But there are also people who are trying to understand, debug and improve models who are not necessarily "building". After all, the scaling laws paper (the original one) was a result of pure analysis of empirical data.
No, it is being able to evaluate models. 5 builders without eval produce zero value. 1 eval person can pick and choose the best model out of a bunch that are open source and commercial and maybe one of them is good enough. Put an eval person together with N builders and you have a chance of making a good enough model.
If you want to know why Hacker News is full of people disappointed or skeptical with AI ask yourself why they put 99.9% of their effort into “zero-shot” when it is clear as day that if you get a few thousand examples and train in that you wipe the floor with “zero-shot”