The summer BERT came out I worked at a company that had models not so good as we have today. I was working on the framework for model training that was Python based but we would pack up Tensorflow models and these would be run inside Scala and made available via a web service.
My guess is the CPU overhead of Python is not significant compared to running an LLM but Python has limited facilities for dealing with concurrency. For a while I was into writing asyncio web servers but I eventually found workloads (an image sorter running the wrong way on an ADSL connection: one process is thinking hard for 2sec, meanwhile images are not downloading) that would tie them into knots. gunicorn and celery and similar things can handle parallelism with multiple processes but if you have a 1GB model you will terribly waste memory.
In Java on the other hand you can have a 1GB model and it is shared by the threads and there is no drama.
I wrote a chess program in Python that was good enough to beat my tester a few times last month and have been wanting to take it to a chess club but my tester tells me it needs to respect time control for that. Also I'd like to support a protocol like XBoard or UCI. Either way it is necessary that the comms thread can interrupt the thinking thread and that's dead easy to do in Java and a huge hassle in Python.
Sure there are threads in Python and if I wanted to screw around with alpha software there is the no-GIL Python but remember this: when you're doing a project which has a high-risk or research component it's a bad time to pick tools that require you to learn things. If you are good at Rust or Go I'd say go with that. But don't pick up a language because you heard somebody else thinks it cool. A lot of people are running big and complex apps on Java but you don't hear about it so much.
If I could boot up Python with --no-gil I might give it a shot but it looks like a hassle to install a no-GIL Python now. And after all that hassle I'd expect it to take years for the details to get worked out.
One reason Java was successful with thread safety was its xenophobia. Sure you can load libs with JNI but it feels like putting your hand in a toilet. If you add threads to POSIX you will always have old libraries not built with threads in mind, if you start out with threads the whole ecosystem is built on the assumption it should work right with threads.
Comments
The summer BERT came out I worked at a company that had models not so good as we have today. I was working on the framework for model training that was Python based but we would pack up Tensorflow models and these would be run inside Scala and made available via a web service.
My guess is the CPU overhead of Python is not significant compared to running an LLM but Python has limited facilities for dealing with concurrency. For a while I was into writing asyncio web servers but I eventually found workloads (an image sorter running the wrong way on an ADSL connection: one process is thinking hard for 2sec, meanwhile images are not downloading) that would tie them into knots. gunicorn and celery and similar things can handle parallelism with multiple processes but if you have a 1GB model you will terribly waste memory.
In Java on the other hand you can have a 1GB model and it is shared by the threads and there is no drama.
I wrote a chess program in Python that was good enough to beat my tester a few times last month and have been wanting to take it to a chess club but my tester tells me it needs to respect time control for that. Also I'd like to support a protocol like XBoard or UCI. Either way it is necessary that the comms thread can interrupt the thinking thread and that's dead easy to do in Java and a huge hassle in Python.
Sure there are threads in Python and if I wanted to screw around with alpha software there is the no-GIL Python but remember this: when you're doing a project which has a high-risk or research component it's a bad time to pick tools that require you to learn things. If you are good at Rust or Go I'd say go with that. But don't pick up a language because you heard somebody else thinks it cool. A lot of people are running big and complex apps on Java but you don't hear about it so much.
Good perspective. No-GIL should make things better (shared memory parallelism), but it's not bulletproof.
If I could boot up Python with --no-gil I might give it a shot but it looks like a hassle to install a no-GIL Python now. And after all that hassle I'd expect it to take years for the details to get worked out.
One reason Java was successful with thread safety was its xenophobia. Sure you can load libs with JNI but it feels like putting your hand in a toilet. If you add threads to POSIX you will always have old libraries not built with threads in mind, if you start out with threads the whole ecosystem is built on the assumption it should work right with threads.