I have a lot of "darkness" around me concerning threading, but isn't it a net win even with the GIL if the threads are mostly waiting for IO to finish?
I think he means attempting to solve the 'symmetric multiprocessing' problem using threads, rather than using threads for sensible things like IO-forking/event-driven, asynchronous stuff.
In the case I had in mind, the threads were CPU-bound and the programmers naively thought that they were running concurrently. For someone coming from a C++ or Java background, that's a natural assumption to make, and Python doesn't put up many (any) red flags to warn you on the way. For example, note that the pydoc for the module threading manages to avoid any mention of the GIL. You can learn how to create and start Python threads without having any idea that they won't run Python code concurrently.
You're right that if the threads are mostly I/O-bound, it can be a net win, although I seem to recall that even in this case, Dave Beazley's work showed that even one CPU-bound thread is enough to spoil the party significantly for the rest.
Comments
I have a lot of "darkness" around me concerning threading, but isn't it a net win even with the GIL if the threads are mostly waiting for IO to finish?
I think he means attempting to solve the 'symmetric multiprocessing' problem using threads, rather than using threads for sensible things like IO-forking/event-driven, asynchronous stuff.
In the case I had in mind, the threads were CPU-bound and the programmers naively thought that they were running concurrently. For someone coming from a C++ or Java background, that's a natural assumption to make, and Python doesn't put up many (any) red flags to warn you on the way. For example, note that the pydoc for the module threading manages to avoid any mention of the GIL. You can learn how to create and start Python threads without having any idea that they won't run Python code concurrently.
You're right that if the threads are mostly I/O-bound, it can be a net win, although I seem to recall that even in this case, Dave Beazley's work showed that even one CPU-bound thread is enough to spoil the party significantly for the rest.