There have been a few times I did things in python that ended-up being a terrible PITA, multiprocessing was one of those. Basically after fork, you should call exec, but multiprocessing doesn't. So many things worked just fine in Linux and then completely fell apart on FreeBSD, OSX, and Windows. I think a lot of this has been fixed since then by using a Manger and a Pool.
I can't speak for Celery, as I've not used it very much.
It may be straightforward to write a functional Redis alternative in pure python using this library, but I would certainly have questions/concerns about performance.
Comments
http://docs.python.org/2/library/multiprocessing.html#sharin...
Why don't anyone build Celery and Redis alternative using this?
Celery uses `billiard`, which is a fork of multiprocessing.
That doesn't help with communicating with a distributed worker pool, though.
There have been a few times I did things in python that ended-up being a terrible PITA, multiprocessing was one of those. Basically after fork, you should call exec, but multiprocessing doesn't. So many things worked just fine in Linux and then completely fell apart on FreeBSD, OSX, and Windows. I think a lot of this has been fixed since then by using a Manger and a Pool.
I can't speak for Celery, as I've not used it very much.
It may be straightforward to write a functional Redis alternative in pure python using this library, but I would certainly have questions/concerns about performance.