asksol on Octobot - A Low-Latency, Highly Parallel Distributed Task Queue Worker

Comment on Octobot - A Low-Latency, Highly Parallel Distributed Task Queue Worker parent

I'm interested in seing the code/config used to benchark Celery. The default settings are not at all optimized for processing lots of small jobs, and you could easily tweak it to get a 100x speed up for that use case, e.g.:

   CELERYD_PREFETCH_MULTIPLIER = 0
   CELERY_DISABLE_RATE_LIMITS = True

Also, channels are not re-used unless you explicitly pass the Publisher, so e.g.

   publisher = task.get_publisher()
   for i in xrange(1000):
       task.apply_async(args=(i, ), publisher=publisher)
   publisher.close()

is known to be a massive speed-up for sending tasks in batch (it seems the creation of channels is very expensive in pyamqplib).

asksol 15y

By the way the performance increase you're seeing with the PyInvoker (from your gist) is most likely because it doesn't have prefetch_count enabled.

Celery enables this so a single worker doesn't suck in a million messages at a time, and to balance the work load between available resources. As noted previously it can be disabled.

Btw, octobot looks great, maybe we can share ideas.

cscottaOP15y

Right on, thanks Ask! I'm checking out some of this right now and might not be able to get through it all today, but will give it a try. Just shot you a couple messages outside of HN - love to talk when you have a chance!

StavrosK 15y

Does CELERYD_PREFETCH_MULTIPLIER do what I think it does (disable prefetching so workers get items as they need them)? I like Celery a lot, having used it for a few days, but the documentation is a bit frustrating :/

Comments