Yup. I'm just wrapping my head around the things like RabbitMQ and Redis and I think I understand what those are for. But can someone explain straight: in which case should I want to use this Octobot?
Octobot (like Celery, Resque, et al) is a worker, meaning that it takes messages from a queue, such as Rabbit or Redis, and processes that message based on a task that you write. Imagine Octobot being used to create thumbnails for Flickr--a job that should be done asynchronously.
Between a list of actions, and the logic that runs those tasks is Octobot. So in your thumbnail example the problem that would be solved would be something along the lines of:
"I've got this code that can create a thumbnail from a given image and let me know if it succeeded or failed, but how do I run this on my backlog of 10million images? I'd need something that can check my list of incoming images and distribute the jobs over X number of computers. At the end it would be great to know my failure rate, and for those failures not to block my ongoing process of creating thumbnails."
So is Octobot there to provide a method of resource allocation? Or is it more of a monitoring app that checks pass/fail of the jobs?
Most web gallery requests go like: (ignoring caching)
Browser -> PHP -> Database
When you upload an image you could simply handle it in-line in the server:
Browser -> PHP -> DB -> Resizer
That means the next page refresh is waiting on that resizer to finish which means long page latency to the most latency-sensitive component imaginable (the fickle user).
So you really want that resize to happen asynchronously, i.e. not wait for the result before showing the page. The roll-your-own method is to put a row in the DB that says "Hey I need to be resized" and have a cron job or somesuch that does the resizing:
Browser -> PHP -> DB
Resizer -> DB
This of course puts all the load on the hardest thing to scale (the DB) so you grow out of it fast. Hence message queues. RabbitMQ, ActiveMQ, Redis, etc are all variants of the queue, so now you have:
Browser -> PHP -> DB -> Queue
And the Queue holds all the resizing jobs that need to be done. You could just modify your cronjob to check the Queue instead of the DB of course.
Octobot (and Celery) is a queue runner that connects to that Queue, reads in the jobs that need to be done and runs them. So instead of a cron job you write your resizer in a way that your queue runner understands and the runner will manage some of the plumbing for you.
So you have
Browser -> PHP -> DB -> Queue
Queue -> Octobot -> Resizer ( -> DB to say it's done perhaps.)
Now that you're decoupled you can add more resizers, webservers, distribute them across multiple systems, expand to EC2 to handle overflow load, etc by leveraging what your queue provides.
Comments
Sorry to be the slow one here... What is a Distributed Task Queue Worker? What problem does it solve?
Yup. I'm just wrapping my head around the things like RabbitMQ and Redis and I think I understand what those are for. But can someone explain straight: in which case should I want to use this Octobot?
Octobot (like Celery, Resque, et al) is a worker, meaning that it takes messages from a queue, such as Rabbit or Redis, and processes that message based on a task that you write. Imagine Octobot being used to create thumbnails for Flickr--a job that should be done asynchronously.
Do I have this correctly?
Between a list of actions, and the logic that runs those tasks is Octobot. So in your thumbnail example the problem that would be solved would be something along the lines of:
"I've got this code that can create a thumbnail from a given image and let me know if it succeeded or failed, but how do I run this on my backlog of 10million images? I'd need something that can check my list of incoming images and distribute the jobs over X number of computers. At the end it would be great to know my failure rate, and for those failures not to block my ongoing process of creating thumbnails."
So is Octobot there to provide a method of resource allocation? Or is it more of a monitoring app that checks pass/fail of the jobs?
To expand the image resizing example:
Most web gallery requests go like: (ignoring caching)
Browser -> PHP -> Database
When you upload an image you could simply handle it in-line in the server:
Browser -> PHP -> DB -> Resizer
That means the next page refresh is waiting on that resizer to finish which means long page latency to the most latency-sensitive component imaginable (the fickle user).
So you really want that resize to happen asynchronously, i.e. not wait for the result before showing the page. The roll-your-own method is to put a row in the DB that says "Hey I need to be resized" and have a cron job or somesuch that does the resizing:
Browser -> PHP -> DB
Resizer -> DB
This of course puts all the load on the hardest thing to scale (the DB) so you grow out of it fast. Hence message queues. RabbitMQ, ActiveMQ, Redis, etc are all variants of the queue, so now you have:
Browser -> PHP -> DB -> Queue
And the Queue holds all the resizing jobs that need to be done. You could just modify your cronjob to check the Queue instead of the DB of course.
Octobot (and Celery) is a queue runner that connects to that Queue, reads in the jobs that need to be done and runs them. So instead of a cron job you write your resizer in a way that your queue runner understands and the runner will manage some of the plumbing for you.
So you have
Browser -> PHP -> DB -> Queue
Queue -> Octobot -> Resizer ( -> DB to say it's done perhaps.)
Now that you're decoupled you can add more resizers, webservers, distribute them across multiple systems, expand to EC2 to handle overflow load, etc by leveraging what your queue provides.
For an example of a perfect fit for a distributed worker, see my blog post:
http://blog.historio.us/asynchronous-processing-using-celery