Any big data application where the majority of the working set changes continuously and requires more processing than just "bucketing".
The example Twitter gives here is for trending topics, but Storm is basically to message queues what Rails is to CRUD web apps, and so you can draw use cases from eg everything Tibco and JMS are used for today.
Storm is nothing you couldn't have done a year ago, or ten years ago, with a message-oriented architecture. But it has a very attractive feature, which is that it bakes in all the fiddley details and problem solving Twitter did while scaling it to their architecture. Systems like this tend to be easy to prototype and a nightmare to mature and manage, so that's not a small feature.
By their very nature, decentralized processing systems have exponential complexity, so this is very welcome to see released. But oh how I hate having to go back into Java :)
The gist is: Storm topologies are just Thrift structures, and Storm can execute processing components in any language by communicating with subprocesses over a simple protocol based on JSON messages over stdin/stdout. Storm has adapters implementing this protocol for Ruby and Python, and it's easy to make a new adapter for any other language. The adapter libraries are ~100 lines of code and have no dependencies other than JSON.
Comments
Any big data application where the majority of the working set changes continuously and requires more processing than just "bucketing".
The example Twitter gives here is for trending topics, but Storm is basically to message queues what Rails is to CRUD web apps, and so you can draw use cases from eg everything Tibco and JMS are used for today.
Storm is nothing you couldn't have done a year ago, or ten years ago, with a message-oriented architecture. But it has a very attractive feature, which is that it bakes in all the fiddley details and problem solving Twitter did while scaling it to their architecture. Systems like this tend to be easy to prototype and a nightmare to mature and manage, so that's not a small feature.
By their very nature, decentralized processing systems have exponential complexity, so this is very welcome to see released. But oh how I hate having to go back into Java :)
Storm can be used with any programming language. I still need to finish the documentation on this, but I have notes here: https://github.com/nathanmarz/storm/wiki/Using-non-JVM-langu...
The gist is: Storm topologies are just Thrift structures, and Storm can execute processing components in any language by communicating with subprocesses over a simple protocol based on JSON messages over stdin/stdout. Storm has adapters implementing this protocol for Ruby and Python, and it's easy to make a new adapter for any other language. The adapter libraries are ~100 lines of code and have no dependencies other than JSON.
Would it be difficult to use protostuff or protobuf instead? Or maybe Thrift is better for this purpose in some way I don't anticipate?
Ahh, that is super sweet then. Missed the Thrift.
Nathan Marz is a Clojurian, large chunks of Storm are written in Clojure. I think the examples are in Java just to not scare off potential users.
Why would you have to go back to Java? It looks like the only requirement here is a JVM.
At least in Rubyland, JVM dependencies have tended to be a win.