Apache Storm[1] isn't exactly like Erlang/OTP, but it definitely aims to achieve similar goals. Though Storm is written in Java and Clojure, the infrastructure it provides is usable across languages thanks to something it calls the "multi-lang protocol".
I work on a Storm library for Python that's called streamparse[2], and the goal of the project is to allow us to easily achieve Erlang OTP-style reliable processing atop open source infrastructure while still writing pure Python code.
I also gave a PyCon talk about streamparse which you can find on YouTube[3]. It describes the motivation for the project -- which is to solve a large-scale real-time analytics problem with Python and do so in a reliable way, while beating Python's multi-core limitations (the GIL, etc.)
I have a more cynical view of the Heron announcement. But let's remember that Twitter ran Storm in production for over 4 years at massive scale, and whatever improvements Heron represents, they thought the abstraction was important enough to reimplement the entire Storm API in the new system.
OTP is described thusly by "Learn You Some Erlang":
"OTP contains functions to safely spawn and initialize processes, send messages to them in a fault-tolerant manner and many other things."
Also, "supervisors are one of the most useful parts of OTP you'll get to use." And, "an OTP application specifically uses OTP behaviours for its processes, and then wraps them in a very specific structure that tells the VM how to set everything up and then tear it down."
Given that, I don't see how you could think Storm isn't at all related. (I never said it was equivalent, just very similar goals.)
not sure if this exists right now, but i would love to see a good tutorial/article on storm showing how to use it to solve real life problems instead of "hey here's a way to split words and count them".
It is a fair criticism of the state of Storm documentation and examples. The book Storm Applied by Manning details tons of Storm usage examples, ranging from the technically advanced to the mundane. I recommend that book to learn more about real world usage (disclosure: I wrote the foreword).
Also, my team is working on an open source example project called birding that uses streamparse and pykafka together to build filtered firehose tweet streams from Twitter's public API. We think this will illustrate both Storm and Kafka really well, and be beyond the typical word count examples. See https://github.com/Parsely/birding to track that effort.
We're using Storm to handle high volume Kafka feeds to process advertising bid requests and other user data in machine learning models. Most of it is proprietary, unfortunately, so I can't write the article you'd like (and that I would have liked before starting this).
So far Storm is working as advertised. Frankly, what it does isn't that terribly difficult, but it's good to have a well-tested implementation of it so we can focus on our business logic.
Comments
Apache Storm[1] isn't exactly like Erlang/OTP, but it definitely aims to achieve similar goals. Though Storm is written in Java and Clojure, the infrastructure it provides is usable across languages thanks to something it calls the "multi-lang protocol".
I work on a Storm library for Python that's called streamparse[2], and the goal of the project is to allow us to easily achieve Erlang OTP-style reliable processing atop open source infrastructure while still writing pure Python code.
I also gave a PyCon talk about streamparse which you can find on YouTube[3]. It describes the motivation for the project -- which is to solve a large-scale real-time analytics problem with Python and do so in a reliable way, while beating Python's multi-core limitations (the GIL, etc.)
[1]: https://storm.apache.org/
[2]: https://github.com/Parsely/streamparse
[3]: https://www.youtube.com/watch?v=ja4Qj9-l6WQ
that reliability wasn't the case for at least one of the internet heavyweights, quite the opposite [1]
[1] http://blog.acolyer.org/2015/06/15/twitter-heron-stream-proc...
I have a more cynical view of the Heron announcement. But let's remember that Twitter ran Storm in production for over 4 years at massive scale, and whatever improvements Heron represents, they thought the abstraction was important enough to reimplement the entire Storm API in the new system.
None of those are equivalent. In fact they don't actually address the issue.
OTP is described thusly by "Learn You Some Erlang":
"OTP contains functions to safely spawn and initialize processes, send messages to them in a fault-tolerant manner and many other things."
Also, "supervisors are one of the most useful parts of OTP you'll get to use." And, "an OTP application specifically uses OTP behaviours for its processes, and then wraps them in a very specific structure that tells the VM how to set everything up and then tear it down."
Given that, I don't see how you could think Storm isn't at all related. (I never said it was equivalent, just very similar goals.)
not sure if this exists right now, but i would love to see a good tutorial/article on storm showing how to use it to solve real life problems instead of "hey here's a way to split words and count them".
It is a fair criticism of the state of Storm documentation and examples. The book Storm Applied by Manning details tons of Storm usage examples, ranging from the technically advanced to the mundane. I recommend that book to learn more about real world usage (disclosure: I wrote the foreword).
Also, my team is working on an open source example project called birding that uses streamparse and pykafka together to build filtered firehose tweet streams from Twitter's public API. We think this will illustrate both Storm and Kafka really well, and be beyond the typical word count examples. See https://github.com/Parsely/birding to track that effort.
We're using Storm to handle high volume Kafka feeds to process advertising bid requests and other user data in machine learning models. Most of it is proprietary, unfortunately, so I can't write the article you'd like (and that I would have liked before starting this).
So far Storm is working as advertised. Frankly, what it does isn't that terribly difficult, but it's good to have a well-tested implementation of it so we can focus on our business logic.