TL;DR: accessing materializations is necessarily a snapshot.
This article reads as though the author hadn't shifted mindset from "the database will solve it for me" to "I'm taking on the relevant subset of problems in my use case". This seems off given that they're trying to sell a streaming product. They claim their product avoids problems by offering "always correct" answers which requires a footnote at the very least but none was given.
Point of note: The consistency guarantee is that upon processing to the same offset in the log that, given that you have taken no other non-constant input, you will have the same computational result as all other processes executing semantically equivalent code.
I take this sort of comment as abusive of the reader:
What does a naive application of eventual consistency have to say about
>
> -- count the records in `data`
> select count(*) from data
>
> It’s not really clear, is it?
A naive application of eventual consistency declares that along some equivalent of a Lamport time stamp across the offsets of shards in the stream, the system will calculate account of records in data as of that offset. Given the ongoing transmission of events that can alter the set data, that value will continue changing as appropriate and in a manner consistent with the data it processes. The new answers will be given when the query is run again or it may even issue an ongoing stream of updates to that value.
I didn't downvote you either. But, in addition to what quodlibetor wrote:
This is a good article from a high-profile author. The way you are criticizing it comes across as ignorant and narcissistic.
"This article reads as though the author hadn't shifted mindset from..."
You are claiming that the author is looking at the problem in the wrong way, and you came to that conclusion before even reading the entire post. It's okay to not be interested in a topic, but stating this in public does not add anything of substance to the discussion.
"I take this sort of comment as abusive of the reader"
This is just offensive, you're basically saying that the comment is stupid. It seems like you are looking for some validation of your intelligence, by the article or by the comment section here.
I'm sorry for expressing ignorant perspectives and behaving narcissistically. My intention was not to call the author stupid or ask for validation of my intellect but I can see how that comes out - thank you for identifying that. I'm sorry and thank you for making it through my comment and being willing to point my failure out to me. I really appreciate it.
I'm actually quite interested in the topic of stream processing and eventual consistency. I was one of the contributors to an open source stream processing based project that won architectural awards at a reasonably big conference, spent a little time working in a machine learning oriented startup based on stream processing with some of the creators of Apache Beam/DataFlow, and now my daily labor is implementing stream processing in another startup. This area of systems design (event sourcing specifically) has been an a bit of professional obsession and labor of love for me over the last five years or so now.
I very clearly failed to express any of that in my post. I further clearly failed to manage my emotional reaction to the article and threw anything useful I may have had an opportunity to add under the bus of my words. I'm sorry for that too.
Again - Thank you very much for helping me grow. I keep working on being a better human and appreciate the help.
I didn't downvote you, but as an outside observer I can see that folks might take issue with the fact that you start with a tl;dr, suggesting that you are summarizing the whole article, followedan in-depth analysis, and then ending by stating that you didn't read the whole thing.
Thank you very much. I was really excited to read the article and became very disappointed but my take was too hot and insufficient. I definitely should have done better. Thank you for expanding my perspective.
Comments
TL;DR: accessing materializations is necessarily a snapshot.
This article reads as though the author hadn't shifted mindset from "the database will solve it for me" to "I'm taking on the relevant subset of problems in my use case". This seems off given that they're trying to sell a streaming product. They claim their product avoids problems by offering "always correct" answers which requires a footnote at the very least but none was given.
Point of note: The consistency guarantee is that upon processing to the same offset in the log that, given that you have taken no other non-constant input, you will have the same computational result as all other processes executing semantically equivalent code.
I take this sort of comment as abusive of the reader:
A naive application of eventual consistency declares that along some equivalent of a Lamport time stamp across the offsets of shards in the stream, the system will calculate account of records in data as of that offset. Given the ongoing transmission of events that can alter the set data, that value will continue changing as appropriate and in a manner consistent with the data it processes. The new answers will be given when the query is run again or it may even issue an ongoing stream of updates to that value.
Maybe it got better as the article went on...
I appreciate that the downvote mechanism is low friction. I wish it were easier to learn and improve from it too.
I didn't downvote you either. But, in addition to what quodlibetor wrote:
This is a good article from a high-profile author. The way you are criticizing it comes across as ignorant and narcissistic.
"This article reads as though the author hadn't shifted mindset from..."
You are claiming that the author is looking at the problem in the wrong way, and you came to that conclusion before even reading the entire post. It's okay to not be interested in a topic, but stating this in public does not add anything of substance to the discussion.
"I take this sort of comment as abusive of the reader"
This is just offensive, you're basically saying that the comment is stupid. It seems like you are looking for some validation of your intelligence, by the article or by the comment section here.
Thank you very much.
I'm sorry for expressing ignorant perspectives and behaving narcissistically. My intention was not to call the author stupid or ask for validation of my intellect but I can see how that comes out - thank you for identifying that. I'm sorry and thank you for making it through my comment and being willing to point my failure out to me. I really appreciate it.
I'm actually quite interested in the topic of stream processing and eventual consistency. I was one of the contributors to an open source stream processing based project that won architectural awards at a reasonably big conference, spent a little time working in a machine learning oriented startup based on stream processing with some of the creators of Apache Beam/DataFlow, and now my daily labor is implementing stream processing in another startup. This area of systems design (event sourcing specifically) has been an a bit of professional obsession and labor of love for me over the last five years or so now.
I very clearly failed to express any of that in my post. I further clearly failed to manage my emotional reaction to the article and threw anything useful I may have had an opportunity to add under the bus of my words. I'm sorry for that too.
Again - Thank you very much for helping me grow. I keep working on being a better human and appreciate the help.
I didn't downvote you, but as an outside observer I can see that folks might take issue with the fact that you start with a tl;dr, suggesting that you are summarizing the whole article, followedan in-depth analysis, and then ending by stating that you didn't read the whole thing.
Thank you very much. I was really excited to read the article and became very disappointed but my take was too hot and insufficient. I definitely should have done better. Thank you for expanding my perspective.