If you have two or more services that need to agree about state, and you have some set of rules that govern what state changes are valid, and you don't want to mess around with any of this "what do we do when we miss and update vs when we get two of the same update" nonsense, and the services aren't in a position to query the same database, then you should really consider a permissioned blockchain. Consensus hard, but it's harder if you're not using tools that understand that what they're going for is consensus.
Why is cryptographically verifying record sequentiality an important property here?
If I offer my customers a source of ordered records, the "trust" in that system is the fact that they pay me to make sure records are ordered. If I sell a fast or slow log database, approximately zero customers in the world care to verify ordering cryptographically.
Or by "blockchain" do you just mean .... records with sequential IDs? Because sequential, guaranteed IDs surface gappiness/idempotency a lot easier than Markov chains over cryptographic primitives.
That also doesn't address the other core problems in the article: the replication (or data retrieval/polling) protocol is a lot more complex than a blockchain's "I can verify and replicate the entire chain state from the beginning of time to you" single behavior. People want more specificity than that.
If your consumers are going to end up in this position:
I do not trust the copy I built, and I have no way to know when it’s wrong, so I will re-derive it from scratch every night, forever.
Then guaranteed sequentiality means that they only have to verify each new block rather than fetch the whole thing every night.
Without it, you have this ever growing probability, which resets to 0 each night, that you unknowingly hold an invalid state. You might've acted on that state and so now when the nightly cleanup runs you have add code to go back out the consequences and instead apply the prosequences.
The complexity you think you're avoiding by not having a consensus protocol you're instead embracing as a data cleanup job, except instead of the same code everywhere, each reader has their own separate implementation.
That all goes away if you just don't process inbound data until you're sure nothing else is going to come along and invalidate it.
Sequential ID's work when there is only one writer and their implementation can be trusted but sometimes we get different events which both say they're number 12 and then we have to go call upstream and learn that so-and-so was on vacation and it won't happen again we promise. It takes days to resolve during which the potential of propagating bad state based on the lack of resolution continues to rise.
Ideally you can just avoid coordination delays entirely by keeping things monotonic and leaning on the CALM theorem but when that's not in the cards it's way better to put that delay on the writer's side, which is what blockchains do. Waiting for the consensus protocol to spit out a block before assuming that a write landed saves so many headaches on the reader side.
99% of the time (and all 3 times in the blog post), there is only one source of truth for any piece of data, and state transitions are completely arbitrary. Blockchain is almost always the wrong solution.
Especially since in most of the cases where it's not-totally-insane to use, the right solution is still the classic distributed database which already existed. In those, the ledger is kept among a predefined/controlled node-membership... as opposed to a bloated mass of workarounds and limitations to make it barely survive being ungovernable.
I've seen some boosters pivot to saying "private blockchain is good", but that's contradictory buzzword nonsense. It's like selling a blog as "single-user Twitter" or advertising a regular car as "user-controlled autonomous vehicle."
I'm not sure what a private blockchain is, but a permissioned blockchain is one where only certain parties have keys that allow them to write blocks. You end up with a message queue optimized to eliminate anything that would lead to the inconsistency nonsense that this article is talking about as soon as it is detected. It then becomes the writer's problem to retransmit in a way that doesn't cause a problem next time, rather than the reader's problem to recover.
The craziness comes in when you'll accept blocks from anybody willing to burn enough electricity to do so, or gamble enough tokens to do so, or whatever other artificial scarcity game people like to play. But if you're only planning to consume data from the eight other companies you do business with then there's no reason to bother with any of that, you can just hard code their public keys into your consensus protocol and you've sidestepped the nonsense.
If you're exchanging public keys to configure your node with in the first place, then presumably you have a trusted side channel for that kind of thing. It's equivalent to the webhook situation re: what if the domain name changes?
If you don't have such a channel then there's always social key recovery schemes, but I don't think there's a particular pattern that you can just adopt. You're now in participate-in-the-research mode.
If the hardcoded domain name changes, then the app needs to be recompiled with new domain name and that's it. The service is instantly restored and no data needs to be updated to reflect the change. In fact, the server side doesn't even need to be made aware there was any change. It just continues serving data as normal.
If you have a blockchain, and the blockchain requires a key to write to, and there's only one key, and the private key gets stolen by malicious email attachment and posted on darkweb - what now? Do you need a whole new blockchain? What happens to data from the old blockchain, how do you verify the new data is correct? Or do you keep the blockchain and use the leaked key to revoke the key and set a new one? How do we know the new key is legitimate?
I genuinely don't know how to recover in this situation and would really like for someone to walk me through the process.
That might be true for 99% of data, but I find that it's the remaining 1% of that data occupies the majority of the time. If you don't have a problem that is solved by a blockchain don't use one, but if you're saying things like this:
I do not trust the copy I built, and I have no way to know when it’s wrong, so I will re-derive it from scratch every night, forever.
Then your life would probably be better if you just had to consume block-at-a-time and not the whole dataset every night. Better to be persistently five minutes behind, then to go all day not knowing whether you're wrong, with a brief moment of certainty each night.
Your nightly reconciliation job just ends up being an inside out version of the consensus protocol that you failed to enforce up front (which may be a necessary evil if you have no influence over the people who publish your data, but let's not let those people off the hook for failing to support incremental verification of sync).
Not 99% of data - 99% of applications. In 99% of applications, 100% of data is such that any piece of this 100% has only one authoritative source. Only in 1% of applications, the percent of data that doesn't have one authoritative source is less than 100%. So the absolute upper bound of when blockchain is even appropriate at all is 1% of applications.
Then your life would probably be better if you just had to consume block-at-a-time and not the whole dataset every night.
Yes, exactly, that's the whole point of the article, the data should be an ordered stream, not asynchronous events. As long as it's an ordered stream, the author's life is peachy. Blockchain gives you an ordered stream, yes. But so does SCROLL. And if you choose SCROLL, you don't have to deal with the plethora of blockchain-specific problems.
Whenever you have a choice between SCROLL and a blockchain, you should always pick SCROLL and never a blockchain. Only if SCROLL won't work for your use case - for example, you actually need a consensus mechanism - you should consider a blockchain.
And no, blindly copying another database and overwriting every discrepancy with their version is NOT a consensus protocol! It's not meant to build a consensus! It's meant to copy data from authoritative source! There's no consensus to be had!
That is true, but I think it's a pattern we need to get away from. It's the source of everything that is wrong with the modern web.
- These single sources of truth become high value targets for corruption.
- Our apps become totally useless if they can't connect to whichever source we've hard-coded as authoritative.
- Even when connectivity is good and upstream sources are trustworthy, we end up creating unnecessary burdens on infrastructure for a connections that spans the globe from a single point to millions of them even when the data we're after is elsewhere on the LAN, ignored because it doesn't have the authoritative hostname.
It's much more fault tolerant to trust the data on the basis of its verifiable properties, not because of where you got it. That way, if the source becomes unavailable or untrustworthy, somebody more relevant can start being the leader (supposing you need a leader at all).
you should always pick SCROLL and never a blockchain
The SCROLL RFC is very honest about how mature it is:
draft-scroll-protocol-00 · request for comments · no implementations in the wild, one document
People have built some truly stupid things on a blockchain, and SCROLL does seem like a good idea for some cases, but I think you're going a bit far by suggesting that a protocol which has never been implemented is always preferable to one with dosens of implementations to chose from.
The real world operates on single sources of truth. The software merely models the real world - as it should. Saying we should move away from this pattern is synonymous with saying we should fundamentally change how are society works. Hint: if your preferred software solution only makes sense if we fundamentally change how society works, maybe it's not the best solution.
If Stripe says you didn't get the payment, then you didn't get the payment. End of story. The payment isn't magically going to appear in your wallet just because you verify some properties of some data.
If your only complaint against SCROLL is that an implementation doesn't exist yet, then how about implementing SCROLL instead of creating yet another blockchain? It's not much work anyway, SCROLL is like 1000 times simpler than even the simplest of blockchains.
I don't think society worked that way until the web made it possible for society to work that way. And it's not working particularly well. It's brittle, coercive, and it asks operators to have a hand in supporting it in any way to support it in every way, raising the challenge of participating to a level where only the largest companies can.
If we think we can fundamentally change it for the better, we should.
I'm not especially enamored of blockchains. I think that partition tolerance is more important than global consistency, so mostly it's CRDT's that I'm trying to apply. I just brought up blockchains because they seemed especially applicable to the problems that SCROLL is trying to solve.
If you have a single source and a reliable connection to it, SCROLL seems great, I just think we should be preparing fallbacks for when those assumptions need to change. Because the real world doesn't operate on single sources of truth. Pick anything that people talk about, and you'll find that they're not on the same page about which sources are trustworthy for that topic. The software world operates on single sources of truth not because the real world does, but because software has so far failed to be useful outside of that domain.
I don't think society worked that way until the web made it possible for society to work that way.
I very much do think so - moreover, it's literally how it worked for decades, if not centuries. Before the internet, there was only one organization that knew whether you paid your taxes, the IRS. There was only one organization that knew your bank balance, the bank. There was only one organization that knew whether you have active water service, the water company. Everybody else - including yourself - had to either guess, or ask the single authoritative source.
Internet changed nothing about this. It only increased the number of queries and shortened response times.
Comments
If you have two or more services that need to agree about state, and you have some set of rules that govern what state changes are valid, and you don't want to mess around with any of this "what do we do when we miss and update vs when we get two of the same update" nonsense, and the services aren't in a position to query the same database, then you should really consider a permissioned blockchain. Consensus hard, but it's harder if you're not using tools that understand that what they're going for is consensus.
Why is cryptographically verifying record sequentiality an important property here?
If I offer my customers a source of ordered records, the "trust" in that system is the fact that they pay me to make sure records are ordered. If I sell a fast or slow log database, approximately zero customers in the world care to verify ordering cryptographically.
Or by "blockchain" do you just mean .... records with sequential IDs? Because sequential, guaranteed IDs surface gappiness/idempotency a lot easier than Markov chains over cryptographic primitives.
That also doesn't address the other core problems in the article: the replication (or data retrieval/polling) protocol is a lot more complex than a blockchain's "I can verify and replicate the entire chain state from the beginning of time to you" single behavior. People want more specificity than that.
If your consumers are going to end up in this position:
Then guaranteed sequentiality means that they only have to verify each new block rather than fetch the whole thing every night.
Without it, you have this ever growing probability, which resets to 0 each night, that you unknowingly hold an invalid state. You might've acted on that state and so now when the nightly cleanup runs you have add code to go back out the consequences and instead apply the prosequences.
The complexity you think you're avoiding by not having a consensus protocol you're instead embracing as a data cleanup job, except instead of the same code everywhere, each reader has their own separate implementation.
That all goes away if you just don't process inbound data until you're sure nothing else is going to come along and invalidate it.
Sequential ID's work when there is only one writer and their implementation can be trusted but sometimes we get different events which both say they're number 12 and then we have to go call upstream and learn that so-and-so was on vacation and it won't happen again we promise. It takes days to resolve during which the potential of propagating bad state based on the lack of resolution continues to rise.
Ideally you can just avoid coordination delays entirely by keeping things monotonic and leaning on the CALM theorem but when that's not in the cards it's way better to put that delay on the writer's side, which is what blockchains do. Waiting for the consensus protocol to spit out a block before assuming that a write landed saves so many headaches on the reader side.
99% of the time (and all 3 times in the blog post), there is only one source of truth for any piece of data, and state transitions are completely arbitrary. Blockchain is almost always the wrong solution.
Especially since in most of the cases where it's not-totally-insane to use, the right solution is still the classic distributed database which already existed. In those, the ledger is kept among a predefined/controlled node-membership... as opposed to a bloated mass of workarounds and limitations to make it barely survive being ungovernable.
I've seen some boosters pivot to saying "private blockchain is good", but that's contradictory buzzword nonsense. It's like selling a blog as "single-user Twitter" or advertising a regular car as "user-controlled autonomous vehicle."
I'm not sure what a private blockchain is, but a permissioned blockchain is one where only certain parties have keys that allow them to write blocks. You end up with a message queue optimized to eliminate anything that would lead to the inconsistency nonsense that this article is talking about as soon as it is detected. It then becomes the writer's problem to retransmit in a way that doesn't cause a problem next time, rather than the reader's problem to recover.
The craziness comes in when you'll accept blocks from anybody willing to burn enough electricity to do so, or gamble enough tokens to do so, or whatever other artificial scarcity game people like to play. But if you're only planning to consume data from the eight other companies you do business with then there's no reason to bother with any of that, you can just hard code their public keys into your consensus protocol and you've sidestepped the nonsense.
What if the keys leak and need to be rotated?
If you're exchanging public keys to configure your node with in the first place, then presumably you have a trusted side channel for that kind of thing. It's equivalent to the webhook situation re: what if the domain name changes?
If you don't have such a channel then there's always social key recovery schemes, but I don't think there's a particular pattern that you can just adopt. You're now in participate-in-the-research mode.
If the hardcoded domain name changes, then the app needs to be recompiled with new domain name and that's it. The service is instantly restored and no data needs to be updated to reflect the change. In fact, the server side doesn't even need to be made aware there was any change. It just continues serving data as normal.
If you have a blockchain, and the blockchain requires a key to write to, and there's only one key, and the private key gets stolen by malicious email attachment and posted on darkweb - what now? Do you need a whole new blockchain? What happens to data from the old blockchain, how do you verify the new data is correct? Or do you keep the blockchain and use the leaked key to revoke the key and set a new one? How do we know the new key is legitimate?
I genuinely don't know how to recover in this situation and would really like for someone to walk me through the process.
That might be true for 99% of data, but I find that it's the remaining 1% of that data occupies the majority of the time. If you don't have a problem that is solved by a blockchain don't use one, but if you're saying things like this:
Then your life would probably be better if you just had to consume block-at-a-time and not the whole dataset every night. Better to be persistently five minutes behind, then to go all day not knowing whether you're wrong, with a brief moment of certainty each night.
Your nightly reconciliation job just ends up being an inside out version of the consensus protocol that you failed to enforce up front (which may be a necessary evil if you have no influence over the people who publish your data, but let's not let those people off the hook for failing to support incremental verification of sync).
Not 99% of data - 99% of applications. In 99% of applications, 100% of data is such that any piece of this 100% has only one authoritative source. Only in 1% of applications, the percent of data that doesn't have one authoritative source is less than 100%. So the absolute upper bound of when blockchain is even appropriate at all is 1% of applications.
Yes, exactly, that's the whole point of the article, the data should be an ordered stream, not asynchronous events. As long as it's an ordered stream, the author's life is peachy. Blockchain gives you an ordered stream, yes. But so does SCROLL. And if you choose SCROLL, you don't have to deal with the plethora of blockchain-specific problems.
Whenever you have a choice between SCROLL and a blockchain, you should always pick SCROLL and never a blockchain. Only if SCROLL won't work for your use case - for example, you actually need a consensus mechanism - you should consider a blockchain.
And no, blindly copying another database and overwriting every discrepancy with their version is NOT a consensus protocol! It's not meant to build a consensus! It's meant to copy data from authoritative source! There's no consensus to be had!
That is true, but I think it's a pattern we need to get away from. It's the source of everything that is wrong with the modern web.
- These single sources of truth become high value targets for corruption.
- Our apps become totally useless if they can't connect to whichever source we've hard-coded as authoritative.
- Even when connectivity is good and upstream sources are trustworthy, we end up creating unnecessary burdens on infrastructure for a connections that spans the globe from a single point to millions of them even when the data we're after is elsewhere on the LAN, ignored because it doesn't have the authoritative hostname.
It's much more fault tolerant to trust the data on the basis of its verifiable properties, not because of where you got it. That way, if the source becomes unavailable or untrustworthy, somebody more relevant can start being the leader (supposing you need a leader at all).
The SCROLL RFC is very honest about how mature it is:
People have built some truly stupid things on a blockchain, and SCROLL does seem like a good idea for some cases, but I think you're going a bit far by suggesting that a protocol which has never been implemented is always preferable to one with dosens of implementations to chose from.
The real world operates on single sources of truth. The software merely models the real world - as it should. Saying we should move away from this pattern is synonymous with saying we should fundamentally change how are society works. Hint: if your preferred software solution only makes sense if we fundamentally change how society works, maybe it's not the best solution.
If Stripe says you didn't get the payment, then you didn't get the payment. End of story. The payment isn't magically going to appear in your wallet just because you verify some properties of some data.
If your only complaint against SCROLL is that an implementation doesn't exist yet, then how about implementing SCROLL instead of creating yet another blockchain? It's not much work anyway, SCROLL is like 1000 times simpler than even the simplest of blockchains.
I don't think society worked that way until the web made it possible for society to work that way. And it's not working particularly well. It's brittle, coercive, and it asks operators to have a hand in supporting it in any way to support it in every way, raising the challenge of participating to a level where only the largest companies can.
If we think we can fundamentally change it for the better, we should.
I'm not especially enamored of blockchains. I think that partition tolerance is more important than global consistency, so mostly it's CRDT's that I'm trying to apply. I just brought up blockchains because they seemed especially applicable to the problems that SCROLL is trying to solve.
If you have a single source and a reliable connection to it, SCROLL seems great, I just think we should be preparing fallbacks for when those assumptions need to change. Because the real world doesn't operate on single sources of truth. Pick anything that people talk about, and you'll find that they're not on the same page about which sources are trustworthy for that topic. The software world operates on single sources of truth not because the real world does, but because software has so far failed to be useful outside of that domain.
I very much do think so - moreover, it's literally how it worked for decades, if not centuries. Before the internet, there was only one organization that knew whether you paid your taxes, the IRS. There was only one organization that knew your bank balance, the bank. There was only one organization that knew whether you have active water service, the water company. Everybody else - including yourself - had to either guess, or ask the single authoritative source.
Internet changed nothing about this. It only increased the number of queries and shortened response times.