99% of the time (and all 3 times in the blog post), there is only one source of truth for any piece of data, and state transitions are completely arbitrary. Blockchain is almost always the wrong solution.
Especially since in most of the cases where it's not-totally-insane to use, the right solution is still the classic distributed database which already existed. In those, the ledger is kept among a predefined/controlled node-membership... as opposed to a bloated mass of workarounds and limitations to make it barely survive being ungovernable.
I've seen some boosters pivot to saying "private blockchain is good", but that's contradictory buzzword nonsense. It's like selling a blog as "single-user Twitter" or advertising a regular car as "user-controlled autonomous vehicle."
I'm not sure what a private blockchain is, but a permissioned blockchain is one where only certain parties have keys that allow them to write blocks. You end up with a message queue optimized to eliminate anything that would lead to the inconsistency nonsense that this article is talking about as soon as it is detected. It then becomes the writer's problem to retransmit in a way that doesn't cause a problem next time, rather than the reader's problem to recover.
The craziness comes in when you'll accept blocks from anybody willing to burn enough electricity to do so, or gamble enough tokens to do so, or whatever other artificial scarcity game people like to play. But if you're only planning to consume data from the eight other companies you do business with then there's no reason to bother with any of that, you can just hard code their public keys into your consensus protocol and you've sidestepped the nonsense.
If you're exchanging public keys to configure your node with in the first place, then presumably you have a trusted side channel for that kind of thing. It's equivalent to the webhook situation re: what if the domain name changes?
If you don't have such a channel then there's always social key recovery schemes, but I don't think there's a particular pattern that you can just adopt. You're now in participate-in-the-research mode.
If the hardcoded domain name changes, then the app needs to be recompiled with new domain name and that's it. The service is instantly restored and no data needs to be updated to reflect the change. In fact, the server side doesn't even need to be made aware there was any change. It just continues serving data as normal.
If you have a blockchain, and the blockchain requires a key to write to, and there's only one key, and the private key gets stolen by malicious email attachment and posted on darkweb - what now? Do you need a whole new blockchain? What happens to data from the old blockchain, how do you verify the new data is correct? Or do you keep the blockchain and use the leaked key to revoke the key and set a new one? How do we know the new key is legitimate?
I genuinely don't know how to recover in this situation and would really like for someone to walk me through the process.
That might be true for 99% of data, but I find that it's the remaining 1% of that data occupies the majority of the time. If you don't have a problem that is solved by a blockchain don't use one, but if you're saying things like this:
I do not trust the copy I built, and I have no way to know when it’s wrong, so I will re-derive it from scratch every night, forever.
Then your life would probably be better if you just had to consume block-at-a-time and not the whole dataset every night. Better to be persistently five minutes behind, then to go all day not knowing whether you're wrong, with a brief moment of certainty each night.
Your nightly reconciliation job just ends up being an inside out version of the consensus protocol that you failed to enforce up front (which may be a necessary evil if you have no influence over the people who publish your data, but let's not let those people off the hook for failing to support incremental verification of sync).
Not 99% of data - 99% of applications. In 99% of applications, 100% of data is such that any piece of this 100% has only one authoritative source. Only in 1% of applications, the percent of data that doesn't have one authoritative source is less than 100%. So the absolute upper bound of when blockchain is even appropriate at all is 1% of applications.
Then your life would probably be better if you just had to consume block-at-a-time and not the whole dataset every night.
Yes, exactly, that's the whole point of the article, the data should be an ordered stream, not asynchronous events. As long as it's an ordered stream, the author's life is peachy. Blockchain gives you an ordered stream, yes. But so does SCROLL. And if you choose SCROLL, you don't have to deal with the plethora of blockchain-specific problems.
Whenever you have a choice between SCROLL and a blockchain, you should always pick SCROLL and never a blockchain. Only if SCROLL won't work for your use case - for example, you actually need a consensus mechanism - you should consider a blockchain.
And no, blindly copying another database and overwriting every discrepancy with their version is NOT a consensus protocol! It's not meant to build a consensus! It's meant to copy data from authoritative source! There's no consensus to be had!
That is true, but I think it's a pattern we need to get away from. It's the source of everything that is wrong with the modern web.
- These single sources of truth become high value targets for corruption.
- Our apps become totally useless if they can't connect to whichever source we've hard-coded as authoritative.
- Even when connectivity is good and upstream sources are trustworthy, we end up creating unnecessary burdens on infrastructure for a connections that spans the globe from a single point to millions of them even when the data we're after is elsewhere on the LAN, ignored because it doesn't have the authoritative hostname.
It's much more fault tolerant to trust the data on the basis of its verifiable properties, not because of where you got it. That way, if the source becomes unavailable or untrustworthy, somebody more relevant can start being the leader (supposing you need a leader at all).
you should always pick SCROLL and never a blockchain
The SCROLL RFC is very honest about how mature it is:
draft-scroll-protocol-00 · request for comments · no implementations in the wild, one document
People have built some truly stupid things on a blockchain, and SCROLL does seem like a good idea for some cases, but I think you're going a bit far by suggesting that a protocol which has never been implemented is always preferable to one with dosens of implementations to chose from.
The real world operates on single sources of truth. The software merely models the real world - as it should. Saying we should move away from this pattern is synonymous with saying we should fundamentally change how are society works. Hint: if your preferred software solution only makes sense if we fundamentally change how society works, maybe it's not the best solution.
If Stripe says you didn't get the payment, then you didn't get the payment. End of story. The payment isn't magically going to appear in your wallet just because you verify some properties of some data.
If your only complaint against SCROLL is that an implementation doesn't exist yet, then how about implementing SCROLL instead of creating yet another blockchain? It's not much work anyway, SCROLL is like 1000 times simpler than even the simplest of blockchains.
I don't think society worked that way until the web made it possible for society to work that way. And it's not working particularly well. It's brittle, coercive, and it asks operators to have a hand in supporting it in any way to support it in every way, raising the challenge of participating to a level where only the largest companies can.
If we think we can fundamentally change it for the better, we should.
I'm not especially enamored of blockchains. I think that partition tolerance is more important than global consistency, so mostly it's CRDT's that I'm trying to apply. I just brought up blockchains because they seemed especially applicable to the problems that SCROLL is trying to solve.
If you have a single source and a reliable connection to it, SCROLL seems great, I just think we should be preparing fallbacks for when those assumptions need to change. Because the real world doesn't operate on single sources of truth. Pick anything that people talk about, and you'll find that they're not on the same page about which sources are trustworthy for that topic. The software world operates on single sources of truth not because the real world does, but because software has so far failed to be useful outside of that domain.
I don't think society worked that way until the web made it possible for society to work that way.
I very much do think so - moreover, it's literally how it worked for decades, if not centuries. Before the internet, there was only one organization that knew whether you paid your taxes, the IRS. There was only one organization that knew your bank balance, the bank. There was only one organization that knew whether you have active water service, the water company. Everybody else - including yourself - had to either guess, or ask the single authoritative source.
Internet changed nothing about this. It only increased the number of queries and shortened response times.
Comments
99% of the time (and all 3 times in the blog post), there is only one source of truth for any piece of data, and state transitions are completely arbitrary. Blockchain is almost always the wrong solution.
Especially since in most of the cases where it's not-totally-insane to use, the right solution is still the classic distributed database which already existed. In those, the ledger is kept among a predefined/controlled node-membership... as opposed to a bloated mass of workarounds and limitations to make it barely survive being ungovernable.
I've seen some boosters pivot to saying "private blockchain is good", but that's contradictory buzzword nonsense. It's like selling a blog as "single-user Twitter" or advertising a regular car as "user-controlled autonomous vehicle."
I'm not sure what a private blockchain is, but a permissioned blockchain is one where only certain parties have keys that allow them to write blocks. You end up with a message queue optimized to eliminate anything that would lead to the inconsistency nonsense that this article is talking about as soon as it is detected. It then becomes the writer's problem to retransmit in a way that doesn't cause a problem next time, rather than the reader's problem to recover.
The craziness comes in when you'll accept blocks from anybody willing to burn enough electricity to do so, or gamble enough tokens to do so, or whatever other artificial scarcity game people like to play. But if you're only planning to consume data from the eight other companies you do business with then there's no reason to bother with any of that, you can just hard code their public keys into your consensus protocol and you've sidestepped the nonsense.
What if the keys leak and need to be rotated?
If you're exchanging public keys to configure your node with in the first place, then presumably you have a trusted side channel for that kind of thing. It's equivalent to the webhook situation re: what if the domain name changes?
If you don't have such a channel then there's always social key recovery schemes, but I don't think there's a particular pattern that you can just adopt. You're now in participate-in-the-research mode.
If the hardcoded domain name changes, then the app needs to be recompiled with new domain name and that's it. The service is instantly restored and no data needs to be updated to reflect the change. In fact, the server side doesn't even need to be made aware there was any change. It just continues serving data as normal.
If you have a blockchain, and the blockchain requires a key to write to, and there's only one key, and the private key gets stolen by malicious email attachment and posted on darkweb - what now? Do you need a whole new blockchain? What happens to data from the old blockchain, how do you verify the new data is correct? Or do you keep the blockchain and use the leaked key to revoke the key and set a new one? How do we know the new key is legitimate?
I genuinely don't know how to recover in this situation and would really like for someone to walk me through the process.
That might be true for 99% of data, but I find that it's the remaining 1% of that data occupies the majority of the time. If you don't have a problem that is solved by a blockchain don't use one, but if you're saying things like this:
Then your life would probably be better if you just had to consume block-at-a-time and not the whole dataset every night. Better to be persistently five minutes behind, then to go all day not knowing whether you're wrong, with a brief moment of certainty each night.
Your nightly reconciliation job just ends up being an inside out version of the consensus protocol that you failed to enforce up front (which may be a necessary evil if you have no influence over the people who publish your data, but let's not let those people off the hook for failing to support incremental verification of sync).
Not 99% of data - 99% of applications. In 99% of applications, 100% of data is such that any piece of this 100% has only one authoritative source. Only in 1% of applications, the percent of data that doesn't have one authoritative source is less than 100%. So the absolute upper bound of when blockchain is even appropriate at all is 1% of applications.
Yes, exactly, that's the whole point of the article, the data should be an ordered stream, not asynchronous events. As long as it's an ordered stream, the author's life is peachy. Blockchain gives you an ordered stream, yes. But so does SCROLL. And if you choose SCROLL, you don't have to deal with the plethora of blockchain-specific problems.
Whenever you have a choice between SCROLL and a blockchain, you should always pick SCROLL and never a blockchain. Only if SCROLL won't work for your use case - for example, you actually need a consensus mechanism - you should consider a blockchain.
And no, blindly copying another database and overwriting every discrepancy with their version is NOT a consensus protocol! It's not meant to build a consensus! It's meant to copy data from authoritative source! There's no consensus to be had!
That is true, but I think it's a pattern we need to get away from. It's the source of everything that is wrong with the modern web.
- These single sources of truth become high value targets for corruption.
- Our apps become totally useless if they can't connect to whichever source we've hard-coded as authoritative.
- Even when connectivity is good and upstream sources are trustworthy, we end up creating unnecessary burdens on infrastructure for a connections that spans the globe from a single point to millions of them even when the data we're after is elsewhere on the LAN, ignored because it doesn't have the authoritative hostname.
It's much more fault tolerant to trust the data on the basis of its verifiable properties, not because of where you got it. That way, if the source becomes unavailable or untrustworthy, somebody more relevant can start being the leader (supposing you need a leader at all).
The SCROLL RFC is very honest about how mature it is:
People have built some truly stupid things on a blockchain, and SCROLL does seem like a good idea for some cases, but I think you're going a bit far by suggesting that a protocol which has never been implemented is always preferable to one with dosens of implementations to chose from.
The real world operates on single sources of truth. The software merely models the real world - as it should. Saying we should move away from this pattern is synonymous with saying we should fundamentally change how are society works. Hint: if your preferred software solution only makes sense if we fundamentally change how society works, maybe it's not the best solution.
If Stripe says you didn't get the payment, then you didn't get the payment. End of story. The payment isn't magically going to appear in your wallet just because you verify some properties of some data.
If your only complaint against SCROLL is that an implementation doesn't exist yet, then how about implementing SCROLL instead of creating yet another blockchain? It's not much work anyway, SCROLL is like 1000 times simpler than even the simplest of blockchains.
I don't think society worked that way until the web made it possible for society to work that way. And it's not working particularly well. It's brittle, coercive, and it asks operators to have a hand in supporting it in any way to support it in every way, raising the challenge of participating to a level where only the largest companies can.
If we think we can fundamentally change it for the better, we should.
I'm not especially enamored of blockchains. I think that partition tolerance is more important than global consistency, so mostly it's CRDT's that I'm trying to apply. I just brought up blockchains because they seemed especially applicable to the problems that SCROLL is trying to solve.
If you have a single source and a reliable connection to it, SCROLL seems great, I just think we should be preparing fallbacks for when those assumptions need to change. Because the real world doesn't operate on single sources of truth. Pick anything that people talk about, and you'll find that they're not on the same page about which sources are trustworthy for that topic. The software world operates on single sources of truth not because the real world does, but because software has so far failed to be useful outside of that domain.
I very much do think so - moreover, it's literally how it worked for decades, if not centuries. Before the internet, there was only one organization that knew whether you paid your taxes, the IRS. There was only one organization that knew your bank balance, the bank. There was only one organization that knew whether you have active water service, the water company. Everybody else - including yourself - had to either guess, or ask the single authoritative source.
Internet changed nothing about this. It only increased the number of queries and shortened response times.