I've wrangled with ledgers on and off for 10+ years now. I can some value in pushing some of the application level concerns to lower down the stack. One could call these as Domain Specific Databases. It'll be interesting to see how this evolves.
Adding a few thoughts off the top of my head.
From my experience I can say that money transfers are rarely, if ever, done atomically between two end user accounts. i.e., <Account A debit, Account B> being one operation. Instead the money first moves from Account A to a pooled account, some business checks/operations take place, and then from a pooled account (could be same or different) to account B. The reason to do so is to run a host of business checks which could take quite some time, in some cases even a day or more. Which is why the two phase account transfer semantic of "Reserve", "Settle" scales so well. One can build all kinds of higher order business flows on top of them. Subscriptions, scheduled payments, and so on.
Account transfers are subject to a bunch of velocity limits such as daily/weekly/monthly count and volume of transactions. It depends on a bunch of factors such as user's KYC levels, fraud rules, customers' age in the system etc., An account starts off at a lower limit and those limits are gradually increased as the account ages in the system. This was a big pain to pull off in scalable fashion. The running counter of limits ends up being bottle neck. For example, a popular merchant account doing a few million transactions a day. Subjecting that account to velocity limit checks million times will mean that merchant account will become a hotspot.
Maybe I should do a blog post sometime, just to give a structure to my brain dump :-P.
Tigerbeetle does support these use cases quite well. It has linked transfers that allows multiple chains of transfers to be linked and all atomically commit or not. Coupled with two-phase commits it provides for some powerful primitives.
TigerBeetle does support two phase reserve/settle semantics. I think we saw a similar problem to what you saw, and wanted to make that primitive first class.
Would love to chat sometime, joran@tigerbeetle.com
Hey Joran, thank you for responding! I now noticed that you do indeed have reserve/settle primitive. I'll surely reach out to you if I have further questions.
What I meant to say was that this is the first time I came across a ledger use case specific DB, so I just coined that phrase, Domain Specific Database (similar to Domain Specific Languages).
And yes, you could say that the industry does suffer from some version of NIH. The concept of ledger after all is thousands of years old and yet software engineers don't bother to learn end up mucking up the implementation.
I attribute it to software engineers who are not willing to step back and learn the domain of money and payments. Whenever money is involved one should only have to look how professional accountants maintain their books and handle it, learn from existing products (such as Stripe[1], Modern Treasury [2]) before running off to roll out a half baked version of double-entry book keeping. The system kinda works initially but when the requests increase the errors begin to accumulate very fast and the businesses end up unable to trace millions of $$ or worse. I've seen it happen way too many times.
Comments
I've wrangled with ledgers on and off for 10+ years now. I can some value in pushing some of the application level concerns to lower down the stack. One could call these as Domain Specific Databases. It'll be interesting to see how this evolves.
Adding a few thoughts off the top of my head.
From my experience I can say that money transfers are rarely, if ever, done atomically between two end user accounts. i.e., <Account A debit, Account B> being one operation. Instead the money first moves from Account A to a pooled account, some business checks/operations take place, and then from a pooled account (could be same or different) to account B. The reason to do so is to run a host of business checks which could take quite some time, in some cases even a day or more. Which is why the two phase account transfer semantic of "Reserve", "Settle" scales so well. One can build all kinds of higher order business flows on top of them. Subscriptions, scheduled payments, and so on.
Account transfers are subject to a bunch of velocity limits such as daily/weekly/monthly count and volume of transactions. It depends on a bunch of factors such as user's KYC levels, fraud rules, customers' age in the system etc., An account starts off at a lower limit and those limits are gradually increased as the account ages in the system. This was a big pain to pull off in scalable fashion. The running counter of limits ends up being bottle neck. For example, a popular merchant account doing a few million transactions a day. Subjecting that account to velocity limit checks million times will mean that merchant account will become a hotspot.
Maybe I should do a blog post sometime, just to give a structure to my brain dump :-P.
Tigerbeetle does support these use cases quite well. It has linked transfers that allows multiple chains of transfers to be linked and all atomically commit or not. Coupled with two-phase commits it provides for some powerful primitives.
I'm sure the team would be keen to talk to you!
Hey Vishnu, Joran from TigerBeetle.
TigerBeetle does support two phase reserve/settle semantics. I think we saw a similar problem to what you saw, and wanted to make that primitive first class.
Would love to chat sometime, joran@tigerbeetle.com
Hey Joran, thank you for responding! I now noticed that you do indeed have reserve/settle primitive. I'll surely reach out to you if I have further questions.
Interesting. Where are these domain specific databases used? And do those industries suffer from NIH (not invented here)?
What I meant to say was that this is the first time I came across a ledger use case specific DB, so I just coined that phrase, Domain Specific Database (similar to Domain Specific Languages).
And yes, you could say that the industry does suffer from some version of NIH. The concept of ledger after all is thousands of years old and yet software engineers don't bother to learn end up mucking up the implementation.
I attribute it to software engineers who are not willing to step back and learn the domain of money and payments. Whenever money is involved one should only have to look how professional accountants maintain their books and handle it, learn from existing products (such as Stripe[1], Modern Treasury [2]) before running off to roll out a half baked version of double-entry book keeping. The system kinda works initially but when the requests increase the errors begin to accumulate very fast and the businesses end up unable to trace millions of $$ or worse. I've seen it happen way too many times.
[1] https://stripe.com/in/guides/atlas/bookkeeping-and-accountin... [2] https://docs.moderntreasury.com/reference/getting-started