This is a great article and exactly mirrors my experience consuming webhooks. My team built almost the same workarounds as the author.

The solution my team eventually settled on to the dedup, buffering, and race problems was:

  * When a webhook came in, store a single copy of the payload (usually json) in some temporary storage and enqueue the id.
  * Any subsequent updates (while the id was enqueued) overwrote that payload and skipped the queue.

So, we were able to ensure we only updated the object once and with the latest state (since the webhook payload always held the entire object state).

We didn't have a great solution for the bootstrap problem though.

I do like the SCROLL proposal, but I wonder about the cost of keeping around log-structured data forever - every log structured DB I know about does compactions for this reason.