The end here reminds me of "The Log: Real-time data's unifying abstraction" [0], which has unfortunately had a bit of link-rot since 2013.
One complication in this approach involves access-windows: What if my system is only supposed to be seeing stuff that happened during two separate weeks in the year, because those are the spans when it was subscribed or authorized?
So the data-host would need to maintain a concept of "connection history" for other services, and also use that to filter/modify its real event stream, inserting artificial "initial state" roll-ups of events that happened in dark periods.
That requirement seems ... uncommon. How many systems/industries have a frequent notion of transient/sliced views of history for their customers?
If that is a real requirement, it seems like it'd be easier to meet by giving customers a realtime-stream/log API whose history starts when they were most recently granted access, and providing them older historical events via a separate API of the classic "ask for a report and we'll get back to you within a day or two with an S3 presigned URL" variety, then synthesizing that huge historical report in batch code that's aware of the subtleties of the customer's visibility windows.
Perhaps less a requirement and more a kind of consistency with prior limitations? For example:
1. I have events in a Calendar service.
2. I want to authorize Reminder service to see upcoming events, so that it can send reminders to attendees in a way the Calendar service does not directly support, e.g. SMS/WhatsApp.
3. With the necessary credentials/SSO, the Reminder service subscribes to Calendar and Calendar periodically POSTSs webhook updates. Reminder needs to recognize when an event is cancelled or rescheduled, so that can alter its reminders.
Do I want to give Reminder potential access to all events ever, or just ones active across the usage period? Meanwhile, the Reminder guys probably don't want to step through the whole Calendar-wide event stream to reconstruct which events haven't finally happened yet.
Fair, and thanks for explaining. Other than my original proposal of a simple webhook stream and a complicated offline backfill, all the good ways to do that with current tools probably involve building support for fine-grained read permissions (Reminder service's windowing) onto the producer (Calendar) side and emitting a tailored reminder stream.
If that challenge is indeed common, then there might be a market/demand/opportunity to implement ... I don't know what to call it, "Postgres RLS for Kafka" or something--a record-level security model for event streams, basically. Event compaction/deduplication would make this hard, though.
Comments
The end here reminds me of "The Log: Real-time data's unifying abstraction" [0], which has unfortunately had a bit of link-rot since 2013.
One complication in this approach involves access-windows: What if my system is only supposed to be seeing stuff that happened during two separate weeks in the year, because those are the spans when it was subscribed or authorized?
So the data-host would need to maintain a concept of "connection history" for other services, and also use that to filter/modify its real event stream, inserting artificial "initial state" roll-ups of events that happened in dark periods.
[0] https://news.ycombinator.com/item?id=6916557
That requirement seems ... uncommon. How many systems/industries have a frequent notion of transient/sliced views of history for their customers?
If that is a real requirement, it seems like it'd be easier to meet by giving customers a realtime-stream/log API whose history starts when they were most recently granted access, and providing them older historical events via a separate API of the classic "ask for a report and we'll get back to you within a day or two with an S3 presigned URL" variety, then synthesizing that huge historical report in batch code that's aware of the subtleties of the customer's visibility windows.
Perhaps less a requirement and more a kind of consistency with prior limitations? For example:
1. I have events in a Calendar service.
2. I want to authorize Reminder service to see upcoming events, so that it can send reminders to attendees in a way the Calendar service does not directly support, e.g. SMS/WhatsApp.
3. With the necessary credentials/SSO, the Reminder service subscribes to Calendar and Calendar periodically POSTSs webhook updates. Reminder needs to recognize when an event is cancelled or rescheduled, so that can alter its reminders.
Do I want to give Reminder potential access to all events ever, or just ones active across the usage period? Meanwhile, the Reminder guys probably don't want to step through the whole Calendar-wide event stream to reconstruct which events haven't finally happened yet.
Fair, and thanks for explaining. Other than my original proposal of a simple webhook stream and a complicated offline backfill, all the good ways to do that with current tools probably involve building support for fine-grained read permissions (Reminder service's windowing) onto the producer (Calendar) side and emitting a tailored reminder stream.
If that challenge is indeed common, then there might be a market/demand/opportunity to implement ... I don't know what to call it, "Postgres RLS for Kafka" or something--a record-level security model for event streams, basically. Event compaction/deduplication would make this hard, though.
https://archive.ph/oIfml swerves the bitrot