Skip to content

Show HN: An ETL engine with hard memory guarantees

github.com/logannye
2 pointslogannyeMDdiscuss
On HN

This is an external-memory ETL engine that enforces hard memory bounds (as opposed to systems that can only try to stay within limits). It's the result of playing around with a recent idea from complexity theory, tinkering to find practical applications of where efficient time-space simulation could be useful.

It enforces a strict memory cap via RAII tracking and scheduling, so you can process arbitrarily large datasets with a fixed, small memory footprint. Operators automatically spill to disk when needed. Might be helpful for serverless and edge environments where you need predictable resource usage.

Comments

No comments yet.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.