Skip to content

Show HN: Blobsearch – Object storage and DuckDB based Elasticsearch alternative

github.com/amr8t
2 pointsconqrrdiscuss
On HN

Hey all, sharing my latest alternative to elasticsearch, cloudwatch etc that either require heavy cloud tax or cost more for a managed solution. This is a known pattern but maybe not well known enough to write logs to s3 as durable storage with Parquet and query rapidly with DuckDB. This has become the primary way I ingest logs for all my side projects and don't ever have to worry about losing logs and possibly free storage.

Features

    Format Agnostic - Works with any JSON log format via configurable field extraction
    Fast - 28K+ entries/sec ingestion
    Efficient - Parquet + Snappy (3.7x compression)
    Quick Queries - DuckDB queries in <50ms on 56K logs
    S3-Compatible - AWS S3, MinIO, DigitalOcean Spaces, R2, etc.
    Partitioned - Hive-style partitioning by date/level (no redundant part suffixes)
    Auto-Flush - Configurable automatic flushing (default: 90s)
    Dedupe - Optional deduplication
Happy to answer any questions about internals

Comments

No comments yet.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.