I've mentioend this whenever iceberg comes up. It's wild how immature the ecosystem is still. Duckdb itself lacks the ability to write iceberg....

https://duckdb.org/docs/stable/extensions/iceberg/overview.h...

Apache iceberg go ? Nope

https://github.com/apache/iceberg-go?tab=readme-ov-file#read...

Basically java iceberg is the only mature way to do this, it's not a very accessible ecosystem.

For a side project I'm using pyiceberg to sink streaming data to iceberg (using DuckDB as the stream processor):

https://sql-flow.com/docs/tutorials/iceberg-sink

It's basically a workaround for DuckDB's lack of native support. I am very happy with the Pyicerbg library as a user, It was very easy and the native Arrow support is a glimpse into the future. Arrow as an interchange format is quite amazing. Just open up the iceberg table and append Arrow dataframes to it!

https://github.com/turbolytics/sql-flow

Arrow is quite spectacular and it's cool to see the industry moving to standardize on it as a dataframe. For example, Clickhouse python also support arrow-based insertion:

https://sql-flow.com/docs/tutorials/clickhouse-sink

This makes the glue code trivial to sink into these different systems as long as arrow is used.