Skip to content

Comment on Polars Cloud: The Distributed Cloud Architecture to Run Polars Anywhereparent

Comments

If you use a feature store to store your DataFrames (most provide APIs for storing Polars, Pandas, PySpark DataFrames in backing Lakehouse/real-time DBs), then you get type checks when writing data to the DataFrame's backing Feature Group (Lakehouse + real-time tables).

Many also add an additional layer of data validation on top of schema validation, using frameworks like Great Expectations. For example, it's not enough to know 'age' is an Integer, it should be an integer in the range 0..150.

Disclaimer: i work for Hopsworks.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.