It’s great to see substrait getting more seriously used!
It has been supported by engines like duckdb but the main serious use case of it I’m aware of is from Apache gluten where it is used to add Velox as the execution engine of Spark.
It’s an ambitious project and certainly has limitations but more projects like this are needed to push it forward.
At Hasura/PromptQL, we attempted to use Substrait IR through Datafusion for representing query engine plans but found that not all semantics were supported.
We ended up having to roll our own [0], but I still think Substrait is a fantastic idea (someone has to solve this problem, eventually) and it's got a good group of minds consistently working on it, so my outlook for it is bright.
Comments
It’s great to see substrait getting more seriously used!
It has been supported by engines like duckdb but the main serious use case of it I’m aware of is from Apache gluten where it is used to add Velox as the execution engine of Spark.
It’s an ambitious project and certainly has limitations but more projects like this are needed to push it forward.
At Hasura/PromptQL, we attempted to use Substrait IR through Datafusion for representing query engine plans but found that not all semantics were supported.
We ended up having to roll our own [0], but I still think Substrait is a fantastic idea (someone has to solve this problem, eventually) and it's got a good group of minds consistently working on it, so my outlook for it is bright.
[0] https://hasura.github.io/ndc-spec/reference/types.html#query...
Yeah there’s definitely a lot work left for substrait and that’s why it makes me happy to see projects like this.
Substrait is the type of project that can only be built by trying to engineer real systems, just like you tried to do.