At Hasura/PromptQL, we attempted to use Substrait IR through Datafusion for representing query engine plans but found that not all semantics were supported.
We ended up having to roll our own [0], but I still think Substrait is a fantastic idea (someone has to solve this problem, eventually) and it's got a good group of minds consistently working on it, so my outlook for it is bright.
Comments
At Hasura/PromptQL, we attempted to use Substrait IR through Datafusion for representing query engine plans but found that not all semantics were supported.
We ended up having to roll our own [0], but I still think Substrait is a fantastic idea (someone has to solve this problem, eventually) and it's got a good group of minds consistently working on it, so my outlook for it is bright.
[0] https://hasura.github.io/ndc-spec/reference/types.html#query...
Yeah there’s definitely a lot work left for substrait and that’s why it makes me happy to see projects like this.
Substrait is the type of project that can only be built by trying to engineer real systems, just like you tried to do.