"you can define assets in pure python using any framework or engine you want."
sounds flexible but what does that actually mean in practice? are there guardrails to keep things interoperable
"engine-agnostic execution"
how that holds up when switching between, say, pandas and spark. are dependencies and semantics actually preserved or is it up to us to manually patch the gaps every time the backend shifts?
Spark is technically not Python, even if we support PySpark with the relevant decorator but it's a very niche use case for us.
As for all the other Python packages, including proprietary ones, the FaaS model is such that you can declare any package you want in a function as node in the pipeline DAG, and any other in another: every function is fully isolated, and you can even selectively use pandas 1 in one, pandas 2 in another, or update the Python interpreter only in node X.
Comments
sounds flexible but what does that actually mean in practice? are there guardrails to keep things interoperable
how that holds up when switching between, say, pandas and spark. are dependencies and semantics actually preserved or is it up to us to manually patch the gaps every time the backend shifts?
Spark is technically not Python, even if we support PySpark with the relevant decorator but it's a very niche use case for us.
As for all the other Python packages, including proprietary ones, the FaaS model is such that you can declare any package you want in a function as node in the pipeline DAG, and any other in another: every function is fully isolated, and you can even selectively use pandas 1 in one, pandas 2 in another, or update the Python interpreter only in node X.
If you're interested in containerization and FaaS abstractions, this is good deep dive: https://arxiv.org/pdf/2410.17465
If you're more the practical type, just try out a few runs in the public sandbox which is free even if we are not GA.