> It is increasingly common to use this kind of “pipeline-as-YAML”
> configuration to piece together a workflow of pre-built
> components 1. Some real examples of this are TFX components,
> scikit-learn Pipelines, or Airflow DAGs.
Airflow DAGs are Python objects, defined in Python code.
There's add-ons that let you define them in yaml. Which I find hilarious since one of the original selling points of Airflow was that the Dags aren't in XML.
I took Adam’s point here to be that an Airflow DAG author primarily concerns themselves with the configuration of those objects, since the underlying components (Celery worker, Python execution process or K8s pod; data warehouse; RPC) have been abstracted in the form of Operators.
Comments
There's add-ons that let you define them in yaml. Which I find hilarious since one of the original selling points of Airflow was that the Dags aren't in XML.
I took Adam’s point here to be that an Airflow DAG author primarily concerns themselves with the configuration of those objects, since the underlying components (Celery worker, Python execution process or K8s pod; data warehouse; RPC) have been abstracted in the form of Operators.