It’s great for handling transformations and since we want to focus on the EL part, we think there’s good synergy there.
Airbyte is already using the DBT CLI internally and as we provide more transformations of the data during syncs, we’ll make it easier to give a better integration with DBT projects downstream:
- Native Transformations as part of sync process: for example, schema migrations for source data changes, un-nesting complex objects columns (from APIs), etc
- Customizable models to override or extend further Airbyte’s proposed transformations to be executed in the same sync pipeline
- Seamless DX between custom downstream transformation and transformations made by Airbyte
- Integration with external orchestrators (Airflow, DBT Cloud jobs) with webhook triggers?
We’re happy to hear more ideas/needs to build this roadmap though!
Comments
I notice dbt integration on your roadmap. As a current Stitch and dbt Cloud customer, can you give some insight into what you're considering there?
Yes, we love DBT too!
It’s great for handling transformations and since we want to focus on the EL part, we think there’s good synergy there.
Airbyte is already using the DBT CLI internally and as we provide more transformations of the data during syncs, we’ll make it easier to give a better integration with DBT projects downstream:
- Native Transformations as part of sync process: for example, schema migrations for source data changes, un-nesting complex objects columns (from APIs), etc
- Customizable models to override or extend further Airbyte’s proposed transformations to be executed in the same sync pipeline
- Seamless DX between custom downstream transformation and transformations made by Airbyte
- Integration with external orchestrators (Airflow, DBT Cloud jobs) with webhook triggers?
We’re happy to hear more ideas/needs to build this roadmap though!
You can have a look at the current state of Airbyte with DBT here: https://docs.airbyte.io/tutorials/connecting-el-with-t-using...