It's nice to see some competition in this space, especially open source. I'm a little confused though I thought Singer was the open source version of Stitch (which you mention briefly): https://www.singer.io. But maybe Singer doesn't have all the UI features that Stitch does and that's where Airbyte is different? I would love to know more about the differences
At GitLab, we're not ready to give up on the Singer spec, community, and ecosystem yet, which is why I've been working on Meltano for the past year: https://meltano.com/
We think that the biggest things holding back Singer are the lack of documentation and tooling around taking existing taps and targets to production, and around building, debugging, maintaining, and testing new or existing high-quality taps and targets.
Meltano itself addresses the first problem, and provides a robust and reliable platform for building, running & orchestrating Singer- and dbt-based ELT pipelines. It's built for developers who are comfortable with CLIs and YML files, and want their pipelines to be defined in a Git repository so that they get the benefits of DevOps best practices like code review and CI/CD.
At the same time, we have been working with some members of the community on a new framework for building taps and targets: https://gitlab.com/meltano/meltano/-/issues/2401, which we have decided to call the Singer SDK: https://gitlab.com/meltano/singer-sdk. We are moving as many Singer specification-specific details around things like incremental state replication and stream/field selection into the framework, so that individual taps only need to worry about getting the data from the source and can be expected to behave more consistently and correctly across the board.
I am one of Airbyte's founder. The initial version of was actually fully based on Singer and this is when we realize it wouldn't be possible for us to depend on it.
Amongst the main reasons: Singer seems to have been abandoned by StitchData (after they got acquired by Talend), the quality of the connectors is too unpredictable, Singer connectors are not usable outside the box.
We would have preferred to use an existing standard if one already existed. It was a tough decision for us to create something from scratch but now we are very satisfied with the decision. It is way easier for the community and for us to build connectors that meet quality standards and we can make it MIT so the community can have control on the evolution of the protocol.
Forgot to mention. We have a compatibility layer with Singer so it possible to run Singer Taps in Airbyte. We have a few sources that are actually some of the high quality Singer's.
Comments
It's nice to see some competition in this space, especially open source. I'm a little confused though I thought Singer was the open source version of Stitch (which you mention briefly): https://www.singer.io. But maybe Singer doesn't have all the UI features that Stitch does and that's where Airbyte is different? I would love to know more about the differences
At GitLab, we're not ready to give up on the Singer spec, community, and ecosystem yet, which is why I've been working on Meltano for the past year: https://meltano.com/
We think that the biggest things holding back Singer are the lack of documentation and tooling around taking existing taps and targets to production, and around building, debugging, maintaining, and testing new or existing high-quality taps and targets.
Meltano itself addresses the first problem, and provides a robust and reliable platform for building, running & orchestrating Singer- and dbt-based ELT pipelines. It's built for developers who are comfortable with CLIs and YML files, and want their pipelines to be defined in a Git repository so that they get the benefits of DevOps best practices like code review and CI/CD.
At the same time, we have been working with some members of the community on a new framework for building taps and targets: https://gitlab.com/meltano/meltano/-/issues/2401, which we have decided to call the Singer SDK: https://gitlab.com/meltano/singer-sdk. We are moving as many Singer specification-specific details around things like incremental state replication and stream/field selection into the framework, so that individual taps only need to worry about getting the data from the source and can be expected to behave more consistently and correctly across the board.
I am one of Airbyte's founder. The initial version of was actually fully based on Singer and this is when we realize it wouldn't be possible for us to depend on it.
Amongst the main reasons: Singer seems to have been abandoned by StitchData (after they got acquired by Talend), the quality of the connectors is too unpredictable, Singer connectors are not usable outside the box.
We would have preferred to use an existing standard if one already existed. It was a tough decision for us to create something from scratch but now we are very satisfied with the decision. It is way easier for the community and for us to build connectors that meet quality standards and we can make it MIT so the community can have control on the evolution of the protocol.
We actually wrote a few articles about it:
https://docs.airbyte.io/faq/differences-with.../singer-vs-ai...
https://airbyte.io/articles/data-engineering-thoughts/airbyt...
Forgot to mention. We have a compatibility layer with Singer so it possible to run Singer Taps in Airbyte. We have a few sources that are actually some of the high quality Singer's.
Stitch is partially open as stitch - many of the integrations they list on their website are hosted versions of OSS singer, but a couple are not OSS.
I'm not sure how Stitch's acquisition will affect Singer contributions and such going forward.
Also, Singer has no UI, it's all CLI.