All of these visual ETL tools are nothing more yhan a patch to the new breed of analytical databases because their support for loading files sucks so badly.
Even the praised dbt is just a hack to make declarative incremental because the SQL capabilities of these databases is also bad and you need to template out your way in.
There will once be a SQL database that will make these things not suck and collapse this entire hot space of startups for data loading and missing features in Snowflake.
Snowflake will go away too if they keep ignoring so many "missing features" startups and not improving their loading or language capabilities. Unless they intend to spin up an ecosystem of "hacks" to complement Snowflake to make it usable, which is another reason for some saner system to displace them.
Disclaimer up front: I work for Snowflake but do not represent them here - I'm not in Product and have very little influence over direction.
What language and loading capabilities would you like to see us add?
I personally doubt we'll ever aim for postgres levels of extensibility, but Java has been in for a while and Python just went public preview - with great reviews from everyone I've spoken to.
Plenty of activity on the loading side too, although we don't really aim for visual ETL - you're pretty much correct that we're happy to let 1000 flowers bloom in that space.
Comments
All of these visual ETL tools are nothing more yhan a patch to the new breed of analytical databases because their support for loading files sucks so badly.
Even the praised dbt is just a hack to make declarative incremental because the SQL capabilities of these databases is also bad and you need to template out your way in.
There will once be a SQL database that will make these things not suck and collapse this entire hot space of startups for data loading and missing features in Snowflake.
Snowflake will go away too if they keep ignoring so many "missing features" startups and not improving their loading or language capabilities. Unless they intend to spin up an ecosystem of "hacks" to complement Snowflake to make it usable, which is another reason for some saner system to displace them.
Disclaimer up front: I work for Snowflake but do not represent them here - I'm not in Product and have very little influence over direction.
What language and loading capabilities would you like to see us add?
I personally doubt we'll ever aim for postgres levels of extensibility, but Java has been in for a while and Python just went public preview - with great reviews from everyone I've spoken to.
Plenty of activity on the loading side too, although we don't really aim for visual ETL - you're pretty much correct that we're happy to let 1000 flowers bloom in that space.
Simple way to consume Oauth2 REST APIs on a schedule.
You mean like https://www.fivetran.com/files and https://docs.airbyte.com/integrations/sources/file/ ? what do you mean by "support for loading files sucks so badly"?