I've been doing ETL-heavy work as (async) Python microservices.
The most important thing is to separate your infrastructure layer from your business logic, and have the infrastructure layer satisfy interfaces that are defines in business logic layer.
That way, when my datastore switches from FTP to S3 or MySQL to Postgres, I can swap out the datastore client, make sure the new one satisfies the interface, and the rest of the application works just the same.
The microservice's database to track and schedule runs is in Postgres, with SELECT FOR UPDATE providing an easy way to treat a Postgres table as a work queue.
You words reminded me that 10-15 years ago ORMs were all the hype, and one of the advertised benefits to use them was that you are abstracted from an underlying database engine, and can flawlessly switch between them.
A bit later, as usually the hype went down and there were a lot of critical articles about ORMs, where they said that well, it is not actually that easy to switch database engines in a real project even with ORMs, and more - ORMs hide great specific powerful features of databases. And finally - how often do you actually jump between database engines, what a strange benefit after all?
I am in no way criticizing your approach, and I clearly understand the rationale and benefits. It was funny, your words reminded me that our industry progresses in an infinite spiral trajectory, and repeats itself on a next level. :)
I know what you mean! In my approach, the ORM would be in the infrastructure layer - I'm assuming I will have to switch it out, if I'm using one at all.
It's the times where I have to migrate from one implementation of an interface to another where I have learned the most, usually meaning "I let the structure of this database get reflected too much in the business logic"/
Comments
I've been doing ETL-heavy work as (async) Python microservices.
The most important thing is to separate your infrastructure layer from your business logic, and have the infrastructure layer satisfy interfaces that are defines in business logic layer.
That way, when my datastore switches from FTP to S3 or MySQL to Postgres, I can swap out the datastore client, make sure the new one satisfies the interface, and the rest of the application works just the same.
The microservice's database to track and schedule runs is in Postgres, with SELECT FOR UPDATE providing an easy way to treat a Postgres table as a work queue.
You words reminded me that 10-15 years ago ORMs were all the hype, and one of the advertised benefits to use them was that you are abstracted from an underlying database engine, and can flawlessly switch between them.
A bit later, as usually the hype went down and there were a lot of critical articles about ORMs, where they said that well, it is not actually that easy to switch database engines in a real project even with ORMs, and more - ORMs hide great specific powerful features of databases. And finally - how often do you actually jump between database engines, what a strange benefit after all?
I am in no way criticizing your approach, and I clearly understand the rationale and benefits. It was funny, your words reminded me that our industry progresses in an infinite spiral trajectory, and repeats itself on a next level. :)
I know what you mean! In my approach, the ORM would be in the infrastructure layer - I'm assuming I will have to switch it out, if I'm using one at all.
It's the times where I have to migrate from one implementation of an interface to another where I have learned the most, usually meaning "I let the structure of this database get reflected too much in the business logic"/