Skip to content

Comment on Ask HN: Is SQL a primary tool of choice for doing ETL pipelines in 2019?parent

Comments

Yes.

Even better, for 2.x I had an "HL7 interface" compiler that would transform the client (partner's) specification into verbose wrappers. The resulting "fluent" code read just like the spec.

Alas, data quality, consistency was always a challenge. Data from multiple sources would come thru the same feed. So often our scrapping (data extraction) code would extract PID (for example) from 3 common places, from most specific to more general, then use null coalescing to choose the best match.

As hinted before, 3.x & XML was handled the same in spirit, but very different code, using query expressions instead of method chaining.

I'd be happy to share whatever I have.

Here's a temp (24 hours) email address if you wish to PM me: vwebaemg@sharklasers.com

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.