I've interacted with Enigma folks throughout the past few years, have always been impressed with their work and methodology. I've had friends who've worked at other massive-public-data-gathering startups, it sounds like a tough business, since collecting/cleaning data is hard, but having data isn't alone a competitive edge. Don't know if Enigma will find success with public data (though their offerings go beyond data, but enterprise platforms apparently), but I've been impressed at the scope of their collection and ability to wrangle data into a standardized structure.
Here's one example: Senate lobbying disclosures. Enigma has taken the original XML data sources and created several flat tables (lobbyists, issues, reports) that can be linked through foreign/primary keys: https://public.enigma.com/browse/lobbyists/09264ee1-792f-445...
I've written my own scripts to clean up and organize this shitshow but it's nice to have Enigma to double-check against, or even get ideas on how to structure things. What's just as impressive to me is the work put in the taxonomy of datasets, e.g. United States > U.S. Senate > Lobbying Reports.
For less data-savvy users, just having a Google-like simple search bar is great for discovery of datasets that contain a term of interest: https://public.enigma.com/search/google
Note: Enigma has had offered this public data for free before, you just had to sign up for an account to even browse the data. This public interface is much nicer, especially for sending people links. Haven't tested out the export functions or the quotas, but in the previous incarnation, free accounts got a huge number of downloads a month.
Comments
I've interacted with Enigma folks throughout the past few years, have always been impressed with their work and methodology. I've had friends who've worked at other massive-public-data-gathering startups, it sounds like a tough business, since collecting/cleaning data is hard, but having data isn't alone a competitive edge. Don't know if Enigma will find success with public data (though their offerings go beyond data, but enterprise platforms apparently), but I've been impressed at the scope of their collection and ability to wrangle data into a standardized structure.
Here's one example: Senate lobbying disclosures. Enigma has taken the original XML data sources and created several flat tables (lobbyists, issues, reports) that can be linked through foreign/primary keys: https://public.enigma.com/browse/lobbyists/09264ee1-792f-445...
Here's what the raw material looks like:
https://www.senate.gov/legislative/Public_Disclosure/LDA_rep...
Excerpt: https://gist.github.com/dannguyen/7588b8334f5c8954d2c2b13bc4...
I've written my own scripts to clean up and organize this shitshow but it's nice to have Enigma to double-check against, or even get ideas on how to structure things. What's just as impressive to me is the work put in the taxonomy of datasets, e.g. United States > U.S. Senate > Lobbying Reports.
For less data-savvy users, just having a Google-like simple search bar is great for discovery of datasets that contain a term of interest: https://public.enigma.com/search/google
Note: Enigma has had offered this public data for free before, you just had to sign up for an account to even browse the data. This public interface is much nicer, especially for sending people links. Haven't tested out the export functions or the quotas, but in the previous incarnation, free accounts got a huge number of downloads a month.