PSA: be very careful with what you query in BigQuery on this dataset! You will be billed $5/TB scanned. I was using this dataset a couple weeks ago for fun, and ran up a $60 bill in about 10 mins. It’s important to understand that using WHERE doesn’t shrink the size of your query; instead your FROM clause should only be limited to a single date or month... not ‘*’
This is EXACTLY why we moved this data into our own database for Crux.run.
We took the JSON export of the Crux database and cleaned and loaded it into a database on a single server.
E.g each of those queries are hitting a single box in us central
Comments
PSA: be very careful with what you query in BigQuery on this dataset! You will be billed $5/TB scanned. I was using this dataset a couple weeks ago for fun, and ran up a $60 bill in about 10 mins. It’s important to understand that using WHERE doesn’t shrink the size of your query; instead your FROM clause should only be limited to a single date or month... not ‘*’
This is EXACTLY why we moved this data into our own database for Crux.run. We took the JSON export of the Crux database and cleaned and loaded it into a database on a single server. E.g each of those queries are hitting a single box in us central
There’s some of that information in the docs : https://docs.crux.run/#/introduction?id
But yeah- BQ can get very expensive very quickly if miss step
Thanks for doing so!