Is there any other field that relies on multiple huge arrays of sensors, distributed around the world, that generate huge amounts of data non-stop? Data that is not only fed into processing pipelines but needs to be explored? Meteorology maybe. Something else?
She actually explains that in the article:
"one object might be anywhere between roughly 50 gigabytes to maybe a terabyte. By the time I'm done reducing that data and imaging it, it'll probably have roughly tripled to quadrupled in size. I tend to have larger surveys, so by the time I'm done I might have several tens of terabytes of data."
their math/programming/modeling skills are not better than the average compsci,physics phd.
No, but the data they handle regularly is just much, much larger than in many other fields.
Earth observation satellites produce an even larger amount of data. It is stored forever in publicly accessible places (a large amount of it, for free).
Earth observation satellites produce an even larger amount of data.
Do they? I don't know and I am really curious: Do you have any source for this?
It is stored forever in publicly accessible places (a large amount of it, for free).
Same for telescope arrays' data, but the same problem: How to get the data from its source into your number crunchers? When I know the pipeline, I can simply calculate the answer, but when I need to explore/play with data, I will inevitably run into IO roadblocks.
Sure they do. Even if you only count optical satellites, each sensor produces an image of about 40.000 x 40.000 pixels every few seconds. There are hundreds of satellites doing that and sending the images to earth. All of these data is archived indefinitely and widely accessible.
How to get the data from its source into your number crunchers?
You do not simply "get" this data. It's simply too large, conceptually infinite. You query it for the parts that you are interested in, and then you extract only those parts.
As an example of free access, see the access hub for the Copernicus program: https://scihub.copernicus.eu/ There is an interface for downloading particular images, and an api to query the archive of images with space/time constraints to obtain a manageable list of URLs to download. You have sentinel-2 for optical images, sentinel-1 for radar images, and sentinel-5p for hyperspectral.
If you want higher resolution images, they are also publicly accessible, but typically not free. There's plenty of commercial satellite imaging providers nowadays, and lots of companies work by extracting information from this huge corpus of data.
genome sequencers are sensors distributed around the work producing 15PB of intermediate data a day. If desired (IE, smebody had the capital and operating) it could be many multiples of that.
Comments
Is there any other field that relies on multiple huge arrays of sensors, distributed around the world, that generate huge amounts of data non-stop? Data that is not only fed into processing pipelines but needs to be explored? Meteorology maybe. Something else?
She actually explains that in the article:
"one object might be anywhere between roughly 50 gigabytes to maybe a terabyte. By the time I'm done reducing that data and imaging it, it'll probably have roughly tripled to quadrupled in size. I tend to have larger surveys, so by the time I'm done I might have several tens of terabytes of data."
No, but the data they handle regularly is just much, much larger than in many other fields.
Earth observation satellites produce an even larger amount of data. It is stored forever in publicly accessible places (a large amount of it, for free).
Do they? I don't know and I am really curious: Do you have any source for this?
Same for telescope arrays' data, but the same problem: How to get the data from its source into your number crunchers? When I know the pipeline, I can simply calculate the answer, but when I need to explore/play with data, I will inevitably run into IO roadblocks.
Sure they do. Even if you only count optical satellites, each sensor produces an image of about 40.000 x 40.000 pixels every few seconds. There are hundreds of satellites doing that and sending the images to earth. All of these data is archived indefinitely and widely accessible.
You do not simply "get" this data. It's simply too large, conceptually infinite. You query it for the parts that you are interested in, and then you extract only those parts.
As an example of free access, see the access hub for the Copernicus program: https://scihub.copernicus.eu/ There is an interface for downloading particular images, and an api to query the archive of images with space/time constraints to obtain a manageable list of URLs to download. You have sentinel-2 for optical images, sentinel-1 for radar images, and sentinel-5p for hyperspectral.
If you want higher resolution images, they are also publicly accessible, but typically not free. There's plenty of commercial satellite imaging providers nowadays, and lots of companies work by extracting information from this huge corpus of data.
genome sequencers are sensors distributed around the work producing 15PB of intermediate data a day. If desired (IE, smebody had the capital and operating) it could be many multiples of that.