Earth observation satellites produce an even larger amount of data. It is stored forever in publicly accessible places (a large amount of it, for free).
Earth observation satellites produce an even larger amount of data.
Do they? I don't know and I am really curious: Do you have any source for this?
It is stored forever in publicly accessible places (a large amount of it, for free).
Same for telescope arrays' data, but the same problem: How to get the data from its source into your number crunchers? When I know the pipeline, I can simply calculate the answer, but when I need to explore/play with data, I will inevitably run into IO roadblocks.
Sure they do. Even if you only count optical satellites, each sensor produces an image of about 40.000 x 40.000 pixels every few seconds. There are hundreds of satellites doing that and sending the images to earth. All of these data is archived indefinitely and widely accessible.
How to get the data from its source into your number crunchers?
You do not simply "get" this data. It's simply too large, conceptually infinite. You query it for the parts that you are interested in, and then you extract only those parts.
As an example of free access, see the access hub for the Copernicus program: https://scihub.copernicus.eu/ There is an interface for downloading particular images, and an api to query the archive of images with space/time constraints to obtain a manageable list of URLs to download. You have sentinel-2 for optical images, sentinel-1 for radar images, and sentinel-5p for hyperspectral.
If you want higher resolution images, they are also publicly accessible, but typically not free. There's plenty of commercial satellite imaging providers nowadays, and lots of companies work by extracting information from this huge corpus of data.
Comments
Earth observation satellites produce an even larger amount of data. It is stored forever in publicly accessible places (a large amount of it, for free).
Do they? I don't know and I am really curious: Do you have any source for this?
Same for telescope arrays' data, but the same problem: How to get the data from its source into your number crunchers? When I know the pipeline, I can simply calculate the answer, but when I need to explore/play with data, I will inevitably run into IO roadblocks.
Sure they do. Even if you only count optical satellites, each sensor produces an image of about 40.000 x 40.000 pixels every few seconds. There are hundreds of satellites doing that and sending the images to earth. All of these data is archived indefinitely and widely accessible.
You do not simply "get" this data. It's simply too large, conceptually infinite. You query it for the parts that you are interested in, and then you extract only those parts.
As an example of free access, see the access hub for the Copernicus program: https://scihub.copernicus.eu/ There is an interface for downloading particular images, and an api to query the archive of images with space/time constraints to obtain a manageable list of URLs to download. You have sentinel-2 for optical images, sentinel-1 for radar images, and sentinel-5p for hyperspectral.
If you want higher resolution images, they are also publicly accessible, but typically not free. There's plenty of commercial satellite imaging providers nowadays, and lots of companies work by extracting information from this huge corpus of data.