Another silly (compression-based) idea I had was to:
- Index say google images, or something else with a large amount of URL -> data
- Find patterns in the indexed data that match patterns in your data, such that storing the URL and an offset into the data (or something more complex) would be smaller than the data chunk you are trying to compress
- Repeat for all chunks
- After you're done you can run it again and again. Infinite compression!
Yes the user has to download WAY more data that what they are trying to extract, and you'd need an insanely large index to be able to compress, but hey it was an idea.
There is a range header in the http specification for resuming downloads at a certain part of the file. Since http is stateless, you can download precisely what you need right away.
Comments
I had the same idea!
Another silly (compression-based) idea I had was to:
- Index say google images, or something else with a large amount of URL -> data
- Find patterns in the indexed data that match patterns in your data, such that storing the URL and an offset into the data (or something more complex) would be smaller than the data chunk you are trying to compress
- Repeat for all chunks
- After you're done you can run it again and again. Infinite compression!
Yes the user has to download WAY more data that what they are trying to extract, and you'd need an insanely large index to be able to compress, but hey it was an idea.
There is a range header in the http specification for resuming downloads at a certain part of the file. Since http is stateless, you can download precisely what you need right away.