Use Tor to proxy. Every time you get a connection error, send a SIGHUP to the Tor daemon, which will tell the Tor daemon to establish a new connection, thereby giving you a new IP. You may not get as good results as a residential proxy, but these proxies are prohibitively expensive if you're doing this for your own personal or academic use, and the Tor method works surprisingly well.
- am not familiar with tor except for the fact that it uses something called onion with multiple hops and is very hard to trace making it privacy friendly
- are we talking here about using tor programmatically?
- what I am referring to on my end is how to go about programmatically downloading a 100 million videos one after the other on s3
- what kind of bot detection / evasion / ip address / proxy rotation mechanisms are needed
- are you familiar with libraries in the pypi ecosystem capable of tackling some of these concerns?
- is aiohttp / asyncio python enough for this paired with multiprocessing or do we have better options?
Comments
- can someone with expertise give us an overview of the architecture involved doing this
- let us say you ran yt-dlp inside python aiohttp
- surely your ll run a limit soon as your ip address will be flagged
- what solutions do we have to auto rotate proxies in python
- are there better, faster and more reliable ways to go about downloading a 100 million videos without getting your ip address blocked?
Use Tor to proxy. Every time you get a connection error, send a SIGHUP to the Tor daemon, which will tell the Tor daemon to establish a new connection, thereby giving you a new IP. You may not get as good results as a residential proxy, but these proxies are prohibitively expensive if you're doing this for your own personal or academic use, and the Tor method works surprisingly well.
- am not familiar with tor except for the fact that it uses something called onion with multiple hops and is very hard to trace making it privacy friendly
- are we talking here about using tor programmatically?
- what I am referring to on my end is how to go about programmatically downloading a 100 million videos one after the other on s3
- what kind of bot detection / evasion / ip address / proxy rotation mechanisms are needed
- are you familiar with libraries in the pypi ecosystem capable of tackling some of these concerns?
- is aiohttp / asyncio python enough for this paired with multiprocessing or do we have better options?
- Use the python Library stem to control the Tor daemon and send sighup commands when necessary
- yt dlp has builtin support for tor
- write the retry logic using your favourite coding AI
- spin up a cheap cloud instance and let it run
Dude, Tor's exit nodes are flagged everywhere.
I downloaded the Laion Disco dataset this way.
Google may supply cache boxes to ISPs which can be manipulated to let download any video you want