Ask YC: Stopping bots at proxy server level
I'm trying to block bad bots/spider on my app server. I'm using nginx as a reverse proxy though and so the IP that appears to my app server is always the proxy.
How can I either capture the actual browser's IP on the app server -or- start stopping bad bots/spiders (not Google or adsense's bot) at the reverse proxy level?
Comments
Answer (not YC provided): nginx passes IPs in X-Real-IP HTTP header. If your bot trap is not customizable, you can set up mod_rpaf Apache module on app instance.