Is there a way to limit the number of concurrent connections per IP address, but have an exception list as well?
Recently, I noticed that several IPs were spamming, scraping, etc. or otherwise overloading our website server with requests. We solved this quickly by installing mod_limitipconn module for apache and set a limit on how many concurrent connections would be allowed per IP address.
However, we now realize that by doing so we may have inadvertently hindered search engine crawlers ability to crawl our site. I did some searching and found a patch for mod_limitipconn here: http://dominia.org/djao/limitipconn.html that would limit by IP, user, vhost or load average and would be perfect but it only works for Apache 1.3 and we are running 2.0.
I’ve done a lot of searching and asking around and no one seems to have a solution for this. It seems very strange to me considering that we can’t be the first website that has had this problem. Is there anyone here with experience dealing with this problem? Any suggestions on how to solve?
