Attack of the Amazon spiders
While investigating a traffic surge at The Quotations Page, I found something interesting: between 4:00 AM and 10:00 AM I received almost 10,000 hits from two particular IP addresses. Things like this happen every day, and it’s usually just a referral spammer or email address harvester. Today’s attack had two interesting qualities, though:
- Every single request was for the robots.txt file. (9875 requests, about one every 5 seconds)
- The two IP addresses in question are 207.171.180.101 and 207.171.167.25, both of which belong to Amazon.com.
The user agent on the requests is libwww-perl/5.35, so apparently somebody at Amazon is having lots of fun with Perl at my expense.
This leads me to wonder if Amazon is experimenting with a new search spider for A9.com. But if they are, it’s gone horribly wrong…
Did you ever get any more information about this?