Dear Santa, thank you for the dolls and pencils and the fish. It's Easter now, so I hope I didn't wake you but... honest, it is an emergency. There's a crack in my wall. Aunt Sharon says it's just an ordinary crack, but I know its not cause at night there's voices so... please please can you send someone to fix it? Or a policeman, or...
Back in a moment.
Thank you Santa.
Back in a moment.
Thank you Santa.
Saturday, December 07
Dealing With Assholes Running Web Spiders On The Internet
Your site can deal with 20 simultaneous threads crawling the entire history of all your content right? Right. I mean, if your server crashes it's not our problem. Also, fuck sending a clear identifier, we're just a Mac running Safari that just happens to be downloading the contents of 70,000 blogs.
Your site can deal with 20 simultaneous threads crawling the entire history of all your content right? Right. I mean, if your server crashes it's not our problem. Also, fuck sending a clear identifier, we're just a Mac running Safari that just happens to be downloading the contents of 70,000 blogs.
Update: We were - and indeed, still are - getting hit by a torrent of garbage requests coming through a group of proxy servers run by an SEO company. I don't know why, but they are now all getting 404.
Update 2: I killed that, and then I discovered we were also getting scanned by a total of 175 different servers at various hosting companies. Those are all blocked at the firewall now, and don't even get the courtesy of a 404.
If you have a problem reaching the site - say it works on mobile but not at home, or vice versa - please let me know. I will double-check the list of blocked IPs tomorrow and convert it to a smaller number of IP ranges.
Posted by: Pixy Misa at
08:38 AM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 187 words, total size 1 kb.
<< Page 1 of 1 >>
37kb generated in CPU 0.0166, elapsed 0.2634 seconds.
49 queries taking 0.2566 seconds, 204 records returned.
Powered by Minx 1.1.6c-pink.
49 queries taking 0.2566 seconds, 204 records returned.
Powered by Minx 1.1.6c-pink.
