Baidu

[Gantry]

Then why do I always fuck it first?

[Reds]

What kind of question is that?

[Shooty]

Berating usually does work.

So does bereting:

[fknmclane]

Forums were indeed slow as shit this morning.

[Gantry]

Yeah the reboot did wonders.  don't know what the issue was, but I falsely accused Baidu.  I apologize to all of China...

[GDavis]

Have you checked the kernel?

[Gantry]

Site's been up 2 minutes and 25 separate connections from Baidu - my uninformed decision is that along with an apache memory leak are causing the issues.

[TβG]

so can you ban whoever that is?

[Gantry]

Not easily, but I'm sure there's a way to tweak the firewall to not allow any access from China. 

[nightwulf]

Code Select Expand


[nightwulf@deenee httpd]# grep "180\.76\.5" access_log | grep "robots"
180.76.5.88 - - [25/Dec/2011:14:54:36 -0600] "GET /robots.txt HTTP/1.1" 404 291
180.76.5.50 - - [26/Dec/2011:21:24:28 -0600] "GET /robots.txt HTTP/1.1" 404 291
180.76.5.101 - - [27/Dec/2011:11:41:30 -0600] "GET /robots.txt HTTP/1.1" 404 291
180.76.5.169 - - [29/Dec/2011:17:30:01 -0600] "GET /robots.txt HTTP/1.1" 404 291


Names changed to protect the innocent. It appears Baidu is at least reading robots.txt. I'll try using that to get them off our backs, though it may take a day or two to see any change. If that doesn't work, I guess we're left with IP banning. Could maybe come up with something based on user-agent, but I don't like the overhead that would create.

[nightwulf]

Robots.txt now politely asking all Baidu spider user-agents (as found here) to fuck off. We'll see if they actually follow it.

[nightwulf]

So far Google and MSN have parsed robots.txt; haven't seen Baidu read it (though there are still 19+ of their spiders spidering). Found a couple old httpd processes going absolutely batshit on CPU usage; restarted httpd and they all came back. 95+% usage at times. Just saw one at 99.5%. Total CPU usage is constantly high, all in httpd processes. Maybe we need to look at the web server ...

[Reds]

Nightwulf, have you tried fucking it?

[Felonious Gunk]

Nightwulf, have you tried fucking it?

Also, is it possible you acted in error by berating it before fucking it?

[Gantry]

So far Google and MSN have parsed robots.txt; haven't seen Baidu read it (though there are still 19+ of their spiders spidering). Found a couple old httpd processes going absolutely batshit on CPU usage; restarted httpd and they all came back. 95+% usage at times. Just saw one at 99.5%. Total CPU usage is constantly high, all in httpd processes. Maybe we need to look at the web server ...

Something is definitely up with apache, last night I noticed total memory usage per top with only 75MB of RAM free.  Restarted httpd and it went to 2.7GB free.  Restarting mysql only freed up 50MB or so.  I think there's a memory leak or something of the sort with apache. 

[TβG]

does the servers have alzheimer's?   

[nightwulf]

Nightwulf, have you tried fucking it?

I wouldn't fuck it with Mike D's dick.

[nightwulf]

Something is definitely up with apache, last night I noticed total memory usage per top with only 75MB of RAM free.  Restarted httpd and it went to 2.7GB free.  Restarting mysql only freed up 50MB or so.  I think there's a memory leak or something of the sort with apache.

Well, I'll look around the internets but I'm more of a "./configure; make; make install" guy, so CentOS is really your ball. In the meantime I'll just restart httpd a couple times a day or so.

[nightwulf]

Baidu spiders appear to be laying off. httpd CPU usage is still high, but significantly lower than it was yesterday. Damn Chinese ...

[Flood]

bump...any news on CPU usage?