14526
Official Announcements / Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Last post by mouser on January 23, 2011, 04:49 PM »So far it's looking like Gothi[c] and I may have identified and remedied the problem:
Looking through that access logs (shout out to my Web Log Filter utility for making it a little easier), it seems like the server slowdown may have been caused by a web spider from russia making huge numbers of calls to the search function, which puts pretty heavy demands on the cpu and memory.
In fairness, the fault may have been mostly ours -- when we moved servers, we changed the forum urls from /Forum/bb to the simpler /forum but we didn't update the robots.txt file which tells robot spiders not to try to spider things like the search page. It's *possible* that this is why the spider was trying to do such massive spidering of search result pages.
This is our working hypothesis -- and we will reopen the search functionality to the public on monday.
Looking through that access logs (shout out to my Web Log Filter utility for making it a little easier), it seems like the server slowdown may have been caused by a web spider from russia making huge numbers of calls to the search function, which puts pretty heavy demands on the cpu and memory.
In fairness, the fault may have been mostly ours -- when we moved servers, we changed the forum urls from /Forum/bb to the simpler /forum but we didn't update the robots.txt file which tells robot spiders not to try to spider things like the search page. It's *possible* that this is why the spider was trying to do such massive spidering of search result pages.
This is our working hypothesis -- and we will reopen the search functionality to the public on monday.

Recent Posts