Author Topic: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline (Read 22,756 times)

mouser · « **on:** January 22, 2011, 10:33 PM »

search is temporarily offline while we investigate.

mouser · « **Reply #1 on:** January 23, 2011, 10:49 PM »

So far it's looking like Gothi[c] and I may have identified and remedied the problem:

Looking through that access logs (shout out to my Web Log Filter utility for making it a little easier), it seems like the server slowdown may have been caused by a web spider from russia making huge numbers of calls to the search function, which puts pretty heavy demands on the cpu and memory.

In fairness, the fault may have been mostly ours -- when we moved servers, we changed the forum urls from /Forum/bb to the simpler /forum but we didn't update the robots.txt file which tells robot spiders not to try to spider things like the search page. It's *possible* that this is why the spider was trying to do such massive spidering of search result pages.

This is our working hypothesis -- and we will reopen the search functionality to the public on monday.

cranioscopical · « **Reply #2 on:** January 24, 2011, 01:12 PM »

Search seems quite fast today. I searched on "align" which produced 7 pages of results in a timely manner.

kyrathaba · « **Reply #3 on:** January 24, 2011, 05:07 PM »

I've been unable to connect to my DC FTP account since 1/21. Tried it again a few minutes ago. Is this related to the server problem? Not a problem, since I can back up my files to my other site. Just curious.

worstje · « **Reply #4 on:** January 24, 2011, 05:19 PM »

Quote from: kyrathaba on January 24, 2011, 05:07 PM

I've been unable to connect to my DC FTP account since 1/21. Tried it again a few minutes ago. Is this related to the server problem? Not a problem, since I can back up my files to my other site. Just curious.

See: moving DCMembers.com to the new server

kyrathaba · « **Reply #5 on:** January 24, 2011, 05:21 PM »

Sorry, the link gives me "An Error Has Occurred!" I take it we're in the process of migrating user accounts to the new server, and may have a few days unavailability?

worstje · « **Reply #6 on:** January 24, 2011, 05:34 PM »

Yeah. Website and such will stay up, but to my knowledge FTP is turned off and SQL if you use that might or might not end up losing some data incurred between the time of backups and the sorts. Or so I understood from Gothi[c] in irc when I asked.

kyrathaba · « **Reply #7 on:** January 24, 2011, 05:38 PM »

No problem. Thanks for the info, worstje!

mouser · « **Reply #8 on:** January 26, 2011, 01:45 AM »

Just a heads up, server may be down briefly in the next 24 hours as we install a cache library and activate some additional code to adapt to server load; should make forum more robust and peppier

mouser · « **Reply #9 on:** January 26, 2011, 09:53 PM »

ok server is going to go down briefly today while we put in some new code.

f0dder · « **Reply #10 on:** January 27, 2011, 05:41 PM »

Quote from: mouser on January 26, 2011, 01:45 AM

Just a heads up, server may be down briefly in the next 24 hours as we install a cache library and activate some additional code to adapt to server load; should make forum more robust and peppier

memcached?

Gothi[c] · « **Reply #11 on:** January 27, 2011, 06:02 PM »

Quote

memcached?

APC

f0dder · « **Reply #12 on:** January 27, 2011, 06:05 PM »

Ah, PHP precompilation - I thought database result caching would result in larger performance wins? (Harder to implement, though

)

mouser · « **Reply #13 on:** January 27, 2011, 07:48 PM »

Quote

I thought database result caching would result in larger performance wins

In addition to PHP opcode cahcing, smf also has some routines for data content caching, which is also happening through APC.

Gothi[c] · « **Reply #14 on:** January 28, 2011, 02:26 AM »

And, on top of that mysql also caches query results.

mouser · « **Reply #15 on:** January 28, 2011, 02:29 AM »

And on top of that Gothic built a little desk and console terminal for Cody to sit at and monitor raw tcp traffic to the server -- when he sees a request come in that he recognizes, he can intercept it and peck out a memorized reply himself. I don't know if that will lead to faster results, but it keeps Cody busy.

f0dder · « **Reply #16 on:** January 28, 2011, 02:30 AM »

Oh, the Cody detail is especially wonderful

cranioscopical · « **Reply #17 on:** January 28, 2011, 03:37 AM »

Quote from: mouser on January 28, 2011, 02:29 AM

And on top of that Gothic built a little desk and console terminal for Cody to sit at

And on top of that, I hope, you will place the miniature mouser mug that you ordered for him!

Ath · « **Reply #18 on:** January 30, 2011, 03:44 PM »

Does this new caching stuff have impact on the refresh/update of the main forum page, at the bottom where the last 10 updated threads are shown? Now if I make a post, it takes a little while and a few screen refreshes for it to show up there

Ath · « **Reply #19 on:** February 03, 2011, 07:57 PM »

And somewhere about 1/2 hour ago, SMF couldn't reach the database, so here I'm back a few minutes later...

mouser · « **Reply #20 on:** February 03, 2011, 07:58 PM »

yep, was offline for a few minutes while we were backing things up. should be ok now.

Ath · « **Reply #21 on:** February 03, 2011, 08:23 PM »

Ok, is there a time-window where backups are being made, so I can avoid hitting the barriers?