topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Thursday March 28, 2024, 3:48 am
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline  (Read 17584 times)

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
search is temporarily offline while we investigate.

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #1 on: January 23, 2011, 04:49 PM »
So far it's looking like Gothi[c] and I may have identified and remedied the problem:

Looking through that access logs (shout out to my Web Log Filter utility for making it a little easier), it seems like the server slowdown may have been caused by a web spider from russia making huge numbers of calls to the search function, which puts pretty heavy demands on the cpu and memory.

In fairness, the fault may have been mostly ours -- when we moved servers, we changed the forum urls from /Forum/bb to the simpler /forum but we didn't update the robots.txt file which tells robot spiders not to try to spider things like the search page.  It's *possible* that this is why the spider was trying to do such massive spidering of search result pages.

This is our working hypothesis -- and we will reopen the search functionality to the public on monday.

cranioscopical

  • Friend of the Site
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4,776
    • View Profile
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #2 on: January 24, 2011, 07:12 AM »
Search seems quite fast today. I searched on "align" which produced 7 pages of results in a timely manner.

kyrathaba

  • N.A.N.Y. Organizer
  • Honorary Member
  • Joined in 2006
  • **
  • Posts: 3,200
    • View Profile
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #3 on: January 24, 2011, 11:07 AM »
I've been unable to connect to my DC FTP account since 1/21.  Tried it again a few minutes ago.  Is this related to the server problem?  Not a problem, since I can back up my files to my other site.  Just curious.

worstje

  • Honorary Member
  • Joined in 2009
  • **
  • Posts: 588
  • The Gent with the White Hat
    • View Profile
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #4 on: January 24, 2011, 11:19 AM »
I've been unable to connect to my DC FTP account since 1/21.  Tried it again a few minutes ago.  Is this related to the server problem?  Not a problem, since I can back up my files to my other site.  Just curious.

See: moving DCMembers.com to the new server

kyrathaba

  • N.A.N.Y. Organizer
  • Honorary Member
  • Joined in 2006
  • **
  • Posts: 3,200
    • View Profile
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #5 on: January 24, 2011, 11:21 AM »
Sorry, the link gives me "An Error Has Occurred!"  I take it we're in the process of migrating user accounts to the new server, and may have a few days unavailability?

worstje

  • Honorary Member
  • Joined in 2009
  • **
  • Posts: 588
  • The Gent with the White Hat
    • View Profile
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #6 on: January 24, 2011, 11:34 AM »
Yeah. Website and such will stay up, but to my knowledge FTP is turned off and SQL if you use that might or might not end up losing some data incurred between the time of backups and the sorts. Or so I understood from Gothi[c] in irc when I asked.

kyrathaba

  • N.A.N.Y. Organizer
  • Honorary Member
  • Joined in 2006
  • **
  • Posts: 3,200
    • View Profile
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #7 on: January 24, 2011, 11:38 AM »
No problem.  Thanks for the info, worstje!

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #8 on: January 25, 2011, 07:45 PM »
Just a heads up, server may be down briefly in the next 24 hours as we install a cache library and activate some additional code to adapt to server load; should make forum more robust and peppier :)

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #9 on: January 26, 2011, 03:53 PM »
ok server is going to go down briefly today while we put in some new code.

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #10 on: January 27, 2011, 11:41 AM »
Just a heads up, server may be down briefly in the next 24 hours as we install a cache library and activate some additional code to adapt to server load; should make forum more robust and peppier :)
memcached?
- carpe noctem

Gothi[c]

  • DC Server Admin
  • Charter Honorary Member
  • Joined in 2006
  • ***
  • Posts: 873
    • View Profile
    • linkerror
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #11 on: January 27, 2011, 12:02 PM »
memcached?

APC

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #12 on: January 27, 2011, 12:05 PM »
Ah, PHP precompilation - I thought database result caching would result in larger performance wins? (Harder to implement, though :))
- carpe noctem

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #13 on: January 27, 2011, 01:48 PM »
I thought database result caching would result in larger performance wins

In addition to PHP opcode cahcing, smf also has some routines for data content caching, which is also happening through APC.

Gothi[c]

  • DC Server Admin
  • Charter Honorary Member
  • Joined in 2006
  • ***
  • Posts: 873
    • View Profile
    • linkerror
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #14 on: January 27, 2011, 08:26 PM »
And, on top of that mysql also caches query results.

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #15 on: January 27, 2011, 08:29 PM »
And on top of that Gothic built a little desk and console terminal for Cody to sit at and monitor raw tcp traffic to the server -- when he sees a request come in that he recognizes, he can intercept it and peck out a memorized reply himself.  I don't know if that will lead to faster results, but it keeps Cody busy.

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #16 on: January 27, 2011, 08:30 PM »
Oh, the Cody detail is especially wonderful :)
- carpe noctem

cranioscopical

  • Friend of the Site
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4,776
    • View Profile
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #17 on: January 27, 2011, 09:37 PM »
And on top of that Gothic built a little desk and console terminal for Cody to sit at

And on top of that, I hope, you will place the miniature mouser mug that you ordered for him!

Ath

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 3,612
    • View Profile
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #18 on: January 30, 2011, 09:44 AM »
Does this new caching stuff have impact on the refresh/update of the main forum page, at the bottom where the last 10 updated threads are shown? Now if I make a post, it takes a little while and a few screen refreshes for it to show up there :o

Ath

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 3,612
    • View Profile
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #19 on: February 03, 2011, 01:57 PM »
And somewhere about 1/2 hour ago, SMF couldn't reach the database, so here I'm back a few minutes later... :huh:

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #20 on: February 03, 2011, 01:58 PM »
yep, was offline for a few minutes while we were backing things up.  should be ok now.

Ath

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 3,612
    • View Profile
    • Donate to Member
Re: Server Slow/Load Issue - Jan 22, 2011 - search temporarily offline
« Reply #21 on: February 03, 2011, 02:23 PM »
Ok, is there a time-window where backups are being made, so I can avoid hitting the barriers? ;)