topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Monday March 18, 2024, 10:53 pm
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: Server - Apache problem running out of memory  (Read 15579 times)

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Server - Apache problem running out of memory
« on: April 19, 2011, 12:57 AM »
I'm not sure how accurate my description of this problem is going to be, but we seem to have a problem on the new server which has caused the server to crash occasionally.  It seems to have to do with running out of memory from Apache processes.

In truth, it's hard for me to even write about the server software that is the backbone of our entire culture without flying into a screaming rage about how incredibly horrible all of it is designed and how it's virtually impossible to figure out what goes wrong and why and how impossible it is to configure this stuff to work well under load.  But hey, it's just the software that powers millions of web sites and servers and runs the entire internet, so why should it matter if it is complete crap.

But anyway.. the best solution we can think of is maybe to set apache maxclients to something like 21 and hope this will keep the server from running out of memory.

Does anyone have any better ideas?  We upgraded the server when we moved to 8gb but the use of vmware to host multiple servers seems to be really taking a toll on available memory.

We never had this problem on the old server.. I'm not sure why we are having it now.

I'm also not sure if this is related to the random but persistent problem we are having with the server (especially member server) timing out on some downloads, or failing with partial files.

Any suggestions or ideas would be welcome.

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Server - Apache problem running out of memory
« Reply #1 on: April 19, 2011, 12:57 AM »
ps. we set maxclients to 21 now to see if that helps.. though it seems to have slowed down web access to the server substantially.

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: Server - Apache problem running out of memory
« Reply #2 on: April 19, 2011, 04:20 AM »
Try another httpd? Like, one that uses async I/O instead of relying on multiple processes?
- carpe noctem

Gothi[c]

  • DC Server Admin
  • Charter Honorary Member
  • Joined in 2006
  • ***
  • Posts: 873
    • View Profile
    • linkerror
    • Donate to Member
Re: Server - Apache problem running out of memory
« Reply #3 on: April 19, 2011, 08:24 AM »
First of all, while mouser states "we don't know" he really means "he doesn't know" :) I tried to explain, but oh well...

The problem is this:

Recently we have been implementing a lot of caching in order to speed things up. The database currently does quite some heavy query caching, and on the apache side we have besides the usual stuff, APC for php opcode caching (which the forum software, smf, can use the API for). All of this, of course comes at a cost of memory.

We had maxclients set quite high (200), which makes things run pretty fast with prefork, since there's always processes around to serve new clients. The problem is that the average apache process size has grown to around 30MB, so worst case scenario is 20*30MB=5.8GB, which is more memory than the server has.
In fact, after the aggressive MySQL caching, there is only 780 MB free for apache to play around with.
This is why I have decreased MaxClients to 20, which will prevent us from running out of memory, at the cost of some slowdown.

So really, all we have to do at this moment is probably just make the MySQL caching a bit less crazy, to make some more room for Apache.

As f0dder points out sort of kind of, the fact that Apache with prefork is process based, it makes memory consumption a bit hard to predict (since the memory size of a process changes by what script is running, moreover, if you use keepalive, it grows with the number of requests a client makes over the same connection to the same client thus making things even more unpredictable).

I wouldn't go as far as switching httpd, but we could switch MPM. There is an experimental event MPM for apache, but I'm not sure how I feel about using an experimental MPM on a production server.

To make matters worse, packet loss at softlayer has been really hurting us.
This is their latest excuse:

    SoftLayer Engineers are aware of the sporadic packetloss and/or connectivity to FCR01.SEA01. Currently engineers are working on resolving this issue; however, there is a chance a reboot to the router will be required. In the event this needs to happen, a notice will be posted here along with any other information gathered.

-- Update --
Service has been restored to customers behind FCR01.SEA01. During the process of working on the router, the issue manifested itself into 100% CPU resulting in upstream and downstream links going down along with routing protocols. Engineers were able to stabilize the router without a reload, and are currently monitoring it to determine if the fix is permanent.

-- UPDATE --
Engineers have determined this is not a permanent fix for the FCR01.SEA01 issue. During the course of troubleshooting this issue with Cisco, it has been determined that the best course of action is to upgrade the router to the latest IOS version. This will be happening at approximately 01:30 CDT.

--UPDATE--
Engineers are continuing to work with Cisco TAC on this issue. At this point, the router has been restored to service at approximately 03:20 CDT. Some customers may continue to experience intermittent packetloss behind FCR01.SEA01.

Working for a hosting provider myself, I understand that stuff happens, but this packet loss stuff with them has been going on ever since we moved to their seattle data center :( every day.
Also, one would think that softlayer is big enough to have a replacement router ready they can just put in (or have a proper redundant network that can route around it for that matter).
« Last Edit: April 19, 2011, 08:35 AM by Gothi[c] »

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Server - Apache problem running out of memory
« Reply #4 on: April 19, 2011, 12:41 PM »
SoftLayer's handling of the issue has been horrible and at this point I wouldn't feel at all comfortable recommending them to anyone as a hosting solution.  In the past i've recommended SoftLayer on the basis that they may be more expensive, but they are more stable and reliable.  But this experience has turned all of that around.

cranioscopical

  • Friend of the Site
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4,776
    • View Profile
    • Donate to Member
Re: Server - Apache problem running out of memory
« Reply #5 on: April 19, 2011, 01:20 PM »


Well, here it's like watching treacle pour over molasses but it certainly makes one appreciate 'normal service'.

I'm sure it'll be fixed one way or another.
Keep smiling!

Stoic Joker

  • Honorary Member
  • Joined in 2008
  • **
  • Posts: 6,646
    • View Profile
    • Donate to Member
Re: Server - Apache problem running out of memory
« Reply #6 on: April 19, 2011, 01:40 PM »
First of all, while mouser states "we don't know" he really means "he doesn't know" :) I tried to explain, but oh well...

Thanks for the clarification, it makes the story much less frightening... :)

I've never worked with Apache (I'm an MS/IIS guy), but I googled the Apache exhausts memory under load thing and it seems a rather common problem. There were some really bad hiccups early this afternoon 12-1 (EDT). And I was wondering if that was SoftLayer, or a lunch activity peek close to 20? (I guess both is possible)

Anyhow, how do the Keep-Alives factor into this, or (are they disabled) don't they?

Shades

  • Member
  • Joined in 2006
  • **
  • Posts: 2,922
    • View Profile
    • Donate to Member
Re: Server - Apache problem running out of memory
« Reply #7 on: April 19, 2011, 04:44 PM »
Maybe a stupid and very obvious one, but:  is it not possible to add more (physical) RAM to the (virtual) server?


Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: Server - Apache problem running out of memory
« Reply #8 on: April 19, 2011, 06:49 PM »
SoftLayer's handling of the issue has been horrible and at this point I wouldn't feel at all comfortable recommending them to anyone as a hosting solution.  In the past i've recommended SoftLayer on the basis that they may be more expensive, but they are more stable and reliable.  But this experience has turned all of that around.

I've had good luck so far with them and no problems. My serves are all in the Texas data centers though.
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

Ath

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 3,610
    • View Profile
    • Donate to Member
Re: Server - Apache problem running out of memory
« Reply #9 on: April 20, 2011, 07:54 AM »
As of about 16 hours the performance is a lot better now.

Refreshing the forum frontpage now is only severely slowed down while waiting for 'static.addtoany.com', but got that solved by adding
127.0.0.1 static.addtoany.com
to my hosts file :D
« Last Edit: April 20, 2011, 07:55 AM by Ath »