What if Dee-nee/Gantry explodes?

[fknmclane]

Not sure about the server but I know your mom goes down every twenty minutes during the week and every fifteen minutes on the weekend to keep up with demand.

[Gantry]

I'll add that to the logbook, thanks

[BDawk]

Can we take a collection for a new server?  How much would you need?

[fknmclane]

Yes.  Let's upgrade this fucker and save Gantry and further bullshit to deal with.

[Gantry]

Looks like the fun started at 1pm on Saturday, was trying to kill httpd processes.  So I guess the questions is do I have a memory leak or does the server sometimes just get overwhelmed to the point where it runs out of RAM? 

It's hard to believe the latter because we went from 2GB to 4GB relatively recently.  Heck I think this server started at 1GB.  But trying down the specifics is beyond my area of expertise.  I'd throw more RAM at it if I thought it would help but DDR2 ram is a ripoff, going from 4GB to 6GB is about $80 as i'm out of slots. 

I guess I'll have to experiment with solutions and more downtime as a result.  YAY

[Gantry]

A new server is not about the cost, but the amount of time it would take to migrate to a new system.  Probably should be done but it's a good 10-20 hours of my & nightwulf's time to get the old server, rbicentral, my punkdatabase wiki, the forums, etc to new hardware and software.  At this point nightwulf has done some much of the legwork on the forums that I'd be lost and have to defer all of that to him, while I focus on getting the rest going. 

If I can throw $80 for a memory upgrade to make our problems go away I would.  I just don't know what the issue is...

[fathedX]

Maybe we can all agree to rendezvous at Dee-Nee 2.0 when we have an outage.  BDawk was nice enough to set this up several years ago when we were having similar outages and I'll bet he doesn't even remember setting it up.

http://deenee.proboards.com/

[BDawk]

I totally forgot I made this.

[Reds]

OBOY fucking killing it over there...

[Gantry]

That does not ring a bell at all...

[fathedX]

You weren't invited.

[fknmclane]

OBOY fucking killing it over there...

OBOY was the best.

[rdub]

Maybe we can all agree to rendezvous at Dee-Nee 2.0 when we have an outage.  BDawk was nice enough to set this up several years ago when we were having similar outages and I'll bet he doesn't even remember setting it up.

http://deenee.proboards.com/

that's fucking great, officially adding to my favorites and you'll find me in "sweaters and such" the next time this thing shits the bed.

[Gantry]

Just going to brainstorm here, with no real train of thought:

1) Quite a few people with this issue, seems to be all apache.
2) If it's not memory leak, I'm confused how a 1pm Saturday load can cause it to go down.  Not exactly peak posting time.
3) There's a mod to limit guest posting
4) Seems like ngnix is WAY better with memory over apache, though I'm not sure it's worth the time to convert.
5) Server memory holding steady at 1.5GB free

[nightwulf]

I have no idea where this all came from. By the time I got home from work Saturday the server was alive but not responding; I left an SSH window up for over an hour and never could get through with even a login prompt.

How up-to-date is apache? Well really anything could cause a memory leak. If we're far behind whatever is current, it may be just a matter of an OS update (which is a lot more involved than it sounds, I know ... )

I absolutely don't think buying more DDR2 is a good idea. At this point a mobo / CPU / DDR3 would be the way to go if any non-free hardware upgrade is done. 16 gigs of RAM for around $80, and moving from the Sempron to something in the AM3 family would be overkill for our needs, but it wouldn't hurt and the power usage might even drop. Not sure what the TDP is on our specific CPU.

Agreed that the biggest cost in a new server is time. Moving the forums shouldn't be too difficult ... really I think most of it could be drag-and-dropped to a new server and most of it would work. The biggest problem I see is incompatibilities if we're on a different version of mysql, but we've gotten through that before.

[rdub]

ya got mud in yer tires

[Gantry]

I have no idea where this all came from. By the time I got home from work Saturday the server was alive but not responding; I left an SSH window up for over an hour and never could get through with even a login prompt.

I know it's there a last resort, but once the OOM killer starts running the system is always dead in the water.  I've never had it actually work, not once.  A DRAC/iLO card that lets you control the BIOS remotely would be very handy in situations like these.   I'm pretty much in the city all weekend so if dee-nee goes down from thursday night through sunday it's not coming back up without taking 90+ minutes out of my day.  And that's assuming I haven't been drinking. 

How up-to-date is apache? Well really anything could cause a memory leak. If we're far behind whatever is current, it may be just a matter of an OS update (which is a lot more involved than it sounds, I know ... )

Apache is 2.0.52 2.2 - looks like there has been a 2.4 release since then though I don't know how many improvements.  I do all the CentOS updates (aka Red Hat Enterprise 5.0) on a regular basis.  I like the ease of use and stability of using yum/rpm to update everything so I don't know how much of a PITA putting a new version in would be.  Probably was just as much or less time to put in nginx which I keep hearing good things about. 

I absolutely don't think buying more DDR2 is a good idea. At this point a mobo / CPU / DDR3 would be the way to go if any non-free hardware upgrade is done. 16 gigs of RAM for around $80, and moving from the Sempron to something in the AM3 family would be overkill for our needs, but it wouldn't hurt and the power usage might even drop. Not sure what the TDP is on our specific CPU.

The reason I'd consider DDR2 is that it take 5 minutes.   I'm not well versed enough in Linux hardware upgrades to know if I can drop in a new mobo/ram/cpu and have it come up without spending all day trying to hash out the drivers.  I'd do that in a heartbeat if that doesn't screw everything up.  Every piece of hardware is integrated on that mobo, so that probably wouldn't be worth it.  The NIC and SATA drivers alone would not be fun, especially with software RAID.

Long term solution is new hardware and OS upgrade, then gradually move everything over.  But heck we haven't even moved dee-nee proper over yet because of the issues we ran into with all our apache rewrite stuff we did.  The combination of stuff running on the two servers:

* Dee-Nee proper
* RBI central sites (probably half a dozen)
* FTP (may not need anymore, doubt anyone updates their RBI Central site)
* Your RBI editor
* 3 mediawiki sites, all with upgraded version
* Forums
* Some other shit I'm forgetting.

Just a big time commitment, though I'm probably willing to take the plunge at this point if these problems persist.   

[nightwulf]

The reason I'd consider DDR2 is that it take 5 minutes.   I'm not well versed enough in Linux hardware upgrades to know if I can drop in a new mobo/ram/cpu and have it come up without spending all day trying to hash out the drivers.  I'd do that in a heartbeat if that doesn't screw everything up.  Every piece of hardware is integrated on that mobo, so that probably wouldn't be worth it.  The NIC and SATA drivers alone would not be fun, especially with software RAID.

Ease of switching motherboards is mostly dependent on your kernel. Back in the day when installing a new distro I'd head straight to kernel.org and download a vanilla kernel, then spend a lot of time configuring in only exactly what I needed. Made for a slim kernel, but any hardware changes required big time changes.

Fortunately I got myself out of that habit and for the most part stick with the stock kernel used by the distro installer. It's far from slim, but these days most of the kernel functionality is added through loadable mods anyway. Those installer kernels are built to be as generic as possible and to facilitate autodetecting a very wide range of hardware. I had a great experience last month swapping the motherboards between my htpc and my slackware server. Other than clearing NVRAM I did nothing for slackware. All hardware was autodetected on boot and the system ran again without the slightest problem.

So if you're using some stock CentOS kernel and haven't compiled a bunch of proprietary drivers, I don't think it'd be a major hassle. If you have, it might.

[Flood]

ya got mud in yer tires

aaaaand I just choked on an almond.

[Gantry]

Stock CentOS, which is stock redhat - everything is modular.  I'm just very weary of screwing up anything related to the SATA drivers & software RAID, though I guess that's what images are for.  This is also precisely why I'd like to go the VM route. 

Then again if we have a nasty memory leak in apache a hardware upgrade may not help.  Though it's probably much tougher to get rid of 16GB of RAM than 4GB. 

This will take consideration, I'll keep my eye on memory footprint (have a top window running 24/7 now) and lets see if this is actually a major problem.  The last 10 outages were all related to the power supply, so maybe this is something that's infrequent enough to deal with.