Linux webserver du jour?

[f0dder]

I'm currently working on a project that among other things includes a part written in Ruby, running on a linux box. Up to now we've been running it under WebRick on the test server, but I'd like moving to a real web daemon - and now I'm wondering what my choices are.

Apache is a no-go. It's not entirely for rational reasons, but I feel it's too big and clunky and dusty.

For my own little webserver I've been using lighttpd which has served me pretty well, and is probably what I'll end up using unless there's better suggestions. This thread is mainly to see if there's something even better, since I haven't shopped around for httpds for several years

Also, is there any particular stuff I should know about having ruby running under a httpd? I managed getting it running on my own server, but have no idea whether it's running interpreted or in jit'ing vm - afaik the default for ruby is interpreted, but there's several VMs available?

[40hz]

Can't really comment on the ramifications for Ruby. (I just declined to get involved in a Ruby based project a short while ago and have very little knowledge about how that language/framework operates.) But I am a fan of lighttpd. Nowadays I'll only go with Apache if I need it's full feature set or need to integrate something that requires it. Otherwise it's lighttpd all the way. And that's better than 80% of the time.

[f0dder]

Can't really comment on the ramifications for Ruby. (I just declined to get involved in a Ruby based project a short while ago and have very little knowledge about how that language/framework operates.)

Relatively rapid development (as long as you follow The True Path), pretty slow execution

I'll wait a bit and see if there's other comments (we have about a month before finaly deployment has to be done), but I'll likely end up with lighttpd.

[Tuxman]

For my own little webserver I've been using lighttpd which has served me pretty well, and is probably what I'll end up using unless there's better suggestions. This thread is mainly to see if there's something even better, since I haven't shopped around for httpds for several years

I use cherokee which supports Ruby rather well and is quite lightweight, too.
http://www.cherokee-project.com/screencasts.html

[f0dder]

Oh, alternative suggestions!

Have you played with other webservers before? It'd be nice hearing why you explicitly chose Cherokee. Is it easy to configure? Got any tips for setting it up for Ruby (specifically RoR) apps?

I've also stumbled across Hiawata and nginx, and heard mentions that lighttpd have unfixed memory leaks. There's so much to investigate, *sigh*

[Tuxman]

I chose cherokee because I like playing with non-mainstream alternatives and it is probably the feature-richest lightweight webserver application available. I am not a Ruby dev at all, so all I can say is that cherokkee works fine. I tested lighttpd, too, but I was missing some features...

Never heard of Hiawata. Any URL?

[f0dder]

http://www.hiawatha-webserver.org .

[Tuxman]

Hmm, looks rather similar to Cherokee but is not so well documented...

[f0dder]

I've been checking out Cherokee, and so far it seems pretty nice - it frustrated me for a while that it seemed to totally ignore vhost setup, until I found out I have to hit "save" in the toolbar - d'oh . The admin interface is definitely pretty comfortable.

Can't figure out what I need to do to get a RoR setup going, though. I've tried following their cookbook, which seems extremely simple and easy to follow - but my rails version (3.0.3) doesn't have a 'scripts/server' script, which Cherokee expects. I've seen "rake rails:update:generate_dispatchers" mentioned somewhere, but rake doesn't know how to build that.

Dunno if I need to install some ruby appserver or something, but I'll do a bit more digging

[Tuxman]

At least it seems to support it at all. 

Keep us informed.

[f0dder]

Why is it that nothing on linux is ever easy or transparent?

So, ignoring httpd-specific modules, it seems that there's basically two ways of running ruby apps from a webserver: with a (fast)cgi process, or some ruby-specific httpd (which means the "main" httpd will basically run as a proxy for this backend).

When adding a Cherokee RoR vhost, it sets up a pool of three handlers ("sources"), each running on a separate port. Each is started as "/var/www/appname/script/server -p portnum". So far, so good - this isn't generated by default and I'm not sure just what to put there...

I've been looking around trying to figure which "glue" to use. So far it seems that Thin might be a good choice, but I'm really not sure. The whole ruby stack honestly confuses me, and there's apparently a lot of choices. A single Thin daemon, a cluster of mongrels, the (standard?) test-while-developing WEBrick, et cetera.

Tested the Thin thingy by simply doing a "thin start" in the rails app root folder, and it seems to work like a perfect drop-in replacement for the standard webrick, I can access the app on the default port 3000 - yay. Haven't set up the 'server' script properly yet, but have manually started a Thin instance running on the port defined in the Cherokee setup. Seems to work; there's some issues wrt. redirecting not-logged-in users (port number is stripped from URL, which it isn't when connecting directly to Thin), but apart from that stuff seems to work. Unfortunately, it seems like my desktop-client JSON calls still aren't gzip'ed, but at least the response headers show I'm getting data from Cherokee and not directly from Thin - so it should be possible to get gzip further down the road.

Might eventually end up with a nicely running system, but this is so darn typical of my experience with linux systems: poor documentation, outdated how-tos, and a lot of manual work for stuff that plainly ought to "just work". This bitching is directed towards the ruby/rails stuff, btw, Cherokee seems like a decent product.

[JavaJones]

There's also Phusion Passenger, though if you're moving away from Apache and don't want to run Nginx it might not be of interest. The same people make "Ruby Enterprise Edition" that's supposed to reduce RoR memory usage 30% (but, I gather, only when used with Passenger). Interesting, I dunno.

The only reason I mention these is my Rails-specific host Rails Playground offers all these, and I figure they might know something about the subject. Their configurations are:
"Pre-configured Ruby on Rails with Apache/Passenger/MySQL, Mongrel/Nginx/MySQL, or Lighttpd + FastCGI Images Available"
which might give some suggestions on good cooperative system configs.

- Oshyan

[f0dder]

Ok, so Cherokee is a nice product and I really like it's config section, but I ran into a bug that apparently hasn't been fixed yet. Short story: host is accessed via a SSH tunnel and /etc/hosts magic - so I connect to "http://myapp.local:4200" on my laptop, which is SSH tunneled to the test-server's own port 80. On HTTP 302 (redirect), the custom port is stripped from the Location response header, which means stuff obviously ends up not working.

Also, Cherokee's default setup of starting an appserver interpreter on demand means that the first time an instance is after server restart, you get a "502 bad gateway" HTTP error. This can be fixed with Cherokee by starting the appserver elsewhere, it's just the default way of doing things that is bad.

I'll check out nginx+passenger now - thanks for the suggestion, JavaJones. It's one of the combos I already saw mentioned elsewhere (and has just been recommended from yet another party), so I guess it's time to give it a try. I'll keep you posted with my results

[Tuxman]

nginx is not documented well, it seems. Is there any comprehensive list of available modules?

[Shades]

@f0dder
If memory serves me right, I believe that 40Hz many moons ago mentioned bitnami that will get you completely configured development environments in the shape of a simple installation file. And guess what, Ruby is among them. 

[f0dder]

If memory serves me right, I believe that 40Hz many moons ago mentioned bitnami that will get you completely configured development environments in the shape of a simple installation file. And guess what, Ruby is among them.

Thanks for the link, but I dunno if a package like that is for me - probably a decent thing for getting a development environment up and running quickly, but for the VPS we're deploying to I'm "stuck" with debian - it's not that bad installing stuff (and configuring it) once you have a clue what you're supposed to installed & configure

nginx is not documented well, it seems. Is there any comprehensive list of available modules?

The website definitely lacks a lot - one of the reasons I checkout out Cherokee before nginx. We'll see how bad it is once I get it installed

[f0dder]

There, nginx+passenger installed. There isn't any ready-made Debian package for nginx with passenger support, and nginx modules have to be built at compile-time... so no sweet auto-binary-install option, instead you use passenger's "build nginx with passenger support" auto buildscript. I generally don't like building from source (except on gentoo, where build-from-source is part of the package management system) - simply because it makes upgrading a bit of a hassle. But the procedure was painless, and after adding six lines of config to nginx.conf, our Rails app is up-and-running.

I still need to fix the /etc/init.d/nginx script and make sure it's added to system startup, something that'd have been pretty automagic if using a normal Debian package install, but I'll live.

Observations so far:

• The "bad gateway" and "missing port" bugs I experienced with Cherokee are gone.
• nginx seems more capable at caching - for some reason, Rails (or whatever part of the stack) inserts "?number" for javascript URLs, which kept Cherokee (at least with default conf) from caching.
• Things fly now, probably thanks to both the caching mentioned above, and Passenger's optimizations.

Next up might be adding Ruby Enterprise Edition to the mix, and I've been recommended looking at Capistrano - that's probably overkill for our app since it's not distributed across multiple machines and probably never will need to be... but I'll check it out nonetheless

[40hz]

Just a note in passing...

Anytime I'm stumped, or frustrated, or in a rush to do something with Linux when it doesn't want to cooperate, I'll pay a visit to www.howtoforge.com and do a little digging. More often than not I'll find my answer there.

Sometimes you need to abstract information from more than one article to get a full solution. But I haven't found much I needed that these folks haven't covered.

Note: HowtoForge isn't ideal if you like a lot of theory. Or if you want in-depth answers to "why" questions. They're long on "nuts & bolts" and short on discussion. But that's not a bad thing when you consider how many places are the exact opposite.  

Good resource! Bookmark it.  

[f0dder]

Thanks for the link, 40hz, I'll be sure to check it out next time I'm frustrated

Getting nginx+passenger running was super smooth. I then proceeded spending the rest of the day doing a few benchmarks (it's good; the major bottleneck now is our Google Maps calls, which we probably can't do much about), fixing up some of my webclient code, bugfixing a bit of my mates' RoR code after hunting ghosts in my code, and then cursing and being close to smash my head in the wall because normal requests went GZIP'ed just fine, while my webservice requests didn't.

Long story short: nginx peculiarity combined with a bug in .NET HttpWebRequest when using HTTP authentication - yummy. Workaround, which also removes an unnecessary server roundtrip: don't use HttpWebRequest.Credentials, but add the HTTP Request "Authorization:" field manually.

[40hz]

@fodder: ever think of writing up what you're doing and submitting it to HTF?

You've definitely got the technical know-how and language skills to show others the way if you don't mind sharing.

Just a thought...  

[f0dder]

@fodder: ever think of writing up what you're doing and submitting it to HTF?

Considering it - I'll be documenting the process for the report anyway, and I'm taking notes as I go along (this thread is part of the notes ).

You've definitely got the technical know-how and language skills to show others the way if you don't mind sharing.

Thanks

[f0dder]

I added Capistrano to the mix yesterday evening, and it's definitely nice being able to type "cap deploy" and have the latest version of the website checked out from subversion + app-server restarted automatically. It might be slightly overkill for out situation, which is a single httpd, single app-server instance (passenger) and a single database, and all running on the same system... but it does automate stuff, and should the system ever need to scale, you can simply add more servers and update the deployment script.

Took a slight bit of work to get up and running, especially since I had my subversion repository running through svn:// - I started by moving that to svn+ssh:// , which required changing a few things here and there. Upside: my svn setup is now pretty secure, and it's not too bad adding new users - their pubkey (and some SSH tunnel command-overriding magic) in ~subversion/.ssh/authorized_keys. I've wanted to do this for a while, but never got around to it, especially not after deciding to switch over to git. My two classmates insisted on svn for this project, so here we are

I also got the time to look into SSH agent-forwarding, which is neeeeeeeeato. I've normally only SSH'ed into one box, but when SSH'ing to another box, or scp'ing or svn+ssh'ing back and forth between systems, it really simplifies things - and is a lot more secure than keeping your private keys (or, *shudder*, plaintext passphrases) on involved machines along the route.

Anyway, the default simple Capistrano deployment procedure documented on their site sucks - it's slow and INSECURE. I didn't realize this until I started wondering why the /var/www/ourapp/releases/* folders were so huge. I realized that they had .svn subfolders... yes, that's right, not only does the default Capistrano setup pull your FULL set of sources from your repository on every deploy, it does a CHECKOUT rather than an EXPORT. This means people can access http://joor.0wned.org/.svn/* files, which leaks information. Oops.

Solution: install the capistrano_rsync_with_remote_cache gem, which does svn checkout (and after that, svn update) to a .rsync_cache folder on the machine you're running the deployment script from, and then does rsync to the servers you're deploying to. This part fixes the slowness, by only transferring the modifications to the server. This STILL transfers the .svn folders, though, until you add the following line to your deployment config file: "set :rsync_options, '-az --delete --exclude=.svn --delete-excluded'" - presto, fast and safe deploys.

It scares me a bit that the defaults are so retarded, and that there's no mention of this on the main site... google was very helpful once I discovered the .svn folders, though, and it took around 20 minutes from beginning researching the problem until the fix was in place and tested

[40hz]

@f0dder: Just stumbled on this post over on Twitter's engineering blog while researching something else:

For a long Time, twitter.com ran on top of Apache and Mongrel, using mod_proxy_balancer. As in the supermarket, Apache would "push" requests to waiting mongrels for processing, using the 'bybusyness' method. Mongrels which had the least number of requests queued received the latest request. Unfortunately, when an incoming request was queued, the balancer process had no way of knowing how far along in each task the mongrels were. Apache would send requests randomly to servers when they were equally loaded, placing fast requests behind slow ones, increasing the latency of each request.

In November we started testing Unicorn, an exciting new Ruby server that takes the Mongrel core and adds Fry's "pull" model. Mobile.twitter.com was our first app to run Unicorn behind Apache, and in January @netik ran the gauntlet to integrate Unicorn into the main Twitter.com infrastructure.

During initial tests, we predicted we would maintain CPU usage and only cut request latency 10-15%. Unicorn surprised us by dropping request latency 30% and significantly lowering CPU usage.

This was interesting enough that it prompted me to take a look at the referenced mongrel-derivative Unicorn:

Unicorn: Rack HTTP server for fast clients and Unix

Unicorn is an HTTP server for Rack applications designed to only serve fast clients on low-latency, high-bandwidth connections and take advantage of features in Unix/Unix-like kernels. Slow clients should only be served by placing a reverse proxy capable of fully buffering both the the request and response in between Unicorn and slow clients.

And its even more interesting cousin:Rainbows!:

Rainbows! - Unicorn for sleepy apps and slow clients

Rainbows! is an HTTP server for sleepy Rack applications. It is based on Unicorn, but designed to handle applications that expect long request/response times and/or slow clients.

For Rack applications not heavily bound by slow external network dependencies, consider Unicorn instead as it simpler and easier to debug.

Of especial interest in Rainbows:

Rainbows! is mainly designed for the odd things Unicorn sucks at:

• Web Sockets (via Sunshowers)
• 3rd-party APIs (to services outside your control/LAN)
• OpenID consumers (to providers outside your control/LAN)
• Reverse proxy implementations with editing/censoring (to upstreams outside your control/LAN)
• Comet
• BOSH (with slow clients)
• HTTP server push
• Long polling
• Reverse AJAX
• real-time upload processing (via upr)

Rainbows! can also be used to service slow clients directly even with fast applications.

Both Unicorn and Rainbows! are more geared towards high-volume production environments, but I thought you might be interested in at least being aware of them. Assuming you're not already.  

From the Unicorn Philosophy page:

Unicorn is not suited for all applications. Unicorn is optimized for applications that are CPU/memory/disk intensive and spend little time waiting on external resources (e.g. a database server or external API).

Unicorn is highly inefficient for Comet/reverse-HTTP/push applications where the HTTP connection spends a large amount of time idle. Nevertheless, the ease of troubleshooting, debugging, and management of Unicorn may still outweigh the drawbacks for these applications.

The Rainbows! aims to fill the gap for odd corner cases where the nginx + Unicorn combination is not enough. While Rainbows! management/administration is largely identical to Unicorn, Rainbows! is far more ambitious and has seen little real-world usage.

Interesting pair of servers...

[f0dder]

Yeah, I came across Unicorn, but figured it was a bit outside our needs.

The web is served by american indians riding unicorns across rainbows! - opensource project names are so fscked. I do think it's kinda funny that we've got Apache, Cherokee and Hiawatha though

nginx + Passenger seems to run fairly well, BUT! - something's retarded somewhere, I'm getting broken pipes to MySQL after the system has been running for a while. The most promising info I've found on the problem so far says it could have something to do with the MySQL adapter in use (and that it's causes by retardedness in Rails). Basically there's ruby/mysql and mysql/ruby (good thing those two have such distinct names to avoid confusion!) where one's written in Ruby (and is buggy), and the other in C (and requires a bunch of manual compilation and setup, but might work).

Once that's fixed, I think I'm pretty much done with the sysadmin part of the project (though I'd like some per-vhost traffic stats, preferably with some real-time rrdtool gathering, rather than lame slow log-file parsing). I'll leave database backups and log rotation to the guys we're handing over to when done

[f0dder]

Still haven't gotten into the MySQL EPIPE error, did some actual coding today, yay!

And bumped into a second .NET HttpWebRequest bug (well, perhaps it's technically following spec, but the behavior is fubar). The concentrated version: you get an exception if you get a 304 (cached) reply for an URL that didn't include a "Date:" header in the original 200 (OK) reply. In my case, it's caused by Phusion Passenger (and Thin, for that matter) not automatically adding the "Date:" header, which WEBrick does.

Damn I like fiddler! It has been invaluable in tracing down wtf was going on.