Main Area and Open Discussion > General Software Discussion
Linux webserver du jour?
f0dder:
@fodder: ever think of writing up what you're doing and submitting it to HTF?-40hz (November 23, 2010, 06:44 PM)
--- End quote ---
Considering it - I'll be documenting the process for the report anyway, and I'm taking notes as I go along (this thread is part of the notes :P).
You've definitely got the technical know-how and language skills to show others the way if you don't mind sharing.-40hz (November 23, 2010, 06:44 PM)
--- End quote ---
Thanks :)
f0dder:
I added Capistrano to the mix yesterday evening, and it's definitely nice being able to type "cap deploy" and have the latest version of the website checked out from subversion + app-server restarted automatically. It might be slightly overkill for out situation, which is a single httpd, single app-server instance (passenger) and a single database, and all running on the same system... but it does automate stuff, and should the system ever need to scale, you can simply add more servers and update the deployment script.
Took a slight bit of work to get up and running, especially since I had my subversion repository running through svn:// - I started by moving that to svn+ssh:// , which required changing a few things here and there. Upside: my svn setup is now pretty secure, and it's not too bad adding new users - their pubkey (and some SSH tunnel command-overriding magic) in ~subversion/.ssh/authorized_keys. I've wanted to do this for a while, but never got around to it, especially not after deciding to switch over to git. My two classmates insisted on svn for this project, so here we are :)
I also got the time to look into SSH agent-forwarding, which is neeeeeeeeato. I've normally only SSH'ed into one box, but when SSH'ing to another box, or scp'ing or svn+ssh'ing back and forth between systems, it really simplifies things - and is a lot more secure than keeping your private keys (or, *shudder*, plaintext passphrases) on involved machines along the route.
Anyway, the default simple Capistrano deployment procedure documented on their site sucks - it's slow and INSECURE. I didn't realize this until I started wondering why the /var/www/ourapp/releases/* folders were so huge. I realized that they had .svn subfolders... yes, that's right, not only does the default Capistrano setup pull your FULL set of sources from your repository on every deploy, it does a CHECKOUT rather than an EXPORT. This means people can access http://joor.0wned.org/.svn/* files, which leaks information. Oops.
Solution: install the capistrano_rsync_with_remote_cache gem, which does svn checkout (and after that, svn update) to a .rsync_cache folder on the machine you're running the deployment script from, and then does rsync to the servers you're deploying to. This part fixes the slowness, by only transferring the modifications to the server. This STILL transfers the .svn folders, though, until you add the following line to your deployment config file: "set :rsync_options, '-az --delete --exclude=.svn --delete-excluded'" - presto, fast and safe deploys.
It scares me a bit that the defaults are so retarded, and that there's no mention of this on the main site... google was very helpful once I discovered the .svn folders, though, and it took around 20 minutes from beginning researching the problem until the fix was in place and tested :)
40hz:
@f0dder: Just stumbled on this post over on Twitter's engineering blog while researching something else:
For a long Time, twitter.com ran on top of Apache and Mongrel, using mod_proxy_balancer. As in the supermarket, Apache would "push" requests to waiting mongrels for processing, using the 'bybusyness' method. Mongrels which had the least number of requests queued received the latest request. Unfortunately, when an incoming request was queued, the balancer process had no way of knowing how far along in each task the mongrels were. Apache would send requests randomly to servers when they were equally loaded, placing fast requests behind slow ones, increasing the latency of each request.
In November we started testing Unicorn, an exciting new Ruby server that takes the Mongrel core and adds Fry's "pull" model. Mobile.twitter.com was our first app to run Unicorn behind Apache, and in January @netik ran the gauntlet to integrate Unicorn into the main Twitter.com infrastructure.
During initial tests, we predicted we would maintain CPU usage and only cut request latency 10-15%. Unicorn surprised us by dropping request latency 30% and significantly lowering CPU usage.
--- End quote ---
This was interesting enough that it prompted me to take a look at the referenced mongrel-derivative Unicorn:
Unicorn: Rack HTTP server for fast clients and Unix
Unicorn is an HTTP server for Rack applications designed to only serve fast clients on low-latency, high-bandwidth connections and take advantage of features in Unix/Unix-like kernels. Slow clients should only be served by placing a reverse proxy capable of fully buffering both the the request and response in between Unicorn and slow clients.
--- End quote ---
And its even more interesting cousin:Rainbows!:
Rainbows! - Unicorn for sleepy apps and slow clients
Rainbows! is an HTTP server for sleepy Rack applications. It is based on Unicorn, but designed to handle applications that expect long request/response times and/or slow clients.
For Rack applications not heavily bound by slow external network dependencies, consider Unicorn instead as it simpler and easier to debug.
--- End quote ---
Of especial interest in Rainbows:
Rainbows! is mainly designed for the odd things Unicorn sucks at:
* Web Sockets (via Sunshowers)
* 3rd-party APIs (to services outside your control/LAN)
* OpenID consumers (to providers outside your control/LAN)
* Reverse proxy implementations with editing/censoring (to upstreams outside your control/LAN)
* Comet
* BOSH (with slow clients)
* HTTP server push
* Long polling
* Reverse AJAX
* real-time upload processing (via upr)
Rainbows! can also be used to service slow clients directly even with fast applications.
--- End quote ---
Both Unicorn and Rainbows! are more geared towards high-volume production environments, but I thought you might be interested in at least being aware of them. Assuming you're not already. ;D
From the Unicorn Philosophy page:
Unicorn is not suited for all applications. Unicorn is optimized for applications that are CPU/memory/disk intensive and spend little time waiting on external resources (e.g. a database server or external API).
Unicorn is highly inefficient for Comet/reverse-HTTP/push applications where the HTTP connection spends a large amount of time idle. Nevertheless, the ease of troubleshooting, debugging, and management of Unicorn may still outweigh the drawbacks for these applications.
The Rainbows! aims to fill the gap for odd corner cases where the nginx + Unicorn combination is not enough. While Rainbows! management/administration is largely identical to Unicorn, Rainbows! is far more ambitious and has seen little real-world usage.
--- End quote ---
Interesting pair of servers... 8)
f0dder:
Yeah, I came across Unicorn, but figured it was a bit outside our needs.
The web is served by american indians riding unicorns across rainbows! - opensource project names are so fscked. I do think it's kinda funny that we've got Apache, Cherokee and Hiawatha though :P
nginx + Passenger seems to run fairly well, BUT! - something's retarded somewhere, I'm getting broken pipes to MySQL after the system has been running for a while. The most promising info I've found on the problem so far says it could have something to do with the MySQL adapter in use (and that it's causes by retardedness in Rails). Basically there's ruby/mysql and mysql/ruby (good thing those two have such distinct names to avoid confusion!) where one's written in Ruby (and is buggy), and the other in C (and requires a bunch of manual compilation and setup, but might work).
Once that's fixed, I think I'm pretty much done with the sysadmin part of the project (though I'd like some per-vhost traffic stats, preferably with some real-time rrdtool gathering, rather than lame slow log-file parsing). I'll leave database backups and log rotation to the guys we're handing over to when done :)
f0dder:
Still haven't gotten into the MySQL EPIPE error, did some actual coding today, yay!
And bumped into a second .NET HttpWebRequest bug (well, perhaps it's technically following spec, but the behavior is fubar). The concentrated version: you get an exception if you get a 304 (cached) reply for an URL that didn't include a "Date:" header in the original 200 (OK) reply. In my case, it's caused by Phusion Passenger (and Thin, for that matter) not automatically adding the "Date:" header, which WEBrick does.
Damn I like fiddler! It has been invaluable in tracing down wtf was going on.
Navigation
[0] Message Index
[*] Previous page
Go to full version