I tried a lot of stuff and finally solved it. My solution is on the Ubuntu Forum, here: One or more of the mounts listed in /etc/fstab/ cannot yet be mounted (Karmic).

I had to re-customize these files since I’m not running with 100% default configuration:

/etc/monit/monit
/etc/monit/monitrc
/etc/dovecot/dovecot.conf
/etc/apache2/apache2.conf
/etc/php/apache2/php.ini

I basically did a manual diff side by side in Emacs and copied my changes over into the new config files. Reboot, no problems. Nice.

I found a good starting point at geokit stubbing for faster tests. I also wanted a way to stub a geocoding failure, and a way to prevent any unit tests from using the real geocoding web service.

Here’s how I did it.

In test/test_helper.rb, add this:

class ActiveSupport::TestCase
  def setup
    GeoKit::Geocoders::MultiGeocoder.stubs(:geocode).raises(RuntimeError,
      'Use mock_geocoding_success! or mock_geocoding_failure! in your test')
  end

  def mock_geocoding_success!
    geocode_payload = GeoKit::GeoLoc.new(:lat => 123.456, :lng => 123.456)
    geocode_payload.success = true
    GeoKit::Geocoders::MultiGeocoder.expects(:geocode).returns(geocode_payload)
  end

  def mock_geocoding_failure!
    geocode_payload = GeoKit::GeoLoc.new
    geocode_payload.success = false
    GeoKit::Geocoders::MultiGeocoder.expects(:geocode).returns(geocode_payload)
  end
end

What this does is to force you to choose either one of those mock_geocoding methods before you call the geocode method. To me this seems like a good idea since the integration tests that exercise the full application stack should probably be written using Cucumber and Webrat (which is what I’m using).

You will probably want to merge my one-line setup method into your existing setup code in test_helper, if any. Also note that this uses Mocha for mocking.

I was on “hardy”, a.k.a. the Hardy Heron release, a.k.a. Ubuntu 8.04 LTS.

I had not bothered to install Ubuntu 8.10 / “Intrepid Ibex” because I didn’t have a reason to when it was release. I now wanted to upgrade to Ubuntu 9.04 “Jaunty Jackalope” which has WordPress 2.7.1, the current release (as of today).

The way to upgrade from 8.04 to 9.04 is to upgrade to 8.10 first. So I did that:

Intrepid Upgrades: Network Upgrade for Ubuntu Servers worked really well. I had to do a little bit of manual file merging as usual (I still don’t understand why dpkg can’t merge changes from the old file into a new file) but that was it. Easy!

When I rebooted the VPS, it kernel panicked: can’t mount the root filesystem. Oh crap. /dev/xvda1 is missing? Really? I told the VPS to hard reboot and it came up fine. But that’s a little scary. (I think this is something more related to my VPS hosting provider than Ubuntu, but I haven’t yet upgraded my laptop VMWare Ubuntu VPS’s yet so I’m not sure.)

The second stage didn’t go so well. I did the same sort of simple upgrade: the Jaunty Network Upgrade for Ubuntu Servers instructions are the same as the ones for Intrepid. Upgrade, edit a couple of config files, reboot. Kernel panic again, same reason, reboot. Should work, right?

It booted, but had no network access. I was able to log in via my VPS hosting provider’s SSH remote console feature, so I was able to see that /etc/init.d/networking was failing to start. It was the same problem that’s described in Ubuntu 9.04 in an OpenVZ VE. Adding that one line to /etc/init.d/networking fixed the problem. Reboot, all better.

So if you’re doing this upgrade on a VPS, make sure you’ve added that little 1-line hack after you do the Jaunty upgrade and before you reboot.

The differences are:

• When you get to the “More Minimizing” section, yum -C grouplist will show a package called “Yum Utilities” which you probably want to leave installed.
• The Deployment_Guide-en-US file is not there so you don’t need to remove it.

That’s it.

I should also note that downloading a 3.9GB DVD ISO image in order to build a ~700MB installed OS may not be very efficient. I didn’t bother looking for a network installer but that might be the way to get this done faster.

noatime

First of all, do yourself a favor and disable atime updates, using the noatime mount option. This yields a huge performance boost.

This is done by adding noatime to the appropriate lines in /etc/fstab (do it once for each ext3 filesystem that’s listed), in the fourth column, which probably says defaults now.

To make this change to a live, running filesystem, remount the drive (adjust this so that the right disk device is specified at the end of the line:

sudo mount -o noatime,nodiratime,remount,rw /dev/xvda1

(My understanding is that the noatime implies the nodiratime option, but I decided to add it just in case this was not true.)

atime is a relative of the well known file modification and creation timestamps, but it tracks access to file data. That means that if you read one byte from a file, even if it’s cached in RAM, you’re actually also triggering a write to the directory entry for that file, so that its atime can be updated. (If you want to slap your forehead now in disbelief, be my guest.) And if you read a ton of little files (which happens rather often in the unix world), that means a ton of writes to update all of their directory entries. You don’t want that, right?

But do you need it? Almost certainly not. It’s required by the POSIX standard, and the need for it to be present and turned on is well debated by people more knowledgeable about this in this thread from the Linux kernel mailing list. The summary of their argument is that it’s the kernel’s job to remain standards compliant, and only the distributor or user has enough information to know that they don’t care about that part of the standard and can safely disable it. I can understand that point of view.

Well, I did the reading, and you can safely disable it, unless you’re using mutt. If you’re using mutt, or if you’re just nervous about disabling something that somebody somewhere says you might maybe need someday, then disable atime for every filesystem that doesn’t have your mail spool on it, and use the relatime mode on that drive. (relatime is a clever hack that simulates atime behavior while skipping the disk write in certain cases.)

Journaling mode

Ext3 is a journaling filesystem, which is generally a good thing. There are three modes of operation for ext3’s journaling functionality, but which to use?

“It depends” is not very satisfying, so an easy rule of thumb would be to use data=journal if you really, really want to ensure the durability of your data, and data=ordered if you can tolerate a teeny tiny chance of data corruption.

I measured all three journaling modes by running time sudo rsnapshot hourly on a VPS that backed up VPSs on the same physical server to a dedicated backup disk. In other words, the source was on the same physical server as the destination but they were on different disks.

rsnapshot uses hard links to share file data across backup sets, so backing up an unchanged directory twice takes a hardly any additional space compared to backing it up once. But it does need to do a bunch of disk reads and writes to make all the linked directory entries when it does this, so there is a fair amount of I/O involved: more than what rsync would need to just update a local directory to match the remote directory, but far less than what would be needed to make a separate copy of every file for each backup.

In abstract terms, the I/O for this backup process involves a lot of small reads and writes, and a very small number of medium or large writes for changed files. All of these occur as fast as the disk can service them, and the disk is quiet aside from this activity.

Here’s what I measured (in three test runs per journal type):

For this application, data=journal takes twice as long as the others, while data=ordered runs just as fast as data=writeback while providing some additional protection.

So data=writeback is useless in my case, and the fact that data=ordered is the default makes sense. You get almost the same level of data protection as with data=journal, but with the performance of data=writeback. Different I/O patterns will give different results, but I suspect that the pattern I tested with is the most common in real server usage. (Note that in ext3’s v1 journal format, data=journal was the only journal behavior.)

My inclination is to stick with the default setting, even using data=ordered on database servers, since the database is doing its own higher-level journaling in the form of a transaction log. I’m basing this recommendation on this detail from the Gentoo article:

When appending data to files, data=ordered mode provides all of the integrity guarantees offered by ext3’s full data journaling mode. However, if part of a file is being overwritten and the system crashes, it’s possible that the region being written will contain a combination of original blocks interspersed with updated blocks.

Since a database transaction log is generally appended to rather than overwritten, my understanding is that it will protect against the above scenario in which data=ordered can cause a mix of old and new data. The database’s data files may have a mix of old and new data, but the transaction log would not show that the transaction have been completed yet, so it would be re-run during recovery and the remaining old data would be removed. I think.

The usage pattern where data that you really care about is overwritten regularly (as opposed to logs, which simply append) is rare in my experience, except in the case of database servers which are covered by their own logs as I just mentioned. So I don’t know of a particular application type that demands the full data journaling mode.

Anyway, I recommend against data=writeback altogether, unless you don’t mind some data corruption if there’s a power failure. The speed gain I measured isn’t worth the risk, in my opinion.

On Linux this is accomplished with hdparm:

sudo hdparm -S 6 /dev/sda

The numeric value is on a nonlinear scale, so man hdparm and read about the -S option to make sure you pick the value you intended.

You can check the current status (active or standby) like this:

sudo hdparm -C /dev/sda

Gotchas:
External drives aren’t managed directly by the kernel, so hdparm won’t work on them. My hardware RAID card won’t pass through these commands. Linux RAID will not pass these commands through, though you may be able to apply the commands individually to RAID set members (which is what I was able to do).

I don’t know how to do this on a per-drive basis on Mac OS X (pmset apparently affects every drive on the system), or at all on Windows. If you know feel free to post a comment explaining how it’s done.

Next, you need a way to compare this list to the list of what you have installed. Here’s a command line that I used:

yum list installed | awk 'split($1,a,".") { if (NR>2){ print a[1] } }' \\
> installed_package_names.txt ; diff installed_package_names.txt \\
minimal_package_names.txt  | grep '<' | colrm 1 2

(The awk command is there to strip out the version number and architecture from the package name.)

Now you can run that command and see a list of package names that are not in your minimal_package_names.txt list. You can switch that grep command so it looks for '>' instead of '<', and see things that you consider minimal which are not currently installed.

Then it's just a matter of "yum install foo" and "yum remove foo". I encourage you to use "yum info foo" to make removal decisions one by one, since someone at the ISP probably took the time to research them and thought you might find them useful. You should probably also remove packages in small groups or one by one, because you might be surprised at the dependencies you find. I was surprised to find that uninstalling postgresql-libs would cause httpd (Apache) to be removed as well.

But if you want to automate it, just tack | xargs yum remove on the end of that command, and it will automatically remove them all at once.

Using this as a starting point, you can change your "minimal" packages list to fit your preferences, or even as a quick and dirty alternative to using Kickstart.

With PHP it’s just another Apache module, but you might need to build a few extra libraries and maybe custom-compile Apache. Once you get it up and running, though, you don’t even need to restart the server when you deploy new code. It’s automatically updated.

With Ruby on Rails, it has been far uglier, especially as you go further back. The standard “Matz Ruby Interpreter” (MRI) doesn’t thread well and is quite remarkably slow, and Ruby + Rails in an MRI process use a lot lot lot of memory. So you don’t really want RoR running inside each Apache process. Folks used to use FastCGI (which should have died over a decade ago, but lingers on like a bad cold) but now use Mongrel, which is conceptually kind of like FastCGI, except that it actually works. Mongrel presents the application via HTTP, which is much easier to understand and integrate with other parts of your architecture (such as a load balancer) than FastCGI.

Whereas in J2EE you’d run one big honkin’ JVM that used lots of memory to load up your code and data structures, but then ran many threads inside that one process, with the limitations of the MRI (green threads and many, many trips into non thread safe C code that requires the use of a “giant lock” that essentially makes it single-threaded), you run one process per thread. That’s like Apache+PHP or OpenSSH or many other unix programs that fork, right? Well, sort of. The issue is that your Ruby code is not seen by the kernel as something that all those forked processes can share; it sees the parsed Ruby code as data, and when the MRI’s garbage collector marks all those objects during garbage collection, it seems this data as being recently changed, differently for each forked process. So not only do you need 30-70MB or more per process, but very little of that is shared between processes. Ouch!

A second problem is that these processes take a while to start up and load the code, so it’s not reasonable to embed the Ruby interpreter in Apache when using Rails; the overhead is just too high. So the Mongrel solution is to pre-launch a bunch of interpreters, and have them just sit there until requests arrive. That’s pretty inefficient from a memory standpoint, but the latency when a request comes in is quite low since there is no initialization needed.

There have been a few interesting alternatives under development: JRuby is very promising, because it reuses all of the investment in VM development that Sun made over the last 10+ years for Java. At this point the JVM is pretty darn good at running many threads across multiple CPU cores, and at garbage collecting efficiently, among other things. These are key weaknesses of MRI, so running Rails on JRuby seems like a huge benefit. I haven’t tried it yet but I suspect that this will become one of the 2 or 3 most common ways to run Rails applications in the near future.

Another interesting alternative was some experimental hacking to MRI’s garbage collector by Hongli Lai, to store its working data separately from the objects being examined, so that preloaded Ruby code would remain shared by many forked interpreter processes over long periods of time. In other words, this is a potentially major memory use savings for Mongrel cluster users, which would in turn allow the sysadmin to run more Mongrels to service more simultaneous requests, or to bump up the database cache, or to increase the size of the running memcached instance. So, this would indirectly be a performance booster, and Ruby could really use that.

This experimentation apparently became Ruby Enterprise Edition, which as of this writing is not available yet. But the other development coming from Hongli Lai’s new company, Phusion, is Passenger, a.k.a. mod_rails.

What’s interesting about mod_rails for the beginning Rails developer is that it is intended to make Rails hosting easier, particularly for shared hosting enviroments, which have been struggling to offer Rails hosting in a uniform and cost-effective fashion. That means that in a short while (weeks?), shared hosting plans for fiddling around with Rails will become much cheaper and more widely available than they are now.

What’s interesting about mod_rails for the experienced sysadmin is that it mimics the min/max process pooling behavior of Apache, and addresses startup overhead in a clever way. It also serves static images via Apache automatically, eliminating the need for a separate block of mod_rewrite rules that must be crafted carefully so as to avoid conflicts with mod_proxy.

The architectural overview is comprehensive and well written, but here’s a summary: The Spawn Server makes a tree of child processes that preloads Ruby, Rails, and your application code for you, and then that is fork()ed to satisfy incoming requests. So the first request after startup incurs startup overhead (in my case, 5 seconds to load the Redmine login page) but subsequent requests get much better response time (.6s to reload that login page).

That seems like a lot of overhead in terms of big Ruby processes. Here’s what I measured just now: 97MB free with just Apache running (no spawn server yet). After the first page view, there was 36MB free, and four new processes: the Spawn Server taking a little over 6MB (rsize), the FrameworkSpawner taking 20MB (rsize), the ApplicationSpawner taking 34MB (rsize), and one Rails process taking 34MB (rsize).

The new “free” value is 36MB. The Buffers and used Swap values remained constant, with only 48KB of swap used. So that means that all four processes, which would seem to need 94MB to run (34+34+20+6), are actually overlapping enough that they are using only 61MB (97-36). And the ApplicationSpawner eventually terminates, leaving 36MB still free, which makes sense – it’s the process that fork()ed the Rails process, so they should ideally be overlapping nearly 100%. I’m surprised that this is so high; based on the GC experimentation that Hongli Lai did, I would have expected them not to overlap as much.

The idle Rails process exits eventually also, controlled by the RailsPoolIdleTime setting. That saves memory but re-introduces the startup overhead. That leaves the FrameworkSpawner and the SpawnServer running, taking about 25MB of memory (quite close to the 20+6 shown by their rsize values).

Let’s compare this memory footprint to a Mongrel cluster. In a Mongrel cluster the processes start up and stay running forever, so the users are unlikely to incur much startup overhead at all, since it’s done long before they visit the application. Some amount of application-specific internal overhead is still an issue, though; that might include gradually filling an initially empty memcached, template compilation and/or caching, etc. As for memory, each Mongrel would need the same 34MB of memory, but there’s no SpawnServer, FrameworkServer, or ApplicationServer, so the extra 25MB of overhead would not be present with a Mongrel cluster.

That means that for a shared hosting setup where many low-traffic Rails sites may be used, or a multifunction server where serving one or more low-traffic Rails applications is just part of the job, mod_rails is a benefit. When the Rails app isn’t being used, it will exit and free up that memory for other processes. The starting and stopping of Rails with mod_rails is automatic and demand-based, so the sysadmin can tune it and forget about it.

On the other hand, a single dedicated server or VPS with a fixed amount of memory serving a single application would be better off with Mongrel, because of the lower memory overhead (25MB less), and the fact that the Mongrel processes start up before users need them and stay running indefinitely. Mongrel clusters could still potentially benefit from the Ruby Enterprise Edition’s garbage collector tweak if forking were used after preloading all of the code.

A single-purpose dedicated server running mod_rails could attain similar performance to a Mongrel cluster by simply setting the RailsPoolIdleTime value to a very high number. Then the Rails processes would hang around, and although you’d pay the price of a 25MB memory overhead, the startup overhead would only be paid by the very first visitor. However, you’d lose the main benefit of mod_rails, which is demand-based pool resizing, particularly if you’re running more than one application, Rails version, or Ruby interpreter version.

In short, I think mod_rails is very nice, and having actually used it I’m impressed with how polished it is for a 1.0 product. But if you’re already running a single application as a Mongrel cluster on a dedicated server, there’s no point in switching.

The first problem (no init script) is easy to solve, but apparently nobody has done so for CentOS 5.1 and published it. So, here is my CentOS 5.1 init script for the Sphinx Search server. It is known to work with version 0.9.8-rc2.

BTW, the alternative solution to the problem of a daemon not having a System V init script is to just put some extra junk in /etc/rc.local. That is the quick and dirty solution, and is undesirable for several reasons:

1. You can’t easily stop or restart the service, because it’s not a service as far as the OS knows; it’s just some junk in a script that got run a while ago.
2. You can’t use chkconfig or its GUI cousin with the creative name, The Services Configuration Tool, to control it and tie it to specific runlevels.

(System V runlevels and init scripts are useful, even if you don’t need all of the runlevel functionality. The stop/start/restart PID stuff is useful by itself.)