But do you know about kill -STOP and kill -CONT?

I’m not sure of the exact mechanism (kernel vs. user process) but everything I’ve used it on on a Mac OS X machine has responded to it in the same way. So you may find a process that doesn’t do this, but I haven’t found one.

Basically kill -STOP 1234 will pause the process with pid 1234, and kill -CONT 1234 will resume it. It’s as if you can sleep individual applications instead of your entire computer.

It’s simple, but you can probably imagine why this would be useful. Want to keep Firefox with 45 tabs open, but you’re not using it for a while and you want to save battery life? Pause it. Pretty much anything like “I don’t want to quit and then re-launch this but I wish it wasn’t using a bunch of CPU time while it idles” is a good candidate.

Remember that this means your app my time out when it is resumed, as if you had slept your laptop. So network connections it had open when you paused it may be broken when you resume it. So pausing Terminal.app with a bunch of open ssh connections to remote servers isn’t going to work too well.

I tried a lot of stuff and finally solved it. My solution is on the Ubuntu Forum, here: One or more of the mounts listed in /etc/fstab/ cannot yet be mounted (Karmic).

I had to re-customize these files since I’m not running with 100% default configuration:

/etc/monit/monit
/etc/monit/monitrc
/etc/dovecot/dovecot.conf
/etc/apache2/apache2.conf
/etc/php/apache2/php.ini

I basically did a manual diff side by side in Emacs and copied my changes over into the new config files. Reboot, no problems. Nice.

I was on “hardy”, a.k.a. the Hardy Heron release, a.k.a. Ubuntu 8.04 LTS.

I had not bothered to install Ubuntu 8.10 / “Intrepid Ibex” because I didn’t have a reason to when it was release. I now wanted to upgrade to Ubuntu 9.04 “Jaunty Jackalope” which has WordPress 2.7.1, the current release (as of today).

The way to upgrade from 8.04 to 9.04 is to upgrade to 8.10 first. So I did that:

Intrepid Upgrades: Network Upgrade for Ubuntu Servers worked really well. I had to do a little bit of manual file merging as usual (I still don’t understand why dpkg can’t merge changes from the old file into a new file) but that was it. Easy!

When I rebooted the VPS, it kernel panicked: can’t mount the root filesystem. Oh crap. /dev/xvda1 is missing? Really? I told the VPS to hard reboot and it came up fine. But that’s a little scary. (I think this is something more related to my VPS hosting provider than Ubuntu, but I haven’t yet upgraded my laptop VMWare Ubuntu VPS’s yet so I’m not sure.)

The second stage didn’t go so well. I did the same sort of simple upgrade: the Jaunty Network Upgrade for Ubuntu Servers instructions are the same as the ones for Intrepid. Upgrade, edit a couple of config files, reboot. Kernel panic again, same reason, reboot. Should work, right?

It booted, but had no network access. I was able to log in via my VPS hosting provider’s SSH remote console feature, so I was able to see that /etc/init.d/networking was failing to start. It was the same problem that’s described in Ubuntu 9.04 in an OpenVZ VE. Adding that one line to /etc/init.d/networking fixed the problem. Reboot, all better.

So if you’re doing this upgrade on a VPS, make sure you’ve added that little 1-line hack after you do the Jaunty upgrade and before you reboot.

The differences are:

• When you get to the “More Minimizing” section, yum -C grouplist will show a package called “Yum Utilities” which you probably want to leave installed.
• The Deployment_Guide-en-US file is not there so you don’t need to remove it.

That’s it.

I should also note that downloading a 3.9GB DVD ISO image in order to build a ~700MB installed OS may not be very efficient. I didn’t bother looking for a network installer but that might be the way to get this done faster.

There are a couple of config file changes that need babysitting but none of them was difficult; I really do wish it would automatically do a three way merge between its old package version, the new version, and my version, and just assume “yes” if they merge cleanly.

Instructions are trivial: see Hardy Upgrades: Network Upgrade for Ubuntu Servers (Recommended).

This also works fine on Xen.

Sometimes this is helpful. Sometimes this is really annoying and incorrect. Assuming you are a moderately well informed sysadmin and know that this message can safely be ignored, you might have been stumped trying to silence it. You may have tried every option in man ssh_options and even some of your own (STFU on?) I think I may be able to help.

First, let’s make sure we understand the situation. OpenSSH is trying to protect you from an exploit. If you are connecting via SSH to www.foobar.com for the first time, you may be vulnerable to a man in the middle attack. If the forward and reverse DNS don’t match, that might be a sign that this sort of attack is being attempted.

If you’re sure you know what you’re doing, though, and your OpenSSH client is warning you about a situation that you already know about, then try the SSH option:

GSSAPIAuthentication no

(I figured this out by using the -vv option to the SSH command line client. It says Next authentication method: gssapi-with-mic right before the error.)

In my case the client and server are already acquainted via mutual public key exchange (client has known_hosts for the server, and server has authorized_keys for the client) so whether or not the DNS entries look hinky is immaterial. I’m in the process of migrating from one hosting environment to another so I can’t “fix” the DNS situation for another couple of weeks. Meanwhile I’m getting these warnings every hour in an email to root, which is annoying, as it currently forwards to me. And since I’m not using GSSAPI, turning it off is fine also.

Please do make sure you know that your situation is secure before disabling warnings like this. OK now I’ve said it enough times, make the annoying warnings go away.

noatime

First of all, do yourself a favor and disable atime updates, using the noatime mount option. This yields a huge performance boost.

This is done by adding noatime to the appropriate lines in /etc/fstab (do it once for each ext3 filesystem that’s listed), in the fourth column, which probably says defaults now.

To make this change to a live, running filesystem, remount the drive (adjust this so that the right disk device is specified at the end of the line:

sudo mount -o noatime,nodiratime,remount,rw /dev/xvda1

(My understanding is that the noatime implies the nodiratime option, but I decided to add it just in case this was not true.)

atime is a relative of the well known file modification and creation timestamps, but it tracks access to file data. That means that if you read one byte from a file, even if it’s cached in RAM, you’re actually also triggering a write to the directory entry for that file, so that its atime can be updated. (If you want to slap your forehead now in disbelief, be my guest.) And if you read a ton of little files (which happens rather often in the unix world), that means a ton of writes to update all of their directory entries. You don’t want that, right?

But do you need it? Almost certainly not. It’s required by the POSIX standard, and the need for it to be present and turned on is well debated by people more knowledgeable about this in this thread from the Linux kernel mailing list. The summary of their argument is that it’s the kernel’s job to remain standards compliant, and only the distributor or user has enough information to know that they don’t care about that part of the standard and can safely disable it. I can understand that point of view.

Well, I did the reading, and you can safely disable it, unless you’re using mutt. If you’re using mutt, or if you’re just nervous about disabling something that somebody somewhere says you might maybe need someday, then disable atime for every filesystem that doesn’t have your mail spool on it, and use the relatime mode on that drive. (relatime is a clever hack that simulates atime behavior while skipping the disk write in certain cases.)

Journaling mode

Ext3 is a journaling filesystem, which is generally a good thing. There are three modes of operation for ext3’s journaling functionality, but which to use?

“It depends” is not very satisfying, so an easy rule of thumb would be to use data=journal if you really, really want to ensure the durability of your data, and data=ordered if you can tolerate a teeny tiny chance of data corruption.

I measured all three journaling modes by running time sudo rsnapshot hourly on a VPS that backed up VPSs on the same physical server to a dedicated backup disk. In other words, the source was on the same physical server as the destination but they were on different disks.

rsnapshot uses hard links to share file data across backup sets, so backing up an unchanged directory twice takes a hardly any additional space compared to backing it up once. But it does need to do a bunch of disk reads and writes to make all the linked directory entries when it does this, so there is a fair amount of I/O involved: more than what rsync would need to just update a local directory to match the remote directory, but far less than what would be needed to make a separate copy of every file for each backup.

In abstract terms, the I/O for this backup process involves a lot of small reads and writes, and a very small number of medium or large writes for changed files. All of these occur as fast as the disk can service them, and the disk is quiet aside from this activity.

Here’s what I measured (in three test runs per journal type):

For this application, data=journal takes twice as long as the others, while data=ordered runs just as fast as data=writeback while providing some additional protection.

So data=writeback is useless in my case, and the fact that data=ordered is the default makes sense. You get almost the same level of data protection as with data=journal, but with the performance of data=writeback. Different I/O patterns will give different results, but I suspect that the pattern I tested with is the most common in real server usage. (Note that in ext3’s v1 journal format, data=journal was the only journal behavior.)

My inclination is to stick with the default setting, even using data=ordered on database servers, since the database is doing its own higher-level journaling in the form of a transaction log. I’m basing this recommendation on this detail from the Gentoo article:

When appending data to files, data=ordered mode provides all of the integrity guarantees offered by ext3’s full data journaling mode. However, if part of a file is being overwritten and the system crashes, it’s possible that the region being written will contain a combination of original blocks interspersed with updated blocks.

Since a database transaction log is generally appended to rather than overwritten, my understanding is that it will protect against the above scenario in which data=ordered can cause a mix of old and new data. The database’s data files may have a mix of old and new data, but the transaction log would not show that the transaction have been completed yet, so it would be re-run during recovery and the remaining old data would be removed. I think.

The usage pattern where data that you really care about is overwritten regularly (as opposed to logs, which simply append) is rare in my experience, except in the case of database servers which are covered by their own logs as I just mentioned. So I don’t know of a particular application type that demands the full data journaling mode.

Anyway, I recommend against data=writeback altogether, unless you don’t mind some data corruption if there’s a power failure. The speed gain I measured isn’t worth the risk, in my opinion.

Next, you need a way to compare this list to the list of what you have installed. Here’s a command line that I used:

yum list installed | awk 'split($1,a,".") { if (NR>2){ print a[1] } }' \\
> installed_package_names.txt ; diff installed_package_names.txt \\
minimal_package_names.txt  | grep '<' | colrm 1 2

(The awk command is there to strip out the version number and architecture from the package name.)

Now you can run that command and see a list of package names that are not in your minimal_package_names.txt list. You can switch that grep command so it looks for '>' instead of '<', and see things that you consider minimal which are not currently installed.

Then it's just a matter of "yum install foo" and "yum remove foo". I encourage you to use "yum info foo" to make removal decisions one by one, since someone at the ISP probably took the time to research them and thought you might find them useful. You should probably also remove packages in small groups or one by one, because you might be surprised at the dependencies you find. I was surprised to find that uninstalling postgresql-libs would cause httpd (Apache) to be removed as well.

But if you want to automate it, just tack | xargs yum remove on the end of that command, and it will automatically remove them all at once.

Using this as a starting point, you can change your "minimal" packages list to fit your preferences, or even as a quick and dirty alternative to using Kickstart.

The first problem (no init script) is easy to solve, but apparently nobody has done so for CentOS 5.1 and published it. So, here is my CentOS 5.1 init script for the Sphinx Search server. It is known to work with version 0.9.8-rc2.

BTW, the alternative solution to the problem of a daemon not having a System V init script is to just put some extra junk in /etc/rc.local. That is the quick and dirty solution, and is undesirable for several reasons:

1. You can’t easily stop or restart the service, because it’s not a service as far as the OS knows; it’s just some junk in a script that got run a while ago.
2. You can’t use chkconfig or its GUI cousin with the creative name, The Services Configuration Tool, to control it and tie it to specific runlevels.

(System V runlevels and init scripts are useful, even if you don’t need all of the runlevel functionality. The stop/start/restart PID stuff is useful by itself.)