Someone else solved this problem three years ago but nobody accepted their patch, and now /etc/bash_completion.d/screen has been overhauled and the patch no longer applies.

I updated the patch so it works and resubmitted it to Ubuntu.

If you don’t wanna wait, grab the code from this gist and do this:

sudo patch /etc/bash_completion.d/screen screen.patch

This will probably work on Debian too since that’s where the completion script came from.

The monit configuration stanza (in monitrc) looks like this:

check process mailtunnel with pidfile /var/run/autossh.pid
  start program = "/root/start_mailtunnel.sh"
  stop program = "/root/stop_mailtunnel.sh"
  if changed pid then alert
  if failed host localhost port 2525 protocol smtp with timeout 30 seconds then alert
  if 3 restarts within 5 cycles then timeout

start_mailtunnel.sh looks like this:

#!/bin/sh
export AUTOSSH_PIDFILE=/var/run/autossh.pid
/usr/bin/autossh -N mailtunnel &

And that -N is the key to making ssh happily connect and not run a remote shell, which would want a tty on stdin. That’s how you tell it to just be a port-forwarding connection.

The ssh mailtunnel config just has some port forwarding rules, and “ExitOnForwardFailure yes”.

Finally, stop_mailtunnel.sh looks like this:

#!/bin/sh
test -e /var/run/autossh.pid && kill `cat /var/run/autossh.pid`

But do you know about kill -STOP and kill -CONT?

I’m not sure of the exact mechanism (kernel vs. user process) but everything I’ve used it on on a Mac OS X machine has responded to it in the same way. So you may find a process that doesn’t do this, but I haven’t found one.

Basically kill -STOP 1234 will pause the process with pid 1234, and kill -CONT 1234 will resume it. It’s as if you can sleep individual applications instead of your entire computer.

It’s simple, but you can probably imagine why this would be useful. Want to keep Firefox with 45 tabs open, but you’re not using it for a while and you want to save battery life? Pause it. Pretty much anything like “I don’t want to quit and then re-launch this but I wish it wasn’t using a bunch of CPU time while it idles” is a good candidate.

Remember that this means your app my time out when it is resumed, as if you had slept your laptop. So network connections it had open when you paused it may be broken when you resume it. So pausing Terminal.app with a bunch of open ssh connections to remote servers isn’t going to work too well.

I tried a lot of stuff and finally solved it. My solution is on the Ubuntu Forum, here: One or more of the mounts listed in /etc/fstab/ cannot yet be mounted (Karmic).

I had to re-customize these files since I’m not running with 100% default configuration:

/etc/monit/monit
/etc/monit/monitrc
/etc/dovecot/dovecot.conf
/etc/apache2/apache2.conf
/etc/php/apache2/php.ini

I basically did a manual diff side by side in Emacs and copied my changes over into the new config files. Reboot, no problems. Nice.

I was on “hardy”, a.k.a. the Hardy Heron release, a.k.a. Ubuntu 8.04 LTS.

I had not bothered to install Ubuntu 8.10 / “Intrepid Ibex” because I didn’t have a reason to when it was release. I now wanted to upgrade to Ubuntu 9.04 “Jaunty Jackalope” which has WordPress 2.7.1, the current release (as of today).

The way to upgrade from 8.04 to 9.04 is to upgrade to 8.10 first. So I did that:

Intrepid Upgrades: Network Upgrade for Ubuntu Servers worked really well. I had to do a little bit of manual file merging as usual (I still don’t understand why dpkg can’t merge changes from the old file into a new file) but that was it. Easy!

When I rebooted the VPS, it kernel panicked: can’t mount the root filesystem. Oh crap. /dev/xvda1 is missing? Really? I told the VPS to hard reboot and it came up fine. But that’s a little scary. (I think this is something more related to my VPS hosting provider than Ubuntu, but I haven’t yet upgraded my laptop VMWare Ubuntu VPS’s yet so I’m not sure.)

The second stage didn’t go so well. I did the same sort of simple upgrade: the Jaunty Network Upgrade for Ubuntu Servers instructions are the same as the ones for Intrepid. Upgrade, edit a couple of config files, reboot. Kernel panic again, same reason, reboot. Should work, right?

It booted, but had no network access. I was able to log in via my VPS hosting provider’s SSH remote console feature, so I was able to see that /etc/init.d/networking was failing to start. It was the same problem that’s described in Ubuntu 9.04 in an OpenVZ VE. Adding that one line to /etc/init.d/networking fixed the problem. Reboot, all better.

So if you’re doing this upgrade on a VPS, make sure you’ve added that little 1-line hack after you do the Jaunty upgrade and before you reboot.

There are a couple of config file changes that need babysitting but none of them was difficult; I really do wish it would automatically do a three way merge between its old package version, the new version, and my version, and just assume “yes” if they merge cleanly.

Instructions are trivial: see Hardy Upgrades: Network Upgrade for Ubuntu Servers (Recommended).

This also works fine on Xen.

My main home server is getting old. It was bought in September of 2004, and featured an Athlon64 3500+ processor, 1GB of RAM, and a heap of hard disks. It’s still very fast as a file server, and has 3GB RAM now, so other responsibilities (backups, virus and spam filtering of email, etc.) are handled quite quickly too. However, I’m starting to do more server virtualization for work purposes (test environments with various OS and app configurations) on this server, so I figured it’d be nice to catch up to a more recent CPU that had multiple cores.

The Socket 939 processor line from AMD is almost phased out now. Upgrading to a new Socket AM2 CPU would mean a new motherboard, and based on the churn in the PC industry, that might lead to some serious cascading upgrades.

For example, the motherboard in this system died a couple of years ago, and it turned out that the PC industry switched to a new type of power connector, so I ended up having to also buy a new power supply also. At the same time, I had to rearrange hard disks because some were attached via a pair of PCI ATA-133 cards, but there were only 3 PCI slots on the new motherboard, and I also had a 3ware 8006-LP (SATA RAID) card, and a video card, fighting for room in those 3 slots. Fun!

The PC enthusiast solution to this sort of problem is apparently to buy a shiny new computer every year, full of brand new parts that match. That would be simple, but would cost about five times as much as what I’ve done. I was keen to keep the cascade of surprise upgrades to a minimum, which meant trying to stay with Socket 939, and maintaining the same clock rate.

Luckily the Opteron 175 fit the bill exactly: 2.2GHz, Socket 939, dual core, and about $150. That’s right in my price range for a non-critical but noticeable speed upgrade.

I bought it from Newegg, who by the way rock utterly. It showed up promptly and I opened it up, and… ugh. A new style of plug for the heatsink fan? Hmm, can I use the old heatsink? Maybe… but the new one looks much fancier. (The new one is on the left and the old one is on the right.)

Note the heat pipe:

It was pretty obvious what the copper tubing was for, but I didn’t know about the cool capillary circulation aspect (no pump!) nor that this had become mainstream technology for low-end coolers. Nifty.

Also notice that the new one (in the first photo alongside the old one) has pre-applied thermal paste. It’s the light gray square in the middle. I wiped most of the paste off of the old one, in anticipation of maybe needing to re-apply new paste and use it again due to the 3 vs. 4 pin problem.

Some reading about this plug switchover led me to believe that the 4th pin was optional and that the new plug was designed to fit into old-style plugs as needed. I tried it and it worked fine.

Here’s the socket with no CPU in it; you can see the white 3-pin connector labeled “CPU FAN” in the upper right corner.

And here is a close-up of the 4-pin cable plugged into the 3-pin connector, overhanging a large screw. Note that the right side of the plug has two plastic ridges designed to force you to plug it in the right way. Clever!

Still, if you fail to plug the CPU fan in and the CPU overheats and dies, you void your warranty, so I’m glad I checked before just plugging it in and hoping it’d work.

I thought about the conventional wisdom, which is first that “box coolers” (the CPU coolers that come in the box with CPUs from the manufacturer) are lame, and second that overclocking an Opteron 175 works well. Given that I have no intention of overclocking this thing, I figured that the box cooler would be fine.

After I put the server back together, I ran two instances of burnK7 (from the cpuburn package) for a little over an hour, to keep both cores super busy. burnK7’s mission in life is to make your CPU temperature as hot as possible. The temperature never exceeded 57C for either core, compared to an idle temperature of 37C. The ambient temperature was 69F, and the case is not particularly fancy (2 fans in, 1 out, plus one in the power supply), so I’d have to say that the box cooler and preinstalled thermal paste are fine.

Overclockers may disagree and favor fancier coolers, but from experience I know that in this server, the hard disks are more likely to overheat on a hot day. I added an external desk fan to this server’s rack last year to resolve that problem, which was caused mainly by poor external air circulation.

A couple of things went more easily than I had expected. First, when researching this upgrade I found that (at least for Ubuntu Gutsy, possibly for other Linux releases or distros) there regular AMD64 kernel has SMP support. So, no OS tweaking was required to support the Opteron 175’s dual cores. Second, VMware Server recognized the second CPU and used both CPUs when I ran some tests on two uniprocessor virtual servers. Interestingly, /proc/cpuinfo on a virtualized Linux server shows the right CPU name for what is a dual-core CPU, but only lists one core, which matches the VMware configuration for that virtual server.

VMware did complain in a couple of cases about the different CPU feature set, when resuming suspended VMs. You can dismiss the message and run the VM, and that works, but it warns each time in my case because they are set up as locked snapshots (always restoring from the point where it was originally suspended, before the CPU upgrade).

This is minor, and the fix to make the warnings go away is quick: resume the VM from the old snapshot (dismissing the warning message), unlock the snapshot, make a new snapshot, lock that, and suspend (or power off) the VM. Now when you resume it in the future the warning will not appear. That’s simple to do, but it wasn’t 100% seamless, so I thought I’d mention it. I am glad that there was a warning, though; that would have helped if there had been an actual problem caused by the CPU change. In my opinion VMware does exactly the right thing.

So, all in all this was a very pleasant upgrade; quick, cheap, and effective (for server workloads which parallelize easily, this server is almost twice as fast as before).

BTW, my non-Rimu home server was the first thing I updated. Notes are here.

The only Xen wrinkle is that you need the libc6-xen package. I don’t know if that’s in Feisty or not. In my case I experienced a slew of scary Segmentation Fault errors in the package configuration stage. I found :this post which contains a solution for the problem. But if possible you might want to try installing that package before upgrading. If that doesn’t work, just use the method described in that post.

After doing that and running dpkg --configure -a, it was okay. It did complain about AppArmor and being unable to do something with modules, but that seems to be not germane to Xen so I ignored it and rebooted, and everything seems fine.

I used the instructions on the GutsyUpgrades page of the Ubuntu help wiki, under the heading “Network upgrade for Ubuntu servers (recommended)”. This is the command-line way of doing things, which is appropriate for administering a remote server via SSH. There are two steps: install the updater, and run the updater. That’s pretty slick.

Unfortunately as usual with Debian based releases there are many cases (a dozen or so) where the simple act of updating a package from an old version to a new version causes the installer to plead for human guidance. I find it irritating that if you change the default configuration at all, it cannot simply merge your changes into the new config file and continue. You have to do that manually. This is tedious, but not a really big issue.

In my case I had to manually merge apache2.conf, php.ini, and dovecot.conf because I had made a single line change in each file.

Upon reboot there were vast numbers of error messages, which I ultimately determined to have been caused by this issue. Removing evms solved the problem, and now there’s a special page about evms and why it’s no longer supported. Based on my reading of that page I suspect that there may be a new version of the Gutsy Gibbon updater that actually helps you by suggesting that you get rid of evms proactively to prevent this problem. Nice! Anyway my fix was simply to wait until the system had booted (slowly, with lots of error messages) and to carefully type in apt-get remove evms which I couldn’t even see as I typed because of the flood of error messages. Once that was done that problem went away completely.

I had a separate problem which was that the kernel has re-named the devices for my hard disks, which made a couple of my drives appear not to be recognized by the system. I use Linux Software RAID for a single mirrored volume, and for some reason it couldn’t use the volume UUID to figure out where the drives had gone. After recompiling a custom kernel using these excellent instructions to try and resolve this issue (which turned out to be a red herring), I found out that I was on the wrong track. (But I also found out that it only takes 56 minutes to compile a kernel on my server, which isn’t too bad.) I read dmesg extremely carefully and figured out what had happened. Some work with mdadm told the software RAID manager how to find the components of my RAID mirror and all was well with that. My understanding is that since /etc/fstab includes UUIDs now, drives moving around shouldn’t cause problems like this, so maybe this is a problem specific to the Software RAID system. Other drives moved but were located and mounted properly, with no need for manual intervention.

Finally, I had to update to VMWare Server 1.0.4, which is the first version that supports the kernel included with Gutsy Gibbon (2.6.22-14-server).

Now that all this is behind me I can say that for servers it’s a decent update; I have no experience with it as a desktop. If you have any doubts you can probably skip this one and wait for the next LTS release. But if you do plan to upgrade, just make sure to remove evms first and look in the /dev/sd_ range for any hard drives that seem to have disappeared.