But do you know about kill -STOP and kill -CONT?

I’m not sure of the exact mechanism (kernel vs. user process) but everything I’ve used it on on a Mac OS X machine has responded to it in the same way. So you may find a process that doesn’t do this, but I haven’t found one.

Basically kill -STOP 1234 will pause the process with pid 1234, and kill -CONT 1234 will resume it. It’s as if you can sleep individual applications instead of your entire computer.

It’s simple, but you can probably imagine why this would be useful. Want to keep Firefox with 45 tabs open, but you’re not using it for a while and you want to save battery life? Pause it. Pretty much anything like “I don’t want to quit and then re-launch this but I wish it wasn’t using a bunch of CPU time while it idles” is a good candidate.

Remember that this means your app my time out when it is resumed, as if you had slept your laptop. So network connections it had open when you paused it may be broken when you resume it. So pausing Terminal.app with a bunch of open ssh connections to remote servers isn’t going to work too well.

I tried a lot of stuff and finally solved it. My solution is on the Ubuntu Forum, here: One or more of the mounts listed in /etc/fstab/ cannot yet be mounted (Karmic).

I had to re-customize these files since I’m not running with 100% default configuration:

/etc/monit/monit
/etc/monit/monitrc
/etc/dovecot/dovecot.conf
/etc/apache2/apache2.conf
/etc/php/apache2/php.ini

I basically did a manual diff side by side in Emacs and copied my changes over into the new config files. Reboot, no problems. Nice.

I was on “hardy”, a.k.a. the Hardy Heron release, a.k.a. Ubuntu 8.04 LTS.

I had not bothered to install Ubuntu 8.10 / “Intrepid Ibex” because I didn’t have a reason to when it was release. I now wanted to upgrade to Ubuntu 9.04 “Jaunty Jackalope” which has WordPress 2.7.1, the current release (as of today).

The way to upgrade from 8.04 to 9.04 is to upgrade to 8.10 first. So I did that:

Intrepid Upgrades: Network Upgrade for Ubuntu Servers worked really well. I had to do a little bit of manual file merging as usual (I still don’t understand why dpkg can’t merge changes from the old file into a new file) but that was it. Easy!

When I rebooted the VPS, it kernel panicked: can’t mount the root filesystem. Oh crap. /dev/xvda1 is missing? Really? I told the VPS to hard reboot and it came up fine. But that’s a little scary. (I think this is something more related to my VPS hosting provider than Ubuntu, but I haven’t yet upgraded my laptop VMWare Ubuntu VPS’s yet so I’m not sure.)

The second stage didn’t go so well. I did the same sort of simple upgrade: the Jaunty Network Upgrade for Ubuntu Servers instructions are the same as the ones for Intrepid. Upgrade, edit a couple of config files, reboot. Kernel panic again, same reason, reboot. Should work, right?

It booted, but had no network access. I was able to log in via my VPS hosting provider’s SSH remote console feature, so I was able to see that /etc/init.d/networking was failing to start. It was the same problem that’s described in Ubuntu 9.04 in an OpenVZ VE. Adding that one line to /etc/init.d/networking fixed the problem. Reboot, all better.

So if you’re doing this upgrade on a VPS, make sure you’ve added that little 1-line hack after you do the Jaunty upgrade and before you reboot.

There are a couple of config file changes that need babysitting but none of them was difficult; I really do wish it would automatically do a three way merge between its old package version, the new version, and my version, and just assume “yes” if they merge cleanly.

Instructions are trivial: see Hardy Upgrades: Network Upgrade for Ubuntu Servers (Recommended).

This also works fine on Xen.

My main home server is getting old. It was bought in September of 2004, and featured an Athlon64 3500+ processor, 1GB of RAM, and a heap of hard disks. It’s still very fast as a file server, and has 3GB RAM now, so other responsibilities (backups, virus and spam filtering of email, etc.) are handled quite quickly too. However, I’m starting to do more server virtualization for work purposes (test environments with various OS and app configurations) on this server, so I figured it’d be nice to catch up to a more recent CPU that had multiple cores.

The Socket 939 processor line from AMD is almost phased out now. Upgrading to a new Socket AM2 CPU would mean a new motherboard, and based on the churn in the PC industry, that might lead to some serious cascading upgrades.

For example, the motherboard in this system died a couple of years ago, and it turned out that the PC industry switched to a new type of power connector, so I ended up having to also buy a new power supply also. At the same time, I had to rearrange hard disks because some were attached via a pair of PCI ATA-133 cards, but there were only 3 PCI slots on the new motherboard, and I also had a 3ware 8006-LP (SATA RAID) card, and a video card, fighting for room in those 3 slots. Fun!

The PC enthusiast solution to this sort of problem is apparently to buy a shiny new computer every year, full of brand new parts that match. That would be simple, but would cost about five times as much as what I’ve done. I was keen to keep the cascade of surprise upgrades to a minimum, which meant trying to stay with Socket 939, and maintaining the same clock rate.

Luckily the Opteron 175 fit the bill exactly: 2.2GHz, Socket 939, dual core, and about $150. That’s right in my price range for a non-critical but noticeable speed upgrade.

I bought it from Newegg, who by the way rock utterly. It showed up promptly and I opened it up, and… ugh. A new style of plug for the heatsink fan? Hmm, can I use the old heatsink? Maybe… but the new one looks much fancier. (The new one is on the left and the old one is on the right.)

Note the heat pipe:

It was pretty obvious what the copper tubing was for, but I didn’t know about the cool capillary circulation aspect (no pump!) nor that this had become mainstream technology for low-end coolers. Nifty.

Also notice that the new one (in the first photo alongside the old one) has pre-applied thermal paste. It’s the light gray square in the middle. I wiped most of the paste off of the old one, in anticipation of maybe needing to re-apply new paste and use it again due to the 3 vs. 4 pin problem.

Some reading about this plug switchover led me to believe that the 4th pin was optional and that the new plug was designed to fit into old-style plugs as needed. I tried it and it worked fine.

Here’s the socket with no CPU in it; you can see the white 3-pin connector labeled “CPU FAN” in the upper right corner.

And here is a close-up of the 4-pin cable plugged into the 3-pin connector, overhanging a large screw. Note that the right side of the plug has two plastic ridges designed to force you to plug it in the right way. Clever!

Still, if you fail to plug the CPU fan in and the CPU overheats and dies, you void your warranty, so I’m glad I checked before just plugging it in and hoping it’d work.

I thought about the conventional wisdom, which is first that “box coolers” (the CPU coolers that come in the box with CPUs from the manufacturer) are lame, and second that overclocking an Opteron 175 works well. Given that I have no intention of overclocking this thing, I figured that the box cooler would be fine.

After I put the server back together, I ran two instances of burnK7 (from the cpuburn package) for a little over an hour, to keep both cores super busy. burnK7’s mission in life is to make your CPU temperature as hot as possible. The temperature never exceeded 57C for either core, compared to an idle temperature of 37C. The ambient temperature was 69F, and the case is not particularly fancy (2 fans in, 1 out, plus one in the power supply), so I’d have to say that the box cooler and preinstalled thermal paste are fine.

Overclockers may disagree and favor fancier coolers, but from experience I know that in this server, the hard disks are more likely to overheat on a hot day. I added an external desk fan to this server’s rack last year to resolve that problem, which was caused mainly by poor external air circulation.

A couple of things went more easily than I had expected. First, when researching this upgrade I found that (at least for Ubuntu Gutsy, possibly for other Linux releases or distros) there regular AMD64 kernel has SMP support. So, no OS tweaking was required to support the Opteron 175’s dual cores. Second, VMware Server recognized the second CPU and used both CPUs when I ran some tests on two uniprocessor virtual servers. Interestingly, /proc/cpuinfo on a virtualized Linux server shows the right CPU name for what is a dual-core CPU, but only lists one core, which matches the VMware configuration for that virtual server.

VMware did complain in a couple of cases about the different CPU feature set, when resuming suspended VMs. You can dismiss the message and run the VM, and that works, but it warns each time in my case because they are set up as locked snapshots (always restoring from the point where it was originally suspended, before the CPU upgrade).

This is minor, and the fix to make the warnings go away is quick: resume the VM from the old snapshot (dismissing the warning message), unlock the snapshot, make a new snapshot, lock that, and suspend (or power off) the VM. Now when you resume it in the future the warning will not appear. That’s simple to do, but it wasn’t 100% seamless, so I thought I’d mention it. I am glad that there was a warning, though; that would have helped if there had been an actual problem caused by the CPU change. In my opinion VMware does exactly the right thing.

So, all in all this was a very pleasant upgrade; quick, cheap, and effective (for server workloads which parallelize easily, this server is almost twice as fast as before).

BTW, my non-Rimu home server was the first thing I updated. Notes are here.

The only Xen wrinkle is that you need the libc6-xen package. I don’t know if that’s in Feisty or not. In my case I experienced a slew of scary Segmentation Fault errors in the package configuration stage. I found :this post which contains a solution for the problem. But if possible you might want to try installing that package before upgrading. If that doesn’t work, just use the method described in that post.

After doing that and running dpkg --configure -a, it was okay. It did complain about AppArmor and being unable to do something with modules, but that seems to be not germane to Xen so I ignored it and rebooted, and everything seems fine.

I used the instructions on the GutsyUpgrades page of the Ubuntu help wiki, under the heading “Network upgrade for Ubuntu servers (recommended)”. This is the command-line way of doing things, which is appropriate for administering a remote server via SSH. There are two steps: install the updater, and run the updater. That’s pretty slick.

Unfortunately as usual with Debian based releases there are many cases (a dozen or so) where the simple act of updating a package from an old version to a new version causes the installer to plead for human guidance. I find it irritating that if you change the default configuration at all, it cannot simply merge your changes into the new config file and continue. You have to do that manually. This is tedious, but not a really big issue.

In my case I had to manually merge apache2.conf, php.ini, and dovecot.conf because I had made a single line change in each file.

Upon reboot there were vast numbers of error messages, which I ultimately determined to have been caused by this issue. Removing evms solved the problem, and now there’s a special page about evms and why it’s no longer supported. Based on my reading of that page I suspect that there may be a new version of the Gutsy Gibbon updater that actually helps you by suggesting that you get rid of evms proactively to prevent this problem. Nice! Anyway my fix was simply to wait until the system had booted (slowly, with lots of error messages) and to carefully type in apt-get remove evms which I couldn’t even see as I typed because of the flood of error messages. Once that was done that problem went away completely.

I had a separate problem which was that the kernel has re-named the devices for my hard disks, which made a couple of my drives appear not to be recognized by the system. I use Linux Software RAID for a single mirrored volume, and for some reason it couldn’t use the volume UUID to figure out where the drives had gone. After recompiling a custom kernel using these excellent instructions to try and resolve this issue (which turned out to be a red herring), I found out that I was on the wrong track. (But I also found out that it only takes 56 minutes to compile a kernel on my server, which isn’t too bad.) I read dmesg extremely carefully and figured out what had happened. Some work with mdadm told the software RAID manager how to find the components of my RAID mirror and all was well with that. My understanding is that since /etc/fstab includes UUIDs now, drives moving around shouldn’t cause problems like this, so maybe this is a problem specific to the Software RAID system. Other drives moved but were located and mounted properly, with no need for manual intervention.

Finally, I had to update to VMWare Server 1.0.4, which is the first version that supports the kernel included with Gutsy Gibbon (2.6.22-14-server).

Now that all this is behind me I can say that for servers it’s a decent update; I have no experience with it as a desktop. If you have any doubts you can probably skip this one and wait for the next LTS release. But if you do plan to upgrade, just make sure to remove evms first and look in the /dev/sd_ range for any hard drives that seem to have disappeared.

The short version:
- Ruby is fun to program in, as you’ve probably heard
- Rails is over-hyped, but it’s still quite good (definitely not perfect)
- I like the productivity of Ruby on Rails but I wouldn’t call it a silver bullet by any means
- Ruby performance was bad and is getting less bad, and can even be good if you do what the experts say
- The real gem (har har) in the Ruby and Rails space is the community itself

Now for the longer version:

It’s hard for me to be objective since I only started programming in Ruby in February and started using Rails in April, but the language and framework don’t strike me as tedious or kludgy at all. I’m definitely enjoying Ruby and making use of its fancier language features, all of which seem to actually work as advertised without any hideous gotchas when you actually try to do something slick.

I particularly enjoy the way Ruby’s collections allow you to write in the style of a functional language; have a look at the Array, Enumerable, and Hash classes to see what I mean. It’s possible to write extremely compact code that morphs a collection into something very different, but without the inscrutability of Perl.

Compared to my experience with mainstream Java and J2EE, I certainly don’t miss Tomcat, Ant, Struts, or JSP Expression Language. (Instead I’m using Mongrel, Rake, Rails, and ERB.) That’s not Java’s fault, and could be alleviated by just picking better tools and frameworks at the start of a project. I don’t miss Hibernate but I don’t have any gripes about it either.

There are features in Java 1.5 that are helpful in alleviating some of the annoyances of programming in Java, but because these are new features, the vast majority of library code in the wild doesn’t use them (similar to the problems with OO being bolted onto Perl and PHP). Also there are political and business deployment roadblocks to using the latest technology: just because you can download an official release of something and it works perfectly for your code, that doesn’t mean that you necessarily have permission to deploy that technology stack onto production servers. Big organizations take years to officially support new releases, so you may be stuck developing against 3 year old products and APIs.

I’m not trying to troll for Java vs Ruby comment flames here. My point is just that Ruby has had features for a relatively long time that Java is just recently adopting (or simulating), and that’s important for more than just petty bragging rights; a feature that’s unavailable to you for your own org’s political or financial reasons might as well not be there at all. (Or maybe it’s worse, if it makes you bitter about your own work environment: “I have to write all this ugly stupid code because Bob is afraid to upgrade the server… grr.”)

I can remember quite a few times when I painted myself into a corner in Java and had to write lots of type casting / checking / converting code, or had to try and work around single inheritance using composition and lots of boilerplate proxy methods, or really wished java.lang.String weren’t final so I could subclass it, etc. Ruby feels a lot more like Perl or JavaScript than Java when it comes to being able to do whatever you want when you’re sure you know what you’re doing, and being able to write weird fancy code that magically makes everything that calls it become dead simple.

I used to be horrified at the idea of messing with somebody else’s classes (as is done by Hash.slice from will_paginate, which I swiped all by itself since I don’t really need will_paginate), but I’m finding that the apocalypse I feared just isn’t there. If you do something wacky, like add a new method to String, the world doesn’t collapse. Or maybe it does, and you decide not to do it that particular way; just do something else. ActiveSupport adds all kinds of stuff to base classes like String, and it’s OK. Example: ActiveSupport::CoreExtensions::String::Inflections.

I’m still adjusting to using a dynamic language: not uber-checking arguments and state all the way down a method call stack (oh no! is it nil?), not locking down everything so it can only work the way I planned, etc. In most cases, the Ruby equivalent of a NullPointerException is fine. I think this is because the built-in error message includes the type and method you were trying to call, so you can usually just read that and see what you did wrong. The only time you’d want to prevent that is if you bury that error deep in your own code, so that the caller would have no idea of what they did that led to it. I’m finding that I’m OK with writing less paranoid code, because the failure will still happen, and be suitably clear in its cause, without my explicitly checking for it.

An issue with Ruby and Rails is that the official documentation is limited. It’s imperative to have 2 or 3 books on hand at all times, in addition to the RDoc Rails documentation. I own three of the Pragmatic Programmer books: Programming Ruby (the “pickaxe book”), Agile Web Development with Rails, 2nd Edition, and Rails Recipes. Just looking at generated API documentation won’t tell you how you’re supposed to use libraries.

The reverse way of looking at that need for documentation is symbol lookup. Rails does some pre-including of library and framework files before loading the application file you’re working on, so unlike Java, you don’t have a mandatory list of imports at the top that tell you where every class you’re working with came from. There are some symbols and functions you can call from instances of particular classes, and they came from somewhere but it’s not obvious where, at least when you’re starting. A good IDE with “find declaration” could fix this, but initially it’s daunting: “what are *all* the options for this method?” “Where is this method defined, even?” etc. This is where the books are less useful and API documentation can help.

I’ve also subscribed to a couple dozen RSS feeds that help me keep up with what the really slick Ruby and Rails people are using, writing, struggling with, etc. Chief among them is probably Err the Blog. The Ruby on Rails podcast is helpful also.

As for productivity, here are 4 specific time-savers I’ve found:

1) TextMate, which is very slick. There are Windows equivalents, though from what I’ve heard, they’re not 100% as good. Point is, a text editor that understands Ruby is critical, and if it helps you jump around in a big tree of files, that’s very helpful.

I’m an Emacs devotee, and TextMate is far better for Ruby and Rails. It’s hard to explain, but let’s just say that TextMate is much deeper and more featureful than it appears at first glance.

I looked at NetBeans which is pretty close to being a very smart Ruby & Rails IDE, but it was still suffering an identity crisis (am I a Java IDE? or a multi-language thing?) that cluttered the UI badly. This is a hot area of innovation right now so any week now TextMate might be eclipsed (or Eclipsed, har har) by something more full-featured. There are a few other options but I didn’t see anything that led me to believe a multi-hour or multi-day eval would be worth it.

2) Locomotive. It’s like MAMP or WAMP or any of those all-in-one application stack installers, except it gives you a working Rails environment. I set up PostgreSQL separately thru MacPorts (a cousin of the FreeBSD ports project), which was somewhat tedious but decently documented.

Getting all the Ruby and Rails stuff compiled from source is no picnic, and frankly unnecessary. On Ubuntu there’s a gotcha: the Ubuntu people and the RubyGems people can’t agree on a file layout, so Ubuntu’s rubygems package sucks. It’s better to install Ruby using apt-get and then install gems manually from the tarball.

3) Rails Plugins. There’s a lot of stuff being built on top of Rails and then improved day by day, and svn-over-WebDAV is the popular sync protocol. There’s a tool called Piston that helps you manage this. Anyway there’s a lot of stuff that is hard to do and commonly needed which is available in plugin form.

4) ZenTest, and more specifically, Autotest. It watches your source tree and runs only the tests in your test suite that need to be run based on what has changed; it’s the dynamic-language version of the incremental compilation that’s present in any good Java IDE.

I had a few packages installed from the Feisty Fawn universe and multiverse repositories (wordpress, clamav) on top of my Edgy Eft installation, and the mixed version setup worked well. Clamav was updated to depend on a library (libc, I think) that was not available in Edgy Eft, and Apt was smart enough to just leave clamav at the existing version. Also, I had made quite a few local customizations to configuration files for dovecot imapd (enabling non-secure IMAP connectons since they only happen over my home LAN), /etc/services (adding VMWare’s admin service and a second Subversion repository), and a couple of others.

I started by changing ‘edgy’ to ‘feisty’ in /etc/apt/sources.list, clearing out the old CD-ROM source line (from my original installation from ISO images) and duplicate universe and multiverse lines (from my mixed installation).

I then did apt-get update, and apt-get -d dist-upgrade (-d means download but don’t install) and let that run (300+ MB of stuff to download, might as well let it run while I sleep).

This morning I ran apt-get dist-upgrade and babysat it. There were a couple of things that it said it would uninstall that I had to reinstall: libapache2-mod-python (needed by Trac), ruby18-elisp (not automatically upgraded to ruby19-elisp, but that was easy to manually install), and libnet-perl, which appears to be broken currently. That last one would seem to be the first of two genuinely broken things in this upgrade, though I can’t think of anything I use that depends on it.

There were a half dozen or so manual config file merges that I had to do; I wish one of the options were to attempt an automatic merge of the existing, customized version and the new, vanilla version. Nothing I changed really required manual intervention; all I did was what patch could have done just as easily. Tedious, but not really a problem. This effort probably added 5 minutes total.

After all that, it rebooted cleanly but Trac doesn’t work anymore. Apparently Trac depends on Clearsilver which reportedly is an antique and obsolete template system that isn’t currently built for Python 2.5, which is part of ubuntu-minimal on Feisty. I have a lead on fixing this manually but I haven’t done it yet. I expect this to be fixed for me in an official package in the next week or less; several folks know about it and say the fix is percolating through the package system already. So I might not bother with the manual fix.

The reboot went fine. I then updated VMware Server, which required a bit of manual assistance, but then it always does. Here’s what I did: VMWare Server in Feisty.

I haven’t upgraded my VPS yet but I’ll do that soon. I want to check with my VPS hosting support folks first to see if they know of any gotchas.