The differences are:

• When you get to the “More Minimizing” section, yum -C grouplist will show a package called “Yum Utilities” which you probably want to leave installed.
• The Deployment_Guide-en-US file is not there so you don’t need to remove it.

That’s it.

I should also note that downloading a 3.9GB DVD ISO image in order to build a ~700MB installed OS may not be very efficient. I didn’t bother looking for a network installer but that might be the way to get this done faster.

Assumptions:

• this server uses an x86_64 CPU architecture, not i386 (or for an ultra-small disk footprint, consider i386 since it doesn’t need duplicate 64 and 32 bit glibc libraries)
• this is a server VPS (running in VMware or Xen) so it isn’t interested in low-level hardware management
• no need for ACPI (sleep/hibernate)
• no need for laptop CPU power reduction
• no need for SMART disk monitoring (since the VPS disk is virtualized)
• no need for MD (software RAID) since it’s a VPS; any RAID is happening at a lower level (host OS / dom0)
• no need to use LVM2 to mirror a logical volume (again because RAID is handled outside of the VPS)
• no need for bluetooth, hot-plug hardware, or PCMCIA
• this server may have more than one CPU (or may be given additional VCPUs later due to load) so multi-CPU support is desired
• this is a headless server so no GUI features are desirable
• there are no legacy services that need RPC
• NFS will not be used
• SELinux will be left in the default configuration (”Enforcing” the “Targeted” policy).

Basic Installation:

Start with the Centos 5.1 x86_64 install DVD.
Boot the DVD.
Select English language and U.S. English keyboard layout.
Choose to Install the OS (not upgrade).
Choose “Remove linux partitions on selected drives and create default layout.”
Select DHCP network configuration, or the static IP address for this server.
(I choose DHCP, and tell the DHCP server to use a specific IP for this host based on its Ethernet MAC address.)
Choose the time zone the server is in (for me this is America/Los Angeles), and enable the “System clock uses UTC” option.
Pick a complex root password (https://grc.com/passwords can generate one for you) and enter it.
When given a chance to install additional tasks, uncheck everything (no additional tasks) and choose the “Customize now” radio button.
In the next screen, go into every group and uncheck everything. (*Nothing* should be checked when you’re done. Be careful not to miss anything!)
Confirm that you want the installer to begin the installation process. (For me this process took about 6 minutes.)
Confirm that you want to reboot, and make sure that the server will boot from the hard disk instead of the installation media.
When the server boots, log in as root. (You can disconnect from the console and use SSH instead at this point if it’s more convenient.)

More Minimizing:
Run this command to tell yum to go grab the latest package info from out on the internet.
yum grouplist
Run this command to make sure you didn’t install anything other than the bare minimum:
yum -C grouplist
You shouldn’t see a section called “Installed Groups:”. If you do see it, it means you missed something you were supposed to disable in the previous section.
In that case, run this to remove it and all the packages in it:
yum -C groupremove SomeGroupName

You can also run this command to count how many packages have been installed already:
yum -C list installed | wc -l
The resulting count of packages installed should be 154.

Next, run “chkconfig --list | grep 3:on” to see what services are enabled.
Several of them can safely be disabled (provided that the assumptions at the top of this guide are true), so run this:
for i in haldaemon lvm2-monitor messagebus netfs; do chkconfig $i off; done
These are useful and should stay enabled:
ip6tables
iptables
kudzu
mcstrans
network
restorecond
sshd
syslog
If for some reason there are others that are still enabled that aren’t on that list, you’ll have to decide for yourself.

Run this command to remove a 9.7MB standalone documentation package that you almost certainly won’t be reading from the server:
yum -C remove Deployment_Guide-en-US

If you wish, run this command to update your installed packages to the latest stable version.
yum update
As of 3/27/2008 this installed a new kernel, updated 19 other packages, and required a 54MB download.
Accept the CentOS package-signing GPG key when asked.
If a kernel update is installed, it would be a good idea to reboot soon to make sure it works.
You can also remove your old kernel (use “rpm -qa | grep kern” to find old ones) to save ~75MB.
yum remove kernel-2.6.18-53.el5 NOTE! ONLY do this if you updated your kernel and have 2 installed now.

Useful Things You May Want To Install:

These packages are very useful for administering servers, deploying software and data to them, and performing backups:
yum install bzip2 lsof man man-pages mlocate quota rsync sysstat vixie-cron wget which

If your server has more than 1 CPU you may wish to install irqbalance, to distribute interrupt servicing duty across CPUs:
yum install irqbalance

This package makes the system boot slightly faster using a very simple, safe technique:
yum install readahead

If you aren’t familiar with the vim text editor, you can install nano, which is less powerful but very easy to use:
yum install nano

Have fun! Hope this helps.

My main home server is getting old. It was bought in September of 2004, and featured an Athlon64 3500+ processor, 1GB of RAM, and a heap of hard disks. It’s still very fast as a file server, and has 3GB RAM now, so other responsibilities (backups, virus and spam filtering of email, etc.) are handled quite quickly too. However, I’m starting to do more server virtualization for work purposes (test environments with various OS and app configurations) on this server, so I figured it’d be nice to catch up to a more recent CPU that had multiple cores.

The Socket 939 processor line from AMD is almost phased out now. Upgrading to a new Socket AM2 CPU would mean a new motherboard, and based on the churn in the PC industry, that might lead to some serious cascading upgrades.

For example, the motherboard in this system died a couple of years ago, and it turned out that the PC industry switched to a new type of power connector, so I ended up having to also buy a new power supply also. At the same time, I had to rearrange hard disks because some were attached via a pair of PCI ATA-133 cards, but there were only 3 PCI slots on the new motherboard, and I also had a 3ware 8006-LP (SATA RAID) card, and a video card, fighting for room in those 3 slots. Fun!

The PC enthusiast solution to this sort of problem is apparently to buy a shiny new computer every year, full of brand new parts that match. That would be simple, but would cost about five times as much as what I’ve done. I was keen to keep the cascade of surprise upgrades to a minimum, which meant trying to stay with Socket 939, and maintaining the same clock rate.

Luckily the Opteron 175 fit the bill exactly: 2.2GHz, Socket 939, dual core, and about $150. That’s right in my price range for a non-critical but noticeable speed upgrade.

I bought it from Newegg, who by the way rock utterly. It showed up promptly and I opened it up, and… ugh. A new style of plug for the heatsink fan? Hmm, can I use the old heatsink? Maybe… but the new one looks much fancier. (The new one is on the left and the old one is on the right.)

Note the heat pipe:

It was pretty obvious what the copper tubing was for, but I didn’t know about the cool capillary circulation aspect (no pump!) nor that this had become mainstream technology for low-end coolers. Nifty.

Also notice that the new one (in the first photo alongside the old one) has pre-applied thermal paste. It’s the light gray square in the middle. I wiped most of the paste off of the old one, in anticipation of maybe needing to re-apply new paste and use it again due to the 3 vs. 4 pin problem.

Some reading about this plug switchover led me to believe that the 4th pin was optional and that the new plug was designed to fit into old-style plugs as needed. I tried it and it worked fine.

Here’s the socket with no CPU in it; you can see the white 3-pin connector labeled “CPU FAN” in the upper right corner.

And here is a close-up of the 4-pin cable plugged into the 3-pin connector, overhanging a large screw. Note that the right side of the plug has two plastic ridges designed to force you to plug it in the right way. Clever!

Still, if you fail to plug the CPU fan in and the CPU overheats and dies, you void your warranty, so I’m glad I checked before just plugging it in and hoping it’d work.

I thought about the conventional wisdom, which is first that “box coolers” (the CPU coolers that come in the box with CPUs from the manufacturer) are lame, and second that overclocking an Opteron 175 works well. Given that I have no intention of overclocking this thing, I figured that the box cooler would be fine.

After I put the server back together, I ran two instances of burnK7 (from the cpuburn package) for a little over an hour, to keep both cores super busy. burnK7’s mission in life is to make your CPU temperature as hot as possible. The temperature never exceeded 57C for either core, compared to an idle temperature of 37C. The ambient temperature was 69F, and the case is not particularly fancy (2 fans in, 1 out, plus one in the power supply), so I’d have to say that the box cooler and preinstalled thermal paste are fine.

Overclockers may disagree and favor fancier coolers, but from experience I know that in this server, the hard disks are more likely to overheat on a hot day. I added an external desk fan to this server’s rack last year to resolve that problem, which was caused mainly by poor external air circulation.

A couple of things went more easily than I had expected. First, when researching this upgrade I found that (at least for Ubuntu Gutsy, possibly for other Linux releases or distros) there regular AMD64 kernel has SMP support. So, no OS tweaking was required to support the Opteron 175’s dual cores. Second, VMware Server recognized the second CPU and used both CPUs when I ran some tests on two uniprocessor virtual servers. Interestingly, /proc/cpuinfo on a virtualized Linux server shows the right CPU name for what is a dual-core CPU, but only lists one core, which matches the VMware configuration for that virtual server.

VMware did complain in a couple of cases about the different CPU feature set, when resuming suspended VMs. You can dismiss the message and run the VM, and that works, but it warns each time in my case because they are set up as locked snapshots (always restoring from the point where it was originally suspended, before the CPU upgrade).

This is minor, and the fix to make the warnings go away is quick: resume the VM from the old snapshot (dismissing the warning message), unlock the snapshot, make a new snapshot, lock that, and suspend (or power off) the VM. Now when you resume it in the future the warning will not appear. That’s simple to do, but it wasn’t 100% seamless, so I thought I’d mention it. I am glad that there was a warning, though; that would have helped if there had been an actual problem caused by the CPU change. In my opinion VMware does exactly the right thing.

So, all in all this was a very pleasant upgrade; quick, cheap, and effective (for server workloads which parallelize easily, this server is almost twice as fast as before).

This time I’m pretty sure I’ve gotten rid of all the badness. I’ve had almost 2 days of continuous uptime, which may not sound like something to get excited about, but it sure beats the 4-6 hour average uptime I’ve had for the past few weeks.

I’ve spent so much time on this that I’m not willing to put in the extra 4-8 hours it would take to really truly isolate this issue. But my best guess is that it was the Canon drivers for my recently purchased Canon PIXMA MX310 multifunction printer. Specifically I think it’s the scanner driver, but I’m not certain. If so, that would mean that the culprits were BJUSBLoad.kext and BJUSBMP.kext in /System/Library/Extensions.

I got fed up a couple of days ago and started using sudo rm -rf on anything that looked suspicious, based on the output of launchctl list. That included completely removing a recently installed copy of VMWare Fusion, and everything that had been installed for my prior printer, which was an HP PSC 1600. I also re-disabled all my fonts, leaving only the required ones (see the prior post for info about that).

The clue came when I had to scan something this past weekend. I had already reinstalled Leopard on an external drive and brought it up to 10.5.2 with the Leopard Graphics Update 1.0, and it was fine. So, I was using the external drive’s clean Leopard install to get work done. (As a re-cap, Safe Boot mode on the internal hard drive had worked fine also. So I knew it had to be something not installed on the external drive yet, that was loaded only in non-Safe Boot mode.)

In order to scan a document I had to install the Canon drivers for the printer and scanner. A couple of hours after doing so, the system froze up. I hadn’t yet made the mental association between the Canon drivers and the subsequent crash, so I kept using the system for a while, even though it was crashing and requiring a reboot every few hours. At that point I decided to punt and go back to using my internal drive, since it was no worse now than the reinstall.

So, given that info, and the fact that I had never installed the HP drivers nor VMWare Fusion on the external drive, I’m pretty sure it was one of the Canon drivers. Otherwise there would have to be something in my user directory (which I copied over from the internal drive) which managed to start up something that was not installed on the active boot disk. That seems unlikely to me, but possible. In the launchd config files I’ve seen, the paths are hard-coded to where the installer puts things, so even though the internal drive was mounted, those things should have failed to start because the actual installed drivers and daemons are not where they were expected to be.

So, which Canon driver is it? Probably the scanner driver. From what I can tell, the printer driver is more or less an outgoing thing: try and print, and it looks on the USB bus for the printer, and prints to it. However, the scanner driver has to deal with the case where the user presses the “Scan” button on the printer’s control panel. If that happens it launches either Image Capture.app or the “MP Navigator EX” application which is Canon’s multifunction dashboard app. My suspicion is that this is what those two kexts are for.

So, if you get this kind of trouble, try blowing away those two kexts (wouldn’t it be grand if Canon provided an uninstall feature for their scanner driver?) and rebooting and see if that fixes it. Better yet, delete them one at a time and reboot each time, and see if you can figure out which one is really the problem.

Update: It was actually an intermittent hardware problem and Apple has replaced the logic board of this computer.

I used the instructions on the GutsyUpgrades page of the Ubuntu help wiki, under the heading “Network upgrade for Ubuntu servers (recommended)”. This is the command-line way of doing things, which is appropriate for administering a remote server via SSH. There are two steps: install the updater, and run the updater. That’s pretty slick.

Unfortunately as usual with Debian based releases there are many cases (a dozen or so) where the simple act of updating a package from an old version to a new version causes the installer to plead for human guidance. I find it irritating that if you change the default configuration at all, it cannot simply merge your changes into the new config file and continue. You have to do that manually. This is tedious, but not a really big issue.

In my case I had to manually merge apache2.conf, php.ini, and dovecot.conf because I had made a single line change in each file.

Upon reboot there were vast numbers of error messages, which I ultimately determined to have been caused by this issue. Removing evms solved the problem, and now there’s a special page about evms and why it’s no longer supported. Based on my reading of that page I suspect that there may be a new version of the Gutsy Gibbon updater that actually helps you by suggesting that you get rid of evms proactively to prevent this problem. Nice! Anyway my fix was simply to wait until the system had booted (slowly, with lots of error messages) and to carefully type in apt-get remove evms which I couldn’t even see as I typed because of the flood of error messages. Once that was done that problem went away completely.

I had a separate problem which was that the kernel has re-named the devices for my hard disks, which made a couple of my drives appear not to be recognized by the system. I use Linux Software RAID for a single mirrored volume, and for some reason it couldn’t use the volume UUID to figure out where the drives had gone. After recompiling a custom kernel using these excellent instructions to try and resolve this issue (which turned out to be a red herring), I found out that I was on the wrong track. (But I also found out that it only takes 56 minutes to compile a kernel on my server, which isn’t too bad.) I read dmesg extremely carefully and figured out what had happened. Some work with mdadm told the software RAID manager how to find the components of my RAID mirror and all was well with that. My understanding is that since /etc/fstab includes UUIDs now, drives moving around shouldn’t cause problems like this, so maybe this is a problem specific to the Software RAID system. Other drives moved but were located and mounted properly, with no need for manual intervention.

Finally, I had to update to VMWare Server 1.0.4, which is the first version that supports the kernel included with Gutsy Gibbon (2.6.22-14-server).

Now that all this is behind me I can say that for servers it’s a decent update; I have no experience with it as a desktop. If you have any doubts you can probably skip this one and wait for the next LTS release. But if you do plan to upgrade, just make sure to remove evms first and look in the /dev/sd_ range for any hard drives that seem to have disappeared.

GFS, ZFS:
I’m also interested in distributed filesystems like GFS. I was under the impression until a few months ago that gross ugly NFS was still the state of the art, but apparently somebody has finally addressed that. Correction: GFS is a SAN filesystem, not a distributed filesystem, so it isn’t really comparable with NFS.
ZFS is also interesting, because RAID has turned out to be complicated and expensive, mainly because it has to provide a transparent abstraction of a simple low-level block device. In classic time-sharing situations where many users run many applications on one volume that made more sense than it does now, where many users share one clustered application that has one dominant usage pattern across a large number of servers, each of which has several disks. The smaller set of requirements means that you can customize the clustered storage behavior and get major cost and performance savings, and ZFS seems to be a really good halfway point between application-level 100% custom clustered storage code, and kernel or hardware-level 100% generic RAID. The goal here would be to have dozens of cheap servers with several cheap drives each connected to cheap (motherboard-based) controllers, with HA and parallelization provided in software.

PostgreSQL:
Also I’m becoming a total PostgreSQL weenie and I want to see how they’re presenting themselves, and who’s making closed-source enterprise add-ons, esp. since Sun is increasingly investing in PostgreSQL (in a way that benefits the PostgreSQL community, not just Sun and Sun’s customers). Sun’s wagon is no longer hitched to Oracle on Solaris on SPARC, and the open stack of PostgreSQL on Linux (or Solaris) on x86 is proving to be extremely cost effective without sacrificing features, data integrity, or SMP scalability (as has been the case with MySQL). I think the “who’s the big fast fancy database of choice” tipping point from Oracle->PostgreSQL is already here and the world just hasn’t fully recognized that yet. The only thing left is built-in multimaster clustering (there’s PGCluster but it doesn’t look like a healthy project, and it’s not built in) but I think that’s just a matter of time now, since it’s such a desirable feature for the kind of customers Sun is courting.