“HI. You’re using libxml2 version 2.6.16 which is over 4 years old and has plenty of bugs. We suggest that for maximum HTML/XML parsing pleasure, you upgrade your version of libxml2 and re-install nokogiri. If you like using libxml2 version 2.6.16, but don’t like this warning, please define the constant I_KNOW_I_AM_USING_AN_OLD_AND_BUGGY_VERSION_OF_LIBXML2 before requring nokogiri.”

Aaron Kalin figured out how to fix this if you’re installing nokogiri as a system gem, but I want to use Bundler and keep my system gems down to the bare minimum. I figured out how to do this under Bundler 0.8.5.

The gems I have installed via plain old “gem install xxx” are bundler08 -v0.8.5, rake -v0.8.7, and ZenTest -v4.2.1. That’s it. Everything else is under vendor/bundler. I’m using RVM and REE 1.8.7 2009.10 as described in my previous post.

Here is a quick summary of Aaron’s blog post; this will update your libxml and libxslt libraries to reasonably recent versions. (Be my guest if you want to use libxml 2.7.7 and libxslt 1.1.26, which are current as of today, but I didn’t test those myself.)

mkdir -p ~/Downloads/XML
cd ~/Downloads/XML
curl -O ftp://xmlsoft.org/libxml2/libxml2-2.7.3.tar.gz
curl -O ftp://xmlsoft.org/libxml2/libxslt-1.1.24.tar.gz
tar xzf libxml2-2.7.3.tar.gz
tar xzf libxslt-1.1.24.tar.gz
cd ~/Downloads/XML/libxml2-2.7.3
./configure
nice make
sudo make install
cd ~/Downloads/XML/libxslt-1.1.24
./configure --with-libxml-prefix=/usr/local --with-libxml-include-prefix=/usr/local/include --with-libxml-libs-prefix=/usr/local/lib
nice make
sudo make install

Now make this YAML file and call it bundler_build_options_macosx_leopard.yml and put it in your RAILS_ROOT:

nokogiri:
  xml2-include: /usr/local/include/libxml2
  xml2-lib: /usr/local/lib

Then, to build nokogiri so that it uses the new libs you just installed:

gem bundle --build-options bundler_build_options_macosx_leopard.yml

You should now have nokogiri installed under vendor/bundler/ruby/1.8/gems, and when you run your app it should not complain about the version of libxml2 anymore.

Today I decided to fix this and looked harder at what dscl would tell me about other hidden accounts. The solution that worked and didn't seem icky to me was this:
sudo dscl . append Users/postgres Password '*'

That sets the password string to *. This allows me to continue to sudo su -l postgres whenever I feel like it, but it isn't shown as an account in the GUI. Hooray!

Autotest.add_hook :initialize do |autotest|
    autotest.sleep=3
    autotest.add_exception(/\\/\.git\\//)
end

require '/Users/jamieflournoy/autotest/sound/sound'
Autotest::Sound.sound_app = '/opt/local/bin/mpg321'
Autotest::Sound.sound_path = "/Users/jamieflournoy/autotest/sound/sound_fx/"

Note that the sound plugin needs a patch:

29c29
<   [:run, :red, :green, :quit, :run_command, :ran_command].each do |hook|
---
>   [:initialize, :red, :green, :quit, :run_command, :ran_command].each do |hook|

You’ll also need to make sure your sound set has an initialize.mp3 instead of a run.mp3.

I also tried a bunch of intermediate stages of downgrading and testing, including 10.5.0, 10.5.1, 10.5.2 without the Leopard Graphics Update, then with it, 10.4.6 (from the DVD), 10.4.11 with all the current stuff, with and without external display, with and without a shim propping up the bottom of the case with a desk fan blowing directly at the underside of the case. No matter what, I can make it crash, but it takes a good half hour or more before it happens.

Plugging in the external display seems to help (it crashes much more quickly with the external display connected). My guess is that the GPU or VRAM is overheating or is just plain faulty, and that increasing the number of pixels they have to drive is what makes them fail faster.

I had suspected that it might be a GPU temperature problem, or even the dreaded Thermal Paste misapplication problem, but using iStat and Bresink’s Hardware Monitor, I wasn’t able to find a correlation between GPU temperature and a GUI crash. But, now that I can prove that it isn’t due to any non-Apple hardware, and that the OS version is irrelevant, this MBP is going back to Apple for repair. Again.

Again? Yes. I had a somewhat similar problem before, but the video artifacts were much more hardware-y in nature. Apple blamed it on 3rd party RAM and told me if I continued to use non-Apple RAM they would charge me for the next logic board, regardless of AppleCare coverage. So I’ve been using Apple RAM, which costs about twice as much as the Patriot RAM I had bought originally.

However, that problem was visible to the Apple Hardware Diagnostic software. This one is not – the diagnostics find nothing wrong (even on the extended diagnostics mode), but in real world usage I get about 2-4 hours of uptime before having to restart. Even, as I said before, under a fresh Tiger reinstall with Apple RAM, and nothing but the power cord plugged in.

Hopefully Apple service folks will be able to reproduce this problem, or diagnose it with some kind of fancier tool that runs for long enough to provoke a failure. I apologize to anyone who has followed in my diagnostic footsteps, which now appear to have been off track.

Update: It was actually an intermittent hardware problem and Apple has replaced the logic board of this computer.

This time I’m pretty sure I’ve gotten rid of all the badness. I’ve had almost 2 days of continuous uptime, which may not sound like something to get excited about, but it sure beats the 4-6 hour average uptime I’ve had for the past few weeks.

I’ve spent so much time on this that I’m not willing to put in the extra 4-8 hours it would take to really truly isolate this issue. But my best guess is that it was the Canon drivers for my recently purchased Canon PIXMA MX310 multifunction printer. Specifically I think it’s the scanner driver, but I’m not certain. If so, that would mean that the culprits were BJUSBLoad.kext and BJUSBMP.kext in /System/Library/Extensions.

I got fed up a couple of days ago and started using sudo rm -rf on anything that looked suspicious, based on the output of launchctl list. That included completely removing a recently installed copy of VMWare Fusion, and everything that had been installed for my prior printer, which was an HP PSC 1600. I also re-disabled all my fonts, leaving only the required ones (see the prior post for info about that).

The clue came when I had to scan something this past weekend. I had already reinstalled Leopard on an external drive and brought it up to 10.5.2 with the Leopard Graphics Update 1.0, and it was fine. So, I was using the external drive’s clean Leopard install to get work done. (As a re-cap, Safe Boot mode on the internal hard drive had worked fine also. So I knew it had to be something not installed on the external drive yet, that was loaded only in non-Safe Boot mode.)

In order to scan a document I had to install the Canon drivers for the printer and scanner. A couple of hours after doing so, the system froze up. I hadn’t yet made the mental association between the Canon drivers and the subsequent crash, so I kept using the system for a while, even though it was crashing and requiring a reboot every few hours. At that point I decided to punt and go back to using my internal drive, since it was no worse now than the reinstall.

So, given that info, and the fact that I had never installed the HP drivers nor VMWare Fusion on the external drive, I’m pretty sure it was one of the Canon drivers. Otherwise there would have to be something in my user directory (which I copied over from the internal drive) which managed to start up something that was not installed on the active boot disk. That seems unlikely to me, but possible. In the launchd config files I’ve seen, the paths are hard-coded to where the installer puts things, so even though the internal drive was mounted, those things should have failed to start because the actual installed drivers and daemons are not where they were expected to be.

So, which Canon driver is it? Probably the scanner driver. From what I can tell, the printer driver is more or less an outgoing thing: try and print, and it looks on the USB bus for the printer, and prints to it. However, the scanner driver has to deal with the case where the user presses the “Scan” button on the printer’s control panel. If that happens it launches either Image Capture.app or the “MP Navigator EX” application which is Canon’s multifunction dashboard app. My suspicion is that this is what those two kexts are for.

So, if you get this kind of trouble, try blowing away those two kexts (wouldn’t it be grand if Canon provided an uninstall feature for their scanner driver?) and rebooting and see if that fixes it. Better yet, delete them one at a time and reboot each time, and see if you can figure out which one is really the problem.

Update: It was actually an intermittent hardware problem and Apple has replaced the logic board of this computer.

These looked very software-ish to me, and forcing a window to repaint (resizing it, etc.) would make them go away. Eventually, the entire screen would freeze except for the cursor, which would remain as a pointer or insert bar despite being moved to various parts of the screen. Given that iTunes or a DVD would keep making noise, I guessed that this was not a systemwide freeze, and I discovered that I could ssh in from another machine and run shutdown -h now or reboot as root and usually get a clean shutdown.

I later learned that I could use pmset hibernatemode 1 to force hibernation instead of normal sleep mode, so that I could un-freeze the GUI by logging in remotely and forcing it to hibernate using pmset sleepnow and then manually waking it up.

All this time I was still trying to figure out how to trigger a failure on demand, and to read log files to find some clues. I didn’t find any log file clues, but I did rule out a hardware problem using the Apple Hardware Diagnostic program that came on the installation DVDs with the computer. The recent 10.5.2 update and exciting-sounding “Leopard Graphics Update 1.0″ didn’t solve the problem. In fact, after the 10.5.2 update and that graphics update, the time between failures dropped to about 5-6 hours.

I learned that opening ~150 windows and triggering Expose (via F9) repeatedly would quickly freeze the GUI. At this point my question was, is this a generic Leopard quality problem or a problem specific to all the junk I have installed? Or perhaps a hardware issue not detectable by the diagnostic software?

Safe boot proved it to be very likely to be a problem with third party software; the problem was gone when Safe Booted. So then it was a matter of selectively disabling various things that Safe Boot disables and booting normally, until the problem goes away.

I wrote a simple script to move all fonts not in the list of fonts installed with Leopard into a folder on my desktop; that didn’t make the problem go away. According to Font Book’s validation feature, I have about 2 dozen damaged fonts (including all of the music notation fonts installed by Sibelius, and Adobe’s Carta font family), but hey, Apple’s own Helvetica LT MM font fails validation too. In fact, all of the built-in “MM” fonts do. It seems that the Font Book validator considers any wacky glyph shapes (like the Multiple Master fonts that make PDFs possible, or music notation symbols, or dingbats) to be invalid.

My next step was to disable any startup items in /System/Library/StartupItems (which was empty) and /Library/StartupItems. There were some old HP printer and scanner drivers, some M-Audio Firewire audio drivers from my Firewire Solo (no Leopard drivers exist yet), Qmaster, Unlockupd, and BRESINKx86Monitoring. All seemed plausible as causes of problems but disabling all of them didn’t fix the problem. Now that I’ve looked up what they each do, though, I’m not going to put them back.

Finally, I disabled the non-Apple stuff in /System/Library/Extensions. There are almost 300 items in there, and most of them sound important. In fact, there’s even Dont Steal Mac OS X.kext which apparently contains a pretty funny easter egg (see the post by “kick52″ on that page). I installed a fresh copy of Leopard from my install DVD onto an external drive and used ls -1 and diff to compare the directory contents. I also performed my torture test and verified that Leopard 10.5.0 all by itself would not freeze when pushed. OK, so it’s definitely not hardware, nor is it a problem inherent to Leopard.

I googled the filenames of the .kext bundles in /System/Library/Extensions on my hard disk that weren’t in the base Leopard install, and found that there were some Parallels kernel extensions in there, from about two years ago when I was using Tiger. Uh oh. So I moved those to another directory. The modules I removed are called helper.kext, hypervisor.kext, Pvsnet.kext, and vmmain.kext. My Parallels.app version is 2.5 v3188, which is pretty old.

UPDATE: I missed one: ConnectUSB. I found it by running kextstat while ssh’d in from another machine, after it froze again a day later. It was listed as com.parallels.kext.ConnectUSB, which stood out among all the com.apple stuff. I had used locate parallels and missed the stuff that shows up with locate Parallels – I didn’t realize it was case sensitive. Hopefully this is the last bit.

The problems seem to be gone now. Woo hoo! I can do my torture test with > 300 finder windows, eight Quicktime movies open and playing, and DVD Player showing a movie behind a partially transparent Terminal window.

UPDATE #2:I continued to have crash problems after this point. I now believe that the Canon MX310 scanner driver was the problem. More details are here.

I could have saved myself a lot of pain with Archive and Install, but I think I’m still coming out ahead due to the time saved by not having to reinstall and relicense dozens of applications. However, for mainstream users (who aren’t willing or able to go through this amount of troubleshooting effort) I strongly recommend using the Archive and Install when installing Leopard on a machine that currently has Tiger installed.

Update #3: It was actually an intermittent hardware problem and Apple has replaced the logic board of this computer.

I upgraded to Mac OS X 10.5 “Leopard” recently. In short, it’s not ready for mainstream use. There are a few nice improvements, but these are balanced by numerous problems that make me wish I had waited until, say, June 2008 or so. If you haven’t upgraded and aren’t sure that you need to, I suggest that you wait a few months, until some of the bugs have been worked out.

The Good Stuff:

Terminal.app now allows tabbed terminals in one window. Terminal also allows window background transparency, which wouldn’t normally be a feature I would care about, except that it makes it quite easy to set transparency at about 85-90% and put a DVD window behind it. This setup is more readable than the arrangement that Trans Lucy provides (it overlays a transparent movie window instead). If you’re like me, sometimes you work better with something busy going on in the background, and this does a good job of leaving text legible while making the movie kinda visible in the background.

Terminal and Safari now allow tabs to be dragged downward off of the tab bar to create a new window for that tab. It looks pretty cool and can be useful at times, mostly in Safari when you’re browsing a bunch of sites and want to pick one and background-load a bunch of links off of that page.

The new Finder fixes some of the old Finder bugs, though it brings new ones. (More about that below.) Basic problems like window resizing not shrinking to fit only the icons that were there, scroll bars displaying even when they are not needed, etc. are now gone. Desktop icons for volumes remember where you left them instead of being automatically placed every time they are mounted. They also position themselves so as to avoid being under the Dock if hiding is off (they don’t reposition themselves automatically as soon as the Dock position changes, but they are aware of where the Dock overhangs when they are choosing where to align).

Similarly, the window manager is smarter about being aware of where the Dock is, and will try to avoid resizing or zooming windows in ways that put part of them under the Dock.

Preview.app is a bit nicer: you can open a bunch of windows at once, even in one-window-per-file mode, and it stacks the windows in sorted order based on filename. The previous version (bundled with Mac OS X 10.4) opened them in essentially random order and would only open a maximum of 20 windows at once, forcing photos 20 and up into a single window with an image list in the sidebar. I don’t know about you but I take a lot more than 20 photos at any given event and it’s annoying to have to keep track of how many Preview windows I have open in order to keep it from merging the excess ones together into a single window. The new Preview fixes that.

My complex development environment worked fine, with one exception (details below). Ruby on Rails, Mongrel, PostreSQL via MacPorts, and autotest all worked fine. That was a big relief.

The Bad Stuff:

There are some incompatibilities that caused me problems early on. Most notably, Application Enhancer (used by the handy Instant Hijack feature of the excellent Audio Hijack Pro) causes a blue screen on boot! Here are the removal instructions. Fortunately I did a backup before installing (really it was just my nightly full backup) or I would have been a lot more freaked out about this.

I experienced seriously poor wireless networking performance. By that I mean very erratic ping times and latency even when sitting right next to the base station. Here are the details and fix; basically it’s a preferences file problem, which can be fixed by deleting the old 10.4-era preferences file. I’m not sure how Apple QA missed this one; I’d think that 100% of Tiger-to-Leopard upgrade cases would experience this problem.

GNU Screen, which I use like crazy, changed somehow, and I can’t figure out what they changed from the documentation. My super slick Screen running shell script, which sets up my whole development environment like a character mode IDE, broke. PATH somehow gets stomped and reset to the standard login-shell PATH, but no other environment variables get touched. I was not able to figure out why Screen was doing this in Leopard but not in Tiger, and eventually punted and put a hack in my .screenrc to re-set the PATH the way I wanted it in each screen. (Previously I had set it before creating any screens and it was inherited by each of them.)

There are UI issues that make existing applications not work right. OmniGraffle, which is a truly fabulous drawing program, had some strange input problems with mouse clicks that made it display error messages. A small point release fixed the problem, and the new OmniGraffle 5.0 beta releases have no issues. Unfortunately, OmniGraffle 5 is Leopard-only, and has a new file format that causes version 4 to warn you about possible unknown incompatibility upon opening the file. Given my current attitude of “don’t upgrade until later” about Leopard for other people, I don’t want to trap my drawings in a new Leopard-only format. So I’m still using 4.2.2 and it’s fine.

Other applications are not so fine. Microsoft Excel has some input issues that cause the keyboard input to suddenly go into limbo for no apparent reason. Clicking onto another application’s window and then back onto Excel fixes it; there is no current fix from Microsoft for this. You just have to accept that sometimes your keyboard input goes away and you have to fiddle for a minute to get it back. I think I’ve figured out that it has something to do with AutoFill and the popup menu of possible completions, so it may be possible to train oneself to not ever do whatever it is that causes the keyboard input to go nuts. But I’d rather just upgrade to Office 2008 whenever it comes out.

Adobe Photoshop CS3 has input problems as well. Most notably, the little numeric text input boxes for various tools work once and then go dead. For example, try transforming a selection and specifying, say, 66% width. Then do the same thing again – can’t do it; the typing goes into nowhere and the resizing never happens. The text is even drawn incorrectly in the little text box. This is a well known problem and according to this guy quoting an Adobe spokesperson, it will be fixed soon. As far as I can tell there is no workaround for this.

Carbon Copy Cloner 3.0.1, which I use for backups, doesn’t work properly on Leopard. Its only function in my world is to sync my whole hard disk every night to an external hard disk. It doesn’t do this correctly, unless you run the program and uncheck/recheck the checkbox next to the scheduled task every time you reboot. Supposedly the 3.0.2 update will fix this, so that the scheduled task properly installs itself without user intervention. For now I have a text file that reminds me to do this in my home directory, and I dragged that icon into my account’s Login Items list in System Preferences so I don’t forget to do it.

There are some race conditions apparent with opening items from Samba valumes. I have a home server with lots of files on it, and some aliases to folders on that server are on my laptop. So, when I log in, I can just open the folder and it mounts the server volume and goes right there. In Leopard, if I open more than one such folder alias, all of them fail, the volume fails to mount, and a stack of error dialog boxes piles up in the middle of my screen. Upon dismissing the last such error dialog box, the Finder crashes. The workaround is to mount the server volume first, or to open only one server-based folder alias first, and then open the rest.

Worse, if you have aliases to .dmg (disk images) on the server, and you try to mount more than one .dmg from a server at the same time (select more than one and open them simultaneously), all of them will fail and the diskimages-helper processes will all get stuck. At that point you must reboot in order to un-wedge the situation; kill -9 kills the processes but any future disk images will get stuck trying to mount in the same fashion. The workaround is to never try and open more than one alias to a disk image from a server at once. You can keep a bunch mounted, but you have to open and mount each one separately.

There are also a lot of cosmetic problems, mostly with the new Finder. The translucent menu bar makes it hard to read things on the menu bar, and just plain looks sloppy with any background other than a white one. I didn’t like the stripes in list view in the finder and using this hint turned that display option off with defaults write com.apple.finder FXListViewStripes -bool FALSE at a command line.

The Finder seems to have completely lost its ability to remember what view type (list, icons, or columns) you wanted for each window; I have frequently found that I’ll set a window to list view and navigate around and when I re-open that folder it’s in icon view, or vice versa. I’m pretty good at detecting patterns of UI behavior so I think I would have grasped any new system Apple has devised for deciding what view a window should have. Whatever the new one is, it sucks.

The Finder still has problems displaying image previews in icon mode. That’s one of my favorite OS X features: the ability to just open a folder full of photos and see previews of all of them at 128×128 pixel size, without using iPhoto or some other photo gallery app. Sadly, the Finder sometimes just stops building previews of the currently displayed set of icons, and you have to scroll around or “select all” and then “select none” to kick it so it notices that it still has some preview-building work to do. I do wish it would cache previews on disk, too; it’s silly that almost every time I visit a folder full of photos, it has to rebuild the previews from scratch. It does cache the previews in memory, but I have digital camera photos from ten years ago, and I don’t understand why the Finder needs to keep rebuilding the previews over and over.

The Finder still resizes the window so that it is partially offscreen when displaying the Toolbar. If you want to search within the current folder, you need to show the Toolbar and enter text in the search box. Okay, but that means that the Sidebar (with all the volumes, on the left side of the window) appears, and for some reason the whole window resizes to preserve the size of the content area, which means that the window grows, substantially. It also preserves the position of the content area, so the window grows on all three sides except the top. If your Finder window was close to being as big as the screen, the new size will be off screen in every direction. You can’t zoom the window without moving it first so that the zoom button is on screen, because the zoom button is now off the left side of the screen. It’s really not very slick, and becomes annoying if you use the Finder very much.

Also, when searching in the Finder using the in-window search function, the scope is always reset to “This Mac” instead of the current folder. Given that there’s a separate Find function, that doesn’t make a lot of sense. Since the search scope controls don’t appear until after you’ve started typing a search string, your only option is to start searching the whole computer and then click the scope button you wanted, which responds slowly since the Finder is busily carrying out the global search you didn’t want. Eventually it throws away the first partially-completed result set and starts over from the current window. Again, it’s frustrating if you search the current folder and its contents very often.

To be honest, though, a lot of these Finder gripes are new UI annoyances exchanged for old Finder UI annoyances. The new Finder is marginally better, except for all that mounting-multiple-disk-image or opening-multiple-folder stuff.

Conclusion:

Wait before upgrading, to give Adobe and Microsoft and Apple time to release some updates. I would say maybe March or June would be a good time to check on these things and see if it’s safe to upgrade. Until then, keep using Tiger; it’s really quite good and doesn’t have these issues.

Android is a full software stack made up of many projects that exist separately from Android. The components of this stack are available under a variety of different terms: Linux is GPL’d; WebKit is LGPL’d; SQLite is in the public domain; FreeType is available via GPL or its own license. There are probably a dozen more little libraries and components that Android builds upon, all of which have licenses that need to be considered. It’s not possible for Google to just put a bow on the whole stack and say it all uses license X.

Google has released the Android SDK, which is the top layer, under the Apache license. According to the OSI list of approved open source licenses, this is an open source license. And yet, the reaction to Android from open source advocates is negative. Why? Well, the concern is that some implementation of Android will add proprietary code and/or remove standard code, fragmenting the platform. Since Google can’t rewrite the licenses of the underlying components of the stack, we’re really talking about fragmentation at the top layer of the stack.

The Apache license allows for parties who download the source code to alter it and then keep the altered source code secret, while distributing a derivative work. Contrast this with the “viral” GNU Public License, which obligates all parties who modify the source to either keep the modified software completely to themselves, or to distribute the source if they distribute a derivative work. Ignoring the case where a licensee simply keeps the derivative work to themselves, the GPL forces a web of innovation and collective advancement, whereas the Apache license encourages a central publishing model, where innovations are kept private and used for competitive gain.

Thus, Android applications designed to be compatible with Google’s platform could be made incompatible with a particular device, by a handset vendor who removes core Android APIs and replaces them with their own closed source alternative. This might seem like a paranoid fantasy of a small clan of open source zealots, but it’s not. This is the same tactic that Apple has successfully used to keep Mac OS X closed. Mac OS X rests upon a large amount of open-source code (some of which is also part of what Android is built upon), while requiring developers to code to Apple’s proprietary Cocoa APIs in order to make Mac apps. You can install Linux on a Mac, but then you lose the ability to run Mac OS X apps. You can build generic Unix applications on a Mac, but they look quite different from a standard Mac app, and lose a lot of Mac-specific functionality. Apple chose to make this possible, but compare this with the iPhone, which uses much of the same software underneath, but (as of this writing) cannot run a generic Unix app because Apple doesn’t want you to do this.

This same sort of situation is possible with Android under this license. Company X grabs the Android sources, dumps a few key APIs (maybe the GUI, network, and process management ones) and suddenly they have their own incompatible platform that can run on the same hardware but can’t share apps with the mainstream. Dump phones on the market (subsidized by monthly fees, as usual) and fund a few key apps (MP3 player, movie player, email/SMS, web browser) and users are stuck with that vendor’s offering, just like they are now. And this is just how telcos and media companies, both of whom are desperately trying to keep hardware and software platforms closed, think. The more closed a platform is, the more secure they feel about their profits, and the more willing they are to invest in it.

The only charitable explanation I can think of for why Google chose this license is the Apple explanation.

If Google were really pandering to the existing mobile carrier crowd, they could simply have released nothing, because another closed platform to build phones with and to write apps for is pointless. There are plenty of existing trunks to be locked in already. We don’t need somebody else to slap together a Linux distro for phones with a closed GUI on it. You can to go LinuxWorld Expo and probably find two dozen companies doing exactly that, and none of them is particularly successful. Google is too smart to add itself to this list of flops. It makes no strategic sense.

More likely, I think, is that Google intends to be Company X in my above scenario, putting themselves in the role of Apple by making an “Android Plus” premium platform that they put on the handsets they’re pushing. In this scenario, you can write your own apps to the reference platform and they’ll run on Google’s favored phone, but Google can still reserve the right to put all sorts of funky stuff on their phones without documenting it or giving developers or users any rights to it, and more importantly, without having to open source their special components so that non-Google devices could use them.

Sure, this is a charitable interpretation. They might just have screwed up royally, buying a company (Android) that wasn’t anything special and releasing something that no one other than journalists will pay attention to. But given that Google is betting $4.6 billion on the 700MHz spectrum auction in the U.S., I’m reluctant to simply write Android off as a “hail mary” acquisition.

There must be a larger strategy here, and I suspect it’s Google putting itself in the shoes of mobile carrier, handset OS maker, and service provider. Somebody else manufactures the handsets, of course, but Google owns and operates the whole experience from end to end other than that. You buy a Google device, pay a monthly Google subscription fee, your bits travel over a Google global network of wireless towers and a wired backbone, and you run apps on your Google phone that interact with Google back-end services. It’s a carbon copy of Apple’s iPhone/iTunes strategy, without AT&T in the picture, and with third party apps allowed on the phone, as long as they work on the already-published Android SDK. All Google has to do is to make a decent looking device and be less customer-hostile than existing U.S. mobile providers, and they’ll do well.

In this light, the partnering talks with existing mobile carriers is puzzling. It’s possible that they’re pitching a strategy that removes the burden of application development from mobile carriers, allowing them to be the billing companies that they really are, and putting Google in the position of being the provider of content and software. The lukewarm responses from these carriers is predictable; for the carriers to be involved as just bandwidth providers and customer billing service providers, there would have to be a careful negotiation of revenue sharing, or else the carriers will simply continue on their current, very lucrative course.

If they are planning to become a carrier themselves, then nothing can actually begin to happen until January, when the 700MHz spectrum auction actually takes place. There’s a closer December 3rd deadline for Google to reveal their plans. At that point we’ll know what they’re up to, assuming they’re actually able to buy the spectrum that would make it possible. Alternatively, this spectrum purchase could be a bargaining chip on Google’s part, which they do not intend to directly utilize themselves. Google would provide the spectrum, the handsets, the OS, the apps, and the services, and the carrier partners would provide the towers, the maintenance staff, sales, and billing.

Given all this, it makes little sense for Google to GPL the Android platform. They need to own it so that they can assure a prospective carrier partner that they will be the ones whose phones are being used by customers, in order to share revenues. If they were to open up the handset market entirely, the carriers would block any new entrants and Android based phones would be doomed.

The short version:
- Ruby is fun to program in, as you’ve probably heard
- Rails is over-hyped, but it’s still quite good (definitely not perfect)
- I like the productivity of Ruby on Rails but I wouldn’t call it a silver bullet by any means
- Ruby performance was bad and is getting less bad, and can even be good if you do what the experts say
- The real gem (har har) in the Ruby and Rails space is the community itself

Now for the longer version:

It’s hard for me to be objective since I only started programming in Ruby in February and started using Rails in April, but the language and framework don’t strike me as tedious or kludgy at all. I’m definitely enjoying Ruby and making use of its fancier language features, all of which seem to actually work as advertised without any hideous gotchas when you actually try to do something slick.

I particularly enjoy the way Ruby’s collections allow you to write in the style of a functional language; have a look at the Array, Enumerable, and Hash classes to see what I mean. It’s possible to write extremely compact code that morphs a collection into something very different, but without the inscrutability of Perl.

Compared to my experience with mainstream Java and J2EE, I certainly don’t miss Tomcat, Ant, Struts, or JSP Expression Language. (Instead I’m using Mongrel, Rake, Rails, and ERB.) That’s not Java’s fault, and could be alleviated by just picking better tools and frameworks at the start of a project. I don’t miss Hibernate but I don’t have any gripes about it either.

There are features in Java 1.5 that are helpful in alleviating some of the annoyances of programming in Java, but because these are new features, the vast majority of library code in the wild doesn’t use them (similar to the problems with OO being bolted onto Perl and PHP). Also there are political and business deployment roadblocks to using the latest technology: just because you can download an official release of something and it works perfectly for your code, that doesn’t mean that you necessarily have permission to deploy that technology stack onto production servers. Big organizations take years to officially support new releases, so you may be stuck developing against 3 year old products and APIs.

I’m not trying to troll for Java vs Ruby comment flames here. My point is just that Ruby has had features for a relatively long time that Java is just recently adopting (or simulating), and that’s important for more than just petty bragging rights; a feature that’s unavailable to you for your own org’s political or financial reasons might as well not be there at all. (Or maybe it’s worse, if it makes you bitter about your own work environment: “I have to write all this ugly stupid code because Bob is afraid to upgrade the server… grr.”)

I can remember quite a few times when I painted myself into a corner in Java and had to write lots of type casting / checking / converting code, or had to try and work around single inheritance using composition and lots of boilerplate proxy methods, or really wished java.lang.String weren’t final so I could subclass it, etc. Ruby feels a lot more like Perl or JavaScript than Java when it comes to being able to do whatever you want when you’re sure you know what you’re doing, and being able to write weird fancy code that magically makes everything that calls it become dead simple.

I used to be horrified at the idea of messing with somebody else’s classes (as is done by Hash.slice from will_paginate, which I swiped all by itself since I don’t really need will_paginate), but I’m finding that the apocalypse I feared just isn’t there. If you do something wacky, like add a new method to String, the world doesn’t collapse. Or maybe it does, and you decide not to do it that particular way; just do something else. ActiveSupport adds all kinds of stuff to base classes like String, and it’s OK. Example: ActiveSupport::CoreExtensions::String::Inflections.

I’m still adjusting to using a dynamic language: not uber-checking arguments and state all the way down a method call stack (oh no! is it nil?), not locking down everything so it can only work the way I planned, etc. In most cases, the Ruby equivalent of a NullPointerException is fine. I think this is because the built-in error message includes the type and method you were trying to call, so you can usually just read that and see what you did wrong. The only time you’d want to prevent that is if you bury that error deep in your own code, so that the caller would have no idea of what they did that led to it. I’m finding that I’m OK with writing less paranoid code, because the failure will still happen, and be suitably clear in its cause, without my explicitly checking for it.

An issue with Ruby and Rails is that the official documentation is limited. It’s imperative to have 2 or 3 books on hand at all times, in addition to the RDoc Rails documentation. I own three of the Pragmatic Programmer books: Programming Ruby (the “pickaxe book”), Agile Web Development with Rails, 2nd Edition, and Rails Recipes. Just looking at generated API documentation won’t tell you how you’re supposed to use libraries.

The reverse way of looking at that need for documentation is symbol lookup. Rails does some pre-including of library and framework files before loading the application file you’re working on, so unlike Java, you don’t have a mandatory list of imports at the top that tell you where every class you’re working with came from. There are some symbols and functions you can call from instances of particular classes, and they came from somewhere but it’s not obvious where, at least when you’re starting. A good IDE with “find declaration” could fix this, but initially it’s daunting: “what are *all* the options for this method?” “Where is this method defined, even?” etc. This is where the books are less useful and API documentation can help.

I’ve also subscribed to a couple dozen RSS feeds that help me keep up with what the really slick Ruby and Rails people are using, writing, struggling with, etc. Chief among them is probably Err the Blog. The Ruby on Rails podcast is helpful also.

As for productivity, here are 4 specific time-savers I’ve found:

1) TextMate, which is very slick. There are Windows equivalents, though from what I’ve heard, they’re not 100% as good. Point is, a text editor that understands Ruby is critical, and if it helps you jump around in a big tree of files, that’s very helpful.

I’m an Emacs devotee, and TextMate is far better for Ruby and Rails. It’s hard to explain, but let’s just say that TextMate is much deeper and more featureful than it appears at first glance.

I looked at NetBeans which is pretty close to being a very smart Ruby & Rails IDE, but it was still suffering an identity crisis (am I a Java IDE? or a multi-language thing?) that cluttered the UI badly. This is a hot area of innovation right now so any week now TextMate might be eclipsed (or Eclipsed, har har) by something more full-featured. There are a few other options but I didn’t see anything that led me to believe a multi-hour or multi-day eval would be worth it.

2) Locomotive. It’s like MAMP or WAMP or any of those all-in-one application stack installers, except it gives you a working Rails environment. I set up PostgreSQL separately thru MacPorts (a cousin of the FreeBSD ports project), which was somewhat tedious but decently documented.

Getting all the Ruby and Rails stuff compiled from source is no picnic, and frankly unnecessary. On Ubuntu there’s a gotcha: the Ubuntu people and the RubyGems people can’t agree on a file layout, so Ubuntu’s rubygems package sucks. It’s better to install Ruby using apt-get and then install gems manually from the tarball.

3) Rails Plugins. There’s a lot of stuff being built on top of Rails and then improved day by day, and svn-over-WebDAV is the popular sync protocol. There’s a tool called Piston that helps you manage this. Anyway there’s a lot of stuff that is hard to do and commonly needed which is available in plugin form.

4) ZenTest, and more specifically, Autotest. It watches your source tree and runs only the tests in your test suite that need to be run based on what has changed; it’s the dynamic-language version of the incremental compilation that’s present in any good Java IDE.

I use it to eliminate the too-many-Terminal-windows problem on my laptop. I also use it to help me achieve some level of continuity on remote hosts, by leaving half-completed sysadmin tasks as-is until hours or days later even if I get interrupted or if the task is really long-running and I need to roam around with my laptop.

Today I decided to invest some time in making my command-line development environment launch with a single script. Here are the details.

Since Screen can preserve context and state only until the host it’s running on is rebooted, I use a text todo list to make sure the big tasks don’t get forgotten or done out of order. On remote hosts I keep this todo list open in a screen by itself, and make other screens for individual steps like “apt-get update” etc. Usually I can go away and come back a couple of days later and pick up where I left off, but if there’s a reboot (which happens occasionally with a VPS I don’t control fully) the todo list is still there as a backup means of remembering where I was.

Locally, though, my todo list is a text file managed in a GUI editor, which lately is TextMate. So all that has to happen throug Screen is the command-line stuff itself.

I used to set things up manually when starting a development session (which could last hours or days, depending on Software Update, travel, etc.). That took a couple of minutes because I had to launch Locomotive, use it to open a Terminal window with the right shell environment, then run screen -e^oo and manually open a bunch of screens, then manually change their titles, then manually run a few tools and stuff.

Now I have all that automated. There’s a shell script that sets the environment up (it’s just the same script that Locomotive makes, but copied out of /tmp into a stable place, since the contents are stable and don’t need to change for each new session). That calls screen -c jamie_screenrc whose contents are below (sanitized a bit):

escape ^oo

screen -t 'Rails project root' 0

# I like to have SVN running at the root of the whole project
# (including specs, notes, etc. with source code changesets)
chdir "../../.."
screen -t svn 1
chdir "Site/trunk/myproject"

# screen/stuff are separated so that when you exit the stuff-ed program,
#  you drop back to the bash shell for that screen instead of immediately
# exiting that screen. This is useful for "^c, up-arrow, enter" restarting of commands

screen -t psql 2
stuff "/opt/local/lib/postgresql82/bin/psql -U myprojectdev\\012"

screen -t autotest 3
stuff "autotest\\012"

screen -t mongrel 4
stuff "./script/server\\012"

screen -t script/console 5
stuff "./script/console\\012"

select 0

It’s “jamie_screenrc” and not ~/.screenrc so I can develop with the same setup on another host just by checking out the code (including this screenrc file) and running screen there. I don’t want this to be my default screen setup every time I run it anyway.

This is “mildly convenient” to quote Professor Frink. But it’s a big deal to me because now the barrier to starting and stopping work is very, very low; if I have a spare couple of minutes and I’m not set up to develop, I can get going in seconds. Likewise I don’t have to put off other things since I know I can stop and do anything else and get right back into flow very quickly.

The next step will probably be to create a RubyOSA script (or maybe just a shell script, if it’s sufficient) to get Firefox and TextMate and OmniGraffle up and running too with the right documents open just where and how I want them. Then I’d be able to run a single script (or click a single icon, in the Dock perhaps) and wait a minute or less and be completely ready to start coding.