With PHP it’s just another Apache module, but you might need to build a few extra libraries and maybe custom-compile Apache. Once you get it up and running, though, you don’t even need to restart the server when you deploy new code. It’s automatically updated.

With Ruby on Rails, it has been far uglier, especially as you go further back. The standard “Matz Ruby Interpreter” (MRI) doesn’t thread well and is quite remarkably slow, and Ruby + Rails in an MRI process use a lot lot lot of memory. So you don’t really want RoR running inside each Apache process. Folks used to use FastCGI (which should have died over a decade ago, but lingers on like a bad cold) but now use Mongrel, which is conceptually kind of like FastCGI, except that it actually works. Mongrel presents the application via HTTP, which is much easier to understand and integrate with other parts of your architecture (such as a load balancer) than FastCGI.

Whereas in J2EE you’d run one big honkin’ JVM that used lots of memory to load up your code and data structures, but then ran many threads inside that one process, with the limitations of the MRI (green threads and many, many trips into non thread safe C code that requires the use of a “giant lock” that essentially makes it single-threaded), you run one process per thread. That’s like Apache+PHP or OpenSSH or many other unix programs that fork, right? Well, sort of. The issue is that your Ruby code is not seen by the kernel as something that all those forked processes can share; it sees the parsed Ruby code as data, and when the MRI’s garbage collector marks all those objects during garbage collection, it seems this data as being recently changed, differently for each forked process. So not only do you need 30-70MB or more per process, but very little of that is shared between processes. Ouch!

A second problem is that these processes take a while to start up and load the code, so it’s not reasonable to embed the Ruby interpreter in Apache when using Rails; the overhead is just too high. So the Mongrel solution is to pre-launch a bunch of interpreters, and have them just sit there until requests arrive. That’s pretty inefficient from a memory standpoint, but the latency when a request comes in is quite low since there is no initialization needed.

There have been a few interesting alternatives under development: JRuby is very promising, because it reuses all of the investment in VM development that Sun made over the last 10+ years for Java. At this point the JVM is pretty darn good at running many threads across multiple CPU cores, and at garbage collecting efficiently, among other things. These are key weaknesses of MRI, so running Rails on JRuby seems like a huge benefit. I haven’t tried it yet but I suspect that this will become one of the 2 or 3 most common ways to run Rails applications in the near future.

Another interesting alternative was some experimental hacking to MRI’s garbage collector by Hongli Lai, to store its working data separately from the objects being examined, so that preloaded Ruby code would remain shared by many forked interpreter processes over long periods of time. In other words, this is a potentially major memory use savings for Mongrel cluster users, which would in turn allow the sysadmin to run more Mongrels to service more simultaneous requests, or to bump up the database cache, or to increase the size of the running memcached instance. So, this would indirectly be a performance booster, and Ruby could really use that.

This experimentation apparently became Ruby Enterprise Edition, which as of this writing is not available yet. But the other development coming from Hongli Lai’s new company, Phusion, is Passenger, a.k.a. mod_rails.

What’s interesting about mod_rails for the beginning Rails developer is that it is intended to make Rails hosting easier, particularly for shared hosting enviroments, which have been struggling to offer Rails hosting in a uniform and cost-effective fashion. That means that in a short while (weeks?), shared hosting plans for fiddling around with Rails will become much cheaper and more widely available than they are now.

What’s interesting about mod_rails for the experienced sysadmin is that it mimics the min/max process pooling behavior of Apache, and addresses startup overhead in a clever way. It also serves static images via Apache automatically, eliminating the need for a separate block of mod_rewrite rules that must be crafted carefully so as to avoid conflicts with mod_proxy.

The architectural overview is comprehensive and well written, but here’s a summary: The Spawn Server makes a tree of child processes that preloads Ruby, Rails, and your application code for you, and then that is fork()ed to satisfy incoming requests. So the first request after startup incurs startup overhead (in my case, 5 seconds to load the Redmine login page) but subsequent requests get much better response time (.6s to reload that login page).

That seems like a lot of overhead in terms of big Ruby processes. Here’s what I measured just now: 97MB free with just Apache running (no spawn server yet). After the first page view, there was 36MB free, and four new processes: the Spawn Server taking a little over 6MB (rsize), the FrameworkSpawner taking 20MB (rsize), the ApplicationSpawner taking 34MB (rsize), and one Rails process taking 34MB (rsize).

The new “free” value is 36MB. The Buffers and used Swap values remained constant, with only 48KB of swap used. So that means that all four processes, which would seem to need 94MB to run (34+34+20+6), are actually overlapping enough that they are using only 61MB (97-36). And the ApplicationSpawner eventually terminates, leaving 36MB still free, which makes sense – it’s the process that fork()ed the Rails process, so they should ideally be overlapping nearly 100%. I’m surprised that this is so high; based on the GC experimentation that Hongli Lai did, I would have expected them not to overlap as much.

The idle Rails process exits eventually also, controlled by the RailsPoolIdleTime setting. That saves memory but re-introduces the startup overhead. That leaves the FrameworkSpawner and the SpawnServer running, taking about 25MB of memory (quite close to the 20+6 shown by their rsize values).

Let’s compare this memory footprint to a Mongrel cluster. In a Mongrel cluster the processes start up and stay running forever, so the users are unlikely to incur much startup overhead at all, since it’s done long before they visit the application. Some amount of application-specific internal overhead is still an issue, though; that might include gradually filling an initially empty memcached, template compilation and/or caching, etc. As for memory, each Mongrel would need the same 34MB of memory, but there’s no SpawnServer, FrameworkServer, or ApplicationServer, so the extra 25MB of overhead would not be present with a Mongrel cluster.

That means that for a shared hosting setup where many low-traffic Rails sites may be used, or a multifunction server where serving one or more low-traffic Rails applications is just part of the job, mod_rails is a benefit. When the Rails app isn’t being used, it will exit and free up that memory for other processes. The starting and stopping of Rails with mod_rails is automatic and demand-based, so the sysadmin can tune it and forget about it.

On the other hand, a single dedicated server or VPS with a fixed amount of memory serving a single application would be better off with Mongrel, because of the lower memory overhead (25MB less), and the fact that the Mongrel processes start up before users need them and stay running indefinitely. Mongrel clusters could still potentially benefit from the Ruby Enterprise Edition’s garbage collector tweak if forking were used after preloading all of the code.

A single-purpose dedicated server running mod_rails could attain similar performance to a Mongrel cluster by simply setting the RailsPoolIdleTime value to a very high number. Then the Rails processes would hang around, and although you’d pay the price of a 25MB memory overhead, the startup overhead would only be paid by the very first visitor. However, you’d lose the main benefit of mod_rails, which is demand-based pool resizing, particularly if you’re running more than one application, Rails version, or Ruby interpreter version.

In short, I think mod_rails is very nice, and having actually used it I’m impressed with how polished it is for a 1.0 product. But if you’re already running a single application as a Mongrel cluster on a dedicated server, there’s no point in switching.

Android is a full software stack made up of many projects that exist separately from Android. The components of this stack are available under a variety of different terms: Linux is GPL’d; WebKit is LGPL’d; SQLite is in the public domain; FreeType is available via GPL or its own license. There are probably a dozen more little libraries and components that Android builds upon, all of which have licenses that need to be considered. It’s not possible for Google to just put a bow on the whole stack and say it all uses license X.

Google has released the Android SDK, which is the top layer, under the Apache license. According to the OSI list of approved open source licenses, this is an open source license. And yet, the reaction to Android from open source advocates is negative. Why? Well, the concern is that some implementation of Android will add proprietary code and/or remove standard code, fragmenting the platform. Since Google can’t rewrite the licenses of the underlying components of the stack, we’re really talking about fragmentation at the top layer of the stack.

The Apache license allows for parties who download the source code to alter it and then keep the altered source code secret, while distributing a derivative work. Contrast this with the “viral” GNU Public License, which obligates all parties who modify the source to either keep the modified software completely to themselves, or to distribute the source if they distribute a derivative work. Ignoring the case where a licensee simply keeps the derivative work to themselves, the GPL forces a web of innovation and collective advancement, whereas the Apache license encourages a central publishing model, where innovations are kept private and used for competitive gain.

Thus, Android applications designed to be compatible with Google’s platform could be made incompatible with a particular device, by a handset vendor who removes core Android APIs and replaces them with their own closed source alternative. This might seem like a paranoid fantasy of a small clan of open source zealots, but it’s not. This is the same tactic that Apple has successfully used to keep Mac OS X closed. Mac OS X rests upon a large amount of open-source code (some of which is also part of what Android is built upon), while requiring developers to code to Apple’s proprietary Cocoa APIs in order to make Mac apps. You can install Linux on a Mac, but then you lose the ability to run Mac OS X apps. You can build generic Unix applications on a Mac, but they look quite different from a standard Mac app, and lose a lot of Mac-specific functionality. Apple chose to make this possible, but compare this with the iPhone, which uses much of the same software underneath, but (as of this writing) cannot run a generic Unix app because Apple doesn’t want you to do this.

This same sort of situation is possible with Android under this license. Company X grabs the Android sources, dumps a few key APIs (maybe the GUI, network, and process management ones) and suddenly they have their own incompatible platform that can run on the same hardware but can’t share apps with the mainstream. Dump phones on the market (subsidized by monthly fees, as usual) and fund a few key apps (MP3 player, movie player, email/SMS, web browser) and users are stuck with that vendor’s offering, just like they are now. And this is just how telcos and media companies, both of whom are desperately trying to keep hardware and software platforms closed, think. The more closed a platform is, the more secure they feel about their profits, and the more willing they are to invest in it.

The only charitable explanation I can think of for why Google chose this license is the Apple explanation.

If Google were really pandering to the existing mobile carrier crowd, they could simply have released nothing, because another closed platform to build phones with and to write apps for is pointless. There are plenty of existing trunks to be locked in already. We don’t need somebody else to slap together a Linux distro for phones with a closed GUI on it. You can to go LinuxWorld Expo and probably find two dozen companies doing exactly that, and none of them is particularly successful. Google is too smart to add itself to this list of flops. It makes no strategic sense.

More likely, I think, is that Google intends to be Company X in my above scenario, putting themselves in the role of Apple by making an “Android Plus” premium platform that they put on the handsets they’re pushing. In this scenario, you can write your own apps to the reference platform and they’ll run on Google’s favored phone, but Google can still reserve the right to put all sorts of funky stuff on their phones without documenting it or giving developers or users any rights to it, and more importantly, without having to open source their special components so that non-Google devices could use them.

Sure, this is a charitable interpretation. They might just have screwed up royally, buying a company (Android) that wasn’t anything special and releasing something that no one other than journalists will pay attention to. But given that Google is betting $4.6 billion on the 700MHz spectrum auction in the U.S., I’m reluctant to simply write Android off as a “hail mary” acquisition.

There must be a larger strategy here, and I suspect it’s Google putting itself in the shoes of mobile carrier, handset OS maker, and service provider. Somebody else manufactures the handsets, of course, but Google owns and operates the whole experience from end to end other than that. You buy a Google device, pay a monthly Google subscription fee, your bits travel over a Google global network of wireless towers and a wired backbone, and you run apps on your Google phone that interact with Google back-end services. It’s a carbon copy of Apple’s iPhone/iTunes strategy, without AT&T in the picture, and with third party apps allowed on the phone, as long as they work on the already-published Android SDK. All Google has to do is to make a decent looking device and be less customer-hostile than existing U.S. mobile providers, and they’ll do well.

In this light, the partnering talks with existing mobile carriers is puzzling. It’s possible that they’re pitching a strategy that removes the burden of application development from mobile carriers, allowing them to be the billing companies that they really are, and putting Google in the position of being the provider of content and software. The lukewarm responses from these carriers is predictable; for the carriers to be involved as just bandwidth providers and customer billing service providers, there would have to be a careful negotiation of revenue sharing, or else the carriers will simply continue on their current, very lucrative course.

If they are planning to become a carrier themselves, then nothing can actually begin to happen until January, when the 700MHz spectrum auction actually takes place. There’s a closer December 3rd deadline for Google to reveal their plans. At that point we’ll know what they’re up to, assuming they’re actually able to buy the spectrum that would make it possible. Alternatively, this spectrum purchase could be a bargaining chip on Google’s part, which they do not intend to directly utilize themselves. Google would provide the spectrum, the handsets, the OS, the apps, and the services, and the carrier partners would provide the towers, the maintenance staff, sales, and billing.

Given all this, it makes little sense for Google to GPL the Android platform. They need to own it so that they can assure a prospective carrier partner that they will be the ones whose phones are being used by customers, in order to share revenues. If they were to open up the handset market entirely, the carriers would block any new entrants and Android based phones would be doomed.

Android is a huge win for developers. The SDK is already available for Windows, Mac-on-Intel, and Linux-on-i386, and it uses technologies that are already mainstream. Based on my previous post I am curious about whether Rhino, Jython, and JRuby will work on the Dalvik VM, but I have no specific reason to believe they won’t. This is exactly the sort of thing I was talking about when I said that layering on top of the JVM or .NET DLR would ease portability; the Dalvik JVM means that you can likely write a Hello World in any language that can compile down to Java bytecode and run it immediately.

OK, so it’s great for developers. So what? Developers don’t control the mobile market; carriers do. Handset makers would probably love to use Linux instead of paying a per-handset license for a closed phone OS; PalmSource/ACCESS and Palm, Inc. have already said they will move in that direction (though Palm, Inc is creating some confusion about whose Linux-based Garnet-compatible runtime will end up on future devices bearing the Palm OS name). But why would carriers want this?

It’s possible that carriers would like to see their value-added apps run on many different handsets without the cost of developing them separately for each handset they sell. Handset makers and mobile OS vendors clearly are making some money from consulting to carriers on these projects (somebody has to tell Sprint how to write the PictureMail app), so actually in this area handset makers would stand to lose money.

What carriers probably would like less about Android is that it would allow Google to bypass the carriers’ value added services and build their own ecosystem of mobile apps for Android-based handsets, which is exactly the point of Android. Who gets the value added dollars from customers? That’s what this is all about. Google is battling ISPs regarding Net Neutrality, and it comes back to the same thing. If a customer is going to pay for a service delivered from a server across a network onto an endpoint, there are at least four parties that want to get paid, and who view the division of revenue as a zero-sum game.

The server folks (Google, Yahoo, Microsoft, Apple, etc.) want to charge you for the subscription to their applications or for individual chunks of content. That charge may take the form of just showing you ads. Then they want to pay a flat rate for the bandwidth across the network to get to you, and will minimize that cost using the massive content delivery networks which they already have in place.

The network folks want to put a toll booth on that network that charges either the end-user or server folks (or both) for transferring paid content, or one that penalizes the end-user (bandwidth shaping) for buying a cheap connectivity plan and then trying to use it for transferring large media files from the server guys. The network folks also still think they can force the Internet to look like Cable TV, by putting up barriers to keep their users from using anybody’s services but their own, so their ISP customers also become their content customers.

The handset OS folks want to be paid to write those apps for the server folks and the network folks who want to also be server folks. They want to encourage developers (ISVs, or server or network big guys) to focus on their platform, thereby making it more attractive to users, thereby making it more valuable so they can charge a larger amount from each phone sold. These folks are directly in competition with Android, whether Google intends to attack them or not.

The handset manufacturers want to minimize the price of their phone (a free OS that supports tons of hardware components that they might use is a good start) and maximize the number of apps that will run on their phone. They should love this, although the most successful high-end handset makers who also use closed OS’s (basically every major handset maker) will not like it as much as the underdogs who sell tons of cheap phones. LG and Samsung say “hooray” while Motorola, RIM, Palm, Sony, and Nokia say “boo-hoo.” Their investment in special fancy phones and fancy apps for their chosen OS is undermined by the prospect of commodified hardware with carrier- or user-installable third party apps.

Users should be happy as well. Developers, as I mentioned, should love this platform, so users should benefit both from more apps and cheaper handsets, and probably also from more service offerings that will work with their handset.

The problem with all this is that as I mentioned in Technical Architecture is a Form of Investing, just because developers like something doesn’t mean it will win. If Google’s aim is to open up and commodify the handset market, they will have to fight the folks that are trying to keep the handset market closed and fragmented. That group includes all of the major U.S. mobile vendors, and the companies who make handset OS’s. The latter group is weak and easily conquered, with the exception of Microsoft; in this space, though, Microsoft is not strong enough to fight Linux and Java. The former group is extremely powerful and will not simply sell handsets that eliminate their chief source of revenue (proprietary value-added services that show up on your mobile phone bill). Nevertheless, these value-added offerings are generally awful and absurdly overpriced, so there is quite a lot of opportunity if someone can break through the carriers’ stranglehold.

The strategy that Google must follow is to convince an underdog mobile carrier to market an Android-based handset to consumers. Google has little strategic advantage to gain from replacing handset OS makers; they are a service and as such need to prevent the network guys from erecting that toll booth in front of Google’s services. To do that they will need to bypass the network guys, and a phone OS isn’t going to do that. Even a handset offering won’t be sufficient; look at Apple’s iPhone bricking debacle for evidence of that. The mobile carriers control the handset makers in the U.S., and Apple has had to learn that the hard way, screwing over their customers who dared to choose another carrier than Apple’s partner. You can bet that Apple wasn’t the driving force behind that decision.

So Google will have to go all the way to partnering with or acquiring a carrier who is currently an underdog and who needs this offering in order to win customers away from the big guys. Alternatively (and less likely, due to the red tape involved) Google will have to become or spin off that underdog carrier themselves as a new carrier.

So, look for the second shoe to drop: not who is going to build a “GooPhone”, but who is going to offer you a mobile plan that lets you use one without placing severe restrictions on what you can run on it.

Second, JRuby 1.1b1 has been released and as expected is considerably faster (see item #5 in this link) than the standard “MRI” runtime. JRuby joins Jython and Rhino in providing a JVM-based runtime for a dynamic language, with features designed to help developers mix and match the dynamic language code with Java code.

See the trend here? Python, Ruby, and JavaScript are emerging as the dynamic languages of the future for .NET and Java developers.

The hard work done by Sun and Microsoft to make their VMs work well is being leveraged by the next wave of languages. Threads, high performance I/O, memory management, and portability are all features that are quite expensive to get right, and the .NET and Java platforms have pretty much achieved that at this point. (Piggybacking newer, higher-level languages on these mature runtimes means that you get a mature new language runtime faster than if each language’s runtime were built from scratch and painstakingly debugged in isolation from the others.)

There are still some hurdles (performance, type safety fears, lack of mass market acceptance, ECMAScript 4 standardization and adoption, etc.), but in 2 or 3 years, things are going to change dramatically in the web application development world. The seeds of this change are already sown, and it’s just a matter of time. Threads, SQL, OOP, and garbage collection are all features of web application architectures that were initially controversial, but have now met with general acceptance. Dynamic languages are clearly the next step.

Obviously, Java and C# are far from dead, and in 10 years people will still be coding in Java and C#, because as with other languages like C and assembly, the newest and highest-level language isn’t automatically right for every project. But if you’re building web applications, most of what your code does falls into the categories of string manipulation, collection operations, or file and socket I/O. Image processing, crypto, full text search, and other CPU-heavy, byte-twiddling features may be part of your application, but you’re not writing the image scaler, RC4 cipher, or inverted index yourself; those are done in a library, probably written in C, and you’re just calling it. So your needs are likely to be similar to the sweet spot of dynamic languages: maximum expressivity and the fancy features to let you write clever code, making you productive and making the code as clean and elegant as possible. In other words, they put developer productivity first (lower labor cost and shorter development schedules) at the expense of runtime performance. Since hardware gets cheaper over time but code gets uglier over time, this is probably the right choice to make for most web application projects.

Another interesting benefit of layering instead of starting over is that the integration between dynamic languages and Java or CLR languages is much nicer than managed vs. unmanaged code in .NET or, even worse, JNI in Java. That is, it won’t be a bloody mess to mix and match code, from a technical feasibility standpoint. This matters, because The Big Rewrite is among the Things You Should Never Do. But little bitty rewrites are fine, especially if you have a thorough test suite to help you avoid breaking things. (By the way, dynamic languages are great for writing automated tests.)

Which of these three (JavaScript, Python, or Ruby) is going to be dominant 5 years from now? I don’t think any of them will be. The dynamic language community is fragmented, and the various vendors and big sponsors of these three languages are fairly entrenched already. Microsoft is investing in all three; Google has standardized on Python and JavaScript to the exclusion of Ruby; Sun has hired the JRuby team; Mozilla is heavily invested in JavaScript; Adobe supports JavaScript in AIR but not Ruby or Python, etc.

In fact, if you encapsulated the glue code sufficiently well, you could mix and match JavaScript, Python, and Ruby in your application, and port your hideous hydra between the JVM and .NET. You would be wasting a lot of effort since the three languages are largely similar, but you could do it. Alternatively, you could create a portability layer between the DLR and the JVM a la WxWindows, and write-once-debug-everywhere in a more productive language than Java.

These are all repugnant ideas, but only because as I write this and as you read this, we probably realize that to attempt this today would be a huge task. But what about in 2010? Probably not so gross. What about an application that could be executed on Silverlight, AIR, Firefox, SWT, and Mono, unmodified? How about a mobile app that runs on smartphones regardless of the runtime (.NET vs. J2SE)? Not gross at all, and not unthinkable if your app is written in JavaScript using some kind of portability layer that doesn’t exist yet.

In the longer term, JavaScript (a.k.a. ECMAScript 4) is likely to become extremely popular. As far as I know it’s not quite a perfect fit for Steve Yegge’s The Next Big Language, but it’s the closest thing there is, and it has two critical advantages over Ruby and Python that will make it successful: C-family syntax (which makes development tools cheaper to build) and effectively unanimous buy-in from vendors and developers.

So, what about the other dynamic languages that people are using in large numbers today? What’s going to happen to ActionScript, CFScript, PHP, and VB?

ActionScript and CFScript are pretty close to JavaScript by design; I’ve read that ActionScript 3 is actually compliant with the ECMAScript 4 draft specfication. It’s pretty clear that Adobe is betting on JavaScript. In the near future (2 or 3 years) I predict that Adobe will rev its products and support ECMAScript 4 across the board.

PHP and VB.NET/VBScript will hang around for a long time because they’re approachable and already very popular, but they’ve already peaked, and will steadily decline as developers switch to C# (on the .NET side) and Rails (on the Linux side), and then JavaScript as soon as a serious web app framework and an ISP-friendly runtime exist. Microsoft will keep investing in VB to keep customers happy; Yahoo will keep investing in PHP because it is so heavily invested in PHP already; new developers will find PHP to be an easy starting point for light duty web development, with tons of documentation and free applications that they can download and hack. But PHP will not inherit the kingdom from C# or Java, and the languages which do achieve mainstream success after C# and Java will do everything that PHP does language-wise, and the market momentum around those languages will make them better than PHP at what PHP does. Developers will ask themselves why they would write the client side and server side in two different languages, especially when the server-side language is more expressive and has better portability and libraries. That’s not true yet, but it will be in a couple of years. In 5 years or so PHP and VBScript will go the way of Perl CGIs: still used, but by a community a tenth of the size it is today.

What about the new Java-based dynamic language, Groovy? Groovy is interesting, but it’s too late. The Java mainstream of vendors and developers only recently managed to convince the world of “serious” C++ developers that automatic garbage collection and JIT compiled bytecodes can actually work in a high traffic context. The next battle, to promote the dynamic language features that Java lacks but which Groovy brings, will take years to fight. Once a developer makes a decision to not use standard Java, Groovy is on a more or less level playing field with the JVM-hosted versions of Python, JavaScript, and Ruby, but each of those languages has far greater adoption than Groovy, and each of them has greater opportunity for leverage on other runtimes than Groovy. For a Java developer, once the door is opened to other languages, the only advantage Groovy has is that its syntax is familiar. Compare this to JavaScript which web developers also need to know how to use; why learn a third language (Groovy) in addition to Java and JavaScript? Over time the simplicity of coding and debugging in JavaScript on client and server, together with dynamic-language productivity, will overcome the momentum of the Java standard, and web developers using server-side Java now will gradually replace it with JavaScript on the JVM. Conservative attitudes in the mainstream Java community (including Fortune 500 companies and the many offshore development firms that write code for them) will make this take quite a while – probably 5 years before JavaScript becomes a common part of the architectures that currently use J2EE, and 10 years before Java goes the way of COBOL (maintained forever but not used for new projects).

So in conclusion, keeping an eye on the future value of a technology, including who’s investing in it and who’s talking about investing in it, is critical to making your own investments today. In five years you’re not going to be using the same technology stack that you are today, and your project’s success and your own salary will be tied in large part to how well you invested today.

I’m reminded of a James Gosling quote disparaging Microsoft tools, particularly Visual Basic: “The easy stuff is easy, but the hard stuff is impossible.” There’s a parallel between VB and Rails in this instance, in that if you only let yourself use the high level tools, the hard stuff is impossible, but the designers specifically tell you to do the hard stuff using a lower level toolset. The controversy that surrounds “X can’t do everything, therefore it sucks” should really be focusing on the feasibility of going through that trapdoor to do things “the hard way”. This is what Delphi did, which is why so many folks chose it over VB; it made the hard stuff easier.

Here’s the task I need to accomplish, for which ActiveRecord is not well suited: complex queries involving SQL functions and multiple-table joins. I want to join a few tables together, order by a SQL function, include with each result row the result of a SQL function that operates on each row, and have all that come back as a graph of high-level objects.

Despite my attempts to use plugins, extend and/or fix bugs in those plugins, and to dig through the ActiveRecord source to figure out what the documentation won’t tell me, I was unable to get it to work. Most of the parts of what I wanted was possible: acts_as_tsearch cleverly weaves SQL functions into a high-level ActiveRecord::Base.find calls; paginating_find provides a very convenient pagination API on top of ActiveRecord::Base.find, and ActiveRecord includes some clever association tricks such as automatic many-to-many relationships (has_and_belongs_to_many), eager loading of associated records using a join (via the :include option to ActiveRecord::Base.find), and a fairly low-level :joins option that lets you add tables to a ‘find’ query which can be used in your :conditions. Problem is, they don’t all work together in a fancy way.

Really, the issue in this case is related to the design choices that went into ActiveRecord.

Some ORMs (object-relational mappers) are designed in a modular fashion: there is a part that helps you describe the relationships between your model objects, a part that helps you construct queries, and a part that does the storage and retrieval. Sometimes there’s another part that uses your description of object relationships to create an empty database with the appropriate data model, or that looks at an existing database and creates an object model that matches it. Sometimes there’s an import/export tool for bulk data loading or dumping as well.

ActiveRecord has the first three functions integrated (which has benefits and drawbacks compared to a more modular approach), has a very isolated schema manipulation module, and has a somewhat isolated data loader tool.

The relationships are explicitly declared in source code using associations: has_one, has_many, belongs_to, and has_and_belongs_to_many. These are pretty fancy and provide some convenience features that make the associations appear as object collections, such that changing the collection and saving it turns into insert/delete/update activity in the database.

Query construction is basically tied to the objects themselves, in a way that greatly simplifies star-join queries, but which handles only the simplest joins across multiple tables, and is barely able to handle self-referential joins at all. So, you can easily load an object (or group of similar objects) and associated objects, but OLAP-style queries (”what are the top 5 states where customers are located who have bought classical CDs within 2 weeks of their release using American Express and had them shipped as gifts via UPS 3-day Select?”) are impossible. Oddly, views, functions, and stored procedures could bridge the gap between real-world data models and ActiveRecord’s limited set of association types, but they are not supported either.

The storage and retrieval code is inseparable from the query code, and so it is not possible to examine and modify the final SQL before it is executed, nor is it possible to provide an arbitrary query and have the results be parsed into an object graph based on the associations you have defined. The code that would allow these features appears to exist and be sufficiently well designed to allow this with a fairly small amount of changes to ActiveRecord. However, it is currently (as of Rails 1.2.3, which is the current release) not part of the documented API and is declared private.

There is a limited facility for constructing simple objects from arbitrary SQL, in find_by_sql. This loses essentially all of the high level functionality of the find method; most notably, it isn’t possible to use find_by_sql results to instantiate an object graph, rather than a flat array of objects (similar to the eager loading feature in the regular find method).

ActiveRecord has fairly good high-level schema creation functionality (”migrations”). Though it lacks concepts for all but the basic database objects, support can be added for foreign key constraints (I kid you not, they aren’t supported by Rails itself!) and views. There’s also a simple way to execute arbitrary SQL. Migrations aren’t technically that amazing, but rather they’re a helpful organizational approach to what can be a really hairy problem: defining a schema and then applying changes to live databases while keeping track of what changes you’ve already applied.

Finally, there is a test data loading facility called Fixtures. The common opinion of Fixtures seems to be that they are broken by design and should be avoided. The main issue I’ve found with them is that the implementation ignores the kind of database design elements that any book on SQL would recommend, such as foreign keys and check constraints. I managed to circumvent this with a combination of a plugin and some customization, described in detail in my previous post, Rails, Fixtures, the Test DB, and Test::Unit. With those changes, all test fixture data is preloaded in the right order (so constraints aren’t violated) before any tests run, and any data alterations within tests are rolled back automatically by Rails.

A secondary issue with Fixtures is that they go directly from YAML text files to SQL INSERT statements, bypassing the ActiveRecord Model classes. ActiveRecord does pretty much rule out any fancy mapping between database tables and objects, so that’s not a problem, but this model-skipping fixture loading implementation means that any code in your model object (validations, before_save filters, etc.) will not be executed when loading fixtures. So fixtures do not work well with the otherwise pervasive Rails design rule of “put all the intelligence in the application”.

Still, despite the commonly-held disdain for using fixtures at all, I find that they can be tamed. In fact I’ve even created a base data facility for loading the fundamental data set that needs to be in the live database (e.g. initial admin user info). My approach is basically to alter fixture behavior to treat it as essentially a bulk data loading tool, and to do the extra housekeeping after loading to make up for the fact that the ActiveRecord model code was bypassed.

As far as I know, there is no bulk data dumping functionality in Rails.

So, to summarize, of the five main ORM features, here’s how ActiveRecord stacks up:

1. Describing Relationships: Easy to understand and use, with lots of slick functionality
2. Querying: Easy to understand and use, but limited to simple join structures, and not possible to customize query building or rewrite SQL before execution
3. Storage and Retrieval: Very easy to use, but only within the limits of the query builder’s features
4. Schema manipulation: Easy to understand and use; limited in functionality but readily extensible; solid third party plugins are available for missing schema objects
5. Bulk Loading and Dumping: Loading is badly designed and implemented, but fixable with some effort; dumping is not offered

Okay, so it definitely makes the easy stuff easy. But what about the rest?

As I observed before, ActiveRecord is not designed as a set of modules that you use to assemble a solution that fits your needs. That’s more of the Java approach to design, and it trades flexibility for convenience. It can be a major pain to assemble a working system out of all of those abstract Java APIs, which are sometimes so comically over-patternized as to draw mockery such as the hilarious “Are Javalanders Happy?” code snippet from Execution in the Kingdom of Nouns. Rails makes the opposite trade-off: sacrifice flexibility and gain a very approachable API.

Unfortunately, the Java approach (too abstract to readily use, but extremely flexible) is easily wrapped with a simpler, more convenient, less customizable API. The Rails approach isn’t internally componentized (have a look at ActiveRecord’s activerecord/base.rb source file in its 2,165-line glory, almost all of which is one class), so if you want to fiddle with its internal behavior, you can’t. So with Rails, it’s all or nothing: high level slickness for simple requirements, or hand-written SQL and hand-coded results mapping for your complex requirements.

As I said at the beginning, though, the key question is not how comprehensive the high level feature set is. More important is the question of how painful things are when you drop down to a lower level for a greater degree of control.

It would be nice if there were a middle level of complexity, between the high-level ‘find’ method and ‘has_xxx’ associations, and raw SQL. There isn’t. I think that the reason there isn’t one is that there is still a persistent belief among many Rails core team members and community members that databases should be stupid: just a persistent hash. Once upon a time I worked that way myself: I didn’t have access to or skill with a SQL RDBMS, and so I solved all of my persistence problems with DBM files, which (using Perl’s Tie::Hash class) are conceptually just persistent hashtables. miniSQL was little more than a SQL query parser on top of that sort of storage engine, and MySQL originally was pretty similar. But big databases have all sorts of useful features that address complicated persistence requirements in a fairly elegant way.

Given that Ruby fans like the idea of domain specific languages, which let you work in a super high level language customized to the problem domain, it’s surprising that Rails groupthink is that SQL is bad. It’s actually a very high level language, and allows a well written database to do some pretty amazing optimization on the fly because it provides a strong layer of abstraction between what you requested and how the storage engine provides it.

No, it’s not dynamic, nor is it pure relational perfection, but it’s pretty darn good. Pre- and post-event validations and arbitrary callbacks to user-specified code, functions providing behavior on top of data… these are all things that Ruby and Rails fans hold in high regard when provided by Ruby and Rails, but which are considered a bad idea at the database layer. As I discussed at length in Rails and the notion of Stupid Databases Being a Good Idea, this is a philosophy rooted in DRY, but it has some major flaws.

Mainly, there is the issue that some things must be done in the data tier, and trying to put them in the application tier doesn’t work. The best example that comes to mind is full text search. Satisfying queries is the database’s job, period. It’s just hideously slow to try and do an inner join in the application across a network link to a database. If you find yourself doing this, that’s a pretty good sign that your architecture is broken. But some queries are too complicated for ActiveRecord, so sometimes you must choose between a series of high level queries whose results are intersected in application code (easy to understand, but extremely inefficient), or hand coded SQL.

Well, SQL is fast and is a high level domain-specific language, so it isn’t actually a bad tool for the job. The problem is that this approach (the trapdoor to the lower level API) is regarded differently by different people. Some see it as a common and reasonable approach to complex requirements; others see it as a bad evil scary thing that should be avoided at all costs, a kludge and a design mistake.

As a result, the low level option in Rails is anemic. It’s there, but you’re not supposed to use it. Ruby’s ActiveRecord Makes Dropping to Raw SQL a Royal Pain (Probably on Purpose) notes that there are no bind variables allowed in ActiveRecord. You may be saying, “No, wait a minute, I’ve used them, that can’t be right.” That’s what I thought. Look at the source; the bind variable functionality is actually a high level feature built on top of drivers that don’t have that feature. Whatever you did at the high level, it’s going to the driver as a single string. Okay, it’s nice that they added that feature, especially since it provides a single point of testing and verification for safe escaping. But that functionality (in sanitize_sql) is not part of the public API. Fortunately that same article provides a workaround that makes sanitize_sql accessible, so you can use bind variables in your hand coded SQL code, and pretend that the driver supports them. But that’s not likely to work forever.

The key problem with ActiveRecord is its least common denominator feature set, based around the least featureful of all popular SQL databases: MySQL. Years ago, MySQL AB (the vendor of the MySQL database) took a strong philosophical stand against pretty much any advanced database features (which their product lacked, and which competing products had), but lately they’ve softened and added those features that they claimed nobody really needed. In the meantime, Rails has been designed with minimal expectations for database sophistication; therefore, the limited functionality of ActiveRecord is fairly complete, assuming you’re using a database with similarly limited functionality.

Triggers, stored procedures, functions, data integrity constraints, nested transactions, and views are all examples of unsupported database functionality. Try and use them via ActiveRecord’s high level API, and you will quickly see how fragile and inflexible ActiveRecord really is. If you shouldn’t need those features in your database, then you shouldn’t need anything that ActiveRecord doesn’t already provide, so it shouldn’t matter that you can’t extend ActiveRecord.

Truly, these are features that you need only in a few small cases in your application, so looking at individual queries they’re needed rarely (which is not the same thing as “never”). But looking at whether you need one or more of them in a given application, they’re needed more often than not. The pain of using hand coded SQL makes this worse: some tricky things could be done either using a view or stored procedure, or using a really slick dynamic SQL statement. Making all of those options painful means that even a clever developer can’t use anything in their bag of tricks to craft an elegant solution.

Unfortunately, non-trivial web applications need things like full text search, complex associations between persistent objects, non-trival summary information about associated objects, and complex reports, and ActiveRecord fails at all of these. These are not just things that big dumb ancient companies that like using Object COBOL think they need; Amazon and eBay need them too.

The acts_as_tsearch plugin is a good case study of ActiveRecord’s design flaws. TSearch2 is the standard PostgreSQL full text search engine, and it’s pretty good in my opinion. It’s also pretty straightforward to use. Unfortunately for developers using Rails, TSearch2 uses SQL functions (mainly to_tsquery and rank_cd). The acts_as_tsearch plugin tries to inject SQL into ActiveRecord’s queries via the high-level find interface, but ultimately fails as soon as you use the :joins or :include options. The problem is that ActiveRecord has a very simplistic idea of how queries and joins work, and so if you need to inject SQL functions to get the job done (as is necessary in TSearch2 queries), too bad. (See also issues 7 and 8 in acts_as_tsearch, in which I describe and attempt to clean up the mess that results when you use find_by_tsearch in non-trivial ways.)

A fellow Rails developer asked me in all seriousness why I wasn’t abandoning the full text search functionality of TSearch2 and just using a completely separate, redundant database product designed exclusively for full text search. Seriously, that is considered the “easy” approach: one database for full text search, and another for ACID/OLTP/CRUD. Honestly if I were going to go down that road I would try hard to just abandon the SQL RDMBS and put everything in the other database, since Lucene and its imitators are capable of far more than just find-text-in-document queries. The pain of duplicating everything, using two query languages, two document representations (in addition to the object representation in Ruby) and writing application-tier query correlation makes the double-DB approach seem very unwise.

It makes far more sense to me to use the SQL RDMBS’s full text search facility, even if there’s a 2x or 3x read performance penalty, because the conceptual simplicity of having one powerful storage tier (instead of two halves cobbled together) eliminates a ton of ugliness in the application, and the SQL RDBMS is going to get clustered for reads anyway. Nevertheless, even if I’m wrong about this case (putting search in the SQL RDBMS instead of in a separate server), there are other cases for needing a smart database that gives you exactly the results you need and lets you push data logic into the data tier.

So, what do I suggest? Abandon Rails? Nope. I still like Ruby a lot, and find Rails very useful. I just think that ActiveRecord needs to support the low-level and middle-level abstractions better.

Specifically, supporting bind variables (either by exposing that sanitize_sql function, or better yet by making drivers and connection adapters support bind variables for real) would make the find_by_sql, select_all, and exec approaches to low-level SQL query execution less painful.

More difficult, and substantially more valuable, would be refactoring ActiveRecord::Base to split it up in the way I described above: association descriptions and unmarshalling code separate from query building code separate from SQL execution and result retrieval code. All of this could remain hidden for most users under the same old slick high-level API, but for advanced requirements, the ability to fiddle with the SQL and still use the built in high-level unmarshalling code to create object graphs from flat result sets would be very powerful, and useful.

I looked at one alternative to ActiveRecord, called Sequel, which overlaps with ActiveRecord only partially. It is a query builder and lazy result proxy, which is actually what I thought ActiveRecord would do when I first started working with Rails. The proxy design means that you can either keep adding constraints or start fetching results, from the same Dataset class. This seems like a pretty good approach, though I haven’t really looked closely to make sure it would fit what ActiveRecord needs.

What Sequel lacks, though, is the unmarshalling side: turning a 2-dimensional (rows of columns) result set into a complex object graph (customers with orders with order lines with products from suppliers stored in warehouses), with user-controlled eager or lazy loading behavior. Ruby is well-suited to a design that would allow user-specified code (i.e., a block) to decompose each row into the object graph associated with that row, leaving the remaining associations on those objects to be lazily provided via future queries.

So, I think there is hope for ActiveRecord, definitely. I considered the idea of rolling a minimal Hibernate clone, or some other sort of challenger to ActiveRecord, but I don’t that ActiveRecord is broken beyond repair. I think the shortest path to a badass Ruby ORM is through improvements (refactoring and abstraction) to ActiveRecord.

So, if you’ve read this far, you probably care about these issues. Here’s my call to action: Please help me make ActiveRecord less like VB and more like Delphi. Who else is interested in helping me with this effort? Are there alternatives that I’ve missed, or components that could be integrated into ActiveRecord to make it better?

Eclipse is an IDE so it’s hard to imagine it not being baroque and difficult to use, requiring weeks of effort to become productive. JBidWatcher has saved me a lot of money on eBay so I could probably put a dollar value on how much it’s worth to endure its bizarre UI. Azureus is fairly fugly also but it does a very good job and has a deep, sophisticated UI that’s fairly easy to understand, so despite the eyesore, it’s at least fairly clear. The common thread among all of these is that they are all written in Java, and that they are so valuable that it’s worthwhile to overlook the ugly UIs.

Now imagine those sorts of trade-offs, but on already difficult to use mobile devices, and aimed at consumers. Are you making a strategically wise choice by sacrificing usability and control over the user interface, and probably access to platform-specific features such as dialing the phone, in order to save money on development? Adam Breindel talks about this in When Building a Smartphone App, Resist the Siren Song of J2ME.

Adam and I worked on a J2ME application and I totally agree with him about the disillusionment of trying to write a single app that would work across phones. Issues include:

• Complex and difficult application installation procedures for end-users: How do you install the JVM on the phone? How do you get the plain J2ME app packaged up so that the phone will accept it? Does the app require manual user configuration before use? Is there a different launching process from other apps?
• Lack of control of the user interface: hardware details such as how many buttons you have, whether there’s a stylus, etc. differ from phone to phone, and the API to let you code once and let J2ME handle the layout for each device leaves your code very disconnected from what’s actually happening on the screen.
• Not being able to use recent J2ME APIs because even the latest phones only support older, more minimal J2ME APIs
• Not being able to do things on a phone that would seem obvious, like dialing the phone, opening a hyperlink in the phone’s browser, sending an SMS, or making a network connection. Either these are entirely impossible or require phone-specific or JVM-specific tools and procedures to sign your application, or having the runtime nag the user to request permission to do something that they just asked the app to do for them.

For all the noise Sun is making about broad J2ME penetration, the developer experience is quite disappointing, and as a result, the user experience is also quite disappointing. You can look up J2ME features and APIs and get excited, but when you actually deploy your app to a handset, it won’t load, or runs terribly slowly, or looks awful, or simply doesn’t do the things that the API says will happen when you call it a certain way. Suddenly the strict J2SE and J2EE logo certification programs make sense, because the J2ME approach of making so much functionality specified but optional leaves developers high and dry. The phone supports J2ME version xyz, but write an app coded to that API that works on the emulator and deploy it to a handset and lo and behold, all those optional APIs turn out to be missing even though the handset is capable of that functionality, and some mandatory API functions are not working. Here be dragons.

Case in point: can’t dial the phone on a Treo 650 (at least, not as of a year ago). The J2ME API tells you how to do it. The PalmOS JVM (made by IBM) lets you make the API call, and returns a successful response. Nothing happens. IBM says they’re aware of this issue. The end. The docs say you can, the code you write says you did, the phone just doesn’t do it.

Case in point: Every time you start an application and it accesses the network for the first time on a Treo 650, the user is nagged for permission to access the network. Quit the app and start again, nagged again. IBM has a tool that you can use to sign the app, but you have to use their VisualAge Micro Edition IDE which costs hundreds of dollars to do that. Try and find and download the trial version. A year ago, it was not possible. So, making that persistent nag go away probably costs several hundred dollars. (I never verified that it actually works, just that IBM says the way to sign the app so that it’s trusted is to do that, and that there was no available free way to get that tool.)

These are minor issues, but they certainly interfere with the quick usage pattern of a mobile app, and make it annoying to use your app. Imagine what that would be like if you had a competitor with a native application for that phone, whose application probably cost them more, but their app is better and the user likes it a lot more.

The important distinction here is not cost, it’s ROI. It costs a lot to develop a similar application for each smartphone platform, using that platform’s native tools. It would seem to cost a lot less to develop a J2ME app. But that’s only if you assume that it’s OK to abandon features and settle for a horrid user experience in the course of development.

It’s likely that your goal as a development team is to develop an app that has a predefined feature set that you know the device can support, and a predefined UI design that your mobile-savvy UI people are sure will go over well with users accustomed to that particular kind of smartphone. In that case you will almost certainly fail to accomplish that goal using J2ME. You have to scale back your goal so that you’re satisfied that you got something kinda like what you wanted working on a bunch of phones, and determined users will probably be able to figure out how to install it and make it work.

I call that phenomenon “write once, be disappointed everywhere.”

Let’s continue talking about cost, though. The native apps may require (or suggest) different programming language skills for different devices. It might seem wise to just write everything in C, but I think that’s a false economy as well. The phone APIs will differ so much that you will really need a native developer for each platform, not a team of generic C developers who will figure out the individual phone stuff and be freely floating resources that you can assign to whatever app version needs their attention. Smartphones may run Linux, may run Windows Mobile, may run PalmOS, may run Symbian… these are different operating systems with very different ideas of how applications run and coexist. The platform specific knowledge (APIs, appropriate UI feel, device capabilities) is probably an order of magnitude harder to learn and maintain than the ability to get an application working in a given programming language.

How much of a great C programmer’s skill is really the C language, and how much of it is proficiency with the available libraries on the platform he or she is accustomed to? I think close to 90% of their professional skill set is platform and library familiarity, and 10% syntax and low-level understanding of how the language actually works.

In light of this (just using C doesn’t mean developers or code are portable across smartphones), consider that there are high level languages available for some smartphones. What if that 90% platform familiarity means they can use a language and/or development environment that makes them 5 or 10 times as productive, after spending just a few days or a couple of weeks learning the language syntax?

After our very disappointing J2ME-on-PalmOS port experience, Adam found Handheld Basic which initially appalled me (oh no, BASIC!) but turned out to be a great choice. It’s a flavor of BASIC, so learning the language didn’t take long, and the support happens to be quite good (lots of code samples) so picking up the library portion of that 90% didn’t take long at all. I imagine that C# on Windows Mobile is similar. As more and more phone start to use Linux as their OS (which will be a particularly huge improvement for PalmOS based phones), you’ll be able to use Python, Ruby, Mono, J2SE (a whole different animal from J2ME), TCL, or pretty much any other high level language available for Linux. At LinuxWorld 2006 I saw a development device running unmodified GNOME desktop apps running on a PalmOS device alongside PalmOS apps. There are more options appearing all the time, and with the exception of Handheld Basic, most of them are ports of familiar, mature, well understood desktop languages, with class libraries relevant to mobile devices.

So, I recommend that you work in whatever high level language lets you do all the platform specific stuff you want, and if that means a different language per phone, that’s actually going to be the least expensive way to get to the apps you actually wanted to build. A big chunk of the scary cost of developing native apps goes away, because native doesn’t necessarily mean abandoning Java for C. (In fact I found Handheld Basic to be a more productive environment for me after a couple of weeks than J2ME was, despite my ~8 years of full time Java experience before starting that project.)

That pretty much means no reuse for you. Sorry, but that’s the deal right now.

If that unique-app-per-platform cost is too scary, consider a few ways to save money:

• For networked apps (aren’t they all?) ask yourself if there’s some logic that could just as easily be done on the server as on the client. Is there some complicated parsing code on the client that could be simplified by changing the response format from the server to something that’s easier to parse?
• Can you remove or alter certain features from a subset of platforms you intend to support, so that your premium supported platforms get your ideal app, whereas a few less popular phones still get a nice app, but perhaps one that doesn’t have every feature available on your premium app. You might be reading this and thinking “but that’s what you said J2ME would force me to do! Why is this any better?” The distinction is that you are in control of the decision of what to leave out to save money, whereas with J2ME it’s the platform vendor who makes that decision for everyone using their platform. If you’re writing an address book, not being able to dial the phone is lethal; not writing the code to let the user attach a photo to the entries is not.
• Can you move some user-facing functionality to the server to make the client simpler? Maybe there are some rarely-used features that could be done via a desktop or mobile web browser, so you can focus on putting the ten-times-a-day features on the handset, and making those features fast and convenient. You probably already made that trade-off in general, but perhaps for some kinds of handset, you’ll move the dividing line a little further, so that for users of particularly rare phones, some moderately frequent features can’t be done on the phone. This could also be a good approach for new handset types: design a “lite” app and a “full” app version, build the “lite” app first on each platform, and let user demand tell you whether the full app is worth it. Your developers can tell you much more accurately how much the incremental functionality would cost, since they already have done the lite version. Maybe you’d provide a VoiceXML interface, or mobile web browser interface, for that feature so that the user can still do whatever the feature requires while they’re far from a desktop PC.

A final consideration: labor. Maybe you have some Java developers, or C developers, and don’t have developers good at Handheld Basic or C#, so you’re not inclined to fire them all and hire new developers to do native apps, you’re thinking maybe C on every handset, or J2ME, is still the right choice. I still say that you should probably use native apps in native high level languages on every smartphone platform. If you’re committed to a strategy of good apps on a bunch of different phones, I think I’ve made clear that native apps are the only way to currently get there; J2ME simply doesn’t let you make good apps. So what’s left is C code written by C developers vs. high level code written by C developers who have to retrain.

As I said above, I don’t see much chance that your developers or your code will be portable across different smartphones if you use C. Maybe you’ll get 5-10% savings that way (some code ported across phones, or some developer hours shuffled between platform teams). But you’d get a 5-10x cost saving from using something very modern and high-level instead of C, and the overhead of training for the language would be very very small as compared to the large and unavoidable overhead of having to learn what the handsets can do and how the APIs for the handset’s OS work.

I suppose that means that if you have a bunch of C developers who lack smartphone skills, you’re in a pickle, but that situation seems kind of unlikely to me (a mobile app company hires a bunch of Unix and Win32 C developers with no mobile phone skills?). More likely is that you’d find a mobile developer who is proficient with one or more handset OSs and the best tools for each one, and they may be of the opinion that C is the best choice since they can reuse skills and some code across platforms. I would say that in that case you need to convince them to (or more likely encourage them to do what they were already considering, which is to) go ahead and find a highly productive high level programming language/environment for each smartphone platform.

A final possibility if you have a ton of platforms to target and a pile of killer C programmers is to try and port something like Python across most of your target platforms, or to make a very high-level API or domain-specific language that runs inside your own custom C portability layer. You might be able to find an open source option that you can invest some developer hours in, so that the actual application-specific code that you write on each platform is minimized and portable. But I suspect that this would still be expensive and would result in some J2ME-like UI abstractions that ended up being very unsatisfying in the end.

Best of luck!

When I was in 5th grade (age 9) learning AppleSoft BASIC on the Apple ][+ in math class, we wrote programs that did this:
10 X = 1
20 PRINT X
30 X = X +1
40 GOTO 20
And we would race each other, starting at the same moment and seeing whose column of increasing numbers on the screen scrolled faster. This was of course stupid because all the computers in the lab were chip-for-chip identical to each other, and probably were all made on the same production run on a single day.

We learned that we could cheat by adding a number larger than one in each loop iteration, which was quickly detected and outlawed. Far more cleverly, someone figured out that you could do something like this:
20 PRINT X: X = X + 1: GOTO 20
The same algorithm yielded better performance if it was all written on one line of code. I could be misremembering (it has been 25 years and I don't have a ][+ handy to verify this on) but it was something like that. Anyway, we learned that the same language runtime on identical hardware using the same algorithm could be made to run faster or slower using simple formatting changes.

So does it make sense to compare different languages this way, which may mean favoring one language's idiomatic code structure while hitting a weak spot of another? This is a common, and in my opinion valid, critique of apples-vs-oranges benchmarks: how do we know that the performance difference isn't due to naive coding or configuration on one side and expert tuning on the other side? For that matter, do we know that the benchmark design isn't selected specifically to highlight exceptionally high performance in one area of a product, to the exclusion of embarassingly slow areas that the benchmark designer would prefer that you not consider?

Thus I claim that this benchmark is approximately as valuable as my 5th grade silly hacks. X=X+1, change the number of iterations to suit your bias, or perhaps just don't bother making them the same because it's meaningless anyway. (Z=X*Y and a matrix multiplication are other parts of this benchmark, but they too are so trivial in concept and implementation as to be equally pointless.)

I'm going to guess that the author of the blog post didn't notice the different in loop iterations, or was looking at the per-second values rather than the total run time. But if we're looking at average performance over time, then how long does it take for the performance to stabilize? Stabi-whatchamaha? Ask Zed Shaw: look at his list of pet peeves, #3.

Do we know that 0.142 seconds is enough to measure “language performance” (really, it’s the performance of a particular runtime environment being measured) including stuff like garbage collection and JIT compilation overhead? If one language’s runtime waits for N iterations before JIT-compiling the code, whereas another runtime waits for 5N, how many total iterations do you need to minimize the effect of that?

What happens in JRuby, Ruby2C, etc.? The poster says the tests were run on a MacBook Pro – what architecture (PPC vs. Intel) were these language runtimes compiled for, with what compiler, blah blah. GCC versions, optimized for certain CPU models, etc. This stuff can make a big difference in CPU benchmarks, which is why proper benchmarks include things like this in their configuration information.

Or are you measuring small script execution time, and >1s runtimes are meaningless for your needs, in which case Java seems painfully slow and Bash lightning fast?

What the heck is being measured by this “microbenchmark”? Language fanboy gullibility?

For a less awful benchmark, have a look at the Computer Language Benchmarks Game: “What fun! Can you manipulate the multipliers and weights to make your favourite language the best programming language in the Benchmarks Game?” At least they realize how not-terribly-useful synthetic CPU benchmarks of language runtimes are.

Anyway, for most applications, if you’re choosing your language based on runtime performance, you’re choosing very poorly. If you’re choosing your language based on a really awful “microbenchmark” comparable in accuracy to the first toy hack of a room full of 9-year-olds, well…

Check out the 4th message in the thread, with the big code sample.

• Table based HTML layout, and no CSS at all? Check. Heck, the table width % values don’t even add up to 100%.
• SQL in the JSP? Check.
• Making a new JDBC connection for each page view, instead of using a connection pool? Check.
• Unescaped strings in the SQL? Check. (Not strings coming from the browser in this particular JSP page, but you don’t know where those strings originate. Why wouldn’t you escape it just in case?)
• Failing to use a prepared statement? Check. (That would also solve the escaping problem.)
• Using a string literal in a SQL or command line context, so you can’t log it beforehand? Check. Even better, the code makes a query string first, prints it (commented out), and then uses a different string in the actual query. Nice!
• Using the JdbcOdbc driver? Check. (From Sun’s JDBC Basics: The JDBC-ODBC Bridge driver provided with JDBC is recommended only for development and testing, or when no other alternative is available.) I’m guessing that the use of ODBC here is the only reason why the database username and password aren’t embedded in the code sample and posted for all to see.
• Empty exception catch block? Check.

I’d have a hard time coming up with a fake example of bad code that was worse.

But wait, what else has this person asked about?

No way. Yes! Error in Socket and File Writing!
The post includes the router’s username and password, and its configuration including:

• Its IP address, and all of the routes it contains, and all of its interfaces and where they go
• A couple of other passwords stored in the router
• A crypto key that appears to be to a VPN (looks like a pre-shared key, meaning not a public key but one that must be kept secret)!

I’m not gonna say “you get what you pay for” since open source software has served me very well, but I will say that you get what you bargain for. If your bargain includes not looking at the work product of the people you hire, which is to say, hiring the cheapest people available and not supervising them, you’re not going to be happy with what you get.

Of course, this could have been written by a U.S. citizen who works in a cube on-site and makes $200/hour. Point is, hire carefully, and supervise your workers. It seems simple when put that way, but it’s amazing how often companies are willing to hire software subcontractors carelessly (solely on price?) and then pay little or no attention to the resulting work, when the arrangement involves offshore outsourcing.

By the way, the IP addresses in the original post (with the awful code sample) are listed next to the name “Areva”, implying that this code is part of a project for Areva. Who is Areva? They make nuclear power plants. Sweet dreams!

When I’ve implemented (in Java) what Rails does for database preparation, I did it like this:

1. Create the test database exactly the same way that the developers’ databases are created: by running the exact same code, pointed at a different database.
2. Load the appropriate sets of data for the test database. “Sets” is plural on purpose; most non-trivial databases include code tables, which constitute base data which are essentially part of the database design itself. Then, test code will want a fixed set of known test data to act upon, so that tests can measure whether the code did the right thing given the test data (the right inputs yield the right outputs).
3. Run the individual tests, providing some way of assuring that changes to the test data are undone before the next test.

At first (11 years ago) I used a hand-maintained SQL DDL file to create the databases. Later I split that up into one file per table, and made a list of the proper ordering of tables during creation (reversible for deletion). Later still, with Hibernate, I ditched the DDL and let a higher-level ORM description of the table do the schema generation (which was painful in Hibernate since it wasn’t made to do that except from the command line, but it was possible to hack it into a state of relative beauty). The test data was always loaded from a bunch of text files that were easy to hand-edit (as opposed to a bunch of SQL INSERT statements).

Running the test with assurance of pristine test data was more or less horrific in a J2EE+Hibernate 2.x environment. The design of Hibernate and JUnit made it difficult to wrap tests in transactions, and the version of MySQL that we were using had no transactional storage engines available at all (MyISAM? Thanks, Red Hat!), so I ended up falling back on an intrusive but relatively high-performance design that required tests to declare if they were going to alter the test data, so that the test teardown method knew it had to reload the test data. Since we were waiting for Hibernate 3.0, MySQL 5.x, and a few other things to become part of our architecture, I left that solution in place and ended up moving on to a new job before fixing it.

Rails initially seemed to nail this problem: the test database is automatically made based on the development database; the data is loaded from YAML files called Fixtures, which feature a very simple and straightforward API, and tests run inside individual transactions. Nice!

Except not. Fixtures are loaded by specifying the tables for which you need test data loaded, and this is done in each Test::Unit::TestCase class, of which I have several hundred. They are stupidly reloaded each time you say a given TestCase is going to use them. Worse, the tables you’re using for this TestCase are emptied out using SQL DELETE statements, but if there is test data in other tables that has foreign key dependencies on the data being deleted, fixture loading will fail. (Rails was not designed for FKs to be enabled in the database, so encountering this this bug is a side effect of enabling them via the plugin.) This deletion behavior is pointless in light of transactions wrapping each test, but if you’re using MySQL MyISAM you can’t use transactions, so it needs to be there for people using MyISAM, which is to say, crazy people who care not for their data.

Since Test::Unit, like Java’s JUnit, lacks a hook for the beginning or end of a given TestCase class’s set of tests, there’s no way to accumulate a list of fixtures created and then delete them and/or reload them at the end. That would at least allow you to undo the creation of the fixtures so that the tables were all empty before the next set of fixtures were loaded. Sadly, Test::Unit is not that clever.

I initially fixed this problem a couple of months ago, using a hack that simply refuses to delete and re-create (test data) fixtures if they’re already loaded. That works since the fixture data progressively accumulates and is always clean since changes within tests are rolled back at the end of those tests.

Upon adding a trigger to a Rails migration and then writing a test case that checked to see if it was working, I found the true ugliness. Rails has Migrations, which in my opinion are an excellent feature that works well, and is a more useful generalization of my ordered-list-o-tables and set of table-definition text files. But… when creating the test database, Rails uses the SchemaDumper’s schema.rb output to create it, instead of using migrations. Talk about principle of least astonishment… I was pretty astonished. We have migrations, which is how we create databases! Great! So let’s use this other thing instead.

Also, SchemaDumper does not in fact dump the schema; it dumps tables and indices only. The RedHillOnRails foreign keys core plugin adds foreign key dumping to this output, but forget about check constraints, triggers, and stored procedures. Those schema objects are ignored, so your test database is not the same as your development (or production) database. Whoops.

I thought of about a dozen ways to deal with this:

1. Abandon triggers and do it all in Rails, make a TODO to fix this later, and get on with feature implementation
2. Add code to the tests to check for the missing schema objects and add them if missing (eww)
3. Replace the db:test:prepare Rake task with one that tells PostgreSQL to copy the database as-is
4. Replace the db:test:prepare Rake task with one that tells PostgreSQL to use pg_dump instead of ActiveRecord::SchemaDumper
5. Hack the PostgreSQL-specific code that SchemaDumper uses to look at the pg_proc and pg_trigger system catalogs and use code similar to the RedHillOnRails Core plugin to dump stored procs and triggers into schema.rb also
6. Just dump using pg_dump into a temp file and parse the output and add that to schema.rb (ewwwwwww)

etc. etc.

I finally found the Migrate Test DB Rake Plugin which simply uses your Rails Migrations to create the test database. Lovely. Except I now had some new problems.

1. rake db:schema:purge for PostgreSQL does dropdb/createdb on the test database to empty it out. That creates a database with no built in procedural langauges, so stored procs won’t work. Adding the language to that database is a DB superuser task, so it couldn’t be done inside of Rake. Fortunately I found that I could solve this via “createlang plpgsql template1″ which puts plpgsql in the template database used for creating new databases. Easy.
2. My never-delete-fixtures code got into a fight with my base-data-loader code. They both used Fixtures to load data, and so the base data fixtures made the never-delete-fixtures code think that the test data was already in. So the tests failed due to lacking test data.

I fixed this initially by modifying my BaseDataLoader class to not load base data if RAILS_ENV is ‘test’, and added code to the Migrate Test DB Plugin to set RAILS_ENV to ‘test’ right before running the migrations on the test database. This is a workaround, really, because it still leaves the base data either missing entirely, or duplicated.

Then I switched to the Preload Fixtures plugin which is nice but still leads to FK related errors. It grab the fixture names from your test/fixtures directory and loads all the files it finds, in the order it found them. That fails since alphabetical order and the required table creation order are different in my case.

Fortunately since I’m using the Migrate Test DB Plugin I can just observe the order in which tables were created and tell the Preload Fixtures plugin to do its work in the same order. This is in my environment.rb because that’s where all my project-wide monkeypatching currently lives. (Cleaning that up and maybe plugin-izing it is a TODO for the future.)

# Due to FKs, gotta specify ordering of fixture preloading here. Why not let migration create_table statements do it?
# (depends on Migrate Test DB Plugin being present; is here for the benefit of the preload_fixtures plugin)
module ActiveRecord::ConnectionAdapters::SchemaStatements
    alias create_table_orig create_table
    def create_table(table_name, options = {}, &block)
        fixture_filename = "#{table_name}.yml"
        if File.file?(File.join([RAILS_ROOT, 'test', 'fixtures' ,fixture_filename]))
            ENV['FIXTURES'] = [ENV['FIXTURES'], fixture_filename].compact.join(',')
            # puts ENV['FIXTURES']
        end
        create_table_orig(table_name, options, &block)
    end
end

Sadly if you run “rake test” it runs ruby as a subprocess in order to do “rake test:units”, “rake test:functionals”, and “rake test:integration”. That means that the migrations are run once (before the tests), but that the preloading is done three times. The second and third times through, though, the preloading fails since it’s trying to delete-then-create each table’s fixtures in table-creation order. So, a patch to preload_fixtures.rb is needed, to ensure that deletes are done first, in the reverse order of table creation. Here’s what the new preload! method looks like:

  def self.preload!
    puts "PRELOADING FIXTURES..."

    require 'active_record/fixtures'
    ActiveRecord::Base.establish_connection(:test)
    fixture_filenames = (ENV['FIXTURES'] ? ENV['FIXTURES'].split(/,/) : Dir.glob(File.join(RAILS_ROOT, 'test', 'fixtures', '*.{yml,csv}')))

    # delete first, in reverse order
    fixture_filenames.reverse.each do |fixture_file|
        table_name = File.basename(fixture_file, '.*') # hack; might not be correct if class name != camelized table name
        ActiveRecord::Base.connection.delete "DELETE FROM #{table_name}", 'Fixture Delete'
    end

    fixture_filenames.each do |fixture_file|
      Fixtures.create_fixtures(File.join(RAILS_ROOT, 'test', 'fixtures'), File.basename(fixture_file, '.*'))
    end
    puts "DONE. Loaded #{Fixtures.all_loaded_fixtures.keys.length} fixtures."
  end

I’m not sure, but I think there’s an assumption in there that the table name is the same as the fixture name. My patch also makes that assumption, which is true in the case of my project. But in your project you might not have done that, so further hackery might be needed.

So, it all seems to work correctly now, and I’m back to working on my trigger code. If this seems like it took a lot of effort, it did, but I think it’ll be worth it once I start using stored procs and triggers more. That phase begins now.

Anyway, EMMA is very conservative about coverage estimates. Consider this single line of Java code:

int x = 1; if (x > 1){throw new RuntimeException();}

When you run it, the assignment will execute, as will the comparison, but the block containing throw will never be reached. So, the line is not fully covered. EMMA looks at bytecodes and maps those back to line numbers, and will mark this line as partially covered.

It’s maddening sometimes, but it’s correct; it’s your problem to figure out how to force all those darned IOExceptions that you know can’t ever happen, or to give up and let some apocalyptic error handling code not be covered. I used to shoot for 85-90% coverage, and just live with some uncovered wacky code when I couldn’t find a reasonable way to trigger hideous errors.

(There’s a little voice in my head that says, “Come on, you slacker, you should make it possible to dynamically break the database configuration and remount the root filesystem as read-only during the test suite so that the error handlers can all be 100% covered!” But that little voice never ships anything, he just sits around and writes more and more paranoid code — no Ariane 5 disaster on my watch! — and gets asymptotically closer to 100% coverage.)

Okay, now the Ruby version:

x = 1; raise RuntimeError if x > 1

Same story, prettier code (cuz it’s Ruby; duh, of course it’s prettier): assignment gets executed, condition is executed, no exception is raised. Rcov says it’s 100% covered.

The difference is C0 coverage (Rcov) vs. C1 coverage (EMMA). EMMA only counts a line as 100% covered if all of the bytecodes compiled from it were executed. Rcov counts a line as 100% covered if execution visited that line at all.

(EMMA uses “basic blocks” instead of bytecodes, so in obscure failure cases the reported line coverage figure is even lower, but that doesn’t affect the conditions for what 100% line coverage requires.)

I’m not sure if it’s feasible to get C1 coverage on the regular Ruby VM, but it would be nice to have. In the meantime, just be aware that 100% C0 coverage is easy to attain with rcov, but doesn’t mean you’re done testing everything.

Of course, 100% C1 coverage doesn’t mean you’re done testing everything either, but it’s closer to meaning that than 100% C0 coverage is.

In fact, it’s good to over-test code that’s already covered once. On any given project in the real world, there is code that will succeed with some values and fail with others. (print x/y, for example.) Just testing with one set of values isn’t enough. I like to create an array of inputs and expected outputs and use Array.each to iterate over them and pass them into a block that calls the code under test and then compares the actual outputs to the expected outputs with assert_equal. It means that I’m following DRY (rather than cutting and pasting several lines of test code and then editing a teeny bit of each pasted chunk… eww) but still testing many different ways.

But, even with a C0 coverage tool, at least you know what hasn’t been touched at all, and you can use your big smart programmer brain to think of a few ways to torture that code. Just remember that you probably should be aiming for an unmeasurable way-over-100% imaginary coverage target, beating the heck out of your fanciest code with many, many different inputs and then moving on to the next chunk of uncovered code only when the last chunk is thoroughly proven to not suck.

Good testing!