I’m not interested whether he likes Rails or not. Lots of people hate Rails, and I don’t care. I’m not going to make a big deal about the fact that he’s comparing a runtime architecture (Apache + mod_perl) with a framework (Ruby on Rails).

Those are insignificant compared to his claim that scalability means “Can deal with workload”. Actually, that’s a description of capacity.

Scalability is a very distinct concept from capacity. Scalability is not a true/false property of a system; there are degrees of scalability, which can be represented in a 2D graph of # of simultaneous requests that you can service with an acceptable response time (X axis), plotted against the resources required to service those requests (on the Y axis). The function f in the y=f(x) equation that is behind that graph is how scalable your application is.

If it’s a straight line, that’s quite good: “linear scalability”. More requests cost the same amount per request as the ones you’re getting now. Double your customers, double your net profits.

If it curves down away from a straight line, that’s even better than linear scalability: you’ve attained an economy of scale, so twice as many requests costs less than twice as much as the amount you’re paying now.

If it curves up away from a straight line, that’s bad, because more load means a greater cost per request. Each new customer makes you less money than the last one. Eventually you will grow to a point where you lose money and your business fails. This is what people are referring to when they say something won’t scale. Linear (or better) scalability curves are what people mean when they say something will scale.

In the worst case, the upward curve is asymptotic to a vertical line. In other words, at some number N of simultaneous requests coming in, you “hit a wall”, and no amount of extra resources will help you. “Allows you to throw money at the problem”, as Mr. Davidson puts it, actually describes all three curves, except for this worst case of curving upward asymptotically. But as long as you don’t hit a wall, “Can deal with workload” is satisfied. The more interesting questions, though, are how much it costs you to add capacity, and whether there’s a certain number of requests above which you start to make or lose money.

Of course, the ideal curves are not what you see in practice. In reality you buy resources in chunks, such as a server or a specific plan of bandwidth, power, and rack space from your colocation provider. The graph looks more like a staircase, and going from N customers to N+1 customers means you have to spend $thousands on new hardware. Each of those chunks represents a certain amount of capacity. Capacity is just a measure of how large each chunk is, or of the largest value of X that your server cluster can support without more resources.

But you can’t just extrapolate from the fact that a single server S will support, say, 100 requests per second, that 100 of them will support 10,000 requests per second. If only it worked that way, capacity planning would be really easy. Sadly, architecting a web application for linear scalability is hard. (It’s doable and the approach is fairly well documented, but it’s not easy.)

I think it’s worth pointing out something now which should be obvious: the slope of a straight line doesn’t change its curvature. If you’re paying a silly amount for each request because you’re using an inefficient architecture that scales linearly, but you’re making an even larger silly amount from your customers, you’re still going to be in business if you have 10x as many customers. You may be leaving money on the table due to inefficient use of resources, but you’re not ruined.

If you’re lucky enough to be in that situation, you can probably hire one or two sysadmin/developer ninjas to optimize your app and change the slope of your line downward. Alternatively, you might decide to increase your profits by just buying more servers and advertising. You could even do both.

Likewise, if you’re losing an average of $5 per customer visit regardless of how many customers you have (with that hard-to-attain linear scalability again), then adding a bunch of servers isn’t going to help you. Sun, Compaq, and Dell sold a ton of server hardware in the late 1990s to companies that didn’t understand this.

In a more realistic scenario, you might be paying a lot for servers but not have much revenue. Improving your application’s efficiency would reduce the cost of your resources somewhat, and you might change from losing money to making money. That’s great, but if your curve bent upward before, it still bends upward, and if you were gonna hit a wall before, you still will. The right decision might be to worry about that later, once you’re profitable. At that time you could afford to change your architecture to scale better. Or, you might choose to invest in the future and focus about scalability improvements and growing your customer base now, losing money now but making piles of cash later when you eventually improve your efficiency.

In conclusion, to tie the two terms together: scalability is a measure of how cost-effectively you can grow (or shrink) your capacity.

And to tie this topic to my ongoing claim that trading runtime language performance for developer productivity is generally a good idea for web apps:

Language performance does not affect whether an application scales or not. It is a coefficient to the cost of capacity.

The cost of capacity affects the slope of your curve, but not the curvature. That’s important. Your architecture and application design are what affect the curvature of your scalability. You need to pay attention to both: the curvature of your scalability function and the cost of capacity will tell you where to invest your developer and sysadmin resources for the best return on investment.

I’m reminded of a James Gosling quote disparaging Microsoft tools, particularly Visual Basic: “The easy stuff is easy, but the hard stuff is impossible.” There’s a parallel between VB and Rails in this instance, in that if you only let yourself use the high level tools, the hard stuff is impossible, but the designers specifically tell you to do the hard stuff using a lower level toolset. The controversy that surrounds “X can’t do everything, therefore it sucks” should really be focusing on the feasibility of going through that trapdoor to do things “the hard way”. This is what Delphi did, which is why so many folks chose it over VB; it made the hard stuff easier.

Here’s the task I need to accomplish, for which ActiveRecord is not well suited: complex queries involving SQL functions and multiple-table joins. I want to join a few tables together, order by a SQL function, include with each result row the result of a SQL function that operates on each row, and have all that come back as a graph of high-level objects.

Despite my attempts to use plugins, extend and/or fix bugs in those plugins, and to dig through the ActiveRecord source to figure out what the documentation won’t tell me, I was unable to get it to work. Most of the parts of what I wanted was possible: acts_as_tsearch cleverly weaves SQL functions into a high-level ActiveRecord::Base.find calls; paginating_find provides a very convenient pagination API on top of ActiveRecord::Base.find, and ActiveRecord includes some clever association tricks such as automatic many-to-many relationships (has_and_belongs_to_many), eager loading of associated records using a join (via the :include option to ActiveRecord::Base.find), and a fairly low-level :joins option that lets you add tables to a ‘find’ query which can be used in your :conditions. Problem is, they don’t all work together in a fancy way.

Really, the issue in this case is related to the design choices that went into ActiveRecord.

Some ORMs (object-relational mappers) are designed in a modular fashion: there is a part that helps you describe the relationships between your model objects, a part that helps you construct queries, and a part that does the storage and retrieval. Sometimes there’s another part that uses your description of object relationships to create an empty database with the appropriate data model, or that looks at an existing database and creates an object model that matches it. Sometimes there’s an import/export tool for bulk data loading or dumping as well.

ActiveRecord has the first three functions integrated (which has benefits and drawbacks compared to a more modular approach), has a very isolated schema manipulation module, and has a somewhat isolated data loader tool.

The relationships are explicitly declared in source code using associations: has_one, has_many, belongs_to, and has_and_belongs_to_many. These are pretty fancy and provide some convenience features that make the associations appear as object collections, such that changing the collection and saving it turns into insert/delete/update activity in the database.

Query construction is basically tied to the objects themselves, in a way that greatly simplifies star-join queries, but which handles only the simplest joins across multiple tables, and is barely able to handle self-referential joins at all. So, you can easily load an object (or group of similar objects) and associated objects, but OLAP-style queries (”what are the top 5 states where customers are located who have bought classical CDs within 2 weeks of their release using American Express and had them shipped as gifts via UPS 3-day Select?”) are impossible. Oddly, views, functions, and stored procedures could bridge the gap between real-world data models and ActiveRecord’s limited set of association types, but they are not supported either.

The storage and retrieval code is inseparable from the query code, and so it is not possible to examine and modify the final SQL before it is executed, nor is it possible to provide an arbitrary query and have the results be parsed into an object graph based on the associations you have defined. The code that would allow these features appears to exist and be sufficiently well designed to allow this with a fairly small amount of changes to ActiveRecord. However, it is currently (as of Rails 1.2.3, which is the current release) not part of the documented API and is declared private.

There is a limited facility for constructing simple objects from arbitrary SQL, in find_by_sql. This loses essentially all of the high level functionality of the find method; most notably, it isn’t possible to use find_by_sql results to instantiate an object graph, rather than a flat array of objects (similar to the eager loading feature in the regular find method).

ActiveRecord has fairly good high-level schema creation functionality (”migrations”). Though it lacks concepts for all but the basic database objects, support can be added for foreign key constraints (I kid you not, they aren’t supported by Rails itself!) and views. There’s also a simple way to execute arbitrary SQL. Migrations aren’t technically that amazing, but rather they’re a helpful organizational approach to what can be a really hairy problem: defining a schema and then applying changes to live databases while keeping track of what changes you’ve already applied.

Finally, there is a test data loading facility called Fixtures. The common opinion of Fixtures seems to be that they are broken by design and should be avoided. The main issue I’ve found with them is that the implementation ignores the kind of database design elements that any book on SQL would recommend, such as foreign keys and check constraints. I managed to circumvent this with a combination of a plugin and some customization, described in detail in my previous post, Rails, Fixtures, the Test DB, and Test::Unit. With those changes, all test fixture data is preloaded in the right order (so constraints aren’t violated) before any tests run, and any data alterations within tests are rolled back automatically by Rails.

A secondary issue with Fixtures is that they go directly from YAML text files to SQL INSERT statements, bypassing the ActiveRecord Model classes. ActiveRecord does pretty much rule out any fancy mapping between database tables and objects, so that’s not a problem, but this model-skipping fixture loading implementation means that any code in your model object (validations, before_save filters, etc.) will not be executed when loading fixtures. So fixtures do not work well with the otherwise pervasive Rails design rule of “put all the intelligence in the application”.

Still, despite the commonly-held disdain for using fixtures at all, I find that they can be tamed. In fact I’ve even created a base data facility for loading the fundamental data set that needs to be in the live database (e.g. initial admin user info). My approach is basically to alter fixture behavior to treat it as essentially a bulk data loading tool, and to do the extra housekeeping after loading to make up for the fact that the ActiveRecord model code was bypassed.

As far as I know, there is no bulk data dumping functionality in Rails.

So, to summarize, of the five main ORM features, here’s how ActiveRecord stacks up:

1. Describing Relationships: Easy to understand and use, with lots of slick functionality
2. Querying: Easy to understand and use, but limited to simple join structures, and not possible to customize query building or rewrite SQL before execution
3. Storage and Retrieval: Very easy to use, but only within the limits of the query builder’s features
4. Schema manipulation: Easy to understand and use; limited in functionality but readily extensible; solid third party plugins are available for missing schema objects
5. Bulk Loading and Dumping: Loading is badly designed and implemented, but fixable with some effort; dumping is not offered

Okay, so it definitely makes the easy stuff easy. But what about the rest?

As I observed before, ActiveRecord is not designed as a set of modules that you use to assemble a solution that fits your needs. That’s more of the Java approach to design, and it trades flexibility for convenience. It can be a major pain to assemble a working system out of all of those abstract Java APIs, which are sometimes so comically over-patternized as to draw mockery such as the hilarious “Are Javalanders Happy?” code snippet from Execution in the Kingdom of Nouns. Rails makes the opposite trade-off: sacrifice flexibility and gain a very approachable API.

Unfortunately, the Java approach (too abstract to readily use, but extremely flexible) is easily wrapped with a simpler, more convenient, less customizable API. The Rails approach isn’t internally componentized (have a look at ActiveRecord’s activerecord/base.rb source file in its 2,165-line glory, almost all of which is one class), so if you want to fiddle with its internal behavior, you can’t. So with Rails, it’s all or nothing: high level slickness for simple requirements, or hand-written SQL and hand-coded results mapping for your complex requirements.

As I said at the beginning, though, the key question is not how comprehensive the high level feature set is. More important is the question of how painful things are when you drop down to a lower level for a greater degree of control.

It would be nice if there were a middle level of complexity, between the high-level ‘find’ method and ‘has_xxx’ associations, and raw SQL. There isn’t. I think that the reason there isn’t one is that there is still a persistent belief among many Rails core team members and community members that databases should be stupid: just a persistent hash. Once upon a time I worked that way myself: I didn’t have access to or skill with a SQL RDBMS, and so I solved all of my persistence problems with DBM files, which (using Perl’s Tie::Hash class) are conceptually just persistent hashtables. miniSQL was little more than a SQL query parser on top of that sort of storage engine, and MySQL originally was pretty similar. But big databases have all sorts of useful features that address complicated persistence requirements in a fairly elegant way.

Given that Ruby fans like the idea of domain specific languages, which let you work in a super high level language customized to the problem domain, it’s surprising that Rails groupthink is that SQL is bad. It’s actually a very high level language, and allows a well written database to do some pretty amazing optimization on the fly because it provides a strong layer of abstraction between what you requested and how the storage engine provides it.

No, it’s not dynamic, nor is it pure relational perfection, but it’s pretty darn good. Pre- and post-event validations and arbitrary callbacks to user-specified code, functions providing behavior on top of data… these are all things that Ruby and Rails fans hold in high regard when provided by Ruby and Rails, but which are considered a bad idea at the database layer. As I discussed at length in Rails and the notion of Stupid Databases Being a Good Idea, this is a philosophy rooted in DRY, but it has some major flaws.

Mainly, there is the issue that some things must be done in the data tier, and trying to put them in the application tier doesn’t work. The best example that comes to mind is full text search. Satisfying queries is the database’s job, period. It’s just hideously slow to try and do an inner join in the application across a network link to a database. If you find yourself doing this, that’s a pretty good sign that your architecture is broken. But some queries are too complicated for ActiveRecord, so sometimes you must choose between a series of high level queries whose results are intersected in application code (easy to understand, but extremely inefficient), or hand coded SQL.

Well, SQL is fast and is a high level domain-specific language, so it isn’t actually a bad tool for the job. The problem is that this approach (the trapdoor to the lower level API) is regarded differently by different people. Some see it as a common and reasonable approach to complex requirements; others see it as a bad evil scary thing that should be avoided at all costs, a kludge and a design mistake.

As a result, the low level option in Rails is anemic. It’s there, but you’re not supposed to use it. Ruby’s ActiveRecord Makes Dropping to Raw SQL a Royal Pain (Probably on Purpose) notes that there are no bind variables allowed in ActiveRecord. You may be saying, “No, wait a minute, I’ve used them, that can’t be right.” That’s what I thought. Look at the source; the bind variable functionality is actually a high level feature built on top of drivers that don’t have that feature. Whatever you did at the high level, it’s going to the driver as a single string. Okay, it’s nice that they added that feature, especially since it provides a single point of testing and verification for safe escaping. But that functionality (in sanitize_sql) is not part of the public API. Fortunately that same article provides a workaround that makes sanitize_sql accessible, so you can use bind variables in your hand coded SQL code, and pretend that the driver supports them. But that’s not likely to work forever.

The key problem with ActiveRecord is its least common denominator feature set, based around the least featureful of all popular SQL databases: MySQL. Years ago, MySQL AB (the vendor of the MySQL database) took a strong philosophical stand against pretty much any advanced database features (which their product lacked, and which competing products had), but lately they’ve softened and added those features that they claimed nobody really needed. In the meantime, Rails has been designed with minimal expectations for database sophistication; therefore, the limited functionality of ActiveRecord is fairly complete, assuming you’re using a database with similarly limited functionality.

Triggers, stored procedures, functions, data integrity constraints, nested transactions, and views are all examples of unsupported database functionality. Try and use them via ActiveRecord’s high level API, and you will quickly see how fragile and inflexible ActiveRecord really is. If you shouldn’t need those features in your database, then you shouldn’t need anything that ActiveRecord doesn’t already provide, so it shouldn’t matter that you can’t extend ActiveRecord.

Truly, these are features that you need only in a few small cases in your application, so looking at individual queries they’re needed rarely (which is not the same thing as “never”). But looking at whether you need one or more of them in a given application, they’re needed more often than not. The pain of using hand coded SQL makes this worse: some tricky things could be done either using a view or stored procedure, or using a really slick dynamic SQL statement. Making all of those options painful means that even a clever developer can’t use anything in their bag of tricks to craft an elegant solution.

Unfortunately, non-trivial web applications need things like full text search, complex associations between persistent objects, non-trival summary information about associated objects, and complex reports, and ActiveRecord fails at all of these. These are not just things that big dumb ancient companies that like using Object COBOL think they need; Amazon and eBay need them too.

The acts_as_tsearch plugin is a good case study of ActiveRecord’s design flaws. TSearch2 is the standard PostgreSQL full text search engine, and it’s pretty good in my opinion. It’s also pretty straightforward to use. Unfortunately for developers using Rails, TSearch2 uses SQL functions (mainly to_tsquery and rank_cd). The acts_as_tsearch plugin tries to inject SQL into ActiveRecord’s queries via the high-level find interface, but ultimately fails as soon as you use the :joins or :include options. The problem is that ActiveRecord has a very simplistic idea of how queries and joins work, and so if you need to inject SQL functions to get the job done (as is necessary in TSearch2 queries), too bad. (See also issues 7 and 8 in acts_as_tsearch, in which I describe and attempt to clean up the mess that results when you use find_by_tsearch in non-trivial ways.)

A fellow Rails developer asked me in all seriousness why I wasn’t abandoning the full text search functionality of TSearch2 and just using a completely separate, redundant database product designed exclusively for full text search. Seriously, that is considered the “easy” approach: one database for full text search, and another for ACID/OLTP/CRUD. Honestly if I were going to go down that road I would try hard to just abandon the SQL RDMBS and put everything in the other database, since Lucene and its imitators are capable of far more than just find-text-in-document queries. The pain of duplicating everything, using two query languages, two document representations (in addition to the object representation in Ruby) and writing application-tier query correlation makes the double-DB approach seem very unwise.

It makes far more sense to me to use the SQL RDMBS’s full text search facility, even if there’s a 2x or 3x read performance penalty, because the conceptual simplicity of having one powerful storage tier (instead of two halves cobbled together) eliminates a ton of ugliness in the application, and the SQL RDBMS is going to get clustered for reads anyway. Nevertheless, even if I’m wrong about this case (putting search in the SQL RDBMS instead of in a separate server), there are other cases for needing a smart database that gives you exactly the results you need and lets you push data logic into the data tier.

So, what do I suggest? Abandon Rails? Nope. I still like Ruby a lot, and find Rails very useful. I just think that ActiveRecord needs to support the low-level and middle-level abstractions better.

Specifically, supporting bind variables (either by exposing that sanitize_sql function, or better yet by making drivers and connection adapters support bind variables for real) would make the find_by_sql, select_all, and exec approaches to low-level SQL query execution less painful.

More difficult, and substantially more valuable, would be refactoring ActiveRecord::Base to split it up in the way I described above: association descriptions and unmarshalling code separate from query building code separate from SQL execution and result retrieval code. All of this could remain hidden for most users under the same old slick high-level API, but for advanced requirements, the ability to fiddle with the SQL and still use the built in high-level unmarshalling code to create object graphs from flat result sets would be very powerful, and useful.

I looked at one alternative to ActiveRecord, called Sequel, which overlaps with ActiveRecord only partially. It is a query builder and lazy result proxy, which is actually what I thought ActiveRecord would do when I first started working with Rails. The proxy design means that you can either keep adding constraints or start fetching results, from the same Dataset class. This seems like a pretty good approach, though I haven’t really looked closely to make sure it would fit what ActiveRecord needs.

What Sequel lacks, though, is the unmarshalling side: turning a 2-dimensional (rows of columns) result set into a complex object graph (customers with orders with order lines with products from suppliers stored in warehouses), with user-controlled eager or lazy loading behavior. Ruby is well-suited to a design that would allow user-specified code (i.e., a block) to decompose each row into the object graph associated with that row, leaving the remaining associations on those objects to be lazily provided via future queries.

So, I think there is hope for ActiveRecord, definitely. I considered the idea of rolling a minimal Hibernate clone, or some other sort of challenger to ActiveRecord, but I don’t that ActiveRecord is broken beyond repair. I think the shortest path to a badass Ruby ORM is through improvements (refactoring and abstraction) to ActiveRecord.

So, if you’ve read this far, you probably care about these issues. Here’s my call to action: Please help me make ActiveRecord less like VB and more like Delphi. Who else is interested in helping me with this effort? Are there alternatives that I’ve missed, or components that could be integrated into ActiveRecord to make it better?

So when I read 7 reasons I switched back to PHP after 2 years on Rails I was a bit surprised, but not much. He’s experienced with PHP (he says he’s written 90,000 lines of code for CDBaby!), and has a huge installed base of code he wrote and understands intimately. He tried Rails, it didn’t work the way he wanted, and he went back to PHP. It was immediately obvious to him that this was what he should continue using.

The most shrill and arrogant among the Rails community have been rather unkind, partly due to this rather poorly written Slashdot headline that misrepresents what Derek says in his article.

I’m using Rails and I’m generally happy with it, though I have had to do some customization of the framework (mostly with pre-existing Rails plugins from like-minded developers) to suit my style. At this time I plan to continue using Rails for my project, and to keep using it in future projects.

But, I thought it would be instructive to summarize the arguments made by the folks slamming Derek in the comments to his article.

• Rails is the correct solution for CDBaby, regardless of what the founder//designer/lead programmer of CDBaby thinks.
• If you don’t agree with Rails’ design, you’re wrong, and you need to change your design if not your whole business to fit Rails.
• PHP is bad and can never result in good code. Conversely, Ruby is good and is always better than PHP.
• Objects are better than SQL and tables. Domain specific languages are great, as long as they’re written in Ruby; using SQL as a domain specific language for data manipulation and querying is not Ruby and therefore is a bad idea. Also, just because one person can code something in PHP in 2 months that runs on one server, instead of two people taking over two years to do it in Rails (which is notoriously hardware-hungry), doesn’t make PHP + SQL better. Because objects make you more productive.
• Just because you wrote a successful online store yourself from scratch, have an O’Reilly column, and then hired a Rails core team member who now works for the same company as most of the rest of the Rails core team including DHH (37Signals), doesn’t mean that you actually had people smart enough to rewrite your app in Rails. You need to prove to the world that what you wanted to do was impossible in Rails before we’ll give you permission to make technical decisions for yourself.
• If you don’t like Rails it must be because Rails is too good for you. Maybe instead of learning from years of real world experience running a business and writing the entire software suite for that business yourself, and hiring one of the best Rails developers in the world for your transition to a new architecture, you should have gone back and gotten a CS degree.

Since this is such an active flamewar with such sloppy readers (responding to the Slashdot headline’s misapprehension of what Derek wrote, instead of what he actually wrote), let me say this clearly:

The above listed points are not my opinions. They are summaries of opinions I find immature and silly.

(I’m assuming someone somewhere will read this post in anger and make themselves look foolish by arguing against these points as if I believed them anyway. I tried to warn ya…)

We can learn something from this. This same flamewar keeps appearing over and over and over, all over the internet, and before that, on BBSs and in print.

The simple fact of human mortality means that most of us are going to be learning and making mistakes that we imagine a wise, seasoned developer wouldn’t make. But that guy retired and is fishing, so it’s up to us to screw up and learn and hopefully do better next time.

For the last few thousand years, we’ve had the advantage of being able to read books, and get wisdom that way. But there are fads, and hyped books that seem to know it all. So you have to read a lot of books, and try a lot of contradictory ways of doing things, to get a mature enough perspective to make wise choices on future projects.

The point that is generally missed by everyone trying to do anything new to them, is that there’s a lot more out there that you don’t understand, and a lot of what you think is your brilliant new invention has been done before. The more you learn, the more humble you become as you learn that there are people WAY smarter that you are, and that there are people who have come before you who you will never catch up to.

In the case of web development, that means that there are people out there who disagree with you, and you might not actually know everything about everything like you think you do. Local experience and immersion in the problem they’re trying to solve makes them much better suited to solving their problem than some armchair quarterback. It’s tempting to fold your arms and feel smug about your quick dismissal of someone else who clearly isn’t as big a genius as you are, but the more certain you are of your own brilliance, the more likely it is that you’re just ignorant. (See also: Unskilled And Unaware Of It.)

In Derek’s case, he clearly jumped feet first into Rails and hired a rockstar developer, and then made the decision given a uniquely advantageous perspective that it just wasn’t working, after giving it a hell of a lot more time to pay off than I would have. (I’m not exactly sitting here scratching my head wondering “how could Derek have been so wrong?”)

What matters a lot more than choice of programming language is the ability to get the project done, meaning tested and correct and launched. Apparently for Derek, PHP is the way to get that done, and Rails ain’t.

Finally, consider the parallels between Derek in this case, and Phil Greenspun in his book (late 90’s). They both use SQL more than the “Objectistas” would like, which is to say, they use it directly and like it. They both use languages that are not the most fancy high level languages available, but they do completely understand how their application works, from UI all the way down to hardware performance, and they make integrated decisions that take business strategy and technological factors into consideration.

This is really critical to understand about why both of these guys are so smart: both of these people put aside dogma and made decisions that were all over the map, sometimes pragmatic and hackish, sometimes very rigorous and disciplined, depending on their assessment of the particular micro-issue they were addressing at the moment. They weren’t subscribing to any particular software development philosophy (Big Ball of Mud, XP, RUP, UML, Agile whatever, waterfall, etc.), nor did they just choose the most hyped architecture of the day and blindly stick with it, but freely mixed and matched whatever they thought was appropriate given their perspective on the situation.

In other words, they didn’t fall into the trap of thinking that there is One True Way to do everything. They improvised. They integrated their own experience and perspective with the wisdom of others, and made the decision that worked for them.

I hate to sound like a moral relativist, because I’m not. But as a practitioner, I’m definitely becoming more and more of a process relativist every day. One of the best ways to make a project fail is to try and force-fit an architecture and/or a methodology onto that project without customizing them to the particular details of the project. The best methodology is called “Roll Your Own.”

And so I leave you with this question, in the hope that it will help you with your current and future projects: Have you made any dogmatic decisions about your process or technology that are hurting your project?

Maybe it’s smarter at this point to keep doing what you’re already doing, but maybe there are some things that you could change. Try to keep your Mind Like Water, strong enough to cut through mountains but flexible enough to flow around obstacles, and making the decisions of which one to do based on the information that you and only you have.

Eclipse is an IDE so it’s hard to imagine it not being baroque and difficult to use, requiring weeks of effort to become productive. JBidWatcher has saved me a lot of money on eBay so I could probably put a dollar value on how much it’s worth to endure its bizarre UI. Azureus is fairly fugly also but it does a very good job and has a deep, sophisticated UI that’s fairly easy to understand, so despite the eyesore, it’s at least fairly clear. The common thread among all of these is that they are all written in Java, and that they are so valuable that it’s worthwhile to overlook the ugly UIs.

Now imagine those sorts of trade-offs, but on already difficult to use mobile devices, and aimed at consumers. Are you making a strategically wise choice by sacrificing usability and control over the user interface, and probably access to platform-specific features such as dialing the phone, in order to save money on development? Adam Breindel talks about this in When Building a Smartphone App, Resist the Siren Song of J2ME.

Adam and I worked on a J2ME application and I totally agree with him about the disillusionment of trying to write a single app that would work across phones. Issues include:

• Complex and difficult application installation procedures for end-users: How do you install the JVM on the phone? How do you get the plain J2ME app packaged up so that the phone will accept it? Does the app require manual user configuration before use? Is there a different launching process from other apps?
• Lack of control of the user interface: hardware details such as how many buttons you have, whether there’s a stylus, etc. differ from phone to phone, and the API to let you code once and let J2ME handle the layout for each device leaves your code very disconnected from what’s actually happening on the screen.
• Not being able to use recent J2ME APIs because even the latest phones only support older, more minimal J2ME APIs
• Not being able to do things on a phone that would seem obvious, like dialing the phone, opening a hyperlink in the phone’s browser, sending an SMS, or making a network connection. Either these are entirely impossible or require phone-specific or JVM-specific tools and procedures to sign your application, or having the runtime nag the user to request permission to do something that they just asked the app to do for them.

For all the noise Sun is making about broad J2ME penetration, the developer experience is quite disappointing, and as a result, the user experience is also quite disappointing. You can look up J2ME features and APIs and get excited, but when you actually deploy your app to a handset, it won’t load, or runs terribly slowly, or looks awful, or simply doesn’t do the things that the API says will happen when you call it a certain way. Suddenly the strict J2SE and J2EE logo certification programs make sense, because the J2ME approach of making so much functionality specified but optional leaves developers high and dry. The phone supports J2ME version xyz, but write an app coded to that API that works on the emulator and deploy it to a handset and lo and behold, all those optional APIs turn out to be missing even though the handset is capable of that functionality, and some mandatory API functions are not working. Here be dragons.

Case in point: can’t dial the phone on a Treo 650 (at least, not as of a year ago). The J2ME API tells you how to do it. The PalmOS JVM (made by IBM) lets you make the API call, and returns a successful response. Nothing happens. IBM says they’re aware of this issue. The end. The docs say you can, the code you write says you did, the phone just doesn’t do it.

Case in point: Every time you start an application and it accesses the network for the first time on a Treo 650, the user is nagged for permission to access the network. Quit the app and start again, nagged again. IBM has a tool that you can use to sign the app, but you have to use their VisualAge Micro Edition IDE which costs hundreds of dollars to do that. Try and find and download the trial version. A year ago, it was not possible. So, making that persistent nag go away probably costs several hundred dollars. (I never verified that it actually works, just that IBM says the way to sign the app so that it’s trusted is to do that, and that there was no available free way to get that tool.)

These are minor issues, but they certainly interfere with the quick usage pattern of a mobile app, and make it annoying to use your app. Imagine what that would be like if you had a competitor with a native application for that phone, whose application probably cost them more, but their app is better and the user likes it a lot more.

The important distinction here is not cost, it’s ROI. It costs a lot to develop a similar application for each smartphone platform, using that platform’s native tools. It would seem to cost a lot less to develop a J2ME app. But that’s only if you assume that it’s OK to abandon features and settle for a horrid user experience in the course of development.

It’s likely that your goal as a development team is to develop an app that has a predefined feature set that you know the device can support, and a predefined UI design that your mobile-savvy UI people are sure will go over well with users accustomed to that particular kind of smartphone. In that case you will almost certainly fail to accomplish that goal using J2ME. You have to scale back your goal so that you’re satisfied that you got something kinda like what you wanted working on a bunch of phones, and determined users will probably be able to figure out how to install it and make it work.

I call that phenomenon “write once, be disappointed everywhere.”

Let’s continue talking about cost, though. The native apps may require (or suggest) different programming language skills for different devices. It might seem wise to just write everything in C, but I think that’s a false economy as well. The phone APIs will differ so much that you will really need a native developer for each platform, not a team of generic C developers who will figure out the individual phone stuff and be freely floating resources that you can assign to whatever app version needs their attention. Smartphones may run Linux, may run Windows Mobile, may run PalmOS, may run Symbian… these are different operating systems with very different ideas of how applications run and coexist. The platform specific knowledge (APIs, appropriate UI feel, device capabilities) is probably an order of magnitude harder to learn and maintain than the ability to get an application working in a given programming language.

How much of a great C programmer’s skill is really the C language, and how much of it is proficiency with the available libraries on the platform he or she is accustomed to? I think close to 90% of their professional skill set is platform and library familiarity, and 10% syntax and low-level understanding of how the language actually works.

In light of this (just using C doesn’t mean developers or code are portable across smartphones), consider that there are high level languages available for some smartphones. What if that 90% platform familiarity means they can use a language and/or development environment that makes them 5 or 10 times as productive, after spending just a few days or a couple of weeks learning the language syntax?

After our very disappointing J2ME-on-PalmOS port experience, Adam found Handheld Basic which initially appalled me (oh no, BASIC!) but turned out to be a great choice. It’s a flavor of BASIC, so learning the language didn’t take long, and the support happens to be quite good (lots of code samples) so picking up the library portion of that 90% didn’t take long at all. I imagine that C# on Windows Mobile is similar. As more and more phone start to use Linux as their OS (which will be a particularly huge improvement for PalmOS based phones), you’ll be able to use Python, Ruby, Mono, J2SE (a whole different animal from J2ME), TCL, or pretty much any other high level language available for Linux. At LinuxWorld 2006 I saw a development device running unmodified GNOME desktop apps running on a PalmOS device alongside PalmOS apps. There are more options appearing all the time, and with the exception of Handheld Basic, most of them are ports of familiar, mature, well understood desktop languages, with class libraries relevant to mobile devices.

So, I recommend that you work in whatever high level language lets you do all the platform specific stuff you want, and if that means a different language per phone, that’s actually going to be the least expensive way to get to the apps you actually wanted to build. A big chunk of the scary cost of developing native apps goes away, because native doesn’t necessarily mean abandoning Java for C. (In fact I found Handheld Basic to be a more productive environment for me after a couple of weeks than J2ME was, despite my ~8 years of full time Java experience before starting that project.)

That pretty much means no reuse for you. Sorry, but that’s the deal right now.

If that unique-app-per-platform cost is too scary, consider a few ways to save money:

• For networked apps (aren’t they all?) ask yourself if there’s some logic that could just as easily be done on the server as on the client. Is there some complicated parsing code on the client that could be simplified by changing the response format from the server to something that’s easier to parse?
• Can you remove or alter certain features from a subset of platforms you intend to support, so that your premium supported platforms get your ideal app, whereas a few less popular phones still get a nice app, but perhaps one that doesn’t have every feature available on your premium app. You might be reading this and thinking “but that’s what you said J2ME would force me to do! Why is this any better?” The distinction is that you are in control of the decision of what to leave out to save money, whereas with J2ME it’s the platform vendor who makes that decision for everyone using their platform. If you’re writing an address book, not being able to dial the phone is lethal; not writing the code to let the user attach a photo to the entries is not.
• Can you move some user-facing functionality to the server to make the client simpler? Maybe there are some rarely-used features that could be done via a desktop or mobile web browser, so you can focus on putting the ten-times-a-day features on the handset, and making those features fast and convenient. You probably already made that trade-off in general, but perhaps for some kinds of handset, you’ll move the dividing line a little further, so that for users of particularly rare phones, some moderately frequent features can’t be done on the phone. This could also be a good approach for new handset types: design a “lite” app and a “full” app version, build the “lite” app first on each platform, and let user demand tell you whether the full app is worth it. Your developers can tell you much more accurately how much the incremental functionality would cost, since they already have done the lite version. Maybe you’d provide a VoiceXML interface, or mobile web browser interface, for that feature so that the user can still do whatever the feature requires while they’re far from a desktop PC.

A final consideration: labor. Maybe you have some Java developers, or C developers, and don’t have developers good at Handheld Basic or C#, so you’re not inclined to fire them all and hire new developers to do native apps, you’re thinking maybe C on every handset, or J2ME, is still the right choice. I still say that you should probably use native apps in native high level languages on every smartphone platform. If you’re committed to a strategy of good apps on a bunch of different phones, I think I’ve made clear that native apps are the only way to currently get there; J2ME simply doesn’t let you make good apps. So what’s left is C code written by C developers vs. high level code written by C developers who have to retrain.

As I said above, I don’t see much chance that your developers or your code will be portable across different smartphones if you use C. Maybe you’ll get 5-10% savings that way (some code ported across phones, or some developer hours shuffled between platform teams). But you’d get a 5-10x cost saving from using something very modern and high-level instead of C, and the overhead of training for the language would be very very small as compared to the large and unavoidable overhead of having to learn what the handsets can do and how the APIs for the handset’s OS work.

I suppose that means that if you have a bunch of C developers who lack smartphone skills, you’re in a pickle, but that situation seems kind of unlikely to me (a mobile app company hires a bunch of Unix and Win32 C developers with no mobile phone skills?). More likely is that you’d find a mobile developer who is proficient with one or more handset OSs and the best tools for each one, and they may be of the opinion that C is the best choice since they can reuse skills and some code across platforms. I would say that in that case you need to convince them to (or more likely encourage them to do what they were already considering, which is to) go ahead and find a highly productive high level programming language/environment for each smartphone platform.

A final possibility if you have a ton of platforms to target and a pile of killer C programmers is to try and port something like Python across most of your target platforms, or to make a very high-level API or domain-specific language that runs inside your own custom C portability layer. You might be able to find an open source option that you can invest some developer hours in, so that the actual application-specific code that you write on each platform is minimized and portable. But I suspect that this would still be expensive and would result in some J2ME-like UI abstractions that ended up being very unsatisfying in the end.

Best of luck!

When I was in 5th grade (age 9) learning AppleSoft BASIC on the Apple ][+ in math class, we wrote programs that did this:
10 X = 1
20 PRINT X
30 X = X +1
40 GOTO 20
And we would race each other, starting at the same moment and seeing whose column of increasing numbers on the screen scrolled faster. This was of course stupid because all the computers in the lab were chip-for-chip identical to each other, and probably were all made on the same production run on a single day.

We learned that we could cheat by adding a number larger than one in each loop iteration, which was quickly detected and outlawed. Far more cleverly, someone figured out that you could do something like this:
20 PRINT X: X = X + 1: GOTO 20
The same algorithm yielded better performance if it was all written on one line of code. I could be misremembering (it has been 25 years and I don't have a ][+ handy to verify this on) but it was something like that. Anyway, we learned that the same language runtime on identical hardware using the same algorithm could be made to run faster or slower using simple formatting changes.

So does it make sense to compare different languages this way, which may mean favoring one language's idiomatic code structure while hitting a weak spot of another? This is a common, and in my opinion valid, critique of apples-vs-oranges benchmarks: how do we know that the performance difference isn't due to naive coding or configuration on one side and expert tuning on the other side? For that matter, do we know that the benchmark design isn't selected specifically to highlight exceptionally high performance in one area of a product, to the exclusion of embarassingly slow areas that the benchmark designer would prefer that you not consider?

Thus I claim that this benchmark is approximately as valuable as my 5th grade silly hacks. X=X+1, change the number of iterations to suit your bias, or perhaps just don't bother making them the same because it's meaningless anyway. (Z=X*Y and a matrix multiplication are other parts of this benchmark, but they too are so trivial in concept and implementation as to be equally pointless.)

I'm going to guess that the author of the blog post didn't notice the different in loop iterations, or was looking at the per-second values rather than the total run time. But if we're looking at average performance over time, then how long does it take for the performance to stabilize? Stabi-whatchamaha? Ask Zed Shaw: look at his list of pet peeves, #3.

Do we know that 0.142 seconds is enough to measure “language performance” (really, it’s the performance of a particular runtime environment being measured) including stuff like garbage collection and JIT compilation overhead? If one language’s runtime waits for N iterations before JIT-compiling the code, whereas another runtime waits for 5N, how many total iterations do you need to minimize the effect of that?

What happens in JRuby, Ruby2C, etc.? The poster says the tests were run on a MacBook Pro – what architecture (PPC vs. Intel) were these language runtimes compiled for, with what compiler, blah blah. GCC versions, optimized for certain CPU models, etc. This stuff can make a big difference in CPU benchmarks, which is why proper benchmarks include things like this in their configuration information.

Or are you measuring small script execution time, and >1s runtimes are meaningless for your needs, in which case Java seems painfully slow and Bash lightning fast?

What the heck is being measured by this “microbenchmark”? Language fanboy gullibility?

For a less awful benchmark, have a look at the Computer Language Benchmarks Game: “What fun! Can you manipulate the multipliers and weights to make your favourite language the best programming language in the Benchmarks Game?” At least they realize how not-terribly-useful synthetic CPU benchmarks of language runtimes are.

Anyway, for most applications, if you’re choosing your language based on runtime performance, you’re choosing very poorly. If you’re choosing your language based on a really awful “microbenchmark” comparable in accuracy to the first toy hack of a room full of 9-year-olds, well…

namespace :spec do
  desc "Add files that DHH doesn't consider to be 'code' to stats"
  task :statsetup do
  require 'code_statistics'

  class CodeStatistics
    alias calculate_statistics_orig calculate_statistics
    def calculate_statistics
      @pairs.inject({}) do |stats, pair|
        if 3 == pair.size
          stats[pair.first] = calculate_directory_statistics(pair[1], pair[2]); stats
        else
          stats[pair.first] = calculate_directory_statistics(pair.last); stats
        end
      end
    end
  end
  ::STATS_DIRECTORIES << ['Views',  'app/views', /\.(rhtml|erb|rb)$/]
  ::STATS_DIRECTORIES << ['Test Fixtures',  'test/fixtures', /\.yml$/]
  ::STATS_DIRECTORIES << ['Email Fixtures',  'test/fixtures', /\.txt$/]
  # note, I renamed all my rails-generated email fixtures to add .txt
  ::STATS_DIRECTORIES << ['Static HTML', 'public', /\.html$/]
  ::STATS_DIRECTORIES << ['Static CSS',  'public', /\.css$/]
  # ::STATS_DIRECTORIES << ['Static JS',  'public', /\.js$/]
  # prototype is ~5384 LOC all by itself - very hard to filter out

  ::CodeStatistics::TEST_TYPES << "Test Fixtures"
  ::CodeStatistics::TEST_TYPES << "Email Fixtures"
  end
end
task :stats => "spec:statsetup"

(This code is based on this code I found that does something related.)

As you can see it doesn’t count JavaScript because the included Prototype code that’s included by default in Rails projects is huge-mungous. You can just uncomment that and see your project grow by 5KLOC if you want. Or you can temporarily delete those files, maybe in a quick ’svn export’ temporary directory, and run rake:stats in there to see what your own JavaScript stuff contributes.

(I tried to get a fancy regex going that excluded the six files that make up Prototype, but I couldn’t get it to work quickly. I don’t care enough about that one issue to keep spending time on it, but if you figure out a clean way to get that done I’d love to see it, just post a comment.)

Hope this proves useful!

This section is at 21:12 in the podcast:

There are a couple of other interesting markets that I found as I was interviewing people for the book. One of the models was that we found that if there’s an entrenched Java shop that has religion, you’re not going to sell them on Rails; don’t try.

But there was a surprising customer set, and that’s the set of customers that’s been very conservative, and they’re just now getting into Java, and to learn Java they find that they have to learn this set of specifications that’s about 2-3 feet high, right? This stack of binders, you know?

“Okay learn this Servlet thing, which means you’ve got to learn this XML thing to deploy it. Oh this XML thing has added this Namespace thing and this Schema thing, and oh you need some kind of middleware, and the middleware is maybe the Spring framework, or Enterprise JavaBeans, so you have to pick which one. And then once you have that picked, then you have to plug the persistence engine into that. And oh, by the way, after you’ve picked one, and hopefully picked right, you have to do the web UI and model view controller layer and then you’re talking about JSPs. And oh what tag library do you want to use with that?” and it is just absolutely overwhelming.

Definitely. I feel lucky that I got started with Java before J2EE, because I kind of understand where JSP and EJB and XML came in and how they grew, and yet there are so many APIs in J2EE that I haven’t ever used, and honestly I don’t think I’ve ever met anybody who used them. It’s a beast. There’s a reason why startups that don’t need asynchronous message queues and “Java Authorization Contract for Containers” and all that stuff are sticking with PHP or Ruby on Rails. There’s just less stuff to learn.

The S stands for Simple, by Pete Lacey.

Regarding your article “Awaiting the Day When Everyone Writes Software”:

Your ignorance of the reality of software development would be excusable if not for the fact that your CV suggests that you should know better. Your defamatory description of programmers smears an entire industry of individuals with a single, pejorative stereotype.

MIT should be ashamed to have someone affiliated with them who is prejudiced against so many of its own graduates. Perhaps you could help the state of software by visiting the computer science department and telling the professors and students that they don’t interact with people, and that software is so ugly because they are drowning in ignorance, complexity, and error. I’m sure they’ll welcome your fresh and insightful help.

Programming is hard, only in small part because of “esoteric programming languages.” More importantly, what programmers actually do includes a substantial amount of coaxing details out of the folks who actually want the software created, trying to devise a conceptual model that would actually work. Only then do we shuffle off into our lonely monastery with pizza and sodas under our arm (or whatever stereotype it is that you have of us) and hunch over keyboards, hoping that we got the esoteric bits right.

Wikipedia’s page on you says that you write “a regular column, entitled “Slipstream”, about new idea[s] in technology for The New York Times on Sunday.” But the idea that we can just talk to a computer in a natural language or domain specific language, instead of this darn esoteric gibberish, is not at all a new idea. It was covered twenty years ago in No Silver Bullet:

The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions. This essence is abstract in that such a conceptual construct is the same under many different representations. It is nonetheless highly precise and richly detailed.

I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems.

If this is true, building software will always be hard. There is inherently no silver bullet.

In other words, it’s the detailed thinking about what the software will do that is the hard part. Adding 2+2 and putting the result onto the screen is easy.

However, you say:

Programmers don’t know what a computer user wants because they spend their days interacting with machines.

I suppose if you redefine “programmer” so that it doesn’t reflect what programmers actually do, you would be correct. (”All programmers communicate only with machines. People are not machines. Therefore, programmers do not communicate with people.”)

But we do work with people, and that’s where the gap exists between a fuzzy idea and a design that a programmer can “hunch over” and type in. Our job is largely to work with people to fill in the many, many gaps between a general idea and a detailed design that can, say, put an International Space Station in orbit so that Mr. Simonyi can visit it.

In fact, since Mr. Simonyi’s days of “punching machine code”, the esoterica of programming has gotten easier and easier, and yet software development remains difficult. Even after the widespread acceptance of COBOL, SQL, and other languages designed to allow Sr. VP Everyman to interact directly with computers in something very similar to plain English (without those darn monks slowing things down and messing it all up with bugs), Sr. VP Everyman decided to hire some folks to hammer out the details and hunch over the computers, interacting with the machines. Those people are called Programmers.

Finally, software is not unique in being the “bottleneck” between wishful thinking and our practical problems. Wouldn’t it be wonderful if we had a wacky gadfly zillionaire / civilian astronaut in training, telling us he had invented “Intentional Engineering”? You just type in a wish for, say, nanobots that will convert heaps of garbage into beautiful energy efficient palaces for the homeless, and it does all the work of those darn grumpy engineers who always want time to design and test things. In the next version, we’ll make the nanobots also accept radioactive waste. Problem solved! You could call your next article “Awaiting the Day When Everyone Designs Nanobots.”