User story: “As a princess I want to confirm my royalty so I get bruised after sleeping on 40 mattresses over a pea”. Just tell real stories”

That sounded so much like a Cucumber feature that I decided to write it as one:

Feature: Physical Sensitivity
  In order to confirm my royalty
  As a princess
  I want to be very delicate

  Scenario: 40 mattresses on a pea
  Given there is a pea on the bed
  And there is a stack of 20 mattresses on the pea
  And there is a stack of 20 featherbeds on the mattresses
  When I try to sleep on top of the stack of featherbeds
  Then I should not be able to sleep

`method_missing': No sham defined for name

It’s discussed in the machinist Google Group as well.

It’s because of a wacky hack that’s part of Machinist that overrides Module.name so you can do Sham.name, but ZenTest expects Module.name to do what it does normally.

I have a fix for this.

I sent notahat a pull request via GitHub, so hopefully it’ll be merged by notahat into the official machinist gem, but until that happens you can try mine. I published the gem as JamieFlournoy-machinist to rubygems.org.

Try ‘gem install JamieFlournoy-machinist -v 1.0.6′ and see if it works for you. It does for me with ZenTest 4.2.1, autotest-rails 4.1.0, and a Rails 2.3.5 project.

(my machinist fork on github)

Igor P’s solution is correct (just install REE 1.8.7-2009.10), but it took a little fiddling to figure out how to get RVM to use the older version of REE. Here’s how to do it:

cd ~/.rvm/archives
wget -q http://rubyforge.org/frs/download.php/66162/ruby-enterprise-1.8.7-2009.10.tar.gz
rvm install ree-1.8.7-2009.10

No, it’s not. Falling back on 500 Server Error for everything outside of the “happy path” through your code is sloppy coding.

I say this for three reasons:

The 5xx response codes are for server-side errors.

If the code is working properly but the request is wrong (no such document, an invalid request, or insufficient user privileges for the requested operation), the 4xx response codes are more appropriate. You should only return a 5xx series error message when your application is broken.

A generic “oops” page hides the cause of the error from the user.

Maybe the user followed a bad link, that goes to a document that has been deleted. They should see a 404 Not Found response, not a 500 Server Error. This is especially important if the user agent is a web service client actively under development. And remember that you can still customize the response body, so a 404 can show a search or site map to help them find the content that was Not Found, and a 403 might suggest that the user upgrade their subscription so they can see the Forbidden content.

Failing to trap bogus input is a recipe for a security vulnerability.

Rails will detect ridiculous input in many cases, but it doesn’t know about your business rules.

For example, what if someone fires up Firebug and figures out your API, and deletes her ex-boyfriend’s account because you forgot to check that the ID being deleted matches the currently logged in user? Oops.

You know who else got pwn3d because they put their security rules in the client? AOL. Don’t be like AOL. They got pwn3d big time. Make sure your controller handles the “unhappy path” too, and responds appropriately.

You make sure your controller handles these cases by writing tests, and checking that the response is correct. To do this, you need unambiguous errors that clearly explain what the controller didn’t like about the request.

Your controller tests should never expect an exception. Exceptions belong inside the application (such as between a model and a controller), not at the boundary.

Instead, you should be using something like assert_response :bad_request, and providing a useful textual message in the page that will help the user understand what they did wrong.

A nice bonus of using the right HTTP error responses is that your tests are more readable. And since controller tests tend to be really verbose already (mine tend to be ~10x as many LOC as the controller itself), the extra readability is really nice.

Here’s the Cucumber step implementation I wrote.

Put this in a new file called features/step_definitions/ym4r_gm_steps.rb :

# Steps for use with the ym4r_gm plugin

Then /^I should see a Google Map$/i do
  within 'head' do
    ['http://maps.google.com/maps', '/javascripts/ym4r-gm.js'].each do |js_url|
      assert_have_xpath "//script[starts-with(@src, '#{js_url}')]"
    end
  end
  assert_select '#map_div'
end

This assumes that you let the plugin installer put the .js file in the default location, and that you have your Google Map inside #map_div.

I found a good starting point at geokit stubbing for faster tests. I also wanted a way to stub a geocoding failure, and a way to prevent any unit tests from using the real geocoding web service.

Here’s how I did it.

In test/test_helper.rb, add this:

class ActiveSupport::TestCase
  def setup
    GeoKit::Geocoders::MultiGeocoder.stubs(:geocode).raises(RuntimeError,
      'Use mock_geocoding_success! or mock_geocoding_failure! in your test')
  end

  def mock_geocoding_success!
    geocode_payload = GeoKit::GeoLoc.new(:lat => 123.456, :lng => 123.456)
    geocode_payload.success = true
    GeoKit::Geocoders::MultiGeocoder.expects(:geocode).returns(geocode_payload)
  end

  def mock_geocoding_failure!
    geocode_payload = GeoKit::GeoLoc.new
    geocode_payload.success = false
    GeoKit::Geocoders::MultiGeocoder.expects(:geocode).returns(geocode_payload)
  end
end

What this does is to force you to choose either one of those mock_geocoding methods before you call the geocode method. To me this seems like a good idea since the integration tests that exercise the full application stack should probably be written using Cucumber and Webrat (which is what I’m using).

You will probably want to merge my one-line setup method into your existing setup code in test_helper, if any. Also note that this uses Mocha for mocking.

However, if you’re upgrading an existing project and you’re sticking with official releases (as opposed to edge Rails) like I am, your code probably needs some tweaking to work with Rails 2.2.2. Mine certainly did. Although there were a lot of failed tests with ugly stacktraces, there were only a few API changes in Rails that needed to be accomodated to fix them all. Here’s a list of the changes that broke my app, and what I had to do to get it working again.

TextHelper.truncate no longer allows you to supply the length parameter as a second argument. You have to use truncate(somestring, :length => 20) instead of truncate(somestring, 20).

I was getting an error that looked like this:

ActionView::TemplateError: wrong number of arguments (3 for 1)

The line in question is:

<%= render(:partial => 'layouts/header') %>

The problem is that the simply_helpful plugin’s behavior is now built-in; just delete the whole vendor/plugins/simply_helpful directory if you were using it, and any existing code that depends on simply_helpful’s API should work again.

I was using the tztime and tzinfo_timezone plugins, which are no longer needed now that time zone support is available via Time.zone in Rails 2.2. Delete the plugins, upgrade the tzinfo gem to 0.3.12 or uninstall any older version of it, and search for TzTime instances in your code and replace any of the occurences of Time.at(some_tztime_variable) that you had been using. TzInfo is still useful in cases such as this:

TZInfo::Timezone.get('UTC')

But since you can now embed the time zone information in a Time instance, you don’t need TzTime objects anymore.

I was using a nice hack from Technoweenie, to allow me to validate forms using the familiar ActiveRecord model validations API even if those forms were not backed by a database table (sometimes the UI just doesn’t look anything like the data model, y’know?). It broke, and I was unable to find any solutions from anyone else, so I wrote my own: EphemeralModel, for Rails 2.2.2 form validation without a DB table.

I was using ActiveRecord::Errors.default_error_messages[:invalid] in a few model validators, which is now deprecated because it’s not internationalized. The new way to ask for the same string in the currently configured language is: I18n.t(:invalid, :scope => 'activerecord.errors.messages')

Finally, the app wouldn’t start because it wants a :secret option to config.action_controller.session in config/environment.rb. I came up with something sufficiently random and now it’s fine.

I’d just like to make one last comment about this: having thorough automated tests is a huge time and sanity saver during this process. If you’re not close to 100% test coverage on your current Rails project, then your house is on fire and you need to address that immediately. Even as I write this in late 2008, I’m talking to prospective clients about contracting work, and some of them don’t do automated testing at all(!). Unless you’re in a mad dash for a demo that will save the company from financial collapse, having automated tests that verify that your code works should be priority #1. Upgrades can wait.

Getting 100% coverage isn’t easy. In general it means you definitely can’t just write a single test case per method and declare victory when it passes. When the number of possible combinations of inputs (method arguments and/or mock objects) and expected outputs (return values, exceptions, and side effects) becomes large, then the potential for copy-and-paste errors in your test code becomes large, and legibility becomes an issue. This is the point at which I find the recent fascination with writing tests in a near-natural-language DSL to be a distraction. It’s orthogonal to the problem I’m dealing with, which is how to comprehensively test the code. In other words, the problem is not making a small number of tests more readable, but concisely expressing a large number of input/output combinations and a large number of different tests, and making it all readable.

I assume that there are others working on this problem too, so I’ll describe some of the things I’ve come up with, and hopefully if you have any good ideas you’ll post them in the comments section.

Test Data Iterators

If you express the set of inputs and outputs as a data structure, such as a Hash of input => expected ouput pairs, or an Array of [input, expected output] Arrays, you can easily do test_data.each do |input,output| ... end and drive tests that way.

(I considered pulling the data itself out of the test code and loading it from a YAML or CSV file or even including a data-only Ruby script, but that seemed like pointless added complexity. If I were working with a professional tester who didn’t want to touch Ruby code, I might change that to make it easier to dump from an Excel spreadsheet into a text file that the tests would use.)

One problem with this approach is that you may have a test_frobnicate method that invokes frobnicate dozens of times. This means that this one test can take a while to run if the tested method takes any amount of time, which is problematic if you just want to re-test that one input/output combo until it works. Your edit/test/debug loop should be ultra fast, and re-testing successful pairs gets in the way of testing the pair you’re not sure about.

More importantly, trying all those combinations in one test method means that if an assertion fails, the failure masks the outcome of the remaining combinations. If there are 25 combinations and combination #2 fails, then a Test::Unit::AssertionFailedError is thrown, and the other 23 of them go unchecked. That’s not ideal, since it can be useful in debugging if you can see that 24/25 failed vs. 1/25 failed.

To solve this, I changed the code so that it iterates over the test data inside the class definition, generating a new test method for each input/output pair:

class_eval do
  test_data_hash.each do |input, output|
    define_method("test_frobnicator_#{input.hash}") do
      check_frobnicator(input, output)
    end
  end
end

That gives you a bunch of test methods that each run the test once with different input and expected output values. Autotest integrates well with this technique, since it will not re-run the test methods which previously passed until after you have fixed all of the ones which failed.

Test Setup Blocks

I also use a with_x_y_z do...end idiom for mock setup when possible. So a test might contain:

def check_index(inputs, expected_outputs)
  if inputs[:logged_in]
    with_fake_login{|user| get :index, inputs[:params] }
  else
    get :index, inputs[:params]
  end

  # ... test outputs against expected output
end

Since this idiom uses blocks, you can nest them to create complicated setup scenarios, pass arguments to the with_blah method to control the behavior of the mock objects, and so on. (You could do this with regular set_up_blah methods but I think this is easier to read.)

Calculated Outputs

Despite their shortcomings, I still use Fixtures (with the PreloadFixtures plugin) and I write unit tests for my models. That means that I can then assume that the models’ behavior is correct in other tests (provided that those unit tests pass). So I try not to put any data that’s already expressed in fixture files into my test method inputs or expected outputs. Instead, I ask for the fixture by its label (widgets(:doodad)) and then use its properties as needed in assertions within the current test. This reduces the size of the input/output data, which makes it easier to read and reduces the amount of effort required to add more combinations later.

Testing Views, Model Associations, and Model Validation

As Bruce Eckel says, If it’s not tested, it’s broken. Or as Peter Drucker said, “What gets measured gets managed.”

I test all of these. I’ve read the arguments of those who say that none of these are necessary, and I disagree. The arugment that testing associations and validations is tantamount to testing Rails itself is just incorrect. Rails’ own unit tests cannot possibly test the validations and associations expressed in every Rails application’s model classes, so Rails application developers still have to do this.

Associations

I can’t tell you how many times I’ve gotten has_one and belongs_to backwards and only found out when ActiveRecord told me that the column didn’t exist, because I told it to look at the table on the wrong side of the association. I feel stupid when I see that I made that mistake, but writing that test takes seconds and once I see the failure it’s easy to fix. Better to catch it and fix it now rather than have a user find it, right?

Validations

The same goes for model validations. Yes, I know that validates_format_of has been tested, but my regexes for username format and email addresses need testing too.

Views

Views should be tested only if you care what your users see.

Somewhere there is probably an organization with no users, where inputs matter but outputs can be anything. Maybe the goal of the project is to burn cash rather than to ship something good. They don’t need to test their views.

I agree that views should be dumbed down as much as possible using helpers, but that’s not enough to assure that they work. If you’re not testing views with automated tests, then how do you test them? With a browser? Every time you change anything, you’re going to exhaustively check every combination of inputs and outputs to make sure that all the dynamic goodies are in the right place? I doubt it. Obviously you do have to check views in a browser to make sure that purely visual aspects are correct, but there are things you can check automatically. You can do quite a lot with assert_select, and for client-side tests, Selenium can do pretty much anything. (It’s probably not worth the effort to have Selenium use the DOM to see that the browser applied CSS in exactly the way that you wanted it to, but if you wanted to, you could.)

I don’t know how to do something similar to coverage testing for views, but a good start is to just check for the effects of everything that the controller stored in instance variables. Maybe those should be displayed as-is in the resulting document, or maybe they should be truncated, or processed by a helper into a different representation.

In any case, you don’t have to test with the same set of inputs as the functional test for the controller; just do white box testing for the minimal set of combinations that should exercise all of the permutations of the view code (which should be a very small number). Testing the helpers separately can cut down on the number of permutations of inputs and outputs also. So if you have a helper that creates a navigation breadcrumb visual element from an array of strings, you could just test that in isolation with 0, 1, 2, and 3 elements, and then check for one version of breadcrumbs when looking for it in a view.

I’m particularly inspired by the claim that Sebastian Delmont of StreetEasy made in the May 31, 2006 episode of the Ruby on Rails Podcast about having no human approval step between a successful test suite run and deployment to production. That’s pretty darn bold, but if you stop and think about it, it is feasible with today’s tools to automate any “white glove test” that a real person would do before approving the release for deployment to production. I’m not that brave yet, but I like the idea of a condition that checks if the changed code has anything to do with a view (ERB, JavaScript, CSS, etc.), and if not, would just rubberstamp it and deploy it. Clearly a change that could require a manual test in multiple browsers wouldn’t be a wise thing to autodeploy, but something like fixing a bug deep in the code shouldn’t require a human to approve it, especially if the autodeploy condition depends on that chunk of code having 100% test coverage following the bug fix.

Your Tips?

I’m always looking for ways to test code with less effort on my part. Please let me know if you have ideas or suggestions.

The coder is the one on the hook for long nights, weekends, and stress-related health problems if the estimates suck. It’s in your interest to exert as much control over estimation and scheduling as you can. If you’re not making the estimate, someone is making it for you.

If you think about it, you’re opting out of an ongoing salary negotiation process. You’re going to work more for the same amount of money, and worse yet, you’re going to have to squeeze that work into the same amount of time, under stressful conditions. Why would you do that?

McConnell’s “Software Estimation” book addresses this problem of wildly inaccurate early estimates quite well. I highly recommend it. In a nutshell, estimates need to carry error bars, so early estimates would be given but with a disclaimer of +/- 100% (or more!), and later estimates get closer to being correct.

You will have to manage expectations, though. Human nature leads to the shortest end of the date range becoming the remembered end date, the guess being remembered as a promise, and the latest feature set being remembered as if it were the original feature set.

This is a big reason to hire a good project manager: they’re there to facilitate the change control process, or planning game, or whatever you call it in your organization when the paying folks want more stuff, and the PM reminds them that need to pay more and wait longer in order to get more out of the dev team. A good project manager can make a huge difference to the stress level and likelihood of success of a project.

It’s psychologically icky for developers to have to participate in the estimation and scheduling process — “eww, it’s a non deterministic domain with lots of ill-defined inputs!” — and I’ve seen some managers play head games like “Oh, you need more time? I thought you were really good at programming… I guess not…” to try and manipulate developers into working nights and weekends out of a sense of guilt or wounded pride. But you need to suck it up and keep providing status updates and reminding people how the process works, because you-the-programmer are the expert on the nature of programming.

Yes, it’s more like Lewis and Clark or the Apollo project than like building a house, and there are just going to be a lot of surprises, but you can still provide a framework and exert a tremendous amount of control over how the team will operate. Things like whether you plan to build a technology proof-of-concept to make sure your architecture doesn’t suck, or whether you intend to spend lots of time writing automated tests and doing daily code reviews in order to control that horrible “bug fixing” phase toward the end are in your hands if you participate in the estimation process. You’re being offered an opportunity to make sure that all those important tasks like performance testing and cross-browser testing and writing an installation guide and an operations manual for the sysadmin get done. It doesn’t matter if the estimates for those tasks are completely wrong at first – what matters is that they’re on the list.

And really, the largest battle you’ll probably have to fight is the one over scope creep: we can’t launch until it has all the original features plus all these additional changes and new features perfectly working, but we need to launch tomorrow. That’s also human nature, and you need to train people to release early and often. Agile methodologies can help here, if you can get your software working and stable from a very early point in the project, and keep it that way. Then you probably have a launchable product very early, and you can just push highly-desired features into post-1.0 releases.

In any case, a list with a bunch of crappy numbers and low confidence factors is a lot better than no list, or a one-item list with a crappy number and an implied 100% confidence factor.

Autotest.add_hook :initialize do |autotest|
    autotest.sleep=3
    autotest.add_exception(/\\/\.git\\//)
end

require '/Users/jamieflournoy/autotest/sound/sound'
Autotest::Sound.sound_app = '/opt/local/bin/mpg321'
Autotest::Sound.sound_path = "/Users/jamieflournoy/autotest/sound/sound_fx/"

Note that the sound plugin needs a patch:

29c29
<   [:run, :red, :green, :quit, :run_command, :ran_command].each do |hook|
---
>   [:initialize, :red, :green, :quit, :run_command, :ran_command].each do |hook|

You’ll also need to make sure your sound set has an initialize.mp3 instead of a run.mp3.