Zombie.js: insanely fast, full-stack, headless testing

Posted on by

101309 15

How It Started

If, like me, you’re developing Web applications that have a lot of client-side JavaScript, you’ve already asked yourself “how the hell do I test this thing?”

To test client-side JavaScript — how it responds to user actions and what changes it makes — you’re going to have to run it in a browser. There are tools that let you do that, I tried a few and they were so bad I considered a career change.

It turns out, though, that you can test a lot of stuff by having a good enough approximation of a browser. You need a reliable HTML parser, a DOM implementation that can manage the document tree in memory, and a JavaScript engine to run the client-side code. A browser that doesn’t render anything: a headless browser.[1]

To run such a headless browser you’re going to need an environment that can load, parse and execute JavaScript real fast. Speed is a feature because there’s no bigger waste of time than waiting for your tests to finish.[2] You’ll also need a DOM implementation you can easily hack, so you can add new features (say, HTML5 Web Storage) and remove features (say, HTML5 Web Storage, because we’re testing compatibility with older browsers).

That got me thinking, “why not Node.js?”  Node is built on Google Chrome’s V8 engine, which is insanely fast and optimized for loading, parsing and executing JavaScript. A perfect fit.

What about the DOM? Fortunately, much of that work was already done by Elijah Insua who created JSDOM, a DOM Level 3 implementation, and Aria Stewart who wrote the HTML5 parser.[3] I then set to build a browser environment on top of that.

Why Zombie.js?  If you’re going to write an insanely fast, headless, browser, how can you not call it Zombie?

Zombie Action

Let’s see the Zombie in action. We’re going to start by setting up a new browser. The Browser object maintains state between page requests, things like cookies, Web Storage, history and (when implemented) caching. We’re typically going to use one browser instance for each session (test scenario):

1
2

var zombie = require("zombie");
var browser = new zombie.Browser;

Next we’re going to point the browser at the Web page we want to test and wait for stuff to happen.

Much of what happens as the browser loads the page is asynchronous: getting the HTTP response, grabbing JavaScript files, running jQuery.onready, firing timers, making AJAX requests for additional contents, etc. We also have to anticipate that our request might error.

Let’s see what that looks like:

3
4
5
6
7
8
9

browser.on("error", function(err) {
  console.log("Error:", err);
});
browser.on("loaded", function() {
  console.log("Loaded:", browser.html());
});
browser.window.location = "http://localhost:3000";

This example illustrates three things you’ll see often when working with Node:

  • Nothing blocks, everything happens asynchronously, which is why you don’t wait for things to happen, you register callbacks.
  • The Browser object is an EventEmitter that can fire different events to multiple listeners. Many Node objects are event emitters: having a uniform API simplifies composing smaller pieces into larger applications.
  • The error event is special. If you don’t catch it, Node terminates and dumps the exception stack trace. It may sound harsh, but it’s a wonderful approach to failure.

Let’s simplify that further by following another Node convention:

3
4
5
6
7

zombie.visit("http://localhost:3000/", function (err, browser) {
  if (err)
    throw(err.message);
  console.log("Loaded:", browser.html());
});

I prefer that, higher level, API.

Now we have a single callback that handles both cases. If there’s an error, the callback is invoked with a single argument containing the error. If there’s no error, the callback is invoked with null, and sometimes and additional value (in this case, a reference to the browser). All it takes is checking the first argument to figure whether we’re handling success or failure.

Again having this convention makes it easier to compose code. For example, let’s write a test case using Vows:[4]

1
2
3
4
5
6
7
8

"this page": {
  topic: function() {
    zombie.visit("http://localhost:8080/", this.callback);
  },
  "should have a title": function(browser) {
    assert.equal(browser.text("title"), "Hi there");
  }
}

The topic function is a combination setup/subject, but instead of returning a value, it wires the Vows callback to the browser visit method. If there’s an error, Vows will catch and report it. Otherwise, it passes the second callback argument (here, browser) to the test case.

Clean, simple and easy to use.

Getting Smarter

Let’s try something more sophisticated. A lot of tests simulate a user clicking links, filling forms, pressing buttons. We’re going to submit a form:
1
2
3
4
5
6
7
8

// Fill email, password and submit form
browser.
  fill("email", "zombie@underworld.dead").
  fill("password", "eat-the-living").
  pressButton("Sign Me Up!", function(err, browser) {
   // Form submitted, new page loaded.
   assert.equal(browser.text("title"), "Welcome to Brains Depot");
  });

This portion of the Zombie API is influenced by Webrat. Chances are you’re familiar with Webrat or one of the many APIs that follow this pattern. As much as possible, Zombie.js tries to behave like tools you’re already know.

Once we get the response back, we’ll want to inspect the page and make sure it has the right contents. We do this after any client-side JavaScript modifications. Since Zombie.js is built on a DOM, you can use any of the DOM methods, for example:

1
2
3
4

item = document.getElementById("item-32");
firstLink = document.links[0].href;
field = document.querySelector(":input[name=email]");
rows = table.querySelectorAll("tr:even");

The last two statements use CSS selectors. If you worked with jQuery before, you’ve practiced using CSS selectors to locate HTML elements.[5] In fact, Zombie uses Sizzle.js, the selector engine that powers jQuery. Like I said, tools you already know.

There’s a lot more to the Zombie API. You can check boxes, click links, fire DOM events, manage cookies and Web storage, play with the system clock, trace requests and more will be coming.

Is It Fast?

Let’s find out. I’ll use a test case I wrote for Rack OAuth 2 Server. We’re actually going to test a Ruby application.[6]

Fire up the practice server, a small Sinatra app that listens on port 8080:

1

$ oauth2-server practice --db test

The test case starts by loading the admin console. Since we’re not signed in, we’re redirected to the authorization page. The practice server has a simple authorization page with two buttons, “grant” and “deny”, and we’re going to click on “grant”. That will redirect us back to the admin console, with an access token that we’re going to store in localStorage. Finally, we’re going to verify that the admin consoles displays a list of client applications (see screenshot).

That probably doesn’t sound like much, but let’s see what’s really going on behind the scenes.

The admin console is a client-side application. It’s a bare HTML page, which loads jQuery, Sammy.js, Underscore.js, Protovis, a bunch of Sammy plugins, and eventually application.js. Once loaded, it changes the location hash to “#/” (the Zombie supports hashchange events), which triggers a Sammy route, which runs some application code that makes an XHR request to the server, retrieves the client list (as JSON), makes another request to load a jQuery template, and finally uses the template to render the data as HTML.

Phfew.

Looking at the Sinatra log I counted 33 HTTP requests from that test.

On my 2.4GHz ’09 MacBook Pro, the entire test runs in just under 0.9 seconds. That includes loading Node, Zombie and its dependencies, compiling the test code from CoffeeScript to JavaScript, hammering Sinatra and dumping output to the console. Because that’s how we run tests in the real world:

1
2
3
4
5

$ time vows spec/admin-spec.coffee --spec
 
real	0m0.909s
user	0m0.763s
sys	0m0.077s

Fast enough for you?

Maybe you noticed that every changelog entry records the number of tests in the Zombie.js test suite, and how long it took to run. That’s how we’ll know if it gets any slower.

So there you are, ready to get started with Zombie.js. Have fun and may your test suite always run fast.

—-

[1] I explained the headless part. As for the full-stack: Zombie.js makes HTTP requests to your test server, exercising every layer of the stack, from HTTP header parsing to database access to response rendering.

[2] I don’t know about you, but if the tests take too long to run, I lose the flow. The speed is not just to spend less time waiting, but to stay in the flow.

[3] Initially looked as Envjs, but it needed a lot of work to fit into the asynchronous model of Node.

[4] I use Zombie in combination with Vows, an asynchronous BDD framework. The “asynchronous” part sounds like buzzword bingo, right until you start using Vows and realize a) it fits your mental model when writing the code you’re about to test, and b) it uses the common idioms and that make test writing easier.

[5] If you worked with Rails, you’ll know assert_select and how to use CSS selectors to test rendered HTML. I released it over 4 years ago, so consider Zombie.js your new & improved, faster & smarter assert_select.

[6] Zombie.js is not just for Node.js, in fact my first two uses cases are for testing Rails and Sinatra applications.

41 thoughts on “Zombie.js: insanely fast, full-stack, headless testing”

  1. I’m currently using ruby/harmony – the advantage of my way
    over yours appears to be that my test suite is self-contained – I
    don’t need to run a daemon, I can override the xhr stuff using
    harmony/johnson and have it execute directly against my rack app
    using rack-client. That’s a big advantage. OTOH my js tests are all
    written in ruby, which is somewhat disappointing. Can’t wait to try
    this and see where it fits in. I’d love to just run js/jasmine-node
    tests.

  3. Assaf, this is incredible. I looked into writing my own framework also, but this is tops. Vanity and now zombie.js are high on my list of tools to use.

  4. Pingback: JavaScript Magazine Blog for JSMag » Blog Archive » News roundup: Zombie.js, evil.js, and even more JavaScript game engines

  5. This looks like really great stuff, mate. Good job!

    I suppose I’m left trying to work out a potential usage case – i.e. when I’d need to use this in my everyday projects.

    I’m so used to whacking open the browser to test my javascript and using the console for everything, so I’ve actually never done any ‘headless’ testing before..

  6. Hi, I’m new to the testing game so apologies if my question sounds stupid.

    I have toyed recently with JsTestdriver which I think is more of a test runner. It seems very good and when I saw zombie.js I thought: instead of using jsdom to simulate a “fake” dom wouldn’t it be cool if zombie could work with JsTestdriver in order to run test in real browsers ?

    Does that make sense ?

  7. Pingback: Spotlight – Recommend Tools For JavaScript Developers: Issue 1

  8. Pingback: The Morning Brew - Chris Alcock » The Morning Brew #762

  9. The problem I always run into with headless running against the webpage itself is, when things go wrong (test fails) it is sometimes hard to determine why. I enjoy selenium for this reason, even if it is a little slower as you are able to take screen shots at the point of failure. Another thing you can do is properly write the code with testing in mind, and just run unit tests against the js code itself in node.js. Then you may only have to worry about smaller edge cases etc. ..

  11. @Morgan: “Full stack” because it’s testing the entire application stack: client side code, HTTP transport, session handling, all the way to the database.

  12. So how does testing your code exclusively against V8 helps much other than ensuring that, well, it works on Chrome?

    I don’t wanna rain on anyone’s parade here. Honest question!

  13. I’m pretty sure Assaf’s aims are like mine – to get fast feedback on a fully integrated stack, a setup that gives you a pretty-good-to-great idea that things are going to work IRL.

    The idea that something is pointless without getting actual feedback Chrome / FF / IE / whatever is, on the typical small webapp, wrong. Not just a little bit wrong, but productivity-killing/rathole/team demotivating kind of wrong.

    A fast-running integration test suite that doesn’t involve a bunch of running daemons is an incredibly powerful tool.

    But let’s say you work at Google or Facebook and the company cares deeply that their apps work in specific browser/OS combos, they want a regression capability that proves this. They have the money and people resources to devote to making an elaborate Selenium setup work. That’s where real browsers and real DOM’s matter.

  14. “But let’s say you work at Google or Facebook and the company cares deeply that their apps work in specific browser/OS combos”

    I don’t agree. Whenever I write JS for browsers, I think I need to be concerned about browser compatibility. It might not be across the whole board given, say, user stats showing there’s little importance in supporting IE6, though surely developing exclusively against Chrome means the other bad extreme in this case.

  15. I wrote Zombie because I’m developing applications that are heavy on client-side JavaScript. We’re not talking some fancy slide effects or drop down menu, but having much of the application logic run in the client.

    I need to make sure the logic is correct, test common and edge cases, and make sure when I change one thing, doesn’t break everything else. Your classic TDD/regression testing, except against logic that runs in the browser.

    Application logic, things like sorting list of items, validating forms, signing up a user, are not browser dependent. And using good libraries, jQuery for example, most of what you write is not browser-sensitive any more.

    I can open a browser to check out the few edge cases, and obviously that it looks good, but for TDD/regression I don’t need to us a browser any more. That’s why Zombie rocks.

  16. “I don’t agree. Whenever I write JS for browsers, I think I need to be concerned about browser compatibility. It might not be across the whole board given, say, user stats showing there’s little importance in supporting IE6, though surely developing exclusively against Chrome means the other bad extreme in this case.”

    Not at all. In my experience, in modern browsers, things tend to work out fine. You’re spending a bunch of time worrying about and area where I claim returns are diminishing. You’re dragging along Selenium infrastructure, setting up CI environments, etc while I’m zooming along with my test suite that takes 15 seconds to run and has no flakiness and minimal maintenance cost.

  21. Looks good – as good as it can be – I keep hoping for a magical js ruby solution that lets me test in js and run zero daemons.

    Guess we need to wait for some kind of js -> ruby bridge for that to be possible…

  24. I appreciate the depth of this article and I’d like to use this, but how do I actually run it? I had to dig into the docs on the git repo to figure out how to install it, but not knowing anything about the node environment, I have no idea how to run a test once I’ve written it.

    Anyone?

  26. “With a startup time in the milliseconds, no need for this to run as a daemon. Fire up, run the tests, shutdown.

    José Valim and Bob Lail are cooking something awesome.”

    Do you know how they plan to avoid the ruby require/startup penalty for every test? I don’t notice them doing anything like forking off a prepared process that hangs around for the duration of the test run…

  28. I’m curious on the status of: “There’s something in the works for Ruby.”

    I want to start working Zombie into my Rails 3 app, and it’d be great to use some existing tools.

  29. I’m skeptical of this setup. I don’t want to go around spreading FUD, but it seems to me running fast full-stack tests in-process, given that you have a Ruby backend, is really only practical if you use something like Harmony (and the tests are therefore ruby-based).

    Requiring Rails libraries is often the most time-consuming part of a test. If your tests are not in ruby, and you’re requiring the backend stack on every test, your test performance is terrible.

    Doing something about that problem usually means running other processes – either a server process or something that requires in your server code and allows cheap forking. But those are adding (unfortunate) complexity.

    I write Selenium-style end-to-end tests in Ruby/Harmony, and keep all my js unit tests in js/jasmine. There’s a clear division of labor, plus my end-to-end tests are fast and there’s just the one single test process (well, for my ruby tests. Then a node run for my js unit tests).

  30. Yeah. FUD.

    Running with Zombie is no different from running without it (say RSpec and Rails integration tests, or Cucumber and Webrat, or Selenium). The cost of firing up Rails is the same, and no one is planning on starting up Rails for every single test.

  32. “The cost of firing up Rails is the same, and no one is planning on starting up Rails for every single test.”

    So are you assuming rails is running in the form of a server process?

    If so how are you planning on doing things like resetting fixture data?

    If not, what’s kicking off the tests – are you running js tests from ruby?

  33. Fixtures belong in the database: you can drop the database, recreate, reload fixtures without having to shutdown the app. You can basically reset the entire state between runs without having to shutdown/restart.

    The Capybara-Zombie project would allow you to write tests directly in Ruby.

    Our application is very JavaScript-heavy, so it makes more sense to write all the tests in JavaScript (or, actually CoffeeScript), but we’ll probably use Ruby to kick start it.

    I’m still experimenting with different options, when I land on something I really like that’s stable, I’ll share it here.

  34. Your needs sound like they match up to mine pretty well. I have my unit tests in pure javascript and then end-to-end happy-path tests in ruby (driving harmony/envjs/johnson). It’s a pretty clean division of labor.

  38. @Mike the Capybara driver could use a better protocol. I’m thinking, switch Capybara to use Bokor, then add commands for everything Capybara needs to do (it has an extensive test suite). This will get us to a full working API.

  40. I’m new to the testing scene, but Zombie sounds like a winner:
    * It’s Linux based (but can also run under Windows)
    * Does not require a Java server (Selenium does)
    * Is “really, really” fast :)
    * Supports sites with AJAX and HTML5

    There is however a big downside compared to WaTir or Selenium: it does NOT have a recorder. This means having to write those tests manually. This got me thinking: what if we could use Selenium’s IDE to record the test and export it in Zombie? Selenium allows new formatters, so building a Zombie formatter should do the trick: you manually run through the test once, get the IDE to export in Zombie.js format and customize as needed.

    Has anyone already tested/built this?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>