Or as the tagline says: “Stop Guessing, Start Testing”. A/Bingo is a plugin for Rails apps that helps you run A/B testing:
A/B tests are always defined in code, and require no setup or configuration. The first time code for a particular test name executes, it does all the setup work for the test and logs the first participant. All subsequent times the test alternatives are read straight from cache, and the only DB access is to score the participant (a fast, optimized UPDATE). A single test name should never be used to refer to a second test.
I started my journey by looking at Google Web Optimizer. GWO is impressive in many ways, what turned me away was the complexity of setting up experiments. Since GWO can’t run code on your server, you have to mark your templates with special script tags, then specify different text/HTML alternatives from the GWO console. There’s additional complexity involving URLs and multivariate experiments.
In contrast, A/Bingo couldn’t be any simpler to use. It’s a Rails plugin, which means your experiments live inside your Rails app, you can even test them as part of the app.
A/Bingo is an extraction from an actual product, which explains why it does one thing, and does it extremely well. The code base is plain simple, always welcome after struggling through over-engineered plugins (cough*authlogic*cough). Simple, not dumb. The API had everything I needed, and pleasantly surprised to find out it uses caching to keep load under control.
Did I mention that it just works?
A couple of helpful tips to use A/Bingo effectively. Depending on the size/complexity of your experiments, you may want to inline alternatives or separate them into partials. And you’ll want to use a query parameter to select a particular variant, that way you can easily experiment from the browser and write test cases for each variant.
I have code that looks something like:
<% intro = params[:intro] || ab_test("intro_size", ["elaborate", "concise"]) %>
<%= render intro %>
You’ll obviously want these experiment reports at the end of each day (A/Bingo uses one-tailed Z-score). I use a cron job that runs daily and emails me all that day’s results:
#!/usr/bin/env ruby
RAILS_ROOT = "..."
require RAILS_ROOT + "/config/environment"
exps = Abingo::Experiment.all.map { |e| [e.test_name, "-" * e.test_name.length, "", e.describe_result_in_words, ""] }
email = TMail::Mail.new
email.subject = "Daily A/Bingo results"
email.to = "..."
email.body = exps.flatten.join("\n")
ActionMailer::Base.deliver email