This is a video of the talk Nathaniel Talbott gave at RubyConf ‘09, introduction Experiment Driven Development and … well, I won’t ruin the surprise.
You can enjoy the Flash video on the Confreaks site.
Download 1280 x 720 HD MP4 or 640 x 360 MP4
Thanks to the Confreaks crew for making this video.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | module Resque module Failure # Store a copy of the failure in Redis, so we have access from the UI. Also # send an email to the developer, so we know something went foul. class Notifier < Base def save # Store in Redis super # Create notification email email = TMail::Mail.new email.to = ".. your email here .." email.subject = "[Error] #{exception}" email.body = <<-EOF Queue: #{queue} Worker: #{worker} #{payload.inspect} #{exception} #{exception.backtrace.join("\n")} EOF Mailer.deliver email rescue end end end end Resque::Failure.backend = Resque::Failure::Notifier if ENV['RAILS_ENV'] == "production" |
I heard this question (in different form) from various people:
Isn’t Apple a massive counterexample to the Minimum Viable Product movement/theory?
Let’s put Customer Development in context. If you’re a startup entering a new market in today’s economy (read: non-bubblical times), your biggest risk is most likely not new technology.
Your biggest risk is running out of money before you find customers for your product (or a product for your customers). Customer Development and (Minimal Viable Product and such) help you mitigate this risk.
Now, what if you’re an rich company that can throw millions on a hobby project, operates in established markets, and have customers lining up months in advance to buy a product you’ve yet to announce? You’ve got bigger chips to fry.
$APPL is a strawman, contrasting it with Customer Development will teach you nothing new or important. You should be looking instead at how successful startups operate in new markets, with unproven products. Take, for example, the story of Apple:
The Apple I and Apple II computers were shown off every two weeks at the club meeting. “Here’s the latest little feature,” we’d say. We’d get some positive feedback going and turn people on. It’s very motivating for a creator to be able to show what’s being created as it goes on. It’s unusual for one of the most successful products of all time, like the Apple II, to be demonstrated throughout its development.
Selling motherboards to enthusiasts is a classic Minimum Viable Product, and so is the Customer Development process that followed:
To Jobs’ utter amazement, Terrell agreed to buy 50 computers for $500 each, cash on delivery. There was only one catch to the $25,000 order: Terrell wanted fully assembled computers.
The trio had originally planned to produce bare circuit boards for $25 each and sell them for $50 to hobbyists who would populate them with the necessary chips and other parts.
I guess, when you look at the full picture and put things in the right context, Apple does come out as a shining example for the Customer Development story.
All the Rails 3 reading materials in one place:
Rails 3 is going to bring a lot of new stuff to the table. Wouldn’t hurt to organize some reading material in categorized and chronological order. So here goes whatever I scavenged so far. Leave comments so I can add more links or if I should mark something as obsolete.
Eric Ries on Vanity Metrics vs. Actionable Metrics:
The only metrics that entrepreneurs should invest energy in collecting are those that help them make decisions. Unfortunately, the majority of data available in off-the-shelf analytics packages are what I call Vanity Metrics. They might make you feel good, but they don’t offer clear guidance for what to do.
When you hear companies doing PR about the billions of messages sent using their product, or the total GDP of their economy, think vanity metrics. But there are examples closer to home. Consider the most basic of all reports: the total number of “hits” to your website. Let’s say you have 10,000. Now what? Do you really know what actions you took in the past that drove those visitors to you, and do you really know which actions to take next? In most cases, I don’t think it’s very helpful.
Now consider the case of an Actionable Metric. Imagine you add a new feature to your website, and you do it using an A/B split-test in which 50% of customers see the new feature and the other 50% don’t. A few days later, you take a look at the revenue you’ve earned from each set of customers, noticing that group B has 20% higher revenue per-customer. Think of all the decisions you can make: obviously, roll out the feature to 100% of your customers; continue to experiment with more features like this one; and realize that you’ve probably learned something that’s particular valuable to your customers.
Stay away from vanity metrics. But do use Vanity to get your actionable metrics.
Had enough of Maven dependency and violence? Javagems.org is … just what it sounds … a gemcutter for the JVM languages. gem install jetty and hack away. This is the first useful, all out replacement of the Maven insanity I’ve seen.
With MavenGem you can enjoy all the Maven dependencies straight from JRuby. It installs Maven packages as gems, no need for a new repository. It won’t be long that we’re all able to bask in the awesomeness of EJB on Rails.
Not satisfied with either one? Try TarPit, the glue for an unholy union of Rake, Hoe and Maven. I think it helps package Maven artifacts into your JRuby app for that great taste that last forever.
The killer new feature of Vanity 1.2 are database metrics. You’ve already got the data, so let’s show it in the dashboard.
Here’s a metric for signups (activation), based on the number of Account records created on each day:
metric "Signup (Activation)" do description "Measures how many people signed up for our awesome service." model Account end
No need to call track! on this metric, and it comes with historical data (assuming you already have accounts in your database).
Some metrics measure values, not occurrences. This example measures user satisfaction by calculating average rating:
metric "Satisfaction Survey" do description "Measures how many people signed up for our awesome service." model Survey, :average=>:rating end
Other aggregates you can use are :minimum, :maximum and :sum.
If you’re only measuring some metrics, use conditions:
metric "Signups to Unlimited" do
description "Signups to our All You Can Eat and Burp Unlimited plan."
model Account, :conditions=>{ :plan_type=>'unlimited' }
end
Even better, named scopes:
metric "Signups to Unlimited" do description "Signups to our All You Can Eat and Burp Unlimited plan." model Account.unlimited end
Or just roll your own:
metric "Signups (Unlimited)" do
description "Signups to our All You Can Eat and Burp Unlimited plan (including upgrades)."
Account.after_save do |account|
track! if account.plan_type_changed? && account.plan_type == 'unlimited'
end
end
Get it now, and get measuring.
This release introduces metrics. Metrics are the gateway drug to better software.
A metric definition is a Ruby file that looks like this:
metric "Signup" do description "Measures how many people signed up for our awesome service." end
Once you have your metric, feed it data by calling the track! method. For example:
class AccountsController < ApplicationController
def create
@person = Person.new(params[:person])
if @person.save
track! :signup # track successful sign up
UserSession.create person
redirect_to root_url
else
render :action=>:new
end
end
end
Define, track, that’s about all you need to know.
Metric data can also come from database queries, log files, Google Analytics, etc. To use external sources you have to write your own metric implementation.
You want to start using metrics from your A/B tests, for example:
ab_test "Pricing options" do metrics :signup alternatives 15, 25, 29 end
You’ll see why this is a good idea soon enough.
Much thanks to Ian Sefferman for finding bugs and helping fix Ruby 1.8.7 and Rails support.
Keep measuring and experimenting,
Assaf
Check out Shuba’s notes from Eric Ries recent talk at MIT.
How to build a lean start-up:
- Continuous deployment.
- Tell a good change from a bad change quickly
- Revert a bad change quickly
- Work in small batches (at IMVU, large batch = 3 days worth of work)
- Break large projects down into small batches
- Have a cluster immune system
- Run tests locally. Everyone gets a complete sandbox
- Continuous integration server – tests to ensure all features that worked before still works
- Incremental deploy – reject changes that move metrics out of bounds
- Alerting and predictive monitoring – wake somebody up if metric goes out of bounds. Use historical trends to predict acceptable bounds.
- Conduct rapid split tests: A/B testing is key to validating hypotheses
- Follow the AAAs of metrics: actionable, accessible and auditable
By popular demand: 1.8.6 and 1.8.7 support, available now in Github master branch.
Metrics let you measure stuff that matters. It’s as simple as defining a metric:
metric "Cheers" do description "They love us, don't they?" end
Tracking it from your code:
track! :cheers
And watching the graph:
This is all in Github right now, but needs final tweaks and proper documentation before I can gem it. Appreciate any help with testing/improvements.
Apropos, have you seen Startup Metrics for Pirates?