When I hear “JUST BANG OUT CODE THAT WORKS” I think of all the apps I don’t use anymore because they gradually lost the ability to iterate.

– Avdi Grimm

§ The Hidden Biases in Big Data. TL;DR there’s a digital divide, and big data is biased towards one side of that divide.

§ Induction — a new kind of tool designed for understanding and communicating relationships in data. Supports PostgreSQL, MySQL, SQLite, Redis, and MongoDB out-of-the-box.

§ Times Haiku, a collection of Haiku found in The New York Times. Love this one:

As an engineer,
I’m sort of a student of
how things fall apart.

§ Eric Bogs:

If Tetris has taught me anything, it is that errors pile up and accomplishments disappear.

§ Rich Rogers:

“A good programmer is someone who looks both ways before crossing a one-way street.” – Doug Linder

Stateless Stop writing classes (or how to avoid unnecessary state)

Stored Everything you need to know about Redis persistence and how it compares to Postgresql and MySQL:

From a more practical point of view Redis provides both AOF and RDB snapshots, that can be enabled simultaneously (this is the advised setup, when in doubt), offering at the same time easy of operations and data durability.

Thinking Learn Unix the Jesse Storimer way:

Read concise programs, cross-reference them with manual pages. Try writing your own stuff. Rinse. Repeat.

RCP over TCP Catch the Replay – Build reliable, traceable, distributed systems with ZeroMQ.

Defaults Lots of useful OS X settings you can enable/disable from the command line.

So true The six stages of debugging.

The kitchen On coffee machines and communities.

QotD Mathias Meyer:

No manual deployment goes unpunished.

Queued There are two things you can do with a relational databases. One, use it as a relational database. Two, try to work around the inconvenience of it being a relational databases. At which point you might be wondering, can I use MySQL as a queue? Yes, but don’t.

Private browsing not always so private. If you’re interested in details, here’s an analysis of private browsing modes in different browsers (PDF).

Ensemble Installing Rails and MySQL Node.js and MongoDB on Ubuntu using Ensemble. Also my introduction to Ensemble Juju. Also, I can’t keep track.

gzip bloom Golomb-coded sets are compressed Bloom filters. Quite useful when you’ve got spare CPU cycles but short on storage.

Signal Turns out Lion ships with a hidden WiFi diagnostic tool (also not-so hidden WiFi connectivity issues) that’s quite nifty.

No browser love Ever wondered why restaurant web sites are so horribly bad? Slate answers.

Pr0n Just as important, how much of the Internet is really porn?

José Valim:

Travis is like Github for CIs. So easy to setup that everyone does it.

It’s the new awesomesauce.

If you need continuous integration for your open source projects than Travis CI is a no brainer. It’s free. It works. It’s trivial to setup. If you don’t know you need CI for your open source project … that’s topic for another post.

It’s “an open-source, distributed build system for the Ruby community”, but if you’re willing to violate these ToS, you can also use it on Node.js projects. See this little hack.

You might have to fiddle with the .travis.yml file, but setting up a build matrix in a text file is orders of magnitude easier than using a Web UI.

Need to test on both Ruby 1.8.7 and Ruby 1.9.2? Run the test suite under both Rails 2 and Rails 3 setups? Test against MongoDB, MySQL, Postgres and Redis? Done, done and done.

The build configuration is checked into source control, that means, anyone can improve it. Right now I can’t get the Vanity tests to pass on 1.8.7 against MySQL, but maybe you can take a look and help?

And — this is my favorite part about Travis CI — anytime you branch or fork a project, you get a new build. Just like that. (If you’re forking, you’ll need to add your fork to Travis, but that’s only a click away)

That means you can work on patches in your own branch and run them through the entire build matrix. Just like that.

When it’s time to contribute your patches to the master branch, you know they won’t break the build. And if there’s need to change the build, you can merge these changes too.

If you’re managing an open source project, you ought to be looking at Travis CI.

I hope soon enough we’ll be able to use it for our internal build system. Until then, my open source projects are in good hands.

Uphill, Both Ways

I got interested in JavaScript when it arrived to the world as Netscape LiveScript. It came in two distinct flavors: client-side JavaScript running in the browser, and not many people remember, Server-Side JavaScript for writing the back-end logic. I believe back then it was called LiveWire.

Next to client-side Java, client-side JavaScript didn’t look like much. DHTML was nice, but nothing compared to the promise of mobile agents roaming around the Interwebs doing all sort of fancy tasks on your behalf. If you knew how to develop — back then JavaScript was marketed for the more casual programmer — you ought to be looking at Java Applets.

Then we let a big company full of engineers perched atop their X Server workstations to manage the UX for Java. We know how that turned out.

On the server-side, JavaScript was limited by a crappy standard library, a sluggish implementation, and the cost of a Netscape Enterprise license.[1] Contrast with Java, which came with a rich standard library, adequate performance, and you could either get an open-source server or were busy developing one.[2]

It’s been longer than a decade since I stopped caring about server-side JavaScript. Now I’m rebooting that passion. So what changed since?

Stuff Got Better

JavaScript is still the same ill-designed language, best known for its high density of “surely this was designed to confuse” language elements. Then again, nothing interesting happens around pure and perfect languages, at best, they evolve into text editors. Fortunately, we’ve got CoffeeScript, a dialect that mixes the better part of Python with the better part of Ruby, and compiles into straightforward JavaScript. It’s incredibly fun to use.

The library situation has improved tremendously. I remember searching for JavaScript code, only to find snippets of badly written DHTML code. Today? Take a look at this incredible list of Node.js modules. Need a Web server framework? Dropbox API? More uptime? Real-time monitoring? Asynchronous BDD? Web scraping?

I went looking for database drivers and found one for MySQL, one for MongoDB and three for Redis. Oh, and a JavaScript implementation of Redis called Awesome.[3]

We’ve got three runtime platforms for choose from: the insanely fast Node.js, the JVM-based Rhino and it’s C-based sibling SpiderMonkey. Competition is healthy. Then again, I’m betting on Node.js. It’s brilliantly fast, fun to use, got a story to tell (everything is asynchronous), but mostly, you can choose from a variety of hosting solutions.[4]

Yeah, we also got push-button hosting for JavaScript.

Looking for a chance to exercise your JavaScript-fu and prove you understand why 0 == []? You can start by writing a Web application using one of many cool frameworks. Or you can package your HTML-based UI into a standalone or mobile application (thanks, Webkit). You could be writing server-side code, or scripting your database: CouchDB and MongoDB both use JavaScript for user code. You can run cronjobs in JavaScript, replace make or managed your EC2 instances.

Better libraries, better performance, and more places to run your code.

It’s A Start

It’s not all peaches and roses though.[5] Most of the libraries out there are young, buggy and incomplete. I got Node.js to segfault a few times. There’s no killer framework on the same caliber as Rails, nor anything that comes close to ActiveSupport and a decent standard runtime library (hmm … that gives me an idea). But then, it’s not much different than what Ruby was five years ago, or Java back in the late 90′s. We’ve all got to start somewhere.

I know I’m going to be doing a lot more JavaScript in 2011. And much of it will come at the expense of what I currently use Ruby for. I already shifted the balance towards applications that use more client-side JavaScript and less server-side Rails. How did it work out? better performance and just as easy to maintain. Next, I’m moving the test suite over to JavaScript thanks to Zombie.js.

So that’s my plan for 2011, and you should follow me on Github and Twitter to find out what happens.

[1] Netscape actually compiled server-side JavaScript into bytecode, so it wasn’t all slow. Back on ’95 hardware it felt slow for what little it did. Java Servlets weren’t speed demons either, but on ’97 hardware and with much more promise, they felt fast enough.

[2] Remember when Java was 10x slower than C, but the language of choice for productive developers? Good times.

[3] I didn’t mention jQuery since this post is about server-side JavaScript. But jQuery did show the world JavaScript code can be clean, simple and productive. I think it deserves credit in changing people’s opinion, including of JavaScript as a server-side language.

[4] Node.js is not the first platform to have a good asynchronous story, it’s just the first platform that many people care to use that has a good asynchronous story. And it’s not so much about the asynchronous part as it is about a good, compelling story. Rhino’s story is “we’re JavaScript only in Java” and SpiderMonkey’s story is “we’re not Rhino”.

[5] Bet you didn’t know JavaScript™ is now trademark of Oracle Corp.

Using MySQL as a NoSQL – A story for exceeding 750,000 qps on a commodity server. Brought to you by the NoMySQL movement.

Miner finds the hottest upcoming Rubygems

jQuery 1.4.3 Offline Learning Kit

Taco Bell Programming — more gets the point across than link bait.

Awesome stuff I never knew CSS could do

YouTube5 replaces the Flash player on YouTube, Vimeo and Facebook

The downside is that they’re never perfect. Other people’s code may be less buggy than mine, but I still run into bugs that I need to patch.  Then there are incompatibility issues, behavior changes from one release to another, and features I have to extend to meet my specific needs.

Some frameworks and libraries are good for that.  Some, whatever you gained from using them is quickly lost when you have to debug their complex code, get surprised by changing behavior, or struggle to extend them. More than once, I wasted more time using a library than it would take to write it myself.

Over time I settled on a few choice frameworks and libraries where the balance was in favor of saving time and building from other people’s expertise. Those may not be the most popular choices out there in Ruby land, but they proved to have the best return-on-investment for me. Here’s a list of my current development stack.

Bundler — I work on several projects at once, maintaining a single environment for all of them is impossible. Bundler makes the Gem management pain go away. Switch to a different project, or even branch within that project, run bundle exec bash and you’re done. Also works great when deploying and getting the right environment in production.

Capistrano — Speaking of deploying, I have some misgivings about Capistrano, but overall it’s great.

Rake — Great for breaking complex jobs into smaller tasks. It’s easier to develop and test when working one task at a time, it also allows complex jobs to pick up they left off, for example, when you bump into a bug or the unexpected.

Unicorn — My production Web server of choice. I also use it for development. The fact that you can run two or more processes at once makes it faster than Webrick, and when you’re doing UI work, waiting for Webrick is cycles.

Resque — If it doesn’t have to be done synchronously, I push it off to a Resque worker.

Pipemaster — Pre-forking server for commands. Think Unicorn (and much of the same code base) but for requests that don’t come through the Web: cron jobs, command line, inbound emails, etc. I also use it to manage my Resque workers, since it’s able to hot-deploy new code with a kill -USR2.

Padrino — A Web framework based on Sinatra. If you like fast, simple and less layers of abstraction between you and HTTP, you ought to check it out.

JSON/JQT — More and more of the UI I’m working on is going down the path of static HTML pages, JSON data and jQuery templates. Static HTML is easy to cache (on server or client), and JSON data means less load and higher latency. It also doubles as an API.

Vanity – I use it for A/B testing and as a metrics dashboard. Some of the metrics come from Google Analytics, some from database queries, or collected by Vanity. Having them all in one place is the easiest way to stay on top of things.

MongoMapper — I recently moved away from MySQL to MongoDB. Lots of reasons. MongoMapper is similar in spirit to ActiveRecord, and the API is similar in many ways, but it’s not a drop-in replacement. I’d say it’s re-imagining what an ORM should be for MongoDB. Love it so far.

SyslogLogger — All the logs from all the processes and up in syslog.

Shoulda — The best ideas of BDD with the bullet-proof simplicity of asserts. I use it for unit, functional and integration tests (the later in combination with Webrat).

OAuth/OAuth2 — You already have a Twitter/Facebook/Google account, why not use it to log on?

jQuery — D’oh.

Necktie came about when I needed to automate server configuration for Apartly. We run on EC2, which means instances for production, for staging, and for short-lived tasks like testing out new configuration or performance benchmarking.

To get the all instances configured the same way, I wrote a Capistrano recipe. I quickly hit the limits of that approach: it would take too long to copy individual files (configs, scripts, etc) from the local box. And I would have to maintain, in addition to the local recipe, a Ruby script to orchestrate everything on the remote machine.

So the next logical step was to pre-package everything — the scripts, configurations, binaries — into a Git repository and have Capistrano clone it on the remote server and run a main script there. It’s similar to cap deploy, except the code you’re deploying and running does things like reloading Nginx with a new configuration, mounting an EBS volume for MySQL, installing cron jobs, stuff like that.

It worked extremely well, but quickly developed into a mishmash of common features and application-specific tasks, part in the application’s own recipe directory, part in a separate Git repository I named necktie. It was begging for extraction.

A few iterations later, and Necktie was born. Necktie is basically a common line tool based on Rake and Git, and a Capistrano task for running it remotely.

Necktie In 15 Seconds

To manage a server configuration, you create a Git repository with a Necktie file in it. You use the same repository to hold other files used for setting up a server: config files, cronjobs, scripts, Ruby gems, etc.

If you have more than a handful of tasks you can split them into individual files and place these in the tasks directory.

A Necktie repository will look something like this:

$ tree .necktie
.necktie/
|-- Necktie
|-- etc
|   |-- cron
|   |   `-- snapshot
|   |-- god.rb
|   |-- init.d
|   |   `-- unicorn
|   `-- nginx
|       `-- unicorn.conf
|-- gems
|   |-- memcache-client-1.7.5.gem
|   |-- mysql-2.8.1.gem
|   `-- unicorn-0.93.3.gem
`-- tasks
 |-- app.rb
 |-- deploy.rb
 `-- db.rb

Separately, I have a Capistrano recipe which contains the following two lines:

require "necktie/capistrano"
set :necktie_url, "git@example.com:necktie"

The first line adds the cap necktie task, the second line tells it where to find the necktie repository. Of course, you should be using your own Git repository.

To configure new servers and upgrade existing ones:

$ git push && cap necktie

Life couldn’t be simpler.

Tasks 101

The heart of your Necktie repository consists of tasks that configure and setup a server instance. You can mix Ruby and shell commands to write tasks like:

# Create /etc/init.d/unicorn so you can sudo service unicorn restart
cp "etc/init.d/unicorn", "/etc/init.d/"
update "/etc/init.d/unicorn", /^ENV=.*$/, "ENV=#{Necktie.env}"
chmod 0755, "/etc/init.d/unicorn"
services.restart "unicorn"

In addition to good old Ruby you’ve got direct access to all the FileUtils methods (cp, mv, chmod, etc), Rake’s sh method and FileList, all of Rush, and a bunch of niceties like read/write/append/update for processing files, install_gem and a thin API around the Linux service command.

Besides bringing fresh servers up to speed, you’ll also want to use Necktie to upgrade production server with least amount of down time. To make that easier (and users happier), write tasks that only execute when necessary.

For example, here’s a task that changes the default memcached configuration to listen on all IPs (I’m running this behind a firewall). It’s designed to work if memcached is not running (and in doing so start it), or if memcached is running with the wrong IP address:

task :memcached do
  # Out of the box, memcached listens to local requests only.
  # Allow access from all servers (in the same security group).
  unless processes.find { |p| p.cmdline[/memcached\s.*-l\s0.0.0.0/] }
    update "/etc/memcached.conf", /^-l 127.0.0.1/, "-l 0.0.0.0"
    services.restart "memcached"
  end
end

The next task updates an Nginx configuration file and reloads its whenever the configuration file from the Necktie repository (that would be “etc/nginx”) is newer than the configuration file installed on the server (that would be “/etc/nginx”):

file "/etc/nginx/sites-available/unicorn.conf"=>"etc/nginx/unicorn.conf" do
  cp "etc/nginx/unicorn.conf", "/etc/nginx/sites-available/"
  ln_sf "/etc/nginx/sites-available/unicorn.conf", "/etc/nginx/sites-enabled/"
  sh "service nginx reload"
end

Here’s another example that will only mount a volume once:

task "/vol" do
  sh "mount /vol"
end

And you can easily express dependencies and use smaller tasks to compose larger tasks:

task :unicorn=>:memcached # unicorn runs our app, so needs memcached
task :nginx=>:unicorn # nginx is front-end to unicorn
task :app=>[:environment, :nginx, :postfix]

Role play

You probably have servers configured for more than one role, at least app server and db server. I recommend using the same set of roles for both Capistrano and Necktie, then you can run commands like this:

$ cap necktie ROLES=app

This instructs Capistrano to run the necktie task against all servers in the role app, and only these servers. Capistrano will invoke the necktie command with the single argument app, telling Necktie to only execute that one task. You just need an aggregate task for each role name.

I have one task called app that orchestrates all the tasks necessary to configure a server for hosting the Web application, and another task called db that sets up MySQL server and EBS snapshots. For convenience I placed them in tasks/app.rb and tasks/db.rb, respectively, and because app is the default role, also defined:

task :default=>:app.

Why Necktie, why not …?

I looked at Puppet, but it speaks some language I don’t understand. I am very well versed in Ruby, Bash and Rake, and that combination works well in so many places, so why not here?

The complexity of your configuration system should be proportional to the complexity of they system you’re configuring. I have to manage a handful of EC2 instances, all running Ubuntu, in three different roles. Chef demanded too much of my time in the kitchen.

Necktie is minimalist, straight Ruby, and uses Git for storage and distribution. Just right.

Image by 2bib.de

.@csharpfritz, @jptoto If you are using EC2 EBS volumes and not taking regular snapshots, you are not being backed up…

ShlomoSwidler

Amazon EBS snapshots are awesome. You can take snapshots to backup your database, quickly move data between staging and production, run analytics on the latest data, resize volumes and much more.

For Apartly, I’m using MySQL, storing data and log files on a mounted EBS volume. EBS offers persistent, reliable storage (with replication for fault tolerance). Still, nothing can prevent data from disappearing, hence backups.

The current setup uses a cron task that runs hourly and creates a snapshot of the EBS volume. It’s a simple procedure involving flushing, locking, and synching, and you can read all about it here. If the database goes catastrophical, I can always go back in history.

Since you don’t want to accumulate snapshots until the end of time (storage is not free), the script keeps hourly snapshots from the past 24 hours and daily snapshots from the past week, and discards the rest.

It’s a very simple script, based on Eric Hammond‘s excellent ec2-consistent-snapshot. Notable differences:

• Using Ruby (tested for 1.9.1 only) instead of Perl. (Mind you, they’re both Unix)
• Discards old snapshots, only keep hourly from last 24 hours and daily for past 7 days.
• Get it: wget http://bit.ly/PTR9K
• Edit settings, then drop into /etc/cron.hourly

Enjoy.

Seek and you shall …  How the brain hard-wires us to love Google, Twitter, and texting:

“The dopamine system does not have satiety built into it,” Berridge explains. “And under certain conditions it can lead us to irrational wants, excessive wants we’d be better off without.” So we find ourselves letting one Google search lead to another, while often feeling the information is not vital and knowing we should stop. “As long as you sit there, the consumption renews the appetite,” he explains.

New in cuke. Five Cucumber/Webrat features you may not know, and the new arguments transform.

The buddy system. Interesting look into how our network affects our health and happiness.

Snapshots. Running MySQL on Amazon EBS. Love being able to backup the database with a snapshot.

Exposé + space. Must have been asleep at class when they covered these Mac keyboard shortcuts. Useful stuff.