How Necktie Happened
Necktie came about when I needed to automate server configuration for Apartly. We run on EC2, which means instances for production, for staging, and for short-lived tasks like testing out new configuration or performance benchmarking.
To get the all instances configured the same way, I wrote a Capistrano recipe. I quickly hit the limits of that approach: it would take too long to copy individual files (configs, scripts, etc) from the local box. And I would have to maintain, in addition to the local recipe, a Ruby script to orchestrate everything on the remote machine.
So the next logical step was to pre-package everything — the scripts, configurations, binaries — into a Git repository and have Capistrano clone it on the remote server and run a main script there. It’s similar to cap deploy, except the code you’re deploying and running does things like reloading Nginx with a new configuration, mounting an EBS volume for MySQL, installing cron jobs, stuff like that.
It worked extremely well, but quickly developed into a mishmash of common features and application-specific tasks, part in the application’s own recipe directory, part in a separate Git repository I named necktie. It was begging for extraction.
A few iterations later, and Necktie was born. Necktie is basically a common line tool based on Rake and Git, and a Capistrano task for running it remotely.
Necktie In 15 Seconds
To manage a server configuration, you create a Git repository with a Necktie file in it. You use the same repository to hold other files used for setting up a server: config files, cronjobs, scripts, Ruby gems, etc.
If you have more than a handful of tasks you can split them into individual files and place these in the tasks directory.
A Necktie repository will look something like this:
$ tree .necktie .necktie/ |-- Necktie |-- etc | |-- cron | | `-- snapshot | |-- god.rb | |-- init.d | | `-- unicorn | `-- nginx | `-- unicorn.conf |-- gems | |-- memcache-client-1.7.5.gem | |-- mysql-2.8.1.gem | `-- unicorn-0.93.3.gem `-- tasks |-- app.rb |-- deploy.rb `-- db.rb
Separately, I have a Capistrano recipe which contains the following two lines:
require "necktie/capistrano" set :necktie_url, "git@example.com:necktie"
The first line adds the cap necktie task, the second line tells it where to find the necktie repository. Of course, you should be using your own Git repository.
To configure new servers and upgrade existing ones:
$ git push && cap necktie
Life couldn’t be simpler.
Tasks 101
The heart of your Necktie repository consists of tasks that configure and setup a server instance. You can mix Ruby and shell commands to write tasks like:
# Create /etc/init.d/unicorn so you can sudo service unicorn restart
cp "etc/init.d/unicorn", "/etc/init.d/"
update "/etc/init.d/unicorn", /^ENV=.*$/, "ENV=#{Necktie.env}"
chmod 0755, "/etc/init.d/unicorn"
services.restart "unicorn"
In addition to good old Ruby you’ve got direct access to all the FileUtils methods (cp, mv, chmod, etc), Rake’s sh method and FileList, all of Rush, and a bunch of niceties like read/write/append/update for processing files, install_gem and a thin API around the Linux service command.
Besides bringing fresh servers up to speed, you’ll also want to use Necktie to upgrade production server with least amount of down time. To make that easier (and users happier), write tasks that only execute when necessary.
For example, here’s a task that changes the default memcached configuration to listen on all IPs (I’m running this behind a firewall). It’s designed to work if memcached is not running (and in doing so start it), or if memcached is running with the wrong IP address:
task :memcached do
# Out of the box, memcached listens to local requests only.
# Allow access from all servers (in the same security group).
unless processes.find { |p| p.cmdline[/memcached\s.*-l\s0.0.0.0/] }
update "/etc/memcached.conf", /^-l 127.0.0.1/, "-l 0.0.0.0"
services.restart "memcached"
end
end
The next task updates an Nginx configuration file and reloads its whenever the configuration file from the Necktie repository (that would be “etc/nginx”) is newer than the configuration file installed on the server (that would be “/etc/nginx”):
file "/etc/nginx/sites-available/unicorn.conf"=>"etc/nginx/unicorn.conf" do cp "etc/nginx/unicorn.conf", "/etc/nginx/sites-available/" ln_sf "/etc/nginx/sites-available/unicorn.conf", "/etc/nginx/sites-enabled/" sh "service nginx reload" end
Here’s another example that will only mount a volume once:
task "/vol" do sh "mount /vol" end
And you can easily express dependencies and use smaller tasks to compose larger tasks:
task :unicorn=>:memcached # unicorn runs our app, so needs memcached task :nginx=>:unicorn # nginx is front-end to unicorn task :app=>[:environment, :nginx, :postfix]
Role play
You probably have servers configured for more than one role, at least app server and db server. I recommend using the same set of roles for both Capistrano and Necktie, then you can run commands like this:
$ cap necktie ROLES=app
This instructs Capistrano to run the necktie task against all servers in the role app, and only these servers. Capistrano will invoke the necktie command with the single argument app, telling Necktie to only execute that one task. You just need an aggregate task for each role name.
I have one task called app that orchestrates all the tasks necessary to configure a server for hosting the Web application, and another task called db that sets up MySQL server and EBS snapshots. For convenience I placed them in tasks/app.rb and tasks/db.rb, respectively, and because app is the default role, also defined:
task :default=>:app.
Why Necktie, why not …?
I looked at Puppet, but it speaks some language I don’t understand. I am very well versed in Ruby, Bash and Rake, and that combination works well in so many places, so why not here?
The complexity of your configuration system should be proportional to the complexity of they system you’re configuring. I have to manage a handful of EC2 instances, all running Ubuntu, in three different roles. Chef demanded too much of my time in the kitchen.
Necktie is minimalist, straight Ruby, and uses Git for storage and distribution. Just right.
Image by 2bib.de
Jørgen Orehøj Erichsen
Assaf