Judging by the lack of posts on the topic of testing and migration, I’m guessing most developers don’t. You try and hope for the best.
I wouldn’t trust my migrations to just work, not when it comes to live data in production. In fact, I just discovered two bugs in mine.
The first bug happened when I tried to split one table in two, to allow users to auto-login from different computers. One table has an optional field, but in the split table, that field is record that only exists if you login. The migration I wrote worked well on the test database, which happened to have login for all accounts.
But it failed when doing a dry-run on the production database. I wouldn’t catch it in time without doing the dry-run test.
The second bug happened when I tried to merge two tables into one. I created a redundancy in the authentication system which evolved over time. The code was growing too big to maintain, so I decided to streamline it and merge the duplicates. So I loaded objects from one table, created them in the other table, and deleted the first set.
That worked fine, but just before deploying the migration, I decided to get rid of some unused fields in the database. And that’s when I broke the migration.
The migration starts by requiring the User model. Then changes the database schema, and then attempts to save new User objects. Because the database schema changes after the User model is loaded, the saving fails without reporting an error. It looks like it works, but there’s no change in the database.
The solution was easy, one line of code courtesy of script/console. After changing the database schema, I force all classes to reload by calling Dispatcher.reset_application!.
Note to self: Always use Dispatcher.reset_application! if you’re changing the database schema and then changing the data.
Fortunately, I discovered both bugs by doing a dry-run against the production database. The migration starts by loading objects into memory, then runs the migration against the database, and tests these objects against the database, printing discrepancies to the console.
I backup the affected tables, run the migration, check the console. If there’s no output, I’m good to go. If I see any errors, I switch back to the old table and fix the migration code.
This also brings me to another topic. Using migrations as the database definition (thanks to Ed Gibbs for the reminder):
There’s a downside to migrations. Over time, your schema definition will be spread across a number of separate migration files, with many files potentially affecting the definition of each table in your schema. When this happens, it becomes difficult to see exactly what each table contains.
I don’t. I treat migrations like patches. You write them to change the existing behavior, and apply the patch to get a new version. And you can’t always go back. If I’m splitting a table to allow multiple values, I can’t switch back without losing the new data.
The patch is the difference, but the current state is in the latest snapshot. That’s what rake db:structure:dump is for. The migration is just code I can test in development environment, and patch the production environment with the confidence it will work.