It may be tempting at the start of a new project to create the first database tables manually, or write SQL scripts that you run manually, especially when you first have to spend a significant amount of time on sifting through all the migration libraries and then some more to get it working properly.

Going through this process did slow me down at the start of the project but I was determined to use a migration tool because hunting inexplicable bugs that only happen in production just to find out there is a definition mismatch between the production and development databases is not fun. Using such a tool also motivates you to write both the setup and teardown steps for each table while the current design is still fresh in your mind.

At first I considered a standalone migration tool because I expect them to be very good at that single task. However, learning the idiosyncrasies of a new tool and trying to make it fit seamlessly into my development workflow seemed like more trouble than it is worth.

I decided to stick with a Common Lisp library and found the following seven that work with PostgreSQL and/or Postmodern:

I quickly discounted Crane and Mito because they are ORM (Object Relational Mapper) libraries which are way more complex than a dedicated migration library. Development on Crane have stalled some time ago and I don’t feel it is mature enough for frictionless use yet. Mito declares itself as being in Alpha state; also not mature enough yet.

I only stumbled onto cl-mgr and Orizuru-orm long after making my decision so I did not investigate them seriously. Orizuru-orm is in any case an ORM which I would have discounted because it is too complex for my needs. CL-mgr looks simple, which is a good thing. It is based on cl-dbi which makes it a good candidate if you foresee switching databases but even if I discovered it sooner I would have discounted it for the same reason as CL-migrations.

CL-migrations looks very promising. It is a simple library focusing only on migrations. It uses clsql to interface with the database which bothered me because I already committed to using Postmodern and I try to avoid adding a lot of unused code to my projects. The positive side is that it interfaces to many different databases so it is a good candidate if you are not committed to using Postmodern. It is also a stable code base with no outstanding bug reports.

The two projects I focused on was Postmodern-passenger-pigeon and Database-migrations because they both use Postmodern for a database interface.

Postmodern-passenger-pigeon was in active development at the time and it seemed safer to use than Database-migrations because it can do dry runs, which is a very nice feature when you are upgrading your production database and face the possibility of losing data when things go awry. Unfortunately I could not get it working within a reasonable amount of time.

I finally settled on Database-migrations. It is a small code base, focused on one task, it is mature and it uses Postmodern so it does not pull in a whole new database interface into my project. There are however some less positive issues.

The first issue is a hindrance during development. Every time the migrations ASDF system (or the file containing it, as ASDF prefers that all systems be defined in a single file) is recompiled it adds all the defined migrations to the migrations list. Though each one will only be applied once to the DB it is still bothersome. One can then clear the list with (setf database-migrations::*migrations* nil) but then only newly modified migration files will be added. The solution then is to touch the .asd file after clearing the migrations list.

The second negative point is quite dangerous. The downgrade function takes a target version as parameter, with a default target of 0. This means that if you execute downgrade without specifying a target version you delete your whole database.

I am currently using Database-migrations and it works well for me. If for some reason I need to switch I will use cl-migrations.

Using Database-migrations

To address the danger of unintentionally deleting my database I created a wrapper function that does both upgrade and downgrade, and it requires a target version number.

Another practical issue I discovered is that upgrades and downgrades happen in the same order as they are defined in the migration file. If you create two tables in a single file where table 2 depends on table 1 then you can not revert / downgrade because Database-migrations will attempt to delete table 1 before table 2. The solution here is to use the def-queries-migration macro (instead of def-query-migration) which defines multiple queries simultaneously . If you get overwhelmed by a single definition that defines multiple tables the other option is to stick with one migration definition per file.