Migrating MongoDB schemas

It's quite common to choose MongoDB for a Node.js API project. We know it's a non-relational, schemaless database and that goes quite well in a Node project. In most cases, we tend to use Mongoose as the ODM for MongoDB. This allows us to have schemas, giving some structure to our database.

As our project evolves, it's possible we need to make breaking changes to our database schemas in order to fix some mistakes, or to improve the schemas for the ever growing amount of data.

While running a small migration to a schema is possible with the MongoDB shell, the same is not advisable to larger migrations. This is where a specialised tool like migrate-mongo comes into play.

Use cases

Let's take a look at some possible use cases for MongoDB migrations:

1. Adding new properties to a document. We often need to add additional properties and while we could use Mongoose's default property to assign a default value, this is only useful when we can use the same value to every document in a collection. When we need to add different values, possibly extracted from other properties, a migration is better qualified for the task.
2. Splitting a property into multiple. A simple example is breaking up a name property into firstName and lastName.
3. Moving a nested schema into its own collection. For the sake of the argument let's say we made a mistake when designing our schema and underestimated the amount of the data we would be processing. We designed a User schema with an array of Subscriptions which is a complex schema by itself. Over time data grows and may risk hitting document size limitations. We can run a migration to break those subscriptions into its own collection.

Using migrate-mongo

The setup is quite simple. All we need is a configuration file where we must declare the connection settings for the Mongo driver and some options for the tool itself. Later on we'll be adding migration files as needed.

To bootstrap the folder structure and configurations run migrate-mongo init. This will create a migrate-mongo-config.js file where you need to adjust the connection string if needed, and the database name.

Creating a migration

The tool also provides us with an utility command to generate a migration file. Running migrate-mongo create MIGRATION_NAME will create a file under the migrations folder. It will look like the following:

module.exports = {
  async up(db, client) {
    // TODO write your migration here.
    // See https://github.com/seppevs/migrate-mongo/#creating-a-new-migration-script
    // Example:
    // await db.collection('albums').updateOne({artist: 'The Beatles'}, {$set: {blacklisted: true}});
  },

  async down(db, client) {
    // TODO write the statements to rollback your migration (if possible)
    // Example:
    // await db.collection('albums').updateOne({artist: 'The Beatles'}, {$set: {blacklisted: false}});
  }
};

To fully configure a migration, we should prepare for the up & down scenarios. That is, when migrating forward and backwards, which is useful when the result of a migration proves troublesome.

Considering we want to add a nickname property to the Users collection, the migration could be as follows:

module.exports = {
  async up(db, client) {
    const cursor = await db.collection('users').find();

    while (await cursor.hasNext()) {
      const doc = await cursor.next();
      const nickname = doc.email.split('@')[0];

      await db.collection('users').updateOne({_id: doc._id}, {$set: {nickname: nickname}});
    }
  },

  async down(db, client) {
    await db.collection('users').updateMany({}, {$unset: {nickname: true}});
  }
};

In short, it loops over all the users and uses the first part of the email address as the nickname.

Running a migration

By running migrate-mongo up, the tool will run through all the migration that haven't been processed and execute them one by one.

On the other hand, running migrate-mongo down will only undo the last applied migration.

You can also check the status of your migrations with migrate-mongo status.

Automating migrations

Although being a very situational topic, dependent on your deployment process, let's consider we are using PM2 to deploy our application.

Our deployment configuration file could look like the following, the key part being on the migrate-mongo up command.

module.exports = {
  deploy: {
    production: {
      // configurations...

      // post-deploy action
      'post-deploy': "yarn install && migrate-mongo up && pm2 startOrReload ecosystem.config.js"
    }
  }
};

Conclusion

While it's not mandatory to use migration tools it certainly becomes useful as our project grows, and our data requirements change.

By using migrate-mongo we get incremental, reversible and version controlled migrations that can be reviewed by the team before being shipped into production.

If you liked this article or have something to add, we are available, as always, via our Support Channel.