Fluid Schema Evolution

Fluid schema evolution is a way to manage the evolution of JSON payloads in a web service or messaging context. Fluid schema evolution works by detecting changes to the JSON schema; snapshotting and versioning schema metadata; then generating migration scripts for the contexts in which the changed objects are used. Working this way means upgrade or downgrade migration scripts are included in the same commit as the schema change. This approach backs the fluid-fallback approach used in Fluid Web Services.

Configuration

Fluid schema evolution needs the JSON definition of the objects used and the context in which they are used:

• To work out if upgrade, downgrade or both migration scripts are needed. For example for a web service serving content only a downgrade script will be required while the opposite is true if the web service has data posted to it.
• To generate migration script templates that cater for the different contexts in which an object is used.

Any higher level representation can be used as the source for this information so long as the JSON schema and usage context can be derived. OpenAPI or Swagger is the likely choice and it is assumed that the fluid utility has been configured with the location of a swagger.json file from this point onwards.

Getting Started

The fluid utility is used to execute fluid commands:

> fluid
Usage: fluid <command>
Available commands:
  check      Checks generate has been called following schema change
  generate   Generates template migration and test scripts
  test       Runs the migration test scripts
  revert     Rolls back the changes made by the last generate

Using the /users endpoint described in Fluid Web Services. The schema for the original version of the user structure is:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": [
    "name",
    "location"
  ],
  "properties": {
    "name": {
      "type": "string"
    },
    "location": {
      "type": "string"
    },
    "hobby": {
      "type": "string"
    },
  }
}

If we check the schema error message:

> fluid check
Failure: Schema changed, generate migration scripts

So lets generate the migration scripts and re-check.

> fluid generate
First schema version, no migration required
> fluid check
Version: 1

This has produced the following directory structure:

/.fluid
  /v1
    meta.json

The updated user structure has the following schema:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "title": "The Root Schema",
  "required": [
    "name",
    "city",
    "hobbies"
  ],
  "properties": {
    "name": {
      "type": "string"
    },
    "city": {
      "type": "string"
    },
    "hobbies": {
      "type": "array",
      "items": {
        "type": "string"
      }
    }
  }
}

Now things get more interesting when we run generate:

> fluid generate
Generated version 2, complete migration and test scripts

The fluid directory now has an new v2 directory

/.fluid
  /v1
    meta.json
  /v2
    meta.json
    migrate.js
    test.js

In good TDD fashion let’s look at the test.js first:

function testMigration() {
  testDowngradeUser();
}
function testDowngradeUser() {
  v2 = {
    "name": "name0",
    "city": "city0",
    "hobbies": [ "hobbies0" ]
  }
  v1 = { 
    "name": "name0",
    "location": "location0",
    "hobby": "hobby0"
  }
  assertEquals(downgradeUser(v2), v1)
}

A good start, lets make the changes to make the test passable …

function testMigration() {
  testUserModel();
}
function testUserMode() {
  v2 = {
    "name": "name0",
    "city": "city0",
    "hobbies": [ "hobbies0", "hobbies1" ]
  }
  v1 = { 
    "name": "name0",
    "location": "city0",
    "hobby": "hobbies0"
  }
  assertEquals(downgradeUser(v2), v1)
}

… and run the tests:

> fluid test
Migration failed:
    :                # failure messages omitted

Take a look at migrate.js:

function downgradeUser(v2) = {
  // Unused v2 properties: city, hobbies
  v1 = {
    "name": v2.name,
    "location": v2.___,
    "hobby": v2.___ 
  }
  return v1;
}

Fill in the missing words …

function downgradeUser(v2) {
    v1 = {
        name: v2.name,
        location: v2.city
    }
    if (v2.hobbies.length > 0) v1.hobby = v2.hobbies[0];
    return v1;
}

… to complete the puzzle:

> fluid test
OK
> fluid check
Version: 2

At this point we have our updated structure and the migration scripts in place so we are in a position to commit.

Wrap Up

Fluid Schema Evolution is essential when delivering Fluid Web Services or Fluid Messaging as it provides the means of fallback needed immediately following the upstream rollout.