Migration

[chubei]

Migration

Migration File Structure

Currently Dozer data are organized as follows (assuming endpoints are trips and zones):

.dozer
  api
    trips.proto
    zones.proto
    descriptor_set.bin
  pipeline
    logs
      trips
        log
        schema.json
      zones
        log
        schema.json

To support migration, we separate api proto files to make endpoints independent from each other, and scope endpoints by migration sequence number:

.dozer
  api
    trips
      v1
        trips.proto
        descriptor_set.bin
    zones
      v1
        zones.proto
        descriptor_set.bin
  pipeline
    logs
      trips
        v1
          log
          schema.json
      zones
        v1
          log
          schema.json

The migration will happen on endpoint level, which means we may have three migrations for trips but only one migration for zones.

dozer migrate

For every endpoint defined in the configuration file, dozer checks if all the following conditions are met:

• pipeline/logs/{endpoint}/v{n}/schema.json exists.
• pipeline/logs/{endpoint}/v{n}/log doesn't exist.
• Schema is exactly the same as would be inferred from the configuration file.

where {n} is the latest migration.

If all the conditions are met, migration is not needed, otherwise a new migration is created.

The reasoning behind these conditions are:

• Dozer app always processes data from scratch, so if the log exists, we have no choice but to migrate.
• When there's no log, but the schema is different from the inferred one, the data source probably has changed since last migration, so we need to migrate.

dozer app run

Dozer app runs on the latest migration, because previous migrations can be invalid when the schema from the data source changes.

Based on the migration condition above, dozer app doesn't need to handle checkpointing, because the latest migration always has a schema and no log.

dozer api run

Endpoint config gets a new parameter version to specify the migration to use. It can either be a number or latest (default).

Dozer api builds cache from the log in the specified migration.

Checkpointing

An offset parameter is added to RwCache::commit to support checkpointing. When dozer api restarts, it reads log from the committed offset and continues to build cache.

Switching

Currently dozer api serves following REST paths (assuming endpoint path is /trips):

• GET /trips
• GET /trips/{pk}
• POST /trips/count
• POST /trips/query
• POST /trips/oapi

And following gRRC methods:

• Trips.count
• Trips.query
• CommonGrpcService.count with endpoint set to trips
• CommonGrpcService.query with endpoint set to trips

To support switching, we additionally serve following REST paths:

• GET /v{n}/trips
• GET /v{n}/trips/{pk}
• POST /v{n}/trips/count
• POST /v{n}/trips/query
• POST /v{n}/trips/oapi

And following gRRC methods:

• TripsV{n}.count
• TripsV{n}.query
• CommonGrpcService.count with endpoint set to trips and migration set to n
• CommonGrpcService.query with endpoint set to trips and migration set to n

The original REST paths and gRPC methods will be redirected to the migration specified in the configuration file.

Dozer api additionally supports creating APIs on the fly by specifying endpoint and migration, and redirect the unscoped REST paths and gRPC methods to any migration.

I'm not sure if this is necessary because the user can just starts a new dozer api instance with a different configuration file, and points the load balancer to the new instance when needed.

[v3g42]

Hey @chubei. Some points for us to think about.

1. As a user, I want to control the version in my URL.

2. As a user, I want to overwrite the existing API definition despite breaking changes.

3. As Dozer, it would be good to suggest best practices as a framework. So does it make sense to suggest something like "semantic versioning" of APIs? Reference - https://cloud.google.com/endpoints/docs/openapi/versioning-an-api

4. Generally thinking from a migrations point of view, retaining the history of API definitions would be good.

5. Schema and API version could differ.

6. This has to play hand in hand with blue/green cache. As a user I should be able to configure when cache switch happens.(Automatic / Manual API trigger or based on a criteria - can we use SQL for this ? )

7. A concept of a namespace to organize APIs into groups would be nice. This is common practice for most API frameworks.

8. We probably need to think about multiple YAML files and merging YAML to get the final definition. Resolve the latest definition based on something ?

[chubei]

As a user, I want to control the version in my URL.

This agrees with the desgin.

As a user, I want to overwrite the existing API definition despite breaking changes.

Our API definition is purely based on schema, so when there's breaking change, we'll create a new migration hence a new set of APIs.

"semantic versioning"

Makes sense. We can follow semantic versioning rules in REST APIs by analyzing schemas.

retaining the history of API definitions would be good

Yes we can bring up API of a history version

Schema and API version could differ

We don't really expose the concept for schema version. There's only API version.

be able to configure when cache switch happens

Sure.

can we use SQL for this

Interesting. We certainly can do this with lambda. Also possible with UDF I guess.

A concept of a namespace to organize APIs into groups would be nice

Don't get it.

multiple YAML files and merging YAML to get the final definition

Sure. If you think about it, our configuration yaml can be arbitarily merged as long as names don't clash.

[chubei]

To summarize previous call:

• We should report error when source schema changes and prompt user to migrate.
• We'll diff schemas upon migration, and automatically generate minor/major version update.
• We'll support application aliasing to let Dozer user specify the version that's exposed to API user.
• We'll add namespaces/scopes to endpoints and allow setting api security on the namespace/scope level.
• Some kind of Git server can be used to manage the Dozer workspace and we merge YAML files on deployment.

@v3g42 Am I missing anything?