Skip to content

Commit

Permalink
Issue #14: Update index node assignments (#20)
Browse files Browse the repository at this point in the history
Motivation
----------
Support changes to index node assignments after an index is created.

Modifications
-------------
Implemented a phase system for mutations so that different replicas can
be moved separately to prevent downtime.

Fixed a flaw in the logic using the /indexStatus API to retrieve node
assignments for automatic replicas.

Created MoveIndexMutation to handle ALTER INDEX statements on 5.5, with
support in IndexManager.

Created a FeatureVersion class to help manage Couchbase versions where
specific features are supported.

Return proper mutations for changes to nodes and num_replicas, in both
the manual and automatic index mutation forms.

Change UpdateIndexMutation to recognize when it's actually a safe
mutation based on replicas being present.

Results
-------
Automatic replica scaling is supported on CB >= 5.0 as an unsafe
operation.

Automatic replica moving is supported on CB >= 5.5 as a safe operation,
moves are skipped with a warning on 5.0 and 5.1.

Manual replica scaling and moving is supported, and is safe so long as
there is at least one replica.

Other update operations, such as index_key and condition, are now safe
operations for any index with at least one replica and running in manual
replica mode.
  • Loading branch information
brantburnett authored Apr 12, 2018
1 parent efdbea0 commit e438845
Show file tree
Hide file tree
Showing 14 changed files with 1,009 additions and 55 deletions.
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,15 +160,15 @@ Replicas are emulated on Couchbase Server 4.X by creating multiple indexes. If

This approach may also be enabled on Couchbase Server 5.X by settings `manual_replica` to true on the index definition.

Note that the `nodes` list is only respected during index creation, indexes will not be moved between nodes if they already exist.
During updates, changes to the `nodes` list may result in indexes being moved from one node to another. So long as there is at least one replica this is considered a safe operation, as each replica will be moved independently leaving other replicas available to serve requests. Changes to the order of the node list are ignored, only adding or removing nodes results in a change.

## Automatic Index Replica Management

On Couchbase Server 5.X, automatic index replica managemnet is the default. In this case, replicas are managed by Couchbase Server directly, rather than by couchbase-index-manager.
On Couchbase Server 5.X, automatic index replica management is the default. In this case, replicas are managed by Couchbase Server directly, rather than by couchbase-index-manager.

Currently, it isn't possible to detect replicas via queries to "system:indexes". Therefore, `num_replica` is only respected during index creation. Changes to `num_replica` on existing indexes will be ignored.
Note that for Couchbase Server 5.0 and 5.1, the `nodes` list is only respected during index creation. Indexes will not be moved between nodes if they already exist. Beginning with Couchbase Server 5.5 an ALTER INDEX command will be used to move replicas between nodes.

Note that the `nodes` list is only respected during index creation, indexes will not be moved between nodes if they already exist.
Because ALTER INDEX cannot currently change the number of replicas, changes to `num_replica` or the number of nodes in `nodes` is an unsafe change that will drop and recreate the index.

## Docker Image

Expand Down
2 changes: 1 addition & 1 deletion app/cli.js
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ function parseBaseOptions(cmd) {
*/
function handleAsync(promise) {
promise.catch((err) => {
console.error(chalk.redBright(err));
console.error(chalk.redBright(err.stack));

process.exit(1);
});
Expand Down
33 changes: 33 additions & 0 deletions app/feature-versions.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
/**
* @typedef Version
* @property {number} major
* @property {number} minor
*/

/**
* Tests for compatibility with various features,
* given a cluster version from clusterCompatibility.
*/
export class FeatureVersions {
/**
* Tests for ALTER INDEX compatibility
*
* @param {Version} version
* @return {boolean}
*/
static alterIndex(version) {
return version &&
(version.major > 5 ||
(version.major == 5 && version.minor >= 5));
}

/**
* Tests for automatic replica compatibility
*
* @param {Version} version
* @return {boolean}
*/
static autoReplicas(version) {
return version && version.major >= 5;
}
}
159 changes: 136 additions & 23 deletions app/index-definition.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ import {IndexDefinitionBase} from './index-definition-base';
import {CreateIndexMutation} from './create-index-mutation';
import {UpdateIndexMutation} from './update-index-mutation';
import {DropIndexMutation} from './drop-index-mutation';
import {MoveIndexMutation} from './move-index-mutation';
import {FeatureVersions} from './feature-versions';

/**
* @typedef LifecycleHash
Expand Down Expand Up @@ -46,7 +48,7 @@ import {DropIndexMutation} from './drop-index-mutation';
/**
* @typedef MutationContext
* @property {array.CouchbaseIndex} currentIndexes
* @property {{major: number, minor: number}} clusterVersion
* @property {?{major: number, minor: number}} clusterVersion
*/

/**
Expand Down Expand Up @@ -230,31 +232,29 @@ export class IndexDefinition extends IndexDefinitionBase {
* @yields {IndexMutation}
*/
* getMutations(context) {
this.normalizeNodeList(context.currentIndexes);

let mutations = [];

if (!this.manual_replica) {
let mutation = this.getMutation(context);
if (mutation) {
yield mutation;
}
mutations.push(...this.getMutation(context));
} else {
for (let i=0; i<=this.num_replica; i++) {
let mutation = this.getMutation(context, i);
if (mutation) {
yield mutation;
}
mutations.push(...this.getMutation(context, i));
}

if (!this.is_primary) {
// Handle dropping replicas if the count is lowered
for (let i=this.num_replica+1; i<=10; i++) {
let mutation = this.getMutation(
context, i, true);

if (mutation) {
yield mutation;
}
mutations.push(...this.getMutation(
context, i, true));
}
}
}

IndexDefinition.phaseMutations(mutations);

yield* mutations;
}

/**
Expand All @@ -263,9 +263,9 @@ export class IndexDefinition extends IndexDefinitionBase {
* @param {?number} replicaNum
* @param {?boolean} forceDrop Always drop, even if lifecycle.drop = false.
* Used for replicas.
* @return {?IndexMutation}
* @yields {IndexMutation}
*/
getMutation(context, replicaNum, forceDrop) {
* getMutation(context, replicaNum, forceDrop) {
let suffix = !replicaNum ?
'' :
`_replica${replicaNum}`;
Expand All @@ -279,18 +279,40 @@ export class IndexDefinition extends IndexDefinitionBase {
if (!currentIndex) {
// Index isn't found
if (!drop) {
return new CreateIndexMutation(this, this.name + suffix,
yield new CreateIndexMutation(this, this.name + suffix,
this.getWithClause(replicaNum));
}
} else if (drop) {
return new DropIndexMutation(this, currentIndex.name);
yield new DropIndexMutation(this, currentIndex.name);
} else if (!this.is_primary && this.requiresUpdate(currentIndex)) {
return new UpdateIndexMutation(this, this.name + suffix,
yield new UpdateIndexMutation(this, this.name + suffix,
this.getWithClause(replicaNum),
currentIndex);
}
} else if (!this.manual_replica && currentIndex.num_replica &&
this.num_replica !== currentIndex.num_replica) {
// Number of replicas changed for an auto replica index
// We must drop and recreate

return undefined;
yield new UpdateIndexMutation(this, this.name + suffix,
this.getWithClause(replicaNum),
currentIndex);
} else if (this.nodes && currentIndex.nodes) {
// Check for required node changes
currentIndex.nodes.sort();

if (this.manual_replica) {
if (this.nodes[replicaNum] !== currentIndex.nodes[0]) {
yield new UpdateIndexMutation(this, this.name + suffix,
this.getWithClause(replicaNum),
currentIndex);
}
} else {
if (!_.isEqual(this.nodes, currentIndex.nodes)) {
yield new MoveIndexMutation(this, this.name + suffix,
!FeatureVersions.alterIndex(context.clusterVersion));
}
}
}
}

/**
Expand Down Expand Up @@ -332,7 +354,8 @@ export class IndexDefinition extends IndexDefinitionBase {

/**
* @private
* Tests to see if a Couchbase index requires updating
* Tests to see if a Couchbase index requires updating,
* ignoring node changes which are handled separately.
*
* @param {CouchbaseIndex} index
* @return {boolean}
Expand Down Expand Up @@ -400,4 +423,94 @@ export class IndexDefinition extends IndexDefinitionBase {

return condition.replace(/'([^']*)'/g, '"$1"');
}

/**
* @private
* Apply phases to the collection of index mutations
*
* @param {array.IndexMutation} mutations
*/
static phaseMutations(mutations) {
// All creates should be in phase one
// All updates should be in one phase each, after creates
// Everything else should be in the last phase
// This is relative to each index definition only

let nextPhase = 1;
for (let mutation of mutations) {
if (mutation instanceof CreateIndexMutation) {
nextPhase = 2;
mutation.phase = 1;
}
}

for (let mutation of mutations) {
if (mutation instanceof UpdateIndexMutation) {
mutation.phase = nextPhase;
nextPhase += 1;
}
}

for (let mutation of mutations) {
if (!(mutation instanceof CreateIndexMutation)
&& !(mutation instanceof UpdateIndexMutation)) {
mutation.phase = nextPhase;
}
}
}

/**
* @private
* Ensures that the node list has port numbers and is sorted in the same
* order as the current indexes. This allows easy matching of existing
* node assignments, reducing reindex load due to minor node shifts.
*
* @param {array.CouchbaseIndex} currentIndexes
*/
normalizeNodeList(currentIndexes) {
if (!this.nodes) {
return;
}

this.nodes = this.nodes.map(ensurePort);
this.nodes.sort();

if (this.manual_replica) {
// We only care about specific node mappings for manual replicas
// For auto replicas we let Couchbase handle it

let newNodeList = [];
let unused = _.clone(this.nodes);

for (let replicaNum=0; replicaNum<=this.num_replica; replicaNum++) {
let suffix = !replicaNum ?
'' :
`_replica${replicaNum}`;

let index = currentIndexes.find((index) => {
return this.isMatch(index, suffix);
});

if (index && index.nodes) {
let unusedIndex = unused.findIndex(
(name) => name === index.nodes[0]);

if (unusedIndex >= 0) {
newNodeList[replicaNum] =
unused.splice(unusedIndex, 1)[0];
}
}
}

// Fill in the remaining nodes that didn't have a match
for (let replicaNum=0; replicaNum<=this.num_replica; replicaNum++) {
if (!newNodeList[replicaNum]) {
newNodeList[replicaNum] =
unused.shift();
}
}

this.nodes = newNodeList;
}
}
}
19 changes: 15 additions & 4 deletions app/index-manager.js
Original file line number Diff line number Diff line change
Expand Up @@ -126,10 +126,19 @@ export class IndexManager {

// Apply hosts from index status API to index information
statuses.forEach((status) => {
let index = indexes.find((index) => index.name === status.index);
// remove (replica X) from the end of the index name
let indexName = /^([^\s]*)/.exec(status.index)[1];

let index = indexes.find((index) => index.name === indexName);
if (index) {
index.nodes = status.hosts;
index.num_replica = status.hosts.length - 1;
if (!index.nodes) {
index.nodes = [];
}

// add any hosts listed to index info
index.nodes.push(...status.hosts);

index.num_replica = index.nodes.length - 1;
}
});

Expand Down Expand Up @@ -161,7 +170,9 @@ export class IndexManager {
*/
moveIndex(indexName, nodes) {
return new Promise((resolve, reject) => {
let statement = `ALTER INDEX \`${indexName}\` WITH `;
let statement = 'ALTER INDEX ' +
`\`${this.bucketName}\`.\`${indexName}\`` +
' WITH ';
statement += JSON.stringify({
action: 'move',
nodes: nodes,
Expand Down
2 changes: 2 additions & 0 deletions app/index-mutation.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
* Abstract base class for index mutations
* @abstract
*
* @property {number} phase Phase for this mutation, default = 1
* @private @property {IndexDefinition} definition
* @private @property {string} name Name of the index to mutate,
* may be different than the name in the definition
Expand All @@ -14,6 +15,7 @@ export class IndexMutation {
constructor(definition, name) {
this.definition = definition;
this.name = name || definition.name;
this.phase = 1;
}

/**
Expand Down
57 changes: 57 additions & 0 deletions app/move-index-mutation.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
import {IndexMutation} from './index-mutation';
import chalk from 'chalk';

/**
* @typedef CouchbaseIndex
* @property {string} name
* @property {array.string} index_key
* @property {?string} condition
*/

/**
* Represents an index mutation which updates an existing index
*/
export class MoveIndexMutation extends IndexMutation {
/**
* @param {IndexDefinition} definition Index definition
* @param {string} name Name of the index to mutate
* @param {boolean} unsupported
* If true, don't actually perform this mutation
*/
constructor(definition, name, unsupported) {
super(definition, name);

this.unsupported = unsupported;
}

/** @inheritDoc */
print(logger) {
const color = this.unsupported ?
chalk.yellowBright :
chalk.cyanBright;

logger.info(color(
` Move: ${this.name}`));

logger.info(color(
` Nodes: ${this.definition.nodes.join()}`));

if (this.unsupported) {
logger.info(color(
` Skip: ALTER INDEX is not supported until CB 5.5`
));
}
}

/** @inheritDoc */
async execute(manager, logger) {
if (!this.unsupported) {
await manager.moveIndex(this.name, this.definition.nodes);
}
}

/** @inheritDoc */
isSafe() {
return true;
}
}
Loading

0 comments on commit e438845

Please sign in to comment.