🗃️ Indexer

Service for indexing RapidPro/TextIt contacts into Elasticsearch.

Deploying

As it is a Go application, it compiles to a binary and that binary along with the config file is all you need to run it on your server. You can find bundles for each platform in the releases directory. You should only run a single instance for a deployment.

It can run in two modes:

1. the default mode, which simply queries the ElasticSearch database, finds the most recently modified contact, then on a schedule queries the contacts_contact table in the database for contacts to add or delete. You should run this as a long running service which constantly keeps ElasticSearch in sync with your contacts.

2. a rebuild mode, started with --rebuild. This builds a brand new index from nothing, querying all contacts on RapidPro. Once complete, this switches out the alias for the contact index with the newly build index. This can be run on a cron (in parallel with the mode above) to rebuild your index occasionally to get rid of bloat.

Configuration

The service uses a tiered configuration system, each option takes precendence over the ones above it:

1. The configuration file
2. Environment variables starting with INDEXER_
3. Command line parameters

We recommend running it with no changes to the configuration and no parameters, using only environment variables to configure it. You can use % rp-indexer --help to see a list of the environment variables and parameters and for more details on each option.

• INDEXER_DB: a URL connection string for your RapidPro database or read replica
• INDEXER_ELASTIC_URL: the URL for your ElasticSearch endpoint

AWS services:

• INDEXER_AWS_ACCESS_KEY_ID: AWS access key id used to authenticate to AWS
• INDEXER_AWS_SECRET_ACCESS_KEY: AWS secret access key used to authenticate to AWS
• INDEXER_AWS_REGION: AWS region (ex: eu-west-1)

Logging and error reporting:

• INDEXER_DEPLOYMENT_ID: used for metrics reporting
• INDEXER_SENTRY_DSN: DSN to use when logging errors to Sentry
• INDEXER_LOG_LEVEL: logging level to use (default is info)

Development

Once you've checked out the code, you can build the service with:

go build github.com/nyaruka/rp-indexer/cmd/rp-indexer

This will create a new executable in $GOPATH/bin called rp-indexer.

To run the tests you need to create the test database:

$ createdb elastic_test

To run all of the tests:

go test ./... -p=1