-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
chore: documentation and startup scripts (#71)
* fix: remove rethrow * chore: add init.db for docker startup * Add docs for postgres * chore: add env var docs * chore: add troubleshooting guide, add monitorStateIntervalInSeconds * chore: switch to using /notarial db * chore: add dotenv and make notify api key non-required docker compose env var * fix: URLs and refer to jobs on this project
- Loading branch information
1 parent
6872136
commit 29c485b
Showing
15 changed files
with
391 additions
and
75 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,151 @@ | ||
| # Troubleshooting | ||
|
|
||
| Use this guide to troubleshoot issues and resolve errors that may occur when notarial-api or notarial-worker is deployed. | ||
|
|
||
| Connect to the database: | ||
| ```sh | ||
| kubectl run -it --rm --env PGPASSWORD='<PASSWORD>' --env PAGER= --image=postgres:16 --restart=Never postgres-client -- psql -h <ENDPOINT_URL> -U master -d notarial | ||
| ``` | ||
|
|
||
| Replace PASSWORD with the password for the database, ENDPOINT_URL with the endpoint URL for the database. | ||
|
|
||
| ## pgboss | ||
|
|
||
| [pgboss](https://github.com/timgit/pg-boss) is used to manage queueing jobs. On application start, pgboss will automatically create necessary tables in the database. | ||
|
|
||
| ### Jobs table | ||
| The jobs table `pgboss.job` is where all the current jobs are stored. Jobs will remain here, until they are completed or failed. Then they will move to `pgboss.archive` | ||
|
|
||
| The jobs table has the following columns: | ||
|
|
||
| ``` | ||
| Column | Type | Collation | Nullable | Default | ||
| --------------+-----------------------------+-----------+----------+----------------------------- | ||
| id | uuid | | not null | gen_random_uuid() | ||
| name | text | | not null | | ||
| priority | integer | | not null | 0 | ||
| data | jsonb | | | | ||
| state | pgboss.job_state | | not null | 'created'::pgboss.job_state | ||
| retrylimit | integer | | not null | 0 | ||
| retrycount | integer | | not null | 0 | ||
| retrydelay | integer | | not null | 0 | ||
| retrybackoff | boolean | | not null | false | ||
| startafter | timestamp with time zone | | not null | now() | ||
| startedon | timestamp with time zone | | | | ||
| singletonkey | text | | | | ||
| singletonon | timestamp without time zone | | | | ||
| expirein | interval | | not null | '00:15:00'::interval | ||
| createdon | timestamp with time zone | | not null | now() | ||
| completedon | timestamp with time zone | | | | ||
| keepuntil | timestamp with time zone | | not null | now() + '14 days'::interval | ||
| on_complete | boolean | | not null | false | ||
| output | jsonb | | | | ||
| ``` | ||
|
|
||
| Columns/values to note are | ||
| - `name`: the name of the job. It can be one of SES_PROCESS, NOTIFY_PROCESS, SES_SEND, NOTIFY_SEND. More detail can be found in [worker/src/README.md](worker/src/README.md). PgBoss will also create some of its own. | ||
| - `state`: the state of the job. Read more about them in [pgboss documentation](https://github.com/timgit/pg-boss/blob/master/docs/readme.md#job-states) | ||
| - `created`: the job has been created | ||
| - `failed`: the job has failed | ||
| - `completed`: the job has been completed (successfully) | ||
| - `active`: the job is currently being processed | ||
| - `data`: the data associated with the job. | ||
| - `output`: the output of the job. This will contain the reference number, or the error message if the job has failed | ||
| - `keepuntil`: the time until the job will be kept in the table. After this time, the job will be moved to `pgboss.archive`. If you need more time to resolve the issue, you can update this value to a later time. | ||
|
|
||
|
|
||
|
|
||
| ## Finding jobs | ||
| To find jobs that have failed, run the following query: | ||
|
|
||
| ```postgresql | ||
| select id, output from pgboss.job where state = 'failed'; | ||
| ``` | ||
|
|
||
| ## Fixing data | ||
| If the retrylimit has not been hit (retrylimit > retrycount) and the retrylimit is not 0, the job will be automatically retried. | ||
|
|
||
| It is recommended you run every query in a transaction, so that you can abort the changes if they are incorrect. | ||
|
|
||
| ```postgresql | ||
| begin; | ||
| -- First run a query to print the current state of the job you are trying to change | ||
| select data from pgboss.job where id = '<id>'; | ||
| update pgboss.job | ||
| set state = 'created', | ||
| completedon = null, | ||
| retrycount = 0, | ||
| state = 'created' | ||
| where id = '<id>'; | ||
| -- Run the query again, to see if you've made the correct changes | ||
| select data from pgboss.job where state = 'failed' and id = '<id>'; | ||
| -- Run the following query to commit the changes | ||
| -- commit; | ||
| -- Run the following to abort the changes | ||
| -- rollback; | ||
| ``` | ||
|
|
||
| The following queries will assume that you are running them in a transaction. | ||
|
|
||
| ## Incorrect data | ||
| If the data is incorrect, you can update the data in the database. All data is stored as jsonb, so you can use [postgresql's jsonb functions](https://www.postgresql.org/docs/current/functions-json.html) to update the data | ||
|
|
||
| ```postgresql | ||
| update pgboss.job | ||
| set data = jsonb_set( | ||
| data, | ||
| '{data, keyToChange}', | ||
| '"<NEW_ANSWER>"' | ||
| ) | ||
| where id = '<id>'; | ||
| ``` | ||
|
|
||
| You may find it easier to copy the data to a text editor, make the changes, and then update the data in the database. | ||
|
|
||
| ```postgresql | ||
| update pgboss.job | ||
| set data = '<NEW_DATA>' | ||
| where id = '<id>'; | ||
| ``` | ||
|
|
||
| ### Retry a job | ||
| If a job has failed, and you want to retry it, you can update the job to `created` state, and reset the `retrycount` to 0. | ||
|
|
||
| ```postgresql | ||
| update pgboss.job | ||
| set state = 'created', | ||
| completedon = null, | ||
| retrycount = 0, | ||
| state = 'created' | ||
| -- output = null | ||
| where id = '<id>'; | ||
| ``` | ||
| You may also want to update output to null, to clear the error message. | ||
|
|
||
| ## Creating a new job | ||
| If the job does not seem to be retrying, or it is easier to just create a new job you need to create a new job, you can do so by running the following query: | ||
|
|
||
| ```postgresql | ||
| insert into pgboss.job (name, data) | ||
| values ('NOTIFY_PROCESS', '{"answers": {}, "metadata": {}, "reference": "123456"}'); | ||
| ``` | ||
|
|
||
| Alternatively, you can copy the data from the failed job, and create a new job with the same data. | ||
|
|
||
| ```postgresql | ||
| insert into pgboss.job (name, data) | ||
| SELECT name, data | ||
| from pgboss.job where id = '<id>'; | ||
| ``` | ||
|
|
||
| ### Moving a job from archive to job | ||
| If a job has been moved to the archive, and you want to retry it, you can move it back to the jobs table. | ||
|
|
||
| ```postgresql | ||
| insert into pgboss.job (name, data) | ||
| SELECT name, data | ||
| from pgboss.archive where id = '<id>'; | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,11 +7,10 @@ if (process.env.NODE_ENV !== "test") { | |
| module.exports = { | ||
| port: "9000", | ||
| env: "development", | ||
| documentPassword: "Sup3rS3cr3tP4ssw0rd", | ||
| submissionAddress: "[email protected]", | ||
| senderEmail: "[email protected]", | ||
| Queue: { | ||
| url: "postgresql://user:root@localhost:5432/queue", | ||
| url: "postgresql://user:root@localhost:5432/notarial", | ||
| defaultOptions: { | ||
| retryBackoff: "true", | ||
| retryLimit: "50", | ||
|
|
||
Oops, something went wrong.