Skip to content

Commit

Permalink
not enough ticks
Browse files Browse the repository at this point in the history
  • Loading branch information
@Vebop
Vebop committed Nov 8, 2024
1 parent b254121 commit ce4d251
Show file tree
Hide file tree
Showing 2 changed files with 30 additions and 30 deletions.
4 changes: 2 additions & 2 deletions doc/operator-guide/monitoring.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@ Monitoring

* Reporting channels

- Users of ConsDB, ConsDBClient (`pqserver`) will usually report via #consolidated-database in rubin-obs.slack.com when they are having issues.
- Users of ConsDB, ConsDBClient (``pqserver``) will usually report via #consolidated-database in rubin-obs.slack.com when they are having issues.
- ConsDB operators should monitor this channel and #ops-usdf, #ops-usdf-alerts for issues and outages reported, as well as escalate verified database issues.

* Database

- The ConsDB team is responsible for verifying whether or not the database is up when issues are reported
- They can check the method reported by the users, check using `psql`/ `pgcli`, and check in the #ops-usdf slack channel for currently reported issues.
- They can check the method reported by the users, check using ``psql``/ ``pgcli``, and check in the #ops-usdf slack channel for currently reported issues.

- Once the ConsDB team has confirmed there is an issue with the database, they should notify #ops-usdf slack channel and USDF DBAs should be responsible for fixing/restarting.

Expand Down
56 changes: 28 additions & 28 deletions doc/operator-guide/schema-migration-process.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ Schema Migration Process

* How to add columns to sdm_schemas

- Edit the repository at https://sdm-schemas.lsst.io/ to apply your schema changes to any of the `cdb_*.yml` schemas.
- Check that valid SQL tables can be created with `felis` (https://felis.lsst.io/user-guide/cli.html#felis-validate) using:
- Edit the repository at https://sdm-schemas.lsst.io/ to apply your schema changes to any of the ``cdb_*.yml`` schemas.
- Check that valid SQL tables can be created with ``felis`` (https://felis.lsst.io/user-guide/cli.html#felis-validate) using:

```
pip install lsst-felis
Expand All @@ -19,58 +19,58 @@ Schema Migration Process
- Alembic (https://alembic.sqlalchemy.org/en/latest/front.html) keeps track of versioning by autogenerated migrations to sync the test stands and summit databases.
- Versioning our database schema changes allows us to apply edits and move the database’s state forward or backward as needed.

- How to create an Alembic migration to deploy your new `sdm_schema` edits:
- How to create an Alembic migration to deploy your new ``sdm_schema`` edits:

1. Create an Alembic migration on your `consdb` ticket branch.
2. Use the script `consdb/alembic-autogenerate.py` to generate Alembic migrations. You’ll need the following environment variables:
1. Create an Alembic migration on your ``consdb`` ticket branch.
2. Use the script ``consdb/alembic-autogenerate.py`` to generate Alembic migrations. You’ll need the following environment variables:

- `SDM_SCHEMAS_DIR` - Points to a local clone of `sdm_schemas` with your `cdb_*` schema changes.
- `POSTGRES_URL` - Defines database URL including authentication, e.g., `"postgresql://DB_NAME:[email protected]/DB_NAME"`.
- ``SDM_SCHEMAS_DIR`` - Points to a local clone of ``sdm_schemas`` with your ``cdb_*`` schema changes.
- ``POSTGRES_URL`` - Defines database URL including authentication, e.g., ``"postgresql://DB_NAME:[email protected]/DB_NAME"``.
.. note: or 'postgresdb01.ls.lsst.org' or something else depending on the location of the db you're trying to access
3. Run `alembic-autogenerate` to create version files in respective database-named directories in `consdb`.
3. Run ``alembic-autogenerate`` to create version files in respective database-named directories in ``consdb``.

- Follow the directions in the header of the script.

4. Manually edit the generated files in `consdb/alembic/<table-name>/` to:
4. Manually edit the generated files in ``consdb/alembic/<table-name>/`` to:

- Remove the `visit1` and `ccdvisit1` views.
- Remove the ``visit1`` and ``ccdvisit1`` views.
- Ensure constraints and renamed columns are correct.

* Test alembic migration and code to populate the new columns/tables at TTS/BTS if Summit schema is changing

You will need to rest the migration on your `consdb` branch on TTS before merging or applying the migration to the Summit.
You will need to rest the migration on your ``consdb`` branch on TTS before merging or applying the migration to the Summit.

1. Update the deployment on the test stand:

Choose the appropriate test stand (TTS, BTS)
Create a branch in `phalanx` and edit the corresponding test stand environment file `phalanx/applications/consdb/values-<test stand>.yaml` to point to your branch.
Create a branch in ``phalanx`` and edit the corresponding test stand environment file ``phalanx/applications/consdb/values-<test stand>.yaml`` to point to your branch.
Coordinate and announce in the appropriate slack channel that you will begin testing your migrations.
Update the consdb deployment in `<url.to.teststand>/argo-cd` to use your `phalanx` branch in the `Target Revision`. Refresh and check pod logs.
Verify the tables that you will be upgrading exist using `psql`
From the `consdb/` directory, (where `alembic.ini` file is) use the alembic commands to upgrade the existing database tables: `alembic upgrade head -n <database name>`
Deploy new consdb software (`hinfo`, `pqserver`) and check the initial logs.
Update the consdb deployment in ``<url.to.teststand>/argo-cd`` to use your ``phalanx`` branch in the ``Target Revision``. Refresh and check pod logs.
Verify the tables that you will be upgrading exist using ``psql``
From the ``consdb/`` directory, (where ``alembic.ini`` file is) use the alembic commands to upgrade the existing database tables: ``alembic upgrade head -n <database name>``
Deploy new consdb software (``hinfo``, ``pqserver``) and check the initial logs.

2. Test with LATISS imaging in ATQueue:

- Access LOVE via `<url.to.teststand>/love` and use the 1Password admin information to sign in, or your SLAC username and password.
- Access LOVE via ``<url.to.teststand>/love`` and use the 1Password admin information to sign in, or your SLAC username and password.
- Navigate to the ATQueue or Auxillary Telescope (AuxTel) Script Queue.
- See (TTS Start Guide)[https://rubinobs.atlassian.net/wiki/spaces/LSSTCOM/pages/53739987/Tucson+Test+Stand+Start+Guide] for guidelines on using the test stands.
- Before editing these scripts, note their starting configurations, as we will return the configuration to that when we are done.
- Take a test/simulated picture with LATISS through the ATQueue using these three scripts:

1. `set_summary_state.py` Change the configuration to enable ATHeaderService and ATCamera.
2. `enable_latiss.py` Remove any existing configuration.
3. `take_image_latiss.py` Update the configuration to remove anything that is not 'nimages' (1) and 'image_type' (ENGTEST)
1. ``set_summary_state.py`` Change the configuration to enable ATHeaderService and ATCamera.
2. ``enable_latiss.py`` Remove any existing configuration.
3. ``take_image_latiss.py`` Update the configuration to remove anything that is not 'nimages' (1) and 'image_type' (ENGTEST)

- Once you have put these three scripts in the queue, click `run`.
- Watch for errors in the Script Queue and the `argo-cd` `consdb` pod logs and `hinfo-latiss` deployment.
- Once you have put these three scripts in the queue, click ``run``.
- Watch for errors in the Script Queue and the ``argo-cd`` ``consdb`` pod logs and ``hinfo-latiss`` deployment.
- Address any errors and retest.
- Check the database by using `psql` commands like `\dt` to display the table names and maybe even `SELECT * from cdb_latiss.exposure where day_obs == <YYYYMMDD>;` to view the most recent data.
- If you have encountered errors in this process, do not proceed to the summit, but address those errors and retest them with your `phalanx` branch pointing to your `consdb` branch with the updates that fix the errors.
- Check the database by using ``psql`` commands like ``\dt`` to display the table names and maybe even ``SELECT * from cdb_latiss.exposure where day_obs == <YYYYMMDD>;`` to view the most recent data.
- If you have encountered errors in this process, do not proceed to the summit, but address those errors and retest them with your ``phalanx`` branch pointing to your ``consdb`` branch with the updates that fix the errors.


- If tests are successful, create a pull request for the Alembic migration in `consdb`. Tag the release according to `standards-practices` guidelines.
- If tests are successful, create a pull request for the Alembic migration in ``consdb``. Tag the release according to ``standards-practices`` guidelines.
- Update your existing phalanx branch to point the environment based deployments to this consdb tag. You are able to retest on the test stand at this point, hopefully there were no changes to your consdb pull request so this step is trivial.

* Deploy migration in synchrony at Summit (if necessary), USDF, and Prompt Release (if necessary)
Expand All @@ -93,10 +93,10 @@ Schema Migration Process

* Summit Deployment Steps

1. Use a branch in `phalanx` to point to the `consdb` tag for deployment.
2. Set the `argo-cd` application `consdb's` target revision to your `phalanx` branch.
1. Use a branch in ``phalanx`` to point to the ``consdb`` tag for deployment.
2. Set the ``argo-cd`` application ``consdb's`` target revision to your ``phalanx`` branch.
3. Refresh the consdb application and review pod logs.
4. Have an image taken with the observing team, then verify database entries with a SQL query or Jupyter notebook.
5. Check your new entries in the database using a jupyter notebook or SQL query in RSP showing your new image has been inserted to the database as expected.

- Once deployment succeeds, set the `Target Revision` in `argo-cd` back to `main` and complete the `phalanx` PR for the tested `consdb` tag.
- Once deployment succeeds, set the ``Target Revision`` in ``argo-cd`` back to ``main`` and complete the ``phalanx`` PR for the tested ``consdb`` tag.

0 comments on commit ce4d251

Please sign in to comment.