From 71c8fb4485fbbf9c9a7a767c7d15c491561eeff2 Mon Sep 17 00:00:00 2001 From: Dawadi Kiran Date: Mon, 2 Sep 2024 15:02:17 -0500 Subject: [PATCH 1/3] Fixes issue #993 - Add SQLDumpRestoration.md file --- SQLDumpRestoration.md | 81 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 SQLDumpRestoration.md diff --git a/SQLDumpRestoration.md b/SQLDumpRestoration.md new file mode 100644 index 00000000..3a672a1f --- /dev/null +++ b/SQLDumpRestoration.md @@ -0,0 +1,81 @@ +## Restoring the Database from SQL Dump + +We generally load a database backup from a JSON file by using the following command. + +``` +docker-compose -f local.yml run --rm django python manage.py loaddata backup.json +``` + +However, if the JSON file is particularly large (>1.5GB), Docker might struggle with this method. In such cases, you can use SQL dump and restore commands as an alternative. + +### Steps for Using SQL Dump and Restore + +1. Begin by starting only the PostgreSQL container. This prevents the Django container from making changes while the PostgreSQL container is starting up. + +``` +docker-compose -f local.yml up postgres +``` + +2. Find the container ID using `docker ps`, then enter the PostgreSQL container to execute commands. + +``` +$ docker ps +CONTAINER ID IMAGE COMMAND +23d33f22cc43 sde_indexing_helper_production_postgres "docker-entrypoint.s…" + +$ docker exec -it 23d33f22cc43 bash +``` + +3. Create a connection to the database. + +``` +psql -U VnUvMKBSdkoFIETgLongnxYHrYVJKufn -d sde_indexing_helper +``` + +4. Ensure that the database `sde_indexing_helper` is empty. + +``` +sde_indexing_helper-# \c +You are now connected to database "sde_indexing_helper" as user "VnUvMKBSdk...". +sde_indexing_helper-# \dt +Did not find any relations. +``` + +If the database is not empty, delete its contents to create a fresh database: + +``` +sde_indexing_helper=# \c postgres +You are now connected to database "postgres" as user "VnUvMKBSdkoFIETgLongnxYHrYVJKufn". +postgres=# DROP DATABASE sde_indexing_helper; +DROP DATABASE +postgres=# CREATE DATABASE sde_indexing_helper; +CREATE DATABASE + +``` + +5. Transfer the backup SQL dump (`backup.sql`) from your local machine to the PostgreSQL container. + +``` +docker cp /local/path/backup.sql 23d33f22cc43:/ +``` + +6. Import the SQL dump into the PostgreSQL container. + +``` +psql -U VnUvMKBSdkoFIETgLongnxYHrYVJKufn -d sde_indexing_helper -f backup.sql +``` + +**Note**: To create a SQL dump of your PostgreSQL database, use the following command: + +``` +pg_dump -U VnUvMKBSdkoFIETgLongnxYHrYVJKufn -W -F p -f backup.sql sde_indexing_helper +``` + +7. Bring up all containers at once, and create a superuser account for logging in. + +``` +docker-compose -f local.yml up +docker-compose -f local.yml run --rm django python manage.py createsuperuser +``` + +8. Log in to the SDE Indexing Helper frontend to ensure that all data has been correctly populated in the UI. \ No newline at end of file From 52521e3076eab9e6a3b265a142238a9b4f281f89 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 2 Sep 2024 20:04:58 +0000 Subject: [PATCH 2/3] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- SQLDumpRestoration.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/SQLDumpRestoration.md b/SQLDumpRestoration.md index 3a672a1f..49e78994 100644 --- a/SQLDumpRestoration.md +++ b/SQLDumpRestoration.md @@ -19,11 +19,11 @@ docker-compose -f local.yml up postgres 2. Find the container ID using `docker ps`, then enter the PostgreSQL container to execute commands. ``` -$ docker ps -CONTAINER ID IMAGE COMMAND +$ docker ps +CONTAINER ID IMAGE COMMAND 23d33f22cc43 sde_indexing_helper_production_postgres "docker-entrypoint.s…" -$ docker exec -it 23d33f22cc43 bash +$ docker exec -it 23d33f22cc43 bash ``` 3. Create a connection to the database. @@ -32,7 +32,7 @@ $ docker exec -it 23d33f22cc43 bash psql -U VnUvMKBSdkoFIETgLongnxYHrYVJKufn -d sde_indexing_helper ``` -4. Ensure that the database `sde_indexing_helper` is empty. +4. Ensure that the database `sde_indexing_helper` is empty. ``` sde_indexing_helper-# \c @@ -78,4 +78,4 @@ docker-compose -f local.yml up docker-compose -f local.yml run --rm django python manage.py createsuperuser ``` -8. Log in to the SDE Indexing Helper frontend to ensure that all data has been correctly populated in the UI. \ No newline at end of file +8. Log in to the SDE Indexing Helper frontend to ensure that all data has been correctly populated in the UI. From a7f823a03996d7b119aeb07beca6725c4005851d Mon Sep 17 00:00:00 2001 From: Dawadi Kiran Date: Mon, 2 Sep 2024 21:34:39 -0500 Subject: [PATCH 3/3] Improve SQLDumpRestoration.md and README.md files --- README.md | 7 +++++-- SQLDumpRestoration.md | 16 ++++++++++------ 2 files changed, 15 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 61cf6b50..cc64ef01 100644 --- a/README.md +++ b/README.md @@ -116,6 +116,10 @@ $ docker cp /path/to/your/backup.json container_name:/path/inside/container/back $ docker-compose -f local.yml run --rm django python manage.py loaddata /path/inside/the/container/backup.json $ docker-compose -f local.yml run --rm django python manage.py migrate ``` +### Restoring the Database from a SQL Dump +If the JSON file is particularly large (>1.5GB), Docker might struggle with this method. In such cases, you can use SQL dump and restore commands as an alternative, as described [here](./SQLDumpRestoration.md). + + ## Additional Commands @@ -191,8 +195,7 @@ Documented [here](https://github.com/NASA-IMPACT/sde-indexing-helper/wiki/How-to ## Adding New Features/Fixes -1. Start with a [GitHub issue](https://github.com/NASA-IMPACT/sde-indexing-helper/issues). -2. Use the GitHub CLI to create branches and pull requests (`gh issue develop -c `). +We welcome contributions to improve the project! Before you begin, please take a moment to review our [Contributing Guidelines](./CONTRIBUTING.md). These guidelines will help you understand the process for submitting new features, bug fixes, and other improvements. ## Job Creation diff --git a/SQLDumpRestoration.md b/SQLDumpRestoration.md index 49e78994..6b4792be 100644 --- a/SQLDumpRestoration.md +++ b/SQLDumpRestoration.md @@ -29,10 +29,14 @@ $ docker exec -it 23d33f22cc43 bash 3. Create a connection to the database. ``` -psql -U VnUvMKBSdkoFIETgLongnxYHrYVJKufn -d sde_indexing_helper +psql -U -d ``` -4. Ensure that the database `sde_indexing_helper` is empty. +**Note**: +- For local deployment, refer to the `.envs/.local/.postgres` file for the `POSTGRES_USER` and `POSTGRES_DB` variables. +- For production deployment, refer to the `.envs/.production/.postgres` file. + +4. Ensure that the database `` is empty. Here's an example: ``` sde_indexing_helper-# \c @@ -44,8 +48,8 @@ Did not find any relations. If the database is not empty, delete its contents to create a fresh database: ``` -sde_indexing_helper=# \c postgres -You are now connected to database "postgres" as user "VnUvMKBSdkoFIETgLongnxYHrYVJKufn". +sde_indexing_helper=# \c postgres //connect to a different database before dropping +You are now connected to database "postgres" as user "VnUvMKBSdk....". postgres=# DROP DATABASE sde_indexing_helper; DROP DATABASE postgres=# CREATE DATABASE sde_indexing_helper; @@ -62,13 +66,13 @@ docker cp /local/path/backup.sql 23d33f22cc43:/ 6. Import the SQL dump into the PostgreSQL container. ``` -psql -U VnUvMKBSdkoFIETgLongnxYHrYVJKufn -d sde_indexing_helper -f backup.sql +psql -U -d -f backup.sql ``` **Note**: To create a SQL dump of your PostgreSQL database, use the following command: ``` -pg_dump -U VnUvMKBSdkoFIETgLongnxYHrYVJKufn -W -F p -f backup.sql sde_indexing_helper +pg_dump -U -W -F p -f backup.sql ``` 7. Bring up all containers at once, and create a superuser account for logging in.