diff --git a/README.md b/README.md index c69ac9fc..ede13f22 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,6 @@ $ docker-compose -f local.yml build ```bash $ docker-compose -f local.yml up ``` - ### Non-Docker Local Setup If you prefer to run the project without Docker, follow these steps: @@ -69,12 +68,23 @@ $ docker-compose -f local.yml run --rm django python manage.py createsuperuser #### Creating Additional Users Create additional users through the admin interface (/admin). +## Database Backup and Restore + +COSMOS provides dedicated management commands for backing up and restoring your PostgreSQL database. These commands handle both compressed and uncompressed backups and work seamlessly in both local and production environments using Docker. + +### Backup Directory Structure -### Database Backup and Restore +All backups are stored in the `/backups` directory at the root of your project. This directory is mounted as a volume in both local and production Docker configurations, making it easy to manage backups across different environments. -COSMOS provides dedicated management commands for backing up and restoring your PostgreSQL database. These commands handle both compressed and uncompressed backups and automatically detect your server environment from your configuration. +- Local development: `./backups/` +- Production server: `/path/to/project/backups/` -#### Creating a Database Backup +If the directory doesn't exist, create it: +```bash +mkdir backups +``` + +### Creating a Database Backup To create a backup of your database: @@ -85,23 +95,24 @@ docker-compose -f local.yml run --rm django python manage.py database_backup # Create an uncompressed backup docker-compose -f local.yml run --rm django python manage.py database_backup --no-compress -# Specify custom output location -docker-compose -f local.yml run --rm django python manage.py database_backup --output /path/to/output.sql +# Specify custom output location within backups directory +docker-compose -f local.yml run --rm django python manage.py database_backup --output my_custom_backup.sql ``` The backup command will automatically: - Detect your server environment (Production/Staging/Local) - Use database credentials from your environment settings - Generate a dated filename if no output path is specified +- Save the backup to the mounted `/backups` directory - Compress the backup by default (can be disabled with --no-compress) -#### Restoring from a Database Backup +### Restoring from a Database Backup -To restore your database from a backup: +To restore your database from a backup, it will need to be in the `/backups` directory. You can then run the following command: ```bash # Restore from a backup (handles both .sql and .sql.gz files) -docker-compose -f local.yml run --rm django python manage.py database_restore path/to/backup.sql[.gz] +docker-compose -f local.yml run --rm django python manage.py database_restore backups/backup_file_name.sql.gz ``` The restore command will: @@ -111,7 +122,7 @@ The restore command will: - Restore all data from the backup - Handle all database credentials from your environment settings -#### Working with Remote Servers +### Working with Remote Servers When working with production or staging servers: @@ -120,44 +131,37 @@ When working with production or staging servers: # For production ssh user@production-server cd /path/to/project - -# For staging -ssh user@staging-server -cd /path/to/project ``` -2. Then run the backup command with the production configuration: +2. Create a backup on the remote server: ```bash docker-compose -f production.yml run --rm django python manage.py database_backup ``` -3. Copy the backup to your local machine: +3. Copy the backup from the remote server's backup directory to your local machine: ```bash -scp user@remote-server:/path/to/backup.sql.gz ./local-backup.sql.gz +scp user@remote-server:/path/to/project/backups/backup_name.sql.gz ./backups/ ``` -4. Finally, restore locally: +4. Restore locally: ```bash -docker-compose -f local.yml run --rm django python manage.py database_restore local-backup.sql.gz +docker-compose -f local.yml run --rm django python manage.py database_restore backups/backup_name.sql.gz ``` -#### Alternative Methods - -While the database_backup and database_restore commands are the recommended approach, there are alternative methods available: +### Alternative Methods -##### Using JSON Fixtures (for smaller datasets) -If you're working with a smaller dataset, you can use Django's built-in fixtures: +While the database_backup and database_restore commands are the recommended approach, you can also use Django's built-in fixtures for smaller datasets: ```bash # Create a backup excluding content types -docker-compose -f production.yml run --rm --user root django python manage.py dumpdata \ +docker-compose -f production.yml run --rm django python manage.py dumpdata \ --natural-foreign --natural-primary \ --exclude=contenttypes --exclude=auth.Permission \ --indent 2 \ - --output /app/backups/prod_backup-$(date +%Y%m%d).json + --output backups/prod_backup-$(date +%Y%m%d).json # Restore from a fixture -docker-compose -f local.yml run --rm django python manage.py loaddata /path/to/backup.json +docker-compose -f local.yml run --rm django python manage.py loaddata backups/backup_name.json ``` Note: For large databases (>1.5GB), the database_backup and database_restore commands are strongly recommended over JSON fixtures as they handle large datasets more efficiently. diff --git a/sde_collections/management/commands/database_backup.py b/sde_collections/management/commands/database_backup.py index 5f6551b3..090de63e 100644 --- a/sde_collections/management/commands/database_backup.py +++ b/sde_collections/management/commands/database_backup.py @@ -4,15 +4,16 @@ Usage: docker-compose -f local.yml run --rm django python manage.py database_backup docker-compose -f local.yml run --rm django python manage.py database_backup --no-compress - docker-compose -f local.yml run --rm django python manage.py database_backup --output /path/to/output.sql + docker-compose -f local.yml run --rm django python manage.py database_backup --output my_backup.sql docker-compose -f production.yml run --rm django python manage.py database_backup + +All backups are stored in the /backups directory, which is mounted as a volume in both local +and production environments. If specifying a custom output path, it will be relative to this directory. """ -import enum import gzip import os import shutil -import socket import subprocess from contextlib import contextmanager from datetime import datetime @@ -21,21 +22,6 @@ from django.core.management.base import BaseCommand -class Server(enum.Enum): - PRODUCTION = "PRODUCTION" - STAGING = "STAGING" - UNKNOWN = "UNKNOWN" - - -def detect_server() -> Server: - hostname = socket.gethostname().upper() - if "PRODUCTION" in hostname: - return Server.PRODUCTION - elif "STAGING" in hostname: - return Server.STAGING - return Server.UNKNOWN - - @contextmanager def temp_file_handler(filename: str): """Context manager to handle temporary files, ensuring cleanup.""" @@ -58,36 +44,45 @@ def add_arguments(self, parser): parser.add_argument( "--output", type=str, - help="Output file path (default: auto-generated based on server name and date)", + help="Output file path (default: auto-generated in /app/backups directory)", ) - def get_backup_filename(self, server: Server, compress: bool, custom_output: str = None) -> tuple[str, str]: + def get_backup_filename(self, compress: bool, custom_output: str = None) -> tuple[str, str]: """Generate backup filename and actual dump path. Args: - server: Server enum indicating the environment compress: Whether the output should be compressed custom_output: Optional custom output path Returns: - tuple[str, str]: A tuple containing (final_filename, temp_filename) - - final_filename: The name of the final backup file (with .gz if compressed) - - temp_filename: The name of the temporary dump file (always without .gz) + tuple[str, str]: A tuple containing: + - final_filename: Full path for the final backup file (with .gz if compressed) + - temp_filename: Full path for the temporary dump file (without .gz) """ + backup_dir = "/app/backups" + os.makedirs(backup_dir, exist_ok=True) + if custom_output: + # If custom_output is relative, make it relative to backup_dir + if not custom_output.startswith("/"): + custom_output = os.path.join(backup_dir, custom_output) + # Ensure the output directory exists output_dir = os.path.dirname(custom_output) if output_dir: os.makedirs(output_dir, exist_ok=True) if compress: - return custom_output + (".gz" if not custom_output.endswith(".gz") else ""), custom_output.removesuffix( + return custom_output + ( + ".gz" if not custom_output.endswith(".gz") else "" + ), custom_output.removesuffix( # noqa ".gz" ) return custom_output, custom_output else: date_str = datetime.now().strftime("%Y%m%d") - temp_filename = f"{server.value.lower()}_backup_{date_str}.sql" + env_name = os.getenv("BACKUP_ENVIRONMENT", "unknown") + temp_filename = os.path.join(backup_dir, f"{env_name}_backup_{date_str}.sql") final_filename = f"{temp_filename}.gz" if compress else temp_filename return final_filename, temp_filename @@ -116,9 +111,15 @@ def compress_file(self, input_file: str, output_file: str) -> None: shutil.copyfileobj(f_in, f_out) def handle(self, *args, **options): - server = detect_server() + if not os.getenv("BACKUP_ENVIRONMENT"): + self.stdout.write( + self.style.WARNING( + "Note: Set BACKUP_ENVIRONMENT in your env if you want automatic environment-based filenames" + ) + ) + compress = not options["no_compress"] - backup_file, dump_file = self.get_backup_filename(server, compress, options.get("output")) + backup_file, dump_file = self.get_backup_filename(compress, options.get("output")) env = os.environ.copy() env["PGPASSWORD"] = settings.DATABASES["default"]["PASSWORD"] @@ -133,10 +134,10 @@ def handle(self, *args, **options): self.stdout.write( self.style.SUCCESS( - f"Successfully created {'compressed ' if compress else ''}backup for {server.value}: {backup_file}" + f"Successfully created {'compressed ' if compress else ''}backup at: backups/{os.path.basename(backup_file)}" # noqa ) ) except subprocess.CalledProcessError as e: - self.stdout.write(self.style.ERROR(f"Backup failed on {server.value}: {str(e)}")) + self.stdout.write(self.style.ERROR(f"Backup failed: {str(e)}")) except Exception as e: self.stdout.write(self.style.ERROR(f"Error during backup process: {str(e)}")) diff --git a/sde_collections/management/commands/database_restore.py b/sde_collections/management/commands/database_restore.py index ece94cce..7410484d 100644 --- a/sde_collections/management/commands/database_restore.py +++ b/sde_collections/management/commands/database_restore.py @@ -2,8 +2,11 @@ Management command to restore PostgreSQL database from backup. Usage: - docker-compose -f local.yml run --rm django python manage.py database_restore path/to/backup.sql[.gz] - docker-compose -f production.yml run --rm django python manage.py database_restore path/to/backup.sql[.gz] + docker-compose -f local.yml run --rm django python manage.py database_restore backups/backup.sql[.gz] + docker-compose -f production.yml run --rm django python manage.py database_restore backups/backup.sql[.gz] + +The backup file should be located in the /backups directory, which is mounted as a volume in both +local and production environments. """ import enum diff --git a/sde_collections/tests/test_database_backup.py b/sde_collections/tests/test_database_backup.py index d8a7be54..680da2f1 100644 --- a/sde_collections/tests/test_database_backup.py +++ b/sde_collections/tests/test_database_backup.py @@ -9,10 +9,7 @@ from django.core.management import call_command from sde_collections.management.commands import database_backup -from sde_collections.management.commands.database_backup import ( - Server, - temp_file_handler, -) +from sde_collections.management.commands.database_backup import temp_file_handler @pytest.fixture @@ -49,18 +46,27 @@ def command(): class TestBackupCommand: - def test_get_backup_filename_compressed(self, command, mock_date): + def test_get_backup_filename_compressed(self, command, mock_date, monkeypatch): """Test backup filename generation with compression.""" - backup_file, dump_file = command.get_backup_filename(Server.STAGING, compress=True) - assert backup_file == "staging_backup_20240115.sql.gz" - assert dump_file == "staging_backup_20240115.sql" + monkeypatch.setenv("BACKUP_ENVIRONMENT", "staging") + backup_file, dump_file = command.get_backup_filename(compress=True) + assert backup_file.endswith("staging_backup_20240115.sql.gz") + assert dump_file.endswith("staging_backup_20240115.sql") - def test_get_backup_filename_uncompressed(self, command, mock_date): + def test_get_backup_filename_uncompressed(self, command, mock_date, monkeypatch): """Test backup filename generation without compression.""" - backup_file, dump_file = command.get_backup_filename(Server.PRODUCTION, compress=False) - assert backup_file == "production_backup_20240115.sql" + monkeypatch.setenv("BACKUP_ENVIRONMENT", "production") + backup_file, dump_file = command.get_backup_filename(compress=False) + assert backup_file.endswith("production_backup_20240115.sql") assert dump_file == backup_file + def test_get_backup_filename_no_environment(self, command, mock_date, monkeypatch): + """Test backup filename generation with no environment set.""" + monkeypatch.delenv("BACKUP_ENVIRONMENT", raising=False) + backup_file, dump_file = command.get_backup_filename(compress=True) + assert backup_file.endswith("unknown_backup_20240115.sql.gz") + assert dump_file.endswith("unknown_backup_20240115.sql") + def test_run_pg_dump(self, command, mock_subprocess, mock_settings): """Test pg_dump command execution.""" env = {"PGPASSWORD": "test_password"} @@ -119,34 +125,18 @@ def test_temp_file_handler_cleanup_on_error(self, tmp_path): raise ValueError("Test error") assert not test_file.exists() - @patch("socket.gethostname") - def test_server_detection(self, mock_hostname): - """Test server environment detection.""" - test_cases = [ - ("PRODUCTION-SERVER", Server.PRODUCTION), - ("STAGING-DB", Server.STAGING), - ("DEV-HOST", Server.UNKNOWN), - ] - - for hostname, expected_server in test_cases: - mock_hostname.return_value = hostname - with patch("sde_collections.management.commands.database_backup.detect_server") as mock_detect: - mock_detect.return_value = expected_server - server = database_backup.detect_server() - assert server == expected_server - @pytest.mark.parametrize( - "compress,hostname", + "compress,env_name", [ - (True, "PRODUCTION-SERVER"), - (False, "STAGING-SERVER"), - (True, "UNKNOWN-SERVER"), + (True, "production"), + (False, "staging"), + (True, "carson_local"), ], ) - def test_handle_integration(self, compress, hostname, mock_subprocess, mock_date, mock_settings): + def test_handle_integration(self, compress, env_name, mock_subprocess, mock_date, mock_settings, monkeypatch): """Test full backup process integration.""" - with patch("socket.gethostname", return_value=hostname): - call_command("database_backup", no_compress=not compress) + monkeypatch.setenv("BACKUP_ENVIRONMENT", env_name) + call_command("database_backup", no_compress=not compress) # Verify correct command execution mock_subprocess.assert_called_once() @@ -154,35 +144,32 @@ def test_handle_integration(self, compress, hostname, mock_subprocess, mock_date # Verify correct filename used cmd_args = mock_subprocess.call_args[0][0] date_str = "20240115" - server_type = hostname.split("-")[0].lower() - expected_base = f"{server_type}_backup_{date_str}.sql" + expected_base = f"{env_name}_backup_{date_str}.sql" + assert cmd_args[-1].endswith(expected_base) + # Verify cleanup attempted if compressed if compress: - assert cmd_args[-1] == expected_base # Temporary file - # Verify cleanup attempted assert not os.path.exists(expected_base) - else: - assert cmd_args[-1] == expected_base - def test_handle_pg_dump_error(self, mock_subprocess, mock_date): + def test_handle_pg_dump_error(self, mock_subprocess, mock_date, monkeypatch): """Test error handling when pg_dump fails.""" mock_subprocess.side_effect = subprocess.CalledProcessError(1, "pg_dump") + monkeypatch.setenv("BACKUP_ENVIRONMENT", "staging") - with patch("socket.gethostname", return_value="STAGING-SERVER"): - call_command("database_backup") + call_command("database_backup") # Verify error handling and cleanup date_str = "20240115" temp_file = f"staging_backup_{date_str}.sql" assert not os.path.exists(temp_file) - def test_handle_compression_error(self, mock_subprocess, mock_date, command): + def test_handle_compression_error(self, mock_subprocess, mock_date, command, monkeypatch): """Test error handling during compression.""" + monkeypatch.setenv("BACKUP_ENVIRONMENT", "staging") # Mock compression to fail command.compress_file = Mock(side_effect=Exception("Compression failed")) - with patch("socket.gethostname", return_value="STAGING-SERVER"): - call_command("database_backup") + call_command("database_backup") # Verify cleanup date_str = "20240115"