Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Back up into folders instead of Tar archives #15

Open
ads103 opened this issue Feb 5, 2024 · 5 comments
Open

Feature Request: Back up into folders instead of Tar archives #15

ads103 opened this issue Feb 5, 2024 · 5 comments

Comments

@ads103
Copy link

ads103 commented Feb 5, 2024

I like to export my backups to an external backup repository using Borg Backup. However, because the backups generated by unraid-appdata.backup are in tar archives, I cannot take advantage of Borg Backup's deduplication features.

I'd like to have the option to back up my appdata as folders, instead of as tar archives. I'd like the result of using this option to be seeing each application's data saved in a folder in the specified backup destination, as opposed to the tar archives that are standard now.

Perhaps rsync could be used for this? Perhaps the main backup routine could resemble something like this?

    /**
     * Create a backup by copying folders using rsync, instead of tar
     * Friendly for deduplicating archivers to handle backups later
     * @param $container array
     * @param $destination string The generated backup folder for this backup run
     * @return bool
     */
    public static function backupCopyContainer($container, $destination) {
        global $abSettings, $dockerClient;

        self::backupLog("Backup {$container['Name']} - Container Volumeinfo: " . print_r($container['Volumes'], true), self::LOGLEVEL_DEBUG);

        $volumes = self::getContainerVolumes($container);

        $containerSettings = $abSettings->getContainerSpecificSettings($container['Name']);
        if ($containerSettings['backupExtVolumes'] == 'no') {
            self::backupLog("Should NOT backup external volumes, sanitizing them...");
            foreach ($volumes as $index => $volume) {
                if (!self::isVolumeWithinAppdata($volume)) {
                    unset($volumes[$index]);
                }
            }
        } else {
            self::backupLog("Backing up EXTERNAL volumes, because its enabled!", self::LOGLEVEL_WARN);
        }

        $tarExcludes = [];
        if (!empty($containerSettings['exclude'])) {
            self::backupLog("Container got excludes! " . PHP_EOL . print_r($containerSettings['exclude'], true), self::LOGLEVEL_DEBUG);
            $excludes = explode("\r\n", $containerSettings['exclude']);
            if (!empty($excludes)) {
                foreach ($excludes as $exclude) {
                    $exclude = rtrim($exclude, "/");
                    if (!empty($exclude)) {
                        if (($volumeKey = array_search($exclude, $volumes)) !== false) {
                            self::backupLog("Exclusion \"$exclude\" matches a container volume - ignoring volume/exclusion pair");
                            unset($volumes[$volumeKey]);
                            continue;
                        }
                        $tarExcludes[] = '--exclude ' . escapeshellarg($exclude);
                    }
                }
            }
        }

        if (empty($volumes)) {
            self::backupLog($container['Name'] . " does not have any volume to back up! Skipping");
            return true;
        }

        self::backupLog("Calculated volumes to back up: " . implode(", ", $volumes));

        $destination = $destination . "/" . $container['Name'] . '.rsync';

        $rsyncOptions       = array_merge($tarExcludes, ['--archive', '--relative']);    // Add excludes to the beginning - https://unix.stackexchange.com/a/33334

        self::backupLog("Target archive: " . $destination, self::LOGLEVEL_DEBUG);

        foreach ($volumes as $volume) {
            $rsyncOptions[] = escapeshellarg($volume);
        }
        $finalRsyncOptions       = implode(" ", $rsyncOptions) . escapeshellarg($destination);

        self::backupLog("Generated rsync command: " . $finalRsyncOptions, self::LOGLEVEL_DEBUG);
        self::backupLog("Backing up " . $container['Name'] . '...');

        $output = $resultcode = null;
        exec("rsync " . $finalRsyncOptions " 2>&1 " . ABSettings::$externalCmdPidCapture, $output, $resultcode);
        self::backupLog("Rsync out: " . implode('; ', $output), self::LOGLEVEL_DEBUG);

        if ($resultcode > 0) {
            self::backupLog("rsync run failed! Rsync said: " . implode('; ', $output), $containerSettings['ignoreBackupErrors'] == 'yes' ? self::LOGLEVEL_INFO : self::LOGLEVEL_ERR);
            return $containerSettings['ignoreBackupErrors'] == 'yes';
        }

        self::backupLog("Backup created without issues");

        if (ABHelper::abortRequested()) {
            return true;
        }
        
        return true;
    }
@Commifreak
Copy link
Owner

This is actually on the ToDo list for the same reason as yours :)

https://forums.unraid.net/topic/132721-plugin-ca-appdata-backup-restore-v25/page/18/#comment-1251220

@ads103
Copy link
Author

ads103 commented Feb 5, 2024

Ah, that's what I get for not looking deep enough. Thank you!

@Commifreak
Copy link
Owner

The recent forum activity revealed, that dedup software like Duplicacy is able to use non-compressed tar's for deduplication (https://forums.unraid.net/topic/137710-plugin-appdatabackup/?do=findComment&comment=1365996). I dont know if Borg is able to do that as well?

If Borg is able (and Duplicacy is), I dont see any advantage of implementing this. At least not now.

Could you check if Borg is capable of this?

@ads103
Copy link
Author

ads103 commented Feb 8, 2024

It looks like Borg's import-tar feature (https://borgbackup.readthedocs.io/en/stable/usage/tar.html) may offer similar functionality. But, it looks like Borgmatic (https://projects.torsion.org/borgmatic-collective/borgmatic) does not expose this functionality.

I'm beginning to feel like it may be more appropriate to ask the Borgmatic developers to expose the Borg's built-in tar-handling feature.

@PeteBa
Copy link

PeteBa commented Apr 19, 2024

+1 for this. (also thanks for the great unraid plugin)

I use urbackup with its cross-platform support and btrf snapshotting for very quick backup times. It does file de-duplication out of the box but not block dedupe. An rsync-like option instead of tar would be great.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants