Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add optional compression and resume support to remote (with ssh) btrfs send/receive #34

Open
Fantu opened this issue Jan 5, 2017 · 1 comment

Comments

@Fantu
Copy link

Fantu commented Jan 5, 2017

Hi, thanks for this good software, it is very useful.
I found some possible improvements to make it optimal also with big remote transfert with very low bandwidth connections and possibile errors.
I think can be very useful add for remote case optional compression send with resume and bandwidth limit support.
This seems possible saving btrfs send output to file, optionally compress it, send it with rsync that can resume partial files and have also bandwidth limit support (that can be also added optional), on remote destination uncompress it (if needed) and do btrfs receive of the file. If fails maintain the "btrfs send" file and resume the transfert (with rsync).
I did some fast manual tests to check if works and how:
I did btrfs send with -f for save to file instead
I compress it to gzip that seems a good compromise between space and used resources/time.
After I send it with rsync --partial for resume support (including simulate common problem), for example I did:
rsync --partial --progress --rsh=ssh fullsend.gz remotehost:/mnt/btests/fullsend.gz
Is possible add also bandwidth limit adding --bwlimit=
After I did uncompress and btrfs receive on destination.
All is ok except the uuid of snapshot for use it for diffs with buttersink, I not found a btrfs-tools command to change uuid same as buttersink done but I suppose that including this improvements in buttersink is possibile maintaing actual buttersink features.

I don't have experience with python, can someone with experience add these features that I think are very useful (or essentials in many cases)?
Optimal result I suppose can be these additional parameters:
-r --resume-support save the btrfs send/receive data to file and make possible transfert it with possibility of resume, useful for remote transfert avoiding to redo all from start in case of error or needed "transfert pause"
-c --compress compress the btrfs send/receive data with gzip, useful for low bandwidth, require the -r
-b --bandwidth-limit limit the remote transfert, require the -r

Thanks for any reply and sorry for my bad english.

@AmesCornish
Copy link
Owner

Fantu, thanks for the suggestions. I can see how it would be useful to have resume and bandwidth limiting. I don't have time to work on this myself right now, but I'll keep it on the list and hopefully I or someone else will tackle it in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants