How to RESTORE an incremental backup file

[Al-NN4ZZ]

This issue was spun off of issue #187. There is an option to create FULL or INCREMENTAL backup files. The incremental backup files only contains the changes made to the profile and local storage since the last full backup.

Here is an example of the FULL backup and 2 INCREMENTAL backup files, note the files sizes. The FULL backup file is much larger than the incremental files as expected.

The Question is: How to RESTORE the data from the incremental files? In a previous issue ( #187) this was explored, however I don't believe the answer to just overwrite the profile is correct. See note below.

There must be a process to APPLY the incremental data to an EXISTING profile that was restored from a FULL backup.

Thanks...Al

[cleidigh]

@Al-NN4ZZ
let me see if I can do some controlled experiments.
from everything I know so far, I still believe "overlaying" an incremental backup
over the last bk is correct. unchanged files do not get overwritten, only the changed files
there is no way to deal with individual file content.
Christopher

[cleidigh]

@Al-NN4ZZ
I just did a small experiment with a partial bk
first I did a fallbackup
then I added another extension and a new e-mail
then I did a partial
I created an empty profile folder
copied to full
then I copied the partial over the full (told Windows to replace files)
reopen Thunderbird with the new profile folder - everything looks the same
what I still have to look into this what is different between exporting your profile
as opposed to a backup
Christopher

[Al-NN4ZZ]

Hi Christopher,
Sorry I didn't follow your test process/results completely, but looking forward to the next update.

Thanks...Al

[cleidigh]

@Al-NN4ZZ
did you have a specific question? I think my experiments prove the method you were looking for.
Christopher

[Al-NN4ZZ]

Christopher,
Let me restate what I think you are saying with an example using my typical data. Here are the steps as I understand them, see my question in STEP 11. I think I must be missing something.

1. Make a FULL backup of the profile which in my case holds 20,000 emails and all of the other account data. This file will be about 8 GB in size.
2. Make an incremental backup file a few days later which only contains about 200 new emails and some other account changes. This file will be about 250 MB in size
3. Make another incremental backup in another few days which contains another 300 more new emails and other account changes. This will will be about 300 MB in size.
4. Assume at this point my working TB profile that contains all the email, folder, and account data becomes corrupted and I want to restore it from the backups
5. Shut down TB -- examine my existing profile file and it will be about 8.5 GB in size.
6. Copy my FULL backup profile file over the existing and active profile. The profile should now be 8 GB in size. Smaller since it doesn't have the most recent emails.
7. Launch TB and verify that the 20,000 older emails and other account data have been restored up to the date of the FULL backup.
8. Shut down TB
9. Copy the FIRST incremental profile backup file over the existing profile ( i.e. the FULL back up file I just copied in step 6 )
10. The profile file will now be 250 MB in size since I overwrote the 8 GB file
11. Launch TB and verify that the 20,000 emails plus the additional emails from the incremental file are restored. Here is where I don't understand the process. If the profile file is now only 250 Mb in size how can it contain all of the 20,000 emails that previously took up 8GB of space.
12. Shut down TB
13. Copy the SECOND incremental profile backup file over the existing profile ( i.e. the FIRST incremental file )
14. Launch TB and verify that the 20,000 emails plus the additional emails from the FIRST and SECOND incremental files are restored.

Please let me know if I've misstated the steps or what I'm missing. I can't understand how overwriting the 8GB backup with the smaller 300MB INCREMENTAL backup file will maintain all of the data since it is all stored in the profile.

Thanks...Al

[LjohnB72]

Gentlemen, I, too, am looking for a simple backup\incremental\restore add-on for TB. Installed I\E\T-NG & started looking for instructions (OLD school). Was dissapointed so came here & found #187, the dialogue didn't help. Came here to #202 & after this dialogue, I have a Q - why is backup\incremental\restore an option in import\export\tools? I don't use import\export\tools, just want backup\incremental\restore, but had to install excess to get a simple function. Apparently, other people are in the same boat. Suggestion: simplify by creating a separate Backup\Incremental\Restore Tool-NG add-on. Why have a Swiss Army Knife when all I want to do is cut up some carrots? If I'm missing something, please school me, I love to learn. And to Al-NN4ZZ, I believe your item #11 question is valid. Ljohn (Christopher - thank you for your persistence)

[Al-NN4ZZ]

Hi @LjohnB72 ,
I'm new to the ImportExportTools NG add-on but based on the way the menu is presented, I suspect the Automated Backup was added later. And that is probably why it appears in the OPTIONs sub menu. I think your suggestion to spin off a separate Automated Backup/Restore add-on is a great idea. The new add-on would just focus on the backup options ( automation, full or incremental, file naming, restore process, etc). The existing ImportExportTools NG add-on could be trimmed down making it just focused on the ImportExport aspects.

With that being said, I'd prefer to get the process for the current incremental restore figured out first. I'm back to doing full backup until it is. And I've also submitted an idea to create a pop-up reminder for when the backup is due. Issue #195 is the enhancement request and it would be very helpful for those of us that want to run regular backups.

Christopher seems to be very busy so I don't expect a new add-on is in the short term. But again It is a good idea and I hope we do see it some day. Hopefully we'll hear from him about the restore process and his thoughts on a new AutomatedBackup add-on.

...Al

[cleidigh]

@Al-NN4ZZ /all
I have been hiding from covid ;-)
I would like to get the question on incremental backups settled, for everyone.
I am going to make a couple examples of the incremental steps.
I'm confident my steps are correct, however, I want to make sure I am not missing anything.
it is hard for me being disabled ( have to do everything by voice) to write long precise entries.
I also sometimes have to leave some things uncorrected Which may add confusion.
I do my best.

I will also comment more on the enhancement for automated schedule pop-up. shortly.
did not want you guys to think I disappeared...
Christopher

[cleidigh]

@LjohnB72
@Al-NN4ZZ
as I am not the original extension author I really cannot comment on why things are as they are.
certainly many of the methods and functions overlap between I/E and backups. all of these functions really should be native. I am not sure if you are aware,but I am recently elected to the Thunderbirds council and am also an outside team member. the plan going forward ultimately is to bring in this functionality internally. this is what team would like, my problem is I do not think I can necessarily accomplish that with my limitations.
while I agree a separate backup extension would be convenient, I would rather do some enhancements to the current extension knowing ultimately it should be internal.
the extension has certainly grown with a wide range of functionality . there are many things to enhance and more importantly issues to fix. as I said above I will try to assess the backup reminders/automated scheduled scenario.
the current menu structure is far from ideal, something I have not addressed. not sure myself, but what about separating the backup functions into a separate Tools Menu?
just a note, I know this extension needs more documentation, I spent a huge amount of time adding a fully localized help page. I need to add to that, but it's very difficult for me.
cheers
Christopher

[cleidigh]

@Al-NN4ZZ
question for you, what software did you use for your dialogue captures?
is it accessible ( keyboard shortcuts)?
I usually have to adjust to full-screen shots as editing images is practically impossible for me.

[Al-NN4ZZ]

Hi Christopher,
First, thanks for your support, and congratulations on being elected to the Thunderbirds council. Very glad to hear about the plan going forward to bring in this functionality internally.

Here is the info on the screen capture tool I use. It is called HyperSnap and I've been using it for more than 15 years. Link here:
http://www.hyperionics.com/

It supports keyboard shortcuts and lets you customize them. There is also a "voice commands" feature.

I'm still on an older version (V6.x) and the latest is V8.X. I should also note that the "help" information above is from the V6 version so there may be different details on the newer V8 versions.

Thanks again and looking forward to hearing the results of your incremental backup tests. Please also let me know if my 14 step process from above is correct or what I'm missing.

...Al

[cleidigh]

@Al-NN4ZZ
@LjohnB72
okay I think I understand where there's some confusion as to why my description of the process might be lacking. so some overall clarification points: (if I point out the obvious I am just trying to be complete, so please do not misinterpret anything ;-)

• the title of the issue actually points out a possible misinterpretation, although I do not think it was intentional.
• the backup is not a file, rather thousands of files.
• the incremental backup is also composed of folders and files which will always be a subset of a full backup. IMPORTANTLY these files are complete files, NOT deltas,patches or anything like that.
• because these are complete files, each changed file that is part of an incremental bk will replace the original file in the full bk

So I think the confusion/incorrect interpretation starts at step 10 where you look at the size of the bk and then have step 11 as a "file overwrite" implying the incremental small "file" overwrites the full bk. what actually happens is only the individual changed files replace the older ones.
the final size of the entire set of folders will be Fbk+Ibk-old files changed.
I have done this several times and it works as expected.

Here are a couple screenshots with descriptions.

this first one is a partial example of the final changes doing a complete comparison with files/folders . I compared to complete backup trees that were done around ten minutes apart. I made no changes to anything. as you can see hundreds of files have changed, most of this is all cache or metadata.

this shows just the main mail folder of a full bk before doing the incremental

this next one shows the changed files (only mail) after receiving a single e-mail. you can see that there are just 3 files changed the actual mbox top-level file along with its index, and state file. these 3 files are complete so they now include all the e-mails not just the new one. so when you do an incremental overw you still have all the e-mails. same for all of the other data.

I hope this clears things up, if not please let me know.

Regarding adding restore like ability, unfortunately I do not think this is possible as it would require being able to intercept the startup to allow retrieving the bk files while TB is running.
that would require internal changes.

Christopher

[Al-NN4ZZ]

Christopher,
Thanks for the details, I think I'm almost there now. I did realize that the profile backup contained multiple folders and files. I believe I know where my misunderstanding of the restore process lies. I originally thought the restore process you were advising was to copy over the ACTIVE profile folder with the backup copies. First with the FULL backup and then with the INCREMENTAL backup. So after the incremental copy the resulting profile would be just contain changed files and folders.

I think I understand the process now. Here is an example:

• The active profile folder is named syx1o14g.default-release
• The full backup file is named syx1o14g.default-release-20210215-1652 (i.e. the date-xxxx is appended to the profile name)
• The incremental backup file is named syx1o14g.default-release-20210225-0556 (i.e. the date-xxxx is appended to the profile name)

Step 1 is to delete the contents (files, folders) of the active profile folder named syx1o14g.default-release

Step 2 is to copy the contents (files, folders) of the FULL backup file to the active profile folder.

Step 3 is to copy the contents (files, folders) of the incremental backup file over the existing content of the profile folder. This would only overwrite any items (files and folders) with an identical name and add any new items (file and folders) that did not exist.

Please advise if this is the correct process. I'm going to do some tests over the next week to see if this works like I expect.

Here is the name of my active profile:

Here is the name of a BACKUP file:

Thanks...Al

P.S. I also now understand why the restore can not be done while TB is running.

[cleidigh]

yes your set of steps above are correct
somehow I have to create more documentation!
let me know how it goes
Christopher

[Al-NN4ZZ]

Christopher,
I have done some testing and it looks like this works. In my testing I restored the backups to a TEST version of the profile folder and compared properties of all the files for each step. I didn't actually restore the backups over my active profile but all of the data looks good. I documented a set of restore steps so in case I ever need to do this for real, I'll have them. I think I covered the bases but let me know if you see anything I missed.

Thanks...Al

Process to restore a corrupted Profile folder from the profile backups when the active file becomes corrupted.

Notes:
The steps below assume you are making Full backups and Incremental backups.
The steps below assume you have a FULL backup file and one or more subsequent INCREMENTAL backup files.
This was done using Thunderbird version 78.7.0 (64 bit)

Steps to restore your Profile from your Full and Incremental backup files.

1 - Locate your active Profile using the HELP/Troubleshooting option in Thunderbird. Click on "Profile Folder/Open Folder" which will bring up it up in Microsoft File Explorer. The folder name will be named syx1o14g.default-release or similar.

2 - Stop Thunderbird. This insures no changes will be made to the Profile during the next steps.

3 - Make a backup copy of the Active Profile Folder (suggest appending -Backup to the file name.). This is just in case you need to restore it. You can delete it later in step 9.

4 - Delete the CONTENTS of the Active Profile Folder but not the folder itself.

5 - Copy the CONTENTS from your FULL backup file to the Active Profile Folder.

6 - Copy the CONTENTS from your FIRST ( i.e. oldest) INCREMENTAL backup file to the Active Profile Folder. Matching files will be updated, new files added, pre-esxisting files will be maintained.

7 - Copy the CONTENTS from any subsequent INCREMENTAL backup files to the Active Profile Folder. Copy them in the order they were created. Matching files will be updated, new files added, pre-esxisting files will be maintained.

8 - Start Thunderbird and verify the contents were restored correctly and everything is working correctly.

9 - Delete the backup copy from step 3 once you are assured the profile has been restored.

[cleidigh]

@Al-NN4ZZ
very glad you feel comfortable with the process, your steps sounds good and we will definitely add this to the wiki and point to it in the help: !
the only thing I might add is that step three should be considered towards optional, but perhaps conservative.
cheers
Christopher

[Al-NN4ZZ]

Christopher,
Thanks and I agree about step 3....I wanted have a backup just in case something went wrong.

There is one more thing I'd like to run by you before I close this out. I would like to do a full backup every 2 weeks and then incremental backups every day between the full backups. I would need to remember to set the option to do the full, then set it back the next day to do the incremental and then after 2 weeks set it back to the full. And repeat. It is a lot to keep up with the manually. I have an enhancement idea that would automate the process. Here is a mockup of the OPTIONS/Backup Scheduling tab to illustrate the process. The change are highlighted in YELLOW.

The idea is to automate the alternating of the full and incremental backups. Basically just set it up and forget it. If you think this is an idea that is feasible, I'll create a new issue to describe the request.

Thanks...Al

[cleidigh]

@Al-NN4ZZ
the alternating setup makes sense, have to figure out the right layout
I can not promise when I could approach this,but I would do experiments
that you could look at first
Christopher

[Al-NN4ZZ]

Christopher,
Thanks, I just created issue #212 for the enhancement and will close this one now.

...Al