After upgrading from 2.4 to 2.5

[amelcon-dot]

Hello,
After some time I have finally upgraded the client from 2.4 to 2.5.2.
First I run a standalone synchronization with --resync to be sure that everything is fresh clean.
Then I run regular standalone synchronization with just --sync. The only thing special that I include in --single-directory. I have noticed that the client is much slower than version 2.4. Before, standalone synchronization took around a couple of minutes in the same directory, and now it goes up to an hour. Is there anything special that need to be configured to get back to similar synchronization tasks as with 2.4 version?. Thanks.

[abraunegg]

@amelcon-dot

After some time I have finally upgraded the client from 2.4 to 2.5.2.

Please use v2.5.3 .. not v2.5.2

I have noticed that the client is much slower than version 2.4. Before, standalone synchronization took around a couple of minutes in the same directory, and now it goes up to an hour.

Please read the release notes - https://github.com/abraunegg/onedrive/releases which state:

The way --single-directory use and data retrieved has changed due to the 100% re-write to support fixing the bugs in v2.4.x around this use, but also to fully support the multi-threaded upload|download. The prior mode would rely on using what is known as the /delta query to track changes over your entire account, then only filter that out based on --single-directory action. in v2.5.x the path as specified is fully tracked, building a tree of your online data so that online changes are not missed.

Client version v2.5.3 also has some significant performance improvements. My advice is you should be using that version instead of v2.5.2.

[amelcon-dot]

OK, thanks, I will do that and let you know.

[amelcon-dot]

Hello,

I am finally in 2.5.3. I copy here the running times I got doing standalone synchronization:

[2024-Nov-25 09:31:24.6641433] Generating a /delta response from the OneDrive API from this Item ID: XXXXXXXXX
[2024-Nov-25 11:24:37.5822693] Finished processing self generated /delta JSON response from the OneDrive API
[2024-Nov-25 11:24:37.7107993] Processing 128797 applicable changes and items received from Microsoft OneDrive
[2024-Nov-25 11:33:24.6954153] Number of items to download from Microsoft OneDrive: 2929

The generation of /delta response takes about 2 hours. Before finishing, it did a final additional /delta response with another 2 hours of time.

I read another post, and this may be related to the use of --single-directory, which in 2.5 branch seems to be a bad choice (I was using this option in 2.4 branch and seemed to be good).

To explain why I was doing this, my configuration of directories in my onedrive is as follows:

1. Some minor folders in root without importance for me.
2. /v1/directory1 in root with hundred thousand of files for work.
3. /v1/directory2 in root with hundred thousand of files for home.

I use two profiles, one to sync /v1/directory1 in work and the second to sync /v1/directory2 in home. That is why I was using previously
--single-directory /v1/directory1 or --single-directory /v1/directory2 in the corresponding profiles, depending on what I wanted to sync (at work or at home).

I read that to sync specific folders it is now better to use sync_list, and it is not a good choice to use --single-directory.

I tried to do this, including in the sync_list of the home profile the entry /v1/directory2. Now I need to do --resync. I get:

Scanning the local file system '~' for new data to upload .
Fetching items from the OneDrive API for Drive ID: XXXXXXXXXXXXXX

I think this Fetching operation is going through the entire onedrive drive, including all three items in my previous list. So it is not just going inside /v1/directory2. I think this will take forever (it will have to go through not only one hundred thousand of files in /v1/directory2, but through two hundred thousand of files also going through /v1/directory1, plus the minor folders).

Is there a way I can optimize this?. I know this sounds complicated, but I thank any recommendation on how I could proceed.

Thank you very much for your help.

[abraunegg]

Is there a way I can optimize this?. I know this sounds complicated, but I thank any recommendation on how I could proceed.

You can optimise this in the following manner:

• Use multiple configuration profiles
  • one to explicitly sync /v1/directory1
  • one to explicitly sync /v1/directory2
  • one to explicitly sync 'minor folders in root without importance'
• Use 'sync_list' for each profile to only sync to|from those locations what you want to sync

Please read https://github.com/abraunegg/onedrive/blob/master/docs/advanced-usage.md#configuring-the-client-to-use-multiple-onedrive-accounts--configurations to configure multiple accounts. Essentially, you should end up with commands | systemd services that do the following:

• onedrive --confdir="/path/to/config/dir/for/directory1" <any other required options here>
• onedrive --confdir="/path/to/config/dir/for/directory2" <any other required options here>
• onedrive --confdir="/path/to/config/dir/for/other/folders" <any other required options here>

The benefits of this approach are:

• Will use 'sync_list' for each configuration, that is independent from each other. Changes to folders / what needs to sync does not impact the other configurations or force a --resync on those when that configuration changes.
• Will only iterate your folders as specified
• Will be able to use /delta so post the initial 'sync' for each configuration to pull in | push changes

[amelcon-dot]

Thank you.
The configuration you propose using --sync-list is working. Here I give some running times for reference, in case they can be of help.

The first --resync operation is very slow:

[2024-Nov-27 09:50:11.9295714] Fetching items from the OneDrive API for Drive ID: XXX
[2024-Nov-27 12:19:01.6133106] Processing 166707 applicable changes and items received from Microsoft OneDrive
[2024-Nov-27 12:40:18.5745757] Number of items to download from Microsoft OneDrive: 88
[2024-Nov-27 12:41:47.4180354] Performing a database consistency and integrity check on locally stored data
[2024-Nov-27 12:42:07.5786488] Scanning the local file system ~ for new data to upload
[2024-Nov-27 12:42:07.5839704] Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
[2024-Nov-27 12:42:08.1751938] Sync with Microsoft OneDrive is complete

So the Fetching operation takes about 2.5 hours. The processing of the changes, about 20 minutes. The rest of the operations are quite fast.

I have in my disk, at the root location, a broken symbolic link. The client gives me this error:

[2024-Nov-27 12:42:07.5839111] ERROR: The local file system returned an error with the following message:
[2024-Nov-27 12:42:07.5839292]   Error Message:    ./OneDrive_JuntaCentro: No such file or directory
[2024-Nov-27 12:42:07.5839370]   Calling Function: scanPathForNewData()

The thing is that I have configured skip_symlinks = "true", and this broken link is outside the --sync-list tree, so I am not sure if this is normal, or something related to a bug.

Subsequent synching operations are faster, and here I give some running times:

[2024-Nov-27 15:04:09.2190644] Performing a full scan of online data to ensure consistent local state
[2024-Nov-27 15:04:09.2190892] Fetching items from the OneDrive API for Drive ID: XXXXX
[2024-Nov-27 15:18:10.9544423] Processing 166597 applicable changes and items received from Microsoft OneDrive
[2024-Nov-27 15:19:27.1380799] Performing a database consistency and integrity check on locally stored data
[2024-Nov-27 15:19:46.7510041] Scanning the local file system '~' for new data to upload
[2024-Nov-27 15:19:46.8950083] Sync with Microsoft OneDrive is complete

Now the Fetching operation is reduced to about 14 minutes. The processing changes operation to around 1.5 minute. Rest of operations are very fast.

A last question. If you can do selective synching with --sync-list, and --single-directory is so slow, what would be the use of --single-directory in a practical situation?.

Thanks again for your help.

[abraunegg]

FYI - for log output, please use 3 x ` at the start and end to correctly 'code-block' log output.

[abraunegg]

@amelcon-dot

[2024-Nov-27 12:19:01.6133106] Processing 166707 applicable changes and items received from Microsoft OneDrive
[2024-Nov-27 12:40:18.5745757] Number of items to download from Microsoft OneDrive: 88

The 'reception' of 166707 items in 20 minutes is slow - but this has nothing to do with the client application.

This has 100% to do with:

• Your Internet access speed / access to the Microsoft OneDrive API
• The processing speed of your system

All of these are 100% beyond this application's control.

[abraunegg]

A last question. If you can do selective synching with --sync-list, and --single-directory is so slow, what would be the use of --single-directory in a practical situation?

You were using --single-directory to sync your entire data.

--single-directory is meant to be used to sync a deep folder in your structure and update this locally.

Consider folk who use --sync only .. manually keeping things up to date.

They update something online and/or locally deep in their folder structure.

This option builds the online state for that exact location, nothing else.

That is the use case, and your use of it to scan your entire top level - whilst it works - is the wrong use.

[abraunegg]

@amelcon-dot
Just to close this out, using a 100K file|folder dataset, with a --resync - total operation time ~16 minutes:

alex@fedora:~/onedrive-master$ time ./onedrive -s --resync --resync-auth
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
D-Bus message bus daemon is available; GUI notifications are now enabled
Deleting the saved application sync status ...
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Configuring Global Azure AD Endpoints
Fetching items from the OneDrive API for Drive ID: b!6YStAVcRrkaXaJQ7jj2CWfQyuny4Y1ROgq7SUuWfQT9l3Nzwzg_8Roub8Te9E-XZ ....................................................................................................................................................................................................................................................................................................................................................................................................... 
Processing 101489 applicable changes and items received from Microsoft OneDrive ............................................................................................................................................................................................................ 
Performing a database consistency and integrity check on locally stored data ............................. 
Scanning the local file system 'OneDrive_100K_Files' for new data to upload .......................... 
New items to upload to Microsoft OneDrive: 1

A database statement execution error occurred: database disk image is malformed

Please restart the application with --resync to potentially fix any local database issues.

ERROR: Unable to perform a database checkpoint: database disk image is malformed

Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching items from the OneDrive API for Drive ID: b!6YStAVcRrkaXaJQ7jj2CWfQyuny4Y1ROgq7SUuWfQT9l3Nzwzg_8Roub8Te9E-XZ .. 
No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive

Sync with Microsoft OneDrive is complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

real    15m41.247s
user    9m35.485s
sys     1m8.029s
alex@fedora:~/onedrive-master$ 

It certainly does not take 2.5 hours to pull down 100K objects.

Please look at your environment, network speed, processing speed of your environment.

[amelcon-dot]

Hello,

Thanks, I understand now the use of --single-directory. It could be indeed useful if you know that you have made changes into a specific folder deep in the folder structure.

The 2.5 hours is still unknown to me. I have verified that the number of files in /v1/directory1 is 107.692 files.

Hardware is not very new, but it has enough memory as to avoid swapping.

>free -h
                 total        used        free       shared      buff/cache   available
Mem:            11Gi       2.3Gi       145Mi       155Mi       9.6Gi            9.3Gi
Swap:          2.0Gi       471Mi       1.5Gi

The only thing I can think of is that I define the sync dir as my entire home Linux dir: sync_dir = "~"
I think this is not important since I understand it is the location where everything will be stored once it is synced, and I include in sync_list the single directory I want to sync.

[abraunegg]

@amelcon-dot

The only thing I can think of is that I define the sync dir as my entire home Linux dir: sync_dir = "~"

Whilst you are free to choose how you use and configure the application, this approach has potentially significant issues for you.

I do not know what you are actually syncing and excluding via 'sync_list' and the 3 x configurations, but there are the things you need to consider with your current approach:

Potential Key Directory Categories in ~

1. Sensitive Directories:

   • ~/.ssh: SSH keys and configuration.
   • ~/.config: Configuration files for various applications.
   • ~/.gnupg: GPG keys and related data.
   • ~/.cache: Cache files, some of which may contain sensitive data.
   • ~/.local/share: Application-specific data, sometimes sensitive.
   • ~/.mozilla / ~/.chrome: Browser data, potentially including cookies and history.
   • ~/.bash_history / ~/.zsh_history: Shell command history.
   • ~/.kde / ~/.gnome2 (or similar): Desktop environment configurations.

2. Large Application Files/Folders:

   • ~/.cache: Can grow significantly for applications like browsers and media players.
   • ~/.local/share: Often includes game data, flatpak applications, etc.
   • ~/snap: Snap application data.
   • ~/.steam / ~/.var/app: Gaming-related data.
   • ~/.docker: Docker image and container data.
   • ~/VirtualBox VMs or similar: Virtual machine data.

3. Document Folders:

   • ~/Documents
   • ~/Downloads
   • ~/Pictures
   • ~/Videos
   • ~/Music

4. Ubuntu-Specific (Snap Files):

   • ~/snap: Contains Snap package data, which can be large and unnecessary for syncing.

While you are free to configure the application however you wish, using your entire home directory (~) as your sync_dir may expose you to several risks and challenges:

1. Sensitive Data Exposure:

   • Your home directory contains sensitive information, including SSH keys (~/.ssh), application configurations (~/.config), and even browser data (~/.mozilla, ~/.chrome).
   • Syncing these directories to a cloud service like OneDrive means they are uploaded and possibly exposed to additional risks if access controls are not perfect.

2. Unnecessary Data Synchronisation:

   • Large application-specific files or folders (e.g., ~/snap, ~/.docker, ~/.cache) that are not typically useful for synchronisation will consume unnecessary bandwidth and storage.
   • This can lead to slower sync operations, higher data usage, and even increased costs if you exceed your cloud storage limits.

3. Data Management Complexity:

   • Syncing your entire home directory introduces complexity in managing what gets synced and what does not.
   • Any accidental changes, deletions, or conflicts could have far-reaching consequences on your system and applications.

Recommended Best Practices:

• Default Sync Directory: The default sync_dir (~/OneDrive) is designed to avoid these issues by isolating synced data from critical system files.
• Alternative Sync Folders: If you wish to sync other directories, consider explicitly setting the sync_dir to a specific folder such as ~/Documents or another dedicated directory. This ensures that only intended files are synchronised.

By configuring the application thoughtfully, you protect your sensitive data and streamline your cloud storage usage, ensuring it works as intended without exposing your system to unnecessary risks.

I strongly encourage you to rethink your implementation and using sync_dir = "~"

[amelcon-dot]

Thanks for the recommendations, I have followed them. It is safer and more elegant as you describe. In my case it did not make much difference, since I have in --sync_list a single directory with the big folder to synchronize /v1/directory1. The performance I get is about the same as before. However, playing around I found two issues:

1. Do not use --resync together with --local-first.
2. If you have a broken symbolic link in your folder structure you get this error:

ERROR: The local file system returned an error with the following message:
[2024-Dec-02 12:10:13.8068441]   Error Message:    ./v1/amelcon/upct/Asignat/Radar/Practicas: No such file or directory
[2024-Dec-02 12:10:13.8068503]   Calling Function: scanPathForNewData()

Then, in --monitor mode the client continues to run, but the sync process for the local folders is actually stopped. You may think that the sync process is finished, but in fact it does not progress beyond the error. If you do not remove the broken symbolic link, it will happen the same in the next sync cycle, stopping again the local folders sync beyond that point.

What it is stranger, is that this error also happens when the broken symbolic link is outside the folder structure specified in --sync_list. This may indicate that the client is doing some operations outside of the folder structure specified in --sync_list.
I wonder if this behavior is normal or not.

Thanks again for your help.

[abraunegg]

@amelcon-dot

If you have a broken symbolic link in your folder structure you get this error

This is because your 'sync_dir' is '~'

The application is scanning your entire home directory and then filtering that based on your 'sync_list' rules.

This is 100% normal behavior.

[amelcon-dot]

Thank you, now everything is clear to me. With sync_dir pointing to a specific folder everything is clearer.
Thanks a lot.