Performance issue with large directory count on Onedrive

[ianc99]

I've a Onedrive account with a lot of directories (~9,500). I only want to sync one of the root directories with Linux though and so I initially tried a sync _list file like this:

!*
#Rpi  

But it was very slow to sync because it scanned all or most of the 9.5k directories anyway and only then disregarded them. I also don't think it worked because "!*" took precedence over the subsequent "#Linux".

I then tried
onedrive --synchronize --single-directory 'Rpi'
Which did work but also took an age because it still appears to be scanning the directories. This can be seen when --verbose is turned on.

Is there a way to avoid this scanning?

Thanks
I

[abraunegg]

@ianc99
Firstly, what application version are you using?

Secondly, your 'sync_list' file is not correct.

!*
#Rpi     <--- This is a comment .. comments are not processed ........

So effectivly, your 'sync_list' in this configuration is that you are ignoring all files .. which is not what you appear to be wanting to do. A valid 'sync_list' for what you are attempting to achieve would be the following:

# Include only the /Rpi directory
/Rpi

Please re-read https://github.com/abraunegg/onedrive/blob/master/docs/USAGE.md#performing-a-selective-sync-via-sync_list-file

But it was very slow to sync because it scanned all or most of the 9.5k directories anyway and only then disregarded them. I also don't think it worked because "!*" took precedence over the subsequent "#Linux".

What hardware are you running this on? What filesystem are you running? Do you have any encryption? What is your Internet speed?

When the application details Processing ABC OneDrive items ... it is processing that number of items|changes sent from OneDrive, as the OneDrive API presents these to the client in 200+ item lots. There is zero API configuration to increase this value, it is hardcoded in the Microsoft API.

Is the processing of this data from OneDrive slow .... well this comes down to your processor (CPU) and hardware (memory, disk speed/SSD speed), filesystem type (XFS, ZFS, EXT4 etc ...) and if you have adequate swap space allocated as to how quick this API data from OneDrive is being processed, combined with your Internet access speed which determines how fast this API data is transferred to you from OneDrive.

While these 'change sets' are being processed, these are evaluated against your 'client side filtering rules' - eg - your 'sync_list' ... this is why you see the items discarded if they do not match your 'client side filtering rules'.

This is a limitation of OneDrive itself ... it was written with the major poor assumption that whatever it (Microsoft OneDrive) sends is what the 'client' wants & will store locally ..... where as many of us in the Open Source Community want a little more 'configuration + control' - and hence, this is why we have all this 'client side filtering' within the application ('sync_list', 'skip_dir, 'skip_file' etc) - but, this still means that all the 'changes' from OneDrive are sent to the client, because OneDrive has zero capability for any server side filtering so that OneDrive only sends the data we want. So everything has to be filtered client side.

I then tried onedrive --synchronize --single-directory 'Rpi' Which did work but also took an age because it still appears to be scanning the directories. This can be seen when --verbose is turned on.

Is there a way to avoid this scanning?

This 'scanning' is done at application startup to ensure that the integrity of your data is correct, and the data that is local matches what you want to be local ....

If you are using --synchronize this will be done at every application use. First run will be more intense, however subsequent runs will be less intense due to leveraging what is known as the /delta change point.

If you are using --monitor this will be done at application startup and then at specific intervals governed by:

• monitor_interval - See: https://github.com/abraunegg/onedrive/blob/master/docs/USAGE.md#monitor_interval
• monitor_fullscan_frequency - See: https://github.com/abraunegg/onedrive/blob/master/docs/USAGE.md#monitor_fullscan_frequency

Additionally avoid using --resync ... as this deletes your sync-state and forces a full sync. Only use --resync if the application tells you to use it post configuration change.

[ianc99]

Wow! Thanks for such a comprehensive response so fast.
Some answers to the Q's you asked - and apologies for not including these up front.

SW version is v2.4.21-1+np1 ( I only downloaded it today :) )
Running on RPi 4b with fully updated Raspbian GNU/Linux 11
Broadband connection is 1Gb/s down & 512Mb/s up
No additional encryption

Your explanations have helped me sort out how I was using it.
Sorted. Thanks

Ian

[abraunegg]

@ianc99

Running on RPi 4b with fully updated Raspbian GNU/Linux 11

Make sure you are using aarch64 and not armhf on your 64Bit processor.