Can one set up sync to ONLY download files of a given type? #2783

hskrieg · 2024-08-21T17:43:38Z

hskrieg
Aug 21, 2024

I've tried setting up the sync_list with only file files I want to sync, but it still seems to sync everything that is not listed in the skip_file list. The reality is that I only want 2 file types out of the whole SharePoint library.

Answered by abraunegg

Aug 21, 2024

@hskrieg

I had been dealing with many other issues earlier, including running an older version of the tool, so that might have been my issue.

As per all documentation, you must use the correct application version which is currently v2.4.25, with v2.5.0 due for release in September after 18 months development. Please ensure you are installing the client the correct way for your distribution - and if you are using Ubuntu or any distribution based on Ubuntu - you must install from the OpenSuSE Build Service repositories.

The reality is that I only want 2 file types out of the whole SharePoint library.

You must first ensure you are using the correct client version.

Once this is done, you m…

View full answer

hskrieg · 2024-08-21T20:19:15Z

hskrieg
Aug 21, 2024
Author

After further reading, it looks like this should be handled by the sync_list. I had been dealing with many other issues earlier, including running an older version of the tool, so that might have been my issue. I'll try again tomorrow.

0 replies

abraunegg · 2024-08-21T21:26:19Z

abraunegg
Aug 21, 2024
Maintainer

@hskrieg

I had been dealing with many other issues earlier, including running an older version of the tool, so that might have been my issue.

As per all documentation, you must use the correct application version which is currently v2.4.25, with v2.5.0 due for release in September after 18 months development. Please ensure you are installing the client the correct way for your distribution - and if you are using Ubuntu or any distribution based on Ubuntu - you must install from the OpenSuSE Build Service repositories.

The reality is that I only want 2 file types out of the whole SharePoint library.

You must first ensure you are using the correct client version.

Once this is done, you must configure the client to use that SharePoint repository - you cannot just add to your 'sync_list' file a file from a SharePoint repository.

I've tried setting up the sync_list with only file files I want to sync, but it still seems to sync everything that is not listed in the skip_file list.

The 'sync_list' file is a exclude all by default file, so this will only download those files explicitly added.

If your client is still downloading all other files - there is a 100% misconfiguration going on.

Start by stopping all active clients, then configuring the client and using --display-config to ensure that the client is configured the correct way, then test by using --dry-run

2 replies

hskrieg Aug 22, 2024
Author

Thank you for the quick update. I have tried to sync with the new configuration, adding a test directory to the SharePoint with a couple of new files (both a .trc and a .ulg file). However, they do not get downloaded. I assume that my configuration is wrong, but I'm not sure where. The intention is to download ALL ulg and trc files in ALL directories under P1.0 Testing Events.

sync_list =
General/P1.0 Test Data/P1.0 Testing Events/*.ulg
*.trc

The new files are in General/P1.0 Test Data/P1.0 Testing Events/SyncTest/

Output for the section using v2.25.0:

./onedrive --confdir="~/.config/SharePoint_P1.0_Hybrid" --synchronize --resync -v+ -v+ --dry-run --single-directory "General/P1.0 Test Data/P1.0 Testing Events"

[DEBUG] ------------------------------------------------------------------
[DEBUG] Processing change 18 of 206
[DEBUG] OneDrive Change: {"@odata.etag":"\"{XXXX},2\"","@odata.type":"#microsoft.graph.driveItem","cTag":"\"c:{XXXX},0\"","eTag":"\"{XXXX},2\"","fileSystemInfo":{"createdDateTime":"2024-08-22T11:31:01Z","lastModifiedDateTime":"2024-08-22T11:35:15Z"},"folder":{"childCount":1},"id":"XXXX52XM","name":"SyncTest","parentReference":{"driveId":"XXXX","driveType":"documentLibrary","id":"XXXX623B","path":"\/drives\/XXXX\/root:\/General\/P1.0 Test Data\/P1.0 Testing Events","siteId":"XXXX"},"size":3088522}
[DEBUG] Not a OneDrive deleted item change
[DEBUG] defaultRootId                                        = XXXXLRRZ
[DEBUG] 'search id'                                          = XXXX623B
[DEBUG] id == defaultRootId                                  = false
[DEBUG] isItemRoot(item)                                     = false
[DEBUG] item['name'].str == 'root'                           = false
[DEBUG] singleDirectoryScope                                 = true
[DEBUG] 'parentReference id'                                 = XXXX623B
[DEBUG] search criteria: syncFolderName                      = P1.0 Testing Events
[DEBUG] search criteria: syncFolderPath                      = /drives/XXXX/root:/General/P1.0 Test Data
[DEBUG] search criteria: syncFolderChildPath                 = /drives/XXXX/root:/General/P1.0 Test Data/P1.0 Testing Events/
[DEBUG] thisItemId                                           = XXXX52XM
[DEBUG] thisItemParentPath                                   = /drives/XXXX/root:/General/P1.0 Test Data/P1.0 Testing Events
[DEBUG] thisItemFullPath                                     = /drives/XXXX/root:/General/P1.0 Test Data/P1.0 Testing Events/SyncTest
[DEBUG] 'item id' matches search 'id'                        = false
[DEBUG] 'parentReference id' matches search 'id'             = true
[DEBUG] 'thisItemParentPath' contains 'syncFolderChildPath'  = false
[DEBUG] 'thisItemParentPath' contains search 'id'            = false
[DEBUG] 'syncFolderChildPath' contains 'thisItemFullPath'    = false
[DEBUG] Change matches search criteria to apply
[DEBUG] The item we are syncing is a folder
[DEBUG] Attempting to calculate local filesystem path for XXXX and XXXX623B
[DEBUG] sync_list item to check: General/P1.0 Test Data/P1.0 Testing Events/SyncTest
[DEBUG] Evaluation against 'sync_list' for this path: General/P1.0 Test Data/P1.0 Testing Events/SyncTest
[DEBUG] [S]exclude           = false
[DEBUG] [S]exludeDirectMatch = false
[DEBUG] [S]excludeMatched    = false
[DEBUG] Evaluation against 'sync_list' entry: General/P1.0 Test Data/P1.0 Testing Events/*.ulg
[DEBUG] Evaluation against 'sync_list' entry: *.trc
[DEBUG] [F]exclude           = false
[DEBUG] [F]exludeDirectMatch = false
[DEBUG] [F]excludeMatched    = false
[DEBUG] Evaluation against 'sync_list' final result: EXCLUDED
Skipping item - excluded by sync_list config: General/P1.0 Test Data/P1.0 Testing Events/SyncTest
[DEBUG] Skipping OneDrive change as this is determined to be unwanted
[DEBUG] ------------------------------------------------------------------
[DEBUG] Processing change 19 of 206
[DEBUG] OneDrive Change: {"@odata.etag":"\"{XXXX},1\"","@odata.type":"#microsoft.graph.driveItem","cTag":"\"c:{XXXX},1\"","eTag":"\"{XXXX},1\"","file":{"hashes":{"quickXorHash":"XXXX"},"mimeType":"application\/octet-stream"},"fileSystemInfo":{"createdDateTime":"2024-08-22T11:32:58Z","lastModifiedDateTime":"2024-08-22T11:32:58Z"},"id":"XXXXCMOO","name":"17_24_14.ulg","parentReference":{"driveId":"XXXX","driveType":"documentLibrary","id":"XXXX52XM","path":"\/drives\/XXXX\/root:\/General\/P1.0 Test Data\/P1.0 Testing Events\/SyncTest","siteId":"XXXX"},"size":3088522}
[DEBUG] Not a OneDrive deleted item change
[DEBUG] defaultRootId                                        = XXXXLRRZ
[DEBUG] 'search id'                                          = XXXX623B
[DEBUG] id == defaultRootId                                  = false
[DEBUG] isItemRoot(item)                                     = false
[DEBUG] item['name'].str == 'root'                           = false
[DEBUG] singleDirectoryScope                                 = true
[DEBUG] 'parentReference id'                                 = XXXX52XM
[DEBUG] search criteria: syncFolderName                      = P1.0 Testing Events
[DEBUG] search criteria: syncFolderPath                      = /drives/XXXX/root:/General/P1.0 Test Data
[DEBUG] search criteria: syncFolderChildPath                 = /drives/XXXX/root:/General/P1.0 Test Data/P1.0 Testing Events/
[DEBUG] thisItemId                                           = XXXXCMOO
[DEBUG] thisItemParentPath                                   = /drives/XXXX/root:/General/P1.0 Test Data/P1.0 Testing Events/SyncTest
[DEBUG] thisItemFullPath                                     = /drives/XXXX/root:/General/P1.0 Test Data/P1.0 Testing Events/SyncTest/17_24_14.ulg
[DEBUG] 'item id' matches search 'id'                        = false
[DEBUG] 'parentReference id' matches search 'id'             = false
[DEBUG] 'thisItemParentPath' contains 'syncFolderChildPath'  = true
[DEBUG] 'thisItemParentPath' contains search 'id'            = false
[DEBUG] 'syncFolderChildPath' contains 'thisItemFullPath'    = false
[DEBUG] Change matches search criteria to apply
[DEBUG] sha256Hash is missing from XXXXCMOO
[DEBUG] Flagging as unwanted: find(item.parentId).length != 0
[DEBUG] Skipping OneDrive change as this is determined to be unwanted

It seems like the new directory gets flagged as unwanted, but I'm not sure why. Once it has been skipped, the new files also get skipped without being checked against the sync_list. The new .trc file is completely ignored.

Running with locally built v2.5.0-rc2 I get

DEBUG: sync_list item to check: General/P1.0 Test Data/P1.0 Testing Events/SyncTest
DEBUG: Evaluation against 'sync_list' for this path: General/P1.0 Test Data/P1.0 Testing Events/SyncTest
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: General/P1.0 Test Data/P1.0 Testing Events/*.ulg
DEBUG: Evaluation against 'sync_list' entry: *.trc
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: EXCLUDED
DEBUG: Skipping item - excluded by sync_list config: General/P1.0 Test Data/P1.0 Testing Events/SyncTest

No new files are mentioned.

Any help would be appreciated.

abraunegg Aug 22, 2024
Maintainer

FYI ..

Output for the section using v2.25.0:

This logout put is 100% not v2.5.0 client code, and 100% matches the current 'master' release of v2.4.25

abraunegg · 2024-08-22T20:22:57Z

abraunegg
Aug 22, 2024
Maintainer

@hskrieg
Please start with the basics. Please provide the output of:

onedrive --version
onedrive --display-config

./onedrive --confdir="~/.config/SharePoint_P1.0_Hybrid" --synchronize --resync -v+ -v+ --dry-run --single-directory "General/P1.0 Test Data/P1.0 Testing Events"

This command is not entirely correct, it can be simplified to the following when using v2.5.x client code:

./onedrive --confdir="~/.config/SharePoint_P1.0_Hybrid" -s --resync -vv --dry-run --single-directory "General/P1.0 Test Data/P1.0 Testing Events"

Running with locally built v2.5.0-rc2 I get

Please do not use RC2 .. however if you are using v2.5.x client version you must first:

Uninstall any v2.4.x client version and system package
Install v2.5.0 RC3 following the details listed in OneDrive Client Version v2.5.0-rc3 is now available for testing #2770

2 replies

hskrieg Aug 22, 2024
Author

> onedrive --version
onedrive v2.4.25-1+np3+5.1

> onedrive --confdir="~/.config/SharePoint_P1.0" --display-config
Configuration file successfully loaded
onedrive version                             = v2.4.25-1+np3+5.1
Config path                                  = /home/hskrieg/.config/SharePoint_P1.0
Config file found in config path             = true
Config option 'drive_id'                     = XXXX
Config option 'sync_dir'                     = /home/hskrieg/SharePoint_P1.0
Config option 'enable_logging'               = false
Config option 'log_dir'                      = /var/log/onedrive/
Config option 'disable_notifications'        = false
Config option 'min_notify_changes'           = 5
Config option 'skip_dir'                     = 
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 6
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = true
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = 
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = common
Config option 'user_agent'                   = 
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 1048576
Config option 'operation_timeout'            = 36000
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 600
Config option 'ip_protocol_version'          = 0
Config option 'sync_root_files'              = false
Selective sync 'sync_list' configured        = true
sync_list contents:
General/P1.0 Test Data/P1.0 Testing Events/*.ulg
*.trc
Config option 'sync_business_shared_folders' = false
Business Shared Folders configured           = false
Config option 'webhook_enabled'              = false

Running with 2.5.0-rc2 was an experiment. Once I get this working right, everyone will probably be using 2.4.25, at least until a later version is released on openSUSE.

hskrieg Aug 22, 2024
Author

Trying to reproduce with a smaller data set such that the logged output is < 30MB. I currently don't get anything in the log that even sees the new SyncTest directory, which I moved further down the tree.

onedrive --confdir="~/.config/SharePoint_P1.0_Hybrid" --synchronize --resync -v+ -v+ --dry-run --single-directory "General/P1.0 Test Data/P1.0 Testing Events/2019 Logs"

onedrive.log

The SyncTest files are in "General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg"

abraunegg · 2024-08-22T20:38:12Z

abraunegg
Aug 22, 2024
Maintainer

@hskrieg
OK .. so your sync list should include the full path you are trying to sync:

/General/P1.0 Test Data/P1.0 Testing Events/SyncTest/*.ulg
*.trc

Please read the v2.5.x documentation regarding using a 'sync_list' file to assist you:

https://github.com/abraunegg/onedrive/blob/onedrive-v2.5.0-release-candidate-3/docs/usage.md#performing-a-selective-synchronisation-via-sync_list-file

Running with 2.5.0-rc2 was an experiment. Once I get this working right, everyone will probably be using 2.4.25, at least until a later version is released on openSUSE.

For performance reasons, and bugs that have been fixed, it is highly recommended that you do not use v2.4.25 code and move to using RC3 until mid-September 2024 when v2.5.0 will be released.

1 reply

hskrieg Aug 22, 2024
Author

Is there a way to pick up ALL new ulg files regardless of what new directories they will be in? My first attempt was with just *.ulg. But the *.trc fails to pick up the new trc file in SyncTest, too.

abraunegg · 2024-08-23T01:46:00Z

abraunegg
Aug 23, 2024
Maintainer

@hskrieg
I have built a SharePoint site to replicate what you are trying to do:

This is the local application configuration:

'sync_list'

# Include all .ulg files in this directory
/General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/*.ulg
# Include .trc files anywhere
*.trc

application version v2.4.25

alex@ubuntu-24-04:~/onedrive$ ./onedrive --synchronize -v --resync
Using 'user' Config Dir: /home/alex/.config/onedrive
Using 'system' Config Dir: /etc/onedrive
Configuration file successfully loaded

The use of --resync will remove your local 'onedrive' client state, thus no record will exist regarding your current 'sync status'
This has the potential to overwrite local versions of files with potentially older versions downloaded from OneDrive which can lead to data loss
If in-doubt, backup your local data first before proceeding with --resync

Are you sure you wish to proceed with --resync? [Y/N] y

Deleting the saved application sync status ...
Checking Application Version ...
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Using Curl defaults for all HTTP operations
Opening the item database ...
All operations will be performed in: /home/alex/OneDrive
Application version: v2.4.25-15-gadad80c
Account Type: documentLibrary
Default Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6qwCLeArSOS6Q6k1P7IEDC
Default Root ID: 01PTBOWVV6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 27471472596678
Fetching details for OneDrive Root
OneDrive Root does not exist in the database. We need to add it.
Added OneDrive Root to the local database
Initializing the Synchronization Engine ...
Syncing changes and items from OneDrive ...
Applying changes of Path ID: 01PTBOWVV6Y2GOVW7725BZO354PWSELRRZ
Updated Remaining Free Space: 27471472596678
Processing 15 OneDrive items to ensure consistent local state due to sync_list being used
Skipping item - excluded by sync_list config: ./Other Data
Creating local directory: ./General
Creating local directory: General/P1.0 Test Data
Creating local directory: General/P1.0 Test Data/P1.0 Testing Events
Creating local directory: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs
Creating local directory: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest
Downloading file General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 
Downloading  95% |oooooooooooooooooooooooooooooooooooooo  |   ETA   00:00:02                                                                                                                   
done.
Downloading file General/P1.0 Test Data/55_44_33_.trc ... done.
Performing a database consistency and integrity check on locally stored data ... 
Uploading differences of ~/OneDrive
Processing .
The directory has not changed
Processing General
The directory has not changed
Processing General/P1.0 Test Data
The directory has not changed
Processing General/P1.0 Test Data/P1.0 Testing Events
The directory has not changed
Processing General/P1.0 Test Data/P1.0 Testing Events/2019 Logs
The directory has not changed
Processing General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest
The directory has not changed
Processing General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg
The file has not changed
Processing General/P1.0 Test Data/55_44_33_.trc
The file has not changed
Uploading new items of ~/OneDrive
Applying changes of Path ID: 01PTBOWVV6Y2GOVW7725BZO354PWSELRRZ
Updated Remaining Free Space: 27471472596678
Processing 15 OneDrive items to ensure consistent local state due to sync_list being used
Skipping item - excluded by sync_list config: ./Other Data
Sync with OneDrive is complete
alex@ubuntu-24-04:~/onedrive$

application version v2.5.0 RC3

alex@ubuntu-24-04:~/onedrive-pr2769$ ./onedrive --synchronize -v --resync
D-Bus message bus daemon is available; GUI notifications are now enabled
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application config and state data: /home/alex/.config/onedrive

DEPRECIATION WARNING: --synchronize has been depreciated in favour of --sync or -s
DEPRECIATION WARNING: Depreciated commands will be removed in a future release.


The usage of --resync will delete your local 'onedrive' client state, thus no record of your current 'sync status' will exist.
This has the potential to overwrite local versions of files with perhaps older versions of documents downloaded from OneDrive, resulting in local data loss.
If in doubt, backup your local data before using --resync

Are you sure you wish to proceed with --resync? [Y/N] y

The item database is incompatible, re-creating database table structures
Deleting the saved application sync status ...
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
Application Version:  onedrive v2.5.0-rc3-24-g2d40551
Account Type:         documentLibrary
Default Drive ID:     b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6qwCLeArSOS6Q6k1P7IEDC
Default Root ID:      01PTBOWVV6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 25584.80 GB (27471472596678 bytes)
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /home/alex/OneDrive
Fetching /delta response from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6qwCLeArSOS6Q6k1P7IEDC
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 15
Skipping item - excluded by sync_list config: ./Other Data
Skipping item - excluded by sync_list config: Other Data/awerqwerqwer
Skipping item - excluded by sync_list config: Other Data/awerqwerqwer/file0.data
Skipping item - excluded by sync_list config: Other Data/awerqwerqwer/file2.data
Skipping item - excluded by sync_list config: Other Data/awerqwerqwer/file1.data
Skipping item - excluded by sync_list config: Other Data/awerqwerqwer/file4.data
Skipping item - excluded by sync_list config: Other Data/awerqwerqwer/file3.data
Finished processing /delta JSON response from the OneDrive API
Processing 7 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Creating local directory: ./General
Creating local directory: ./General/P1.0 Test Data
Creating local directory: ./General/P1.0 Test Data/P1.0 Testing Events
Creating local directory: ./General/P1.0 Test Data/P1.0 Testing Events/2019 Logs
Creating local directory: ./General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest
Number of items to download from Microsoft OneDrive: 2
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 0%   |  ETA    --:--:--
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 5%   |  ETA    00:00:00
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 10%  |  ETA    00:00:00
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 15%  |  ETA    00:00:00
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 20%  |  ETA    00:00:00
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 25%  |  ETA    00:00:00
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 30%  |  ETA    00:00:00
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 35%  |  ETA    00:00:00
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 40%  |  ETA    00:00:00
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 45%  |  ETA    00:00:00
Downloading file: General/P1.0 Test Data/55_44_33_.trc ... done
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 55%  |  ETA    00:00:00
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 60%  |  ETA    00:00:00
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 65%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 70%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 75%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 80%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 85%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 90%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 95%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 100% | DONE in 00:00:01
Downloading file: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... done
Performing a database consistency and integrity check on locally stored data
Processing DB entries for this Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6qwCLeArSOS6Q6k1P7IEDC
Processing: ~/OneDrive
The directory has not changed
Processing: General
The directory has not changed
Processing: General/P1.0 Test Data
The directory has not changed
Processing: General/P1.0 Test Data/P1.0 Testing Events
The directory has not changed
Processing: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs
The directory has not changed
Processing: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest
The directory has not changed
Processing: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg
The file has not changed
Processing: General/P1.0 Test Data/55_44_33_.trc
The file has not changed
Scanning the local file system '~/OneDrive' for new data to upload
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching /delta response from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6qwCLeArSOS6Q6k1P7IEDC
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 0
Finished processing /delta JSON response from the OneDrive API
No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive

Sync with Microsoft OneDrive is complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

In both instances:

The .ulg file in the required directory is downloaded
The .trc file in any location is downloaded

alex@ubuntu-24-04:~/OneDrive$ tree
.
└── General
    └── P1.0 Test Data
        ├── 55_44_33_.trc
        └── P1.0 Testing Events
            └── 2019 Logs
                └── SyncTest
                    └── 17_24_14.ulg

6 directories, 2 files
alex@ubuntu-24-04:~/OneDrive$

2 replies

hskrieg Aug 23, 2024
Author

Thank you for taking the time to dig through this. I'm still having issues. The files themselves are being tagged as "included for sync", but the SyncTest directory is not.
(Using v2.5.0-rc3)
Changed sync_list to
*.ulg
*.trc
trying to get any of these files anywhere.

~/onedrive/onedrive --confdir="~/.config/SharePoint_P1.0" --synchronize --resync -vv | tee onedrive.log

DEBUG: sync_dir: runtimeSyncDirectory set to: /home/hskrieg/SharePoint_P1.0
DEBUG: --------------- Application Runtime Configuration ---------------
DEBUG: Application version                          = onedrive v2.5.0-rc3
DEBUG: Compiled with                                = LDC 2098
DEBUG: User Application Config path                 = /home/hskrieg/.config/SharePoint_P1.0
DEBUG: System Application Config path               = /etc/onedrive
DEBUG: Applicable Application 'config' location     = /home/hskrieg/.config/SharePoint_P1.0/config
DEBUG: Configuration file found in config location  = true - using 'config' file values to override application defaults
DEBUG: Applicable 'sync_list' location              = /home/hskrieg/.config/SharePoint_P1.0/sync_list
DEBUG: Applicable 'items.sqlite3' location          = /home/hskrieg/.config/SharePoint_P1.0/items.sqlite3
DEBUG: Config option 'drive_id'                     = b!uxuyoD2U_EOCvGWIJyi2s6jGQMlwFlVGkkEIuG9gSFCM_QTjDaZFSavZKVk6Oppk
DEBUG: Config option 'sync_dir'                     = ~/SharePoint_P1.0
DEBUG: Config option 'enable_logging'               = false
DEBUG: Config option 'log_dir'                      = /var/log/onedrive
DEBUG: Config option 'disable_notifications'        = false
DEBUG: Config option 'skip_dir'                     = 
DEBUG: Config option 'skip_dir_strict_match'        = false
DEBUG: Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial
DEBUG: Config option 'skip_dotfiles'                = false
DEBUG: Config option 'skip_symlinks'                = false
DEBUG: Config option 'monitor_interval'             = 300
DEBUG: Config option 'monitor_log_frequency'        = 12
DEBUG: Config option 'monitor_fullscan_frequency'   = 12
DEBUG: Config option 'read_only_auth_scope'         = false
DEBUG: Config option 'dry_run'                      = false
DEBUG: Config option 'upload_only'                  = false
DEBUG: Config option 'download_only'                = true
DEBUG: Config option 'local_first'                  = false
DEBUG: Config option 'check_nosync'                 = false
DEBUG: Config option 'check_nomount'                = false
DEBUG: Config option 'resync'                       = true
DEBUG: Config option 'resync_auth'                  = false
DEBUG: Config option 'cleanup_local_files'          = false
DEBUG: Config option 'classify_as_big_delete'       = 1000
DEBUG: Config option 'disable_upload_validation'    = false
DEBUG: Config option 'disable_download_validation'  = false
DEBUG: Config option 'bypass_data_preservation'     = false
DEBUG: Config option 'no_remote_delete'             = false
DEBUG: Config option 'remove_source_files'          = false
DEBUG: Config option 'sync_dir_permissions'         = 700
DEBUG: Config option 'sync_file_permissions'        = 600
DEBUG: Config option 'space_reservation'            = 52428800
DEBUG: Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
DEBUG: Config option 'azure_ad_endpoint'            = 
DEBUG: Config option 'azure_tenant_id'              = 
DEBUG: Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.0-rc3
DEBUG: Config option 'force_http_11'                = false
DEBUG: Config option 'debug_https'                  = false
DEBUG: Config option 'rate_limit'                   = 1048576
DEBUG: Config option 'operation_timeout'            = 36000
DEBUG: Config option 'dns_timeout'                  = 60
DEBUG: Config option 'connect_timeout'              = 10
DEBUG: Config option 'data_timeout'                 = 60
DEBUG: Config option 'ip_protocol_version'          = 0
DEBUG: Config option 'threads'                      = 8
DEBUG: Compile time option --enable-notifications   = false
DEBUG: 
DEBUG: Selective sync 'sync_list' configured        = true
DEBUG: sync_list config option 'sync_root_files'    = false
DEBUG: sync_list contents:
DEBUG: *.ulg
DEBUG: *.trc
DEBUG: 
DEBUG: Config option 'sync_business_shared_items'   = false
DEBUG: 
DEBUG: Config option 'webhook_enabled'              = false
DEBUG: 
DEBUG: --------------------DEVELOPER_OPTIONS----------------------------
DEBUG: Config option 'force_children_scan'          = false
DEBUG: 
DEBUG: -----------------------------------------------------------------

Searching for SyncTest:

DEBUG: defaultRootId                                        = 014E7E2IV6Y2GOVW7725BZO354PWSELRRZ
DEBUG: 'search id'                                          = 014E7E2IUH7FCCNPUO6JBLHXUI2ICR52XM
DEBUG: id == defaultRootId                                  = false
DEBUG: isItemRoot(onedriveJSONItem)                         = false
DEBUG: onedriveJSONItem['name'].str == 'root'               = false
DEBUG: itemHasParentReferenceId                             = true
DEBUG: sync_list item to check: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest
DEBUG: Evaluation against 'sync_list' for this path: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: *.ulg
DEBUG: Evaluation against 'sync_list' entry: *.trc
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: EXCLUDED
DEBUG: Skipping item - excluded by sync_list config: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest
DEBUG: Initial JSON item processing time: 126 μs and 7 hnsecs
DEBUG: ------------------------------------------------------------------
DEBUG: Processing OneDrive Item 20 of 209 from API Response Bundle 33
DEBUG: This item is not a OneDrive deletion change
DEBUG: defaultRootId                                        = 014E7E2IV6Y2GOVW7725BZO354PWSELRRZ
DEBUG: 'search id'                                          = 014E7E2IRN5RVTAAUDLVDJHL75IP57CMOO
DEBUG: id == defaultRootId                                  = false
DEBUG: isItemRoot(onedriveJSONItem)                         = false
DEBUG: onedriveJSONItem['name'].str == 'root'               = false
DEBUG: itemHasParentReferenceId                             = true
DEBUG: skip_file item to check (file name only - parent path not in database): /17_24_14.ulg
DEBUG: skip_file evaluation for: /17_24_14.ulg
DEBUG: skip_file evaluation for: /17_24_14.ulg
DEBUG: Result: false
DEBUG: sync_list item to check: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg
DEBUG: Evaluation against 'sync_list' for this path: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg
DEBUG: [S]exclude           = false
DEBUG: [S]exludeDirectMatch = false
DEBUG: [S]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' entry: *.ulg
DEBUG: Evaluation against 'sync_list' result: wildcard pattern match
DEBUG: Evaluation against 'sync_list' entry: *.trc
DEBUG: [F]exclude           = false
DEBUG: [F]exludeDirectMatch = false
DEBUG: [F]excludeMatched    = false
DEBUG: Evaluation against 'sync_list' final result: included for sync
DEBUG: Adding this Raw JSON OneDrive Item to jsonItemsToProcess array for further processing
DEBUG: Initial JSON item processing time: 155 μs and 3 hnsecs

But down in the Processing portion:

DEBUG: ------------------------------------------------------------------
DEBUG: Processing OneDrive JSON item 427 of 433 as part of JSON Item Batch 2 of 2
DEBUG: Raw JSON OneDrive Item (Batched Item): {.... /root:\/General\/P1.0 Test Data\/P1.0 Testing Events\/2019 Logs\/SyncTest","size":3088522}
DEBUG: Flagging as unwanted: thisItemDriveId (), thisItemParentId (014E7E2IUH7FCCNPUO6JBLHXUI2ICR52XM) not in local database
DEBUG: Reason: Parent ID is not in the DB .. 
DEBUG: Skipping OneDrive change as this is determined to be unwanted
DEBUG: Batched JSON item processing time: 60 μs

Is this due to the fact that the SyncTest directory is new? But your log shows that it creates it without issue.

hskrieg Aug 23, 2024
Author

The initial download was done using a LOT of skip_file entries to pare down to mostly the ulg and trc files, so all of the directories existed, except for the new SyncTest.

abraunegg · 2024-08-23T19:50:54Z

abraunegg
Aug 23, 2024
Maintainer

@hskrieg

DEBUG: sync_list contents:
DEBUG: *.ulg
DEBUG: *.trc

In deleting all local content, and running ./onedrive --synchronize -v --resync I agree with you that the files that are supposed to be downloaded are not. This is a gap in v2.5.0 RC3 (thus a bug)

Testing v2.4.25, the exact same issue occurs, thus is not a regression.

Will resolve this with RC3 in the next few days.

1 reply

abraunegg Aug 23, 2024
Maintainer

I have self raised #2784 to track this issue

abraunegg · 2024-08-24T02:16:12Z

abraunegg
Aug 24, 2024
Maintainer

@hskrieg
I need to run through some more exhaustive scenarios before committing this to RC3, however:

alex@ubuntu-24-04:~/onedrive-pr2769$ ./onedrive --synchronize -v --resync --resync-auth
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application config and state data: /home/alex/.config/onedrive

DEPRECIATION WARNING: --synchronize has been depreciated in favour of --sync or -s
DEPRECIATION WARNING: Depreciated commands will be removed in a future release.

Deleting the saved application sync status ...
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
Application Version:  onedrive v2.5.0-rc3-24-g2d40551
Account Type:         documentLibrary
Default Drive ID:     b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6qwCLeArSOS6Q6k1P7IEDC
Default Root ID:      01PTBOWVV6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 25584.80 GB (27471466617656 bytes)
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /home/alex/OneDrive
Fetching /delta response from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6qwCLeArSOS6Q6k1P7IEDC
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 19
Skipping item - excluded by sync_list config: ./Other Data
Skipping item - excluded by sync_list config: Other Data/awerqwerqwer
Skipping item - excluded by sync_list config: Other Data/awerqwerqwer/file0.data
Skipping item - excluded by sync_list config: Other Data/awerqwerqwer/file2.data
Skipping item - excluded by sync_list config: Other Data/awerqwerqwer/file1.data
Skipping item - excluded by sync_list config: Other Data/awerqwerqwer/file4.data
Skipping item - excluded by sync_list config: Other Data/awerqwerqwer/file3.data
Skipping item - excluded by sync_list config: ./General
Skipping item - excluded by sync_list config: General/P1.0 Test Data
Skipping item - excluded by sync_list config: General/P1.0 Test Data/P1.0 Testing Events
Skipping item - excluded by sync_list config: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs
Skipping item - excluded by sync_list config: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest
Including file - included by sync_list config: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg
Parental Path structure needs to be created to support included file: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest
Including file - included by sync_list config: General/P1.0 Test Data/55_44_33_.trc
Skipping item - excluded by sync_list config: General/P1.0 Test Data/Another Folder
Including file - included by sync_list config: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg
Parental Path structure needs to be created to support included file: General/P1.0 Test Data/Another Folder
Skipping item - excluded by sync_list config: ./read_write-fedora.txt
Skipping item - excluded by sync_list config: ./read_write.txt
Finished processing /delta JSON response from the OneDrive API
Processing 3 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Number of items to download from Microsoft OneDrive: 3
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 0%   |  ETA    --:--:--
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 0%   |  ETA    --:--:--
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 5%   |  ETA    00:00:19
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 10%  |  ETA    00:00:09
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 15%  |  ETA    00:00:06
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 5%   |  ETA    00:00:19
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 20%  |  ETA    00:00:04
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 25%  |  ETA    00:00:03
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 30%  |  ETA    00:00:03
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 15%  |  ETA    00:00:09
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 35%  |  ETA    00:00:02
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 20%  |  ETA    00:00:06
Downloading file: General/P1.0 Test Data/55_44_33_.trc ... done
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 25%  |  ETA    00:00:04
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 30%  |  ETA    00:00:03
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 40%  |  ETA    00:00:02
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 35%  |  ETA    00:00:05
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 40%  |  ETA    00:00:04
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 45%  |  ETA    00:00:03
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 45%  |  ETA    00:00:03
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 50%  |  ETA    00:00:03
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 50%  |  ETA    00:00:02
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 55%  |  ETA    00:00:02
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 55%  |  ETA    00:00:02
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 60%  |  ETA    00:00:02
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 65%  |  ETA    00:00:02
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 60%  |  ETA    00:00:02
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 70%  |  ETA    00:00:02
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 65%  |  ETA    00:00:02
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 70%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 75%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 75%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 80%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 80%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 85%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 90%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 95%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 90%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... 100% | DONE in 00:00:02
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 95%  |  ETA    00:00:01
Downloading: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... 100% | DONE in 00:00:02
Downloading file: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... done
Downloading file: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg ... done
Performing a database consistency and integrity check on locally stored data
Processing DB entries for this Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6qwCLeArSOS6Q6k1P7IEDC
Processing: ~/OneDrive
The directory has not changed
Processing: General
The directory has not changed
Processing: General/P1.0 Test Data
The directory has not changed
Processing: General/P1.0 Test Data/P1.0 Testing Events
The directory has not changed
Processing: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs
The directory has not changed
Processing: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest
The directory has not changed
Processing: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg
The file has not changed
Processing: General/P1.0 Test Data/Another Folder
The directory has not changed
Processing: General/P1.0 Test Data/Another Folder/aa_dd_cc.ulg
The file has not changed
Processing: General/P1.0 Test Data/55_44_33_.trc
The file has not changed
Scanning the local file system '~/OneDrive' for new data to upload
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching /delta response from the OneDrive API for Drive ID: b!knrQ8K6YIkqi-IrggfGWyi6Bu-HVzjxMpA7qvZX7Gy6qwCLeArSOS6Q6k1P7IEDC
Processing API Response Bundle: 1 - Quantity of 'changes|items' in this bundle to process: 0
Finished processing /delta JSON response from the OneDrive API
No changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive

Sync with Microsoft OneDrive is complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

0 replies

abraunegg · 2024-08-24T23:40:49Z

abraunegg
Aug 24, 2024
Maintainer

@hskrieg
I have gone through some further validation testing, and this fix has now been added to RC3.

Please use these instructions to rebuild RC3 correctly:

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/2769/head:pr2769
git checkout pr2769
./configure --enable-debug; make clean; make;

To run the PR, you need to run the client from the PR build directory:

./onedrive <any options needed>

When running this updated PR, your client version should be: onedrive v2.5.0-rc3-25-g7042f53 or greater.

If you can please test and confirm that would be greatly appreciated.

2 replies

hskrieg Aug 26, 2024
Author

Still with the full data direcories I ran:
./onedrive --confdir="~/.config/SharePoint_P1.0" --display-config

Reading configuration file: /home/hskrieg/.config/SharePoint_P1.0/config
Configuration file successfully loaded
Application version                          = onedrive v2.5.0-rc3-36-gf527eca
Compiled with                                = LDC 2098
User Application Config path                 = /home/hskrieg/.config/SharePoint_P1.0
System Application Config path               = /etc/onedrive
Applicable Application 'config' location     = /home/hskrieg/.config/SharePoint_P1.0/config
Configuration file found in config location  = true - using 'config' file values to override application defaults
Applicable 'sync_list' location              = /home/hskrieg/.config/SharePoint_P1.0/sync_list
Applicable 'items.sqlite3' location          = /home/hskrieg/.config/SharePoint_P1.0/items.sqlite3
Config option 'drive_id'                     = b!
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial|~*|.~*|*.tmp|*.mov|*.mp4|*.jpg|*.jpeg|*.gif|*.png|*.nef|*.zip|*.tgz|*.pdf|*.doc|*.docx|*.xls|*.xlsx|*.pptx|*.ods|*.csv|*.py20*|Icon*|*.mdat
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 12
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = true
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = 
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.0-rc3-36-gf527eca
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 2097152
Config option 'operation_timeout'            = 36000
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 0
Config option 'threads'                      = 8
Compile time option --enable-notifications   = false

Selective sync 'sync_list' configured        = true
sync_list config option 'sync_root_files'    = false
sync_list contents:
*.ulg
*.trc
Config option 'sync_business_shared_items'   = false

Config option 'webhook_enabled'              = false

./onedrive --confdir="~/.config/SharePoint_P1.0" --sync --resync -v --dry-run --single-directory "General/P1.0 Test Data/P1.0 Testing Events" | tee onedrive.log

Snippets, because the log is still large:

Reading configuration file: /home/hskrieg/.config/SharePoint_P1.0/config
Configuration file successfully loaded
Using 'user' configuration path for application config and state data: /home/hskrieg/.config/SharePoint_P1.0
DRY-RUN Configured. Output below shows what 'would' have occurred.
DRY-RUN: No database copy created for --dry-run due to --resync also being used

The usage of --resync will delete your local 'onedrive' client state, thus no record of your current 'sync status' will exist.
This has the potential to overwrite local versions of files with perhaps older versions of documents downloaded from OneDrive, resulting in local data loss.
If in doubt, backup your local data before using --resync

Are you sure you wish to proceed with --resync? [Y/N] 
DRY RUN: Not removing hash files as --dry-run has been used
DRY RUN: Not creating backup config file as --dry-run has been used
DRY RUN: Not updating hash files as --dry-run has been used
Deleting the saved application sync status ...
DRY RUN: Not removing the saved application sync status
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
User Configured Rate Limit: 2097152
Quota information is restricted or not available for this drive.
ERROR: OneDrive quota information is being restricted. Please fix by speaking to your OneDrive / Office 365 Administrator.
Application Version:  onedrive v2.5.0-rc3-36-gf527eca
Account Type:         documentLibrary
Default Drive ID:     b!
Default Root ID:      014E7E2IV6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space:       Not Available
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /home/hskrieg/SharePoint_P1.0
The OneDrive Client was asked to search for this directory online and create it if it's not located: ./General/P1.0 Test Data/P1.0 Testing Events
Syncing changes from this selected path: General/P1.0 Test Data/P1.0 Testing Events
Generating a /delta response from the OneDrive API for Drive ID: b!uxuyoD2U_EOCvGWIJyi2s6jGQMlwFlVGkkEIuG9gSFCM_QTjDaZFSavZKVk6Oppk
Adding OneDrive root details for processing
Adding OneDrive folder details for processing
Adding 155 OneDrive items for processing from the OneDrive 'root' Folder
Skipping path - excluded by sync_list config: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs
Skipping path - excluded by sync_list config: General/P1.0 Test Data/P1.0 Testing Events/2020 Logs

... (All other directories directory P1.0 Testing Events)

Finished processing self generated /delta JSON response from the OneDrive API
Processing 1 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Skipping path - excluded by sync_list config: General/P1.0 Test Data/P1.0 Testing Events
Flagging item for local delete as item exists in database: General/P1.0 Test Data/P1.0 Testing Events
Items to potentially delete locally: 1
Skipping local deletion activity as --download-only has been used
Skipping local deletion for directory General/P1.0 Test Data/P1.0 Testing Events
Performing a database consistency and integrity check on locally stored data
Processing DB entries for this Drive ID: b!
Quota information is restricted or not available for this drive.

Sync with Microsoft OneDrive is complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

No directories under the 2019 Logs or 2020 Logs are looked at. SyncTest is not mentioned at all. The same with the -vv flag.

hskrieg Aug 26, 2024
Author

If, however, I drop the --single-directory flag, I am now getting the .ulg and .trc files downloaded! It just takes a little longer to go through everything else.
DEBUG: Downloading file: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/17_24_14.ulg ... done
DEBUG: Downloading file: General/P1.0 Test Data/P1.0 Testing Events/2019 Logs/SyncTest/9-26-22.trc ... done

Thank you!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can one set up sync to ONLY download files of a given type? #2783

{{title}}

Replies: 8 comments 10 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Can one set up sync to ONLY download files of a given type? #2783

hskrieg Aug 21, 2024

Replies: 8 comments · 10 replies

hskrieg Aug 21, 2024 Author

abraunegg Aug 21, 2024 Maintainer

hskrieg Aug 22, 2024 Author

abraunegg Aug 22, 2024 Maintainer

abraunegg Aug 22, 2024 Maintainer

hskrieg Aug 22, 2024 Author

hskrieg Aug 22, 2024 Author

abraunegg Aug 22, 2024 Maintainer

hskrieg Aug 22, 2024 Author

abraunegg Aug 23, 2024 Maintainer

'sync_list'

application version v2.4.25

application version v2.5.0 RC3

hskrieg Aug 23, 2024 Author

hskrieg Aug 23, 2024 Author

abraunegg Aug 23, 2024 Maintainer

abraunegg Aug 23, 2024 Maintainer

abraunegg Aug 24, 2024 Maintainer

abraunegg Aug 24, 2024 Maintainer

hskrieg Aug 26, 2024 Author

hskrieg Aug 26, 2024 Author

hskrieg
Aug 21, 2024

Replies: 8 comments 10 replies

hskrieg
Aug 21, 2024
Author

abraunegg
Aug 21, 2024
Maintainer

hskrieg Aug 22, 2024
Author

abraunegg Aug 22, 2024
Maintainer

abraunegg
Aug 22, 2024
Maintainer

hskrieg Aug 22, 2024
Author

hskrieg Aug 22, 2024
Author

abraunegg
Aug 22, 2024
Maintainer

hskrieg Aug 22, 2024
Author

abraunegg
Aug 23, 2024
Maintainer

hskrieg Aug 23, 2024
Author

hskrieg Aug 23, 2024
Author

abraunegg
Aug 23, 2024
Maintainer

abraunegg Aug 23, 2024
Maintainer

abraunegg
Aug 24, 2024
Maintainer

abraunegg
Aug 24, 2024
Maintainer

hskrieg Aug 26, 2024
Author

hskrieg Aug 26, 2024
Author