Sync Pending - Windows 10

[afzl-wtu]

This Sync Pending status is very irritating. You can see folder is uploaded to server but it is still showing me it is pending in status.
Also please clean those garbage files with pending status. They keep getting focus. PLease remove .sync_b39e158b5df3.db and other log files in director or do not show pending status.

[azukaar]

#4201

[dirrgang]

While possibly related to #4201, it would appear to be a slightly different manifestation. I have the same issue: Files are synchronised properly (as far as I can tell), but all files are shown as sync pending.

Windows 11, Nextcloud Desktop-Client Version 3.4.2 (Windows).
I have uploaded the debug archive.

[dirrgang]

Issue persists with client version 3.4.3

[github-actions]

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

[dirrgang]

I managed to fix it by removing Nextcloud and all Nextcloud files located in AppData as well as the sync folder.

[pzwahlen]

Hi devs!

Is there anything we can do to help figuring out what is happening here ? I have this problem on new client installations so I am not sure it depends on old files in appdata{local,roaming}.

I am running 3.4.4 on Win10 21H2 against the official docker image for 23.0.3.

By playing with "Always keep on this device" and then "Free up space", some files would go back to the normal "Available when online" (cloud status icon) and some others would go back to "Sync pending". For those, it would be interesting to know if space is actually freed or not.

Happy to help with debug here and "keep up with the great work!"

[pzwahlen]

A quick update to let everyone know that 3.5RC3 doesn't help with this problem.

[pzwahlen]

I think this bug is the same as #4383, except I can see it with many other filetypes. It also seems identical to #3301 which was automatically closed because of no response.

[virtaj]

Having same issue with Windows 11. Desktop client 3.4.4 and Nextcloud 23.0.3 on docker.

[CoYoNq]

Same issue here, but only with files which size is < 5kb and VFS enabled.
IE, i can create a file (a simple txt with only some chars on it), get it synced, but when i sent it to server, sync state on file still "pending". It only happends with files less than 5kb size. If i add some bytes to the file, it ends getting the right state!
This happends with any kind of file.
I was trying to track this error but with no luck (check full logs of server and client, but no errors found)
Everything performs great (lots of files syncing with no issue, only this little ones fails)
My setup:
Self hosted Nextcloud Hub II (23.0.3) server
20 Win 10 clients (with 3.4.4 version) in an exclusive LAN
About 270gb of files (with any kind of files)

[pzwahlen]

Now that I read this, I think you are right and this happens on small files. Is there a desktop debug procedure one can follow in order to better understand what is happening ?

[dnwhoop02]

Same issue here, and it definitely only happens with small files. In my case there are about 1,600 files under 5KB that stay in the "waiting to sync" state. These are files that are on the local hard drive, and at some point I told the client to "free up space". The client freed up all of the bigger files, but left these on the hard drive stuck in this state. The client periodically tries to sync the 1,600 files again, which is slightly annoying, but the bigger problem with this is that every time I make real changes to other files, or create new files (triggering a sync), the client tries to sync the 1,600 small files first, then it finally gets to the one or two files that actually needed to be sync'd. I've found that if I right click on these small files and select "Always keep on this device", the status changes to solid green checkmark, then I can right-click them again and clear the checkmark next to "Always keep on this device", and the status changes to green/white checkmark (as it should), and then the sync client will stop worrying about that particular file. I just found out that I can open the Nextcloud folder and type size:tiny in the search box and it'll find all the files under 15K, so I can try the "Always keep..." workaround on all of them...

[pzwahlen]

Hi,

Now that 3.5.0 is out it would be great to tackle this issue. In order to validate and confirm the link with file size, I have done the following.

Wrote a simple Powershell script that writes files with random content. File size is incremented from 0 to 8k in steps of 8 bytes and the file name is the actual content size written in HEX

$outDir="$PSScriptRoot\nc4226";
if (!(Test-Path -path $outDir)) {new-item -path $outDir -itemtype directory}

for ($i=0; $i -le 8192; $i=$i+8 ) {
  $fSize = $i;
  $fName = $i.ToString("X4");
  $fContent = [Byte[]]::new($fSize);
  $rng = [System.Random]::new();
  $rng.NextBytes($fContent);
  [System.IO.File]::WriteAllBytes("$outDir\$fName", $fContent);
}

You can download a zip archive of the output here: https://cloud-dev.lesz.ch/s/BYT4ktYnqKAkXTL

When I drop this folder within the NC folder (with Virtual Filesystem enabled) and wait for the initial sync, the folder appears as locally available, as expected:

All files within the folder are also locally available, as expected:

Now if I "Free up space" on the folder, I can see that the empty file gets the correct status but all larger files get the wrong "sync in progress" status:

I can easily sort the folder by file size and we can see that all files below 4k get the wrong status and all files >4k are showing the correct status:

Also, any sync activity from this point will trigger a resync of all the file <4k in size (except the empty file).

[CoYoNq]

Any news regarding this? New Win client (3.5.1) still showing this bug.
I still trying to track this issue, but can't find any relevant info in server or client log.
The impact in global performance (all machines in my scenario are VMs, so, resources can be easylly monitoreable) of a continuous trigger to sync this tiny files is really high. NC server and clients are affected by this "load spikes".

[andreaslink-de]

I just want to confirm, I have the exact same problem under Windows 10, running newest Win Client 3.5.1. In my case it is related to all my files <=4 kB and it is very annoying, especially when there is a git or subversion folder included, I had more than 10k files continuously being synced in a loop generating quite some load on the system.

My workaround was now to fully delete the connection, set it up again and do not mark anything with "Always keep on this device". Then I can at least work without always pausing the sync.

Looking very much for a solution.

[pzwahlen]

Dear @camilasan @mgallien @allexzander @matthieugallien

According to the repo history you seem to be the devs making changes to the Windows "vfs" and "cfapi" code and therefore you might want to have a look at this bug which (I think) is affecting the entire userbase (Windows+VFS).

I have created a simple reproducer in the issue thread.

Also, I tried to find places where the code would depend on a 4k buffer/size limit and I could only find those two:

desktop/shell_integration/windows/NCUtil/CommunicationSocket.cpp

Line 25 in 4efe00f

#define DEFAULT_BUFLEN 4096

desktop/src/gui/folderwatcher_win.cpp

Line 61 in 8f7646d

const size_t fileNameBufferSize = 4096;

Could one of them be part of the issue ?

Cheers!

[pzwahlen]

I have just updated my client to 3.5.2 and the problem is still there.

I don't know yet if it will help, but I am attaching 2 debug log files. These where generated by doing a "Free up local space", first on a <4k file (NOT.txt) and then a >4k (OK.txt) file

free_NOK.txt
free_OK.txt

I couldn't spot an obvious error/difference but expert eyes are needed here.

[Dremor]

Interesting note that may or may no be related: even the excluded files get a "pending sync" status icon. Disabling VFS give the correct behaviour.

[pzwahlen]

I'm just giving a gentle bump to this issue. Could someone add it to the "clients team" board here: https://github.com/orgs/nextcloud/projects/48

This might give it some additional visibility.

[mgallien]

to be honest, I am working on this but because of poor documentation from Microsoft I am struggling to find the exact wrong API call within the desktop client code
but be sure that this is known and not forgotten

[pzwahlen]

Thx @mgallien, knowing that someone knowledgeable is working on this has made my day! I think we all appreciate the hard work here.

Cheers!

[BasicUser22]

Hello,
I've got the same problem on about 53000 files.
Nextcloud version

client : 3.5.2
server : 24.0.3
Operating system and version

client : windows 10.0.19044
server : Debian 11
Apache : Apache/2.4.54 (Debian)
PHP version : PHP 8.1.8 (cli) (built: Jul 11 2022 08:55:24) (NTS)

The issue you are facing : I have to keep some files on my desktop because they can’t be “Free up”

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it :
I don’t know actually how to create the problem
But, if I take a problematic file and copy it in a new folder, the problem remains
In the sync.log file (AppData/Roaming/Nextcloud) :

• the initial copy leaves the following trace => the file is stored on the desktop and synchronized

timestamp duration file instruction dir modtime etag size fileId status errorString http result code other size other modtime X-Request-ID 18:09:22 <file> 8 1 1659990112 5e7d27e78bcf2bd1c22a1ee0999d8adc 1682 00452258oc6lwp0up92s 4 201 0 0 b503a6c9-70ae-4072-9620-80484283c7db

• when I try to free up the space, I’ve got the following trace :
  [ debug default C:\Users\sysadmin\AppData\Local\Temp\2\windows-10816\client-building\desktop\src\libsync\propagateupload.cpp:300 ] [ OCC::PropagateUploadFileCommon::slotComputeContentChecksum ]: Trying to compute the checksum of the file 2022-08-09 20:09:22:925 [ debug default C:\Users\sysadmin\AppData\Local\Temp\2\windows-10816\client-building\desktop\src\libsync\propagateupload.cpp:301 ] [ OCC::PropagateUploadFileCommon::slotComputeContentChecksum ]: Still trying to understand if this is the local file or the uploaded one

I also notice that the desktop client stops suddenly and repeatedly

[roberix]

Hi. I can confirm that is still an issue in 3.7.1 also for me.

I got another weird behavior but I don't know whether this is releated. Once I open a MS office file the sync status also gets pending and never returns to green. I only open the file but not changing anything. Happens also with files larger than 4k.

If we can help in anyway to test further this issue please let me know.

Regards

Rob

[gschenck]

I can confirm that this is still an issue in 3.7.1.

[anonbergine]

Still an issue in 3.7.3

[Jaytie2k7]

I'm also dealing with this bug, mostly with all sorts of files below 1Mb.
I also encounter it, without virtual folders enabled.
I guess there also isn't a workaround is it?
I'm about to roll out Nextcloud for a group of people as a backup solution. This stops me from going on right now. I really hope it gets fixed soon. Thank you for you efford!

[mgallien]

I'm also dealing with this bug, mostly with all sorts of files below 1Mb. I also encounter it, without virtual folders enabled. I guess there also isn't a workaround is it? I'm about to roll out Nextcloud for a group of people as a backup solution. This stops me from going on right now. I really hope it gets fixed soon. Thank you for you efford!

the currently recommended workaround is really to use classic sync and selectively selecting the folders needed
we are going to have another try at fixing it soon

[Jaytie2k7]

the currently recommended workaround is really to use classic sync and selectively selecting the folders needed we are going to have another try at fixing it soon

Sadly, as stated above, I'm facing the same issue when using the classic sync method. (see screenshot) Or is there even another way of syncing?

Glad to hear, that you guys are on it!

[pzwahlen]

we are going to have another try at fixing it soon

Hi @mgallien, it is refreshing to read that this bug will get some attention from "the people who know".

[pzwahlen]

Sadly, as stated above, I'm facing the same issue when using the classic sync method. (see screenshot) Or is there even another way of syncing?

Hi @Jaytie2k7, I'm not sure i understand what you mean. This bug is really about hydration/dehydration of small files, with dehydratin of files <4k bringing them into a constant "to be sync" status.

I think your screenshot highlights that some file (whatever their size) are missign the overlay icon, which I would say is a completely different issue.

[roberix]

Sadly, as stated above, I'm facing the same issue when using the classic sync method. (see screenshot) Or is there even another way of syncing?

Hi @Jaytie2k7, I'm not sure i understand what you mean. This bug is really about hydration/dehydration of small files, with dehydratin of files <4k bringing them into a constant "to be sync" status.

I think your screenshot highlights that some file (whatever their size) are missign the overlay icon, which I would say is a completely different issue.

Hi.

I face this issue also with larger files. Especially msoffice files which stays in sync pending once files is opened (no change). Maybe 2 different issue though and we should separate them – hard for me to judge.

Regards

rob

[Jaytie2k7]

I think your screenshot highlights that some file (whatever their size) are missign the overlay icon, which I would say is a completely different issue.

The screenshot shows a testfolder after switching to the classic sync method. NC client says it's in sync while it isn't.
The files lack the overlay-icon because they are not getting synced. These files are actually missing on the serverside (not visible in the WebUI)
Also they are all smaller than 1mb, which seems to be the case mostly.

Edit: I have to appologize, my initial problem was more related to 5094
Got that fixed now.

But I still have the bug, which is described here, where I cannot free up space / file is in sync pedning.

[pzwahlen]

the currently recommended workaround is really to use classic sync and selectively selecting the folders needed
we are going to have another try at fixing it soon

Just a quick note: it looks like the latest nightly (which includes #5474) solves the problem!

[mgallien]

@pzwahlen thanks for the notice
I completely overlooked closing this issue when the PR was merged

[ErikUden]

the currently recommended workaround is really to use classic sync and selectively selecting the folders needed
we are going to have another try at fixing it soon

Just a quick note: it looks like the latest nightly (which includes #5474) solves the problem!

Sorry to bother you and @mgallien, but when can we expect those being implemented on the beta channel :D? I just downloaded the newest version (3.7.4) and I am on Windows-10.0.19045 and sadly this update didn't fix the error for me!

It still shows "All synced" with just about every file having the "pending sync" symbol next to it! Force sync etc. doesn't help

In a week? A month? All is fine by me, I just wanna know at what point, if the error still persists even after an updated client, I should start worrying again :D

[pzwahlen]

@ErikUden can you take one of those small file and do the following:

• hydrate it (ie right-click->'make always available locally')
• dehydrate it (ie right-click->'free up local space')

In my case, the small file ends up with the correct "cloud" icon/status and not the "sync pending" status anymore.

You may have to do this for all the files problematic files.

[mgallien]

@ErikUden the fix is already part of 3.7.4
that said, the state of the file on storage is invalid and may indeed require you to make the steps suggested by @pzwahlen

[rose-a]

I can confirm this has fixed the issue for me in 3.7.4

[welcometopwn]

This is still an issue for me. The above steps mentioned by @pzwahlen works for files, but the folders still say "sync pending". Removing and re-adding them doesn't work either.

Edit: I'm running 3.8.0 on Windows 11

[Floflobel]

The problem seems to be still present in 3.8.0.

[afzl-wtu]

Issue is present still in latest version.

[ErikUden]

Issue is present still in latest version.

Issues with the database cannot be fixed via a client update. If a previous client (or precious server version) caused server-side issues, then a client update won't fix it.

The new client ensures the same issue won't happen again.

Those who already have the issue will need to fix it themselves and then be glad it won't occur again.

[Floflobel]

Those who already have the issue will need to fix it themselves and then be glad it won't occur again.

Unless I'm mistaken I haven't seen a solution to fix the problem.

[afzl-wtu]

Issue is present still in latest version.

Issues with the database cannot be fixed via a client update. If a previous client (or precious server version) caused server-side issues, then a client update won't fix it.

The new client ensures the same issue won't happen again.

Those who already have the issue will need to fix it themselves and then be glad it won't occur again.

How to fix it?

[mgallien]

the issue is that the state of some virtual files may be incorrect
the solution (apart from running a client version including the fix) is to make files be always offline then wait for the sync to be done then make them online only (and wait for the sync to be done)
starting from now, they should be fine and show the proper status
I do not know if the windows API would allow us to detect the invalid state of some virtual files and fix them

[welcometopwn]

I did once again try to fix it but some of the folders still have the sync pending icon. I even disabled the virtual files, let everything sync and enabled it again.

[anonbergine]

I've managed to fix most of the files using the "fix" discussed above, but it does not work for folders. I still have some folder with forever-sync issue.

[mgallien]

@anonbergine this may be due to ignored files
you can probably get clue by checking the activity list for ignored files/patterns
or you try to dig info about ignored files/folders from the client logs

[mgallien]

those ignored files may cause the client to report the folder as being only partially synced (hence the pending sync icon in the files explorer)