[Bug]: Files over 10 MB break when uploading - Missing metadata for chunked upload - affects version Nextcloud client version 3.10.0

[asheroto]

⚠️ Before submitting, please verify the following: ⚠️

• This is a bug, not a question or a configuration issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
• I agree to follow Nextcloud's Code of Conduct

Bug description

Similar to nextcloud/server#38801, but appears to be separate, as it only affects the client and not the server. Not sure if related to to the PR #3600?

There a bug with Nextcloud client version 3.10.0 when uploading files greater than 10 MB. The file gets to 10 MB and then breaks. Nextcloud client logs say...

Server replied "412 Precondition failed" to "PUT https://domain/remote.php/dav/uploads/username/etc/etc" (Missing metadata for chunked upload)|412|0|0|cc29eb65-2435-4fac-9caa-5b7c2d7f1643|

If the Nextcloud client 3.10.0 is uninstalled and version 3.9.4 is installed then the issue goes away and uploads work perfectly.

Background:

• Not using Cloudflare
• Using apache2, php8.2-fpm
• Using S3 for storage
• Using server side encryption
• Encrypting the home storage

For reference, .user.ini is...

memory_limit=1G
upload_max_filesize=256M
post_max_size=256M
max_input_time=1800
max_execution_time=1800

This workaround was mentioned on nextcloud/server#38801 but doesn't fix this bug. It doesn't matter whether chunked file uploads are set to 0 or any other value, the error still occurs.

Thanks for your time on this. For now the workaround seems to be to downgrade to 3.9.4.

Steps to reproduce

1. Nextcloud 27.1.0 server, Nextcloud client 3.10.0, S3 storage.
2. Sync a file greater than 10 MB
3. "Missing metadata for chunked upload" appears

Expected behavior

Syncing should be successful

Which files are affected by this bug

Not sure if this means one of my files or one of the source code files?

Operating system

Windows

Which version of the operating system you are running.

Windows 11

Package

Other

Nextcloud Server version

23.1.0

Nextcloud Desktop Client version

3.10.0

Is this bug present after an update or on a fresh install?

Updated from a minor version (ex. 3.4.2 to 3.4.4)

Are you using the Nextcloud Server Encryption module?

Encryption is Enabled

Are you using an external user-backend?

• Default internal user-backend
• LDAP/ Active Directory
• SSO - SAML
• Other

Nextcloud Server logs

No error messages for this issue appear in the server nextcloud.log file when using the default log level. I can enable debug mode if needed, but this appears to be a client issue not a server issue. I tried downgrading to an older server version but still had the same issue.

Additional info

I can create a debug archive if needed. The issue went away after downgrading the client.

[OrvilleQ]

Same issue here.

• server 27.0.2
• client 3.10.0

Can use web interface to upload the same file without issue.

[claucambra]

Hi @asheroto a debug archive would indeed be highly appreciated, thanks!

[OrvilleQ]

412Debug.zip

@claucambra Here's mine.

[S1mpl]

Hi @claucambra when uploading files larger than 10 MB, they are divided into chunks; when sending a chunk, empty metadata uploadId and uploadPath are sent

[claucambra]

Hi @claucambra when uploading files larger than 10 MB, they are divided into chunks; when sending a chunk, empty metadata uploadId and uploadPath are sent

Reading the code this is not the issue. The server checks for these values but the uploadId is created by the server itself and the uploadPath is pulled from the Destination header sent with the chunk, which we do definitely set. Rather the issue seems to be that the destination header was set to a percent-encoded path that the server deemed invalid leading to the precondition failed error

I can confirm that in my own testing with #6084 chunked upload works; once the CI builds the AppImage you will be able to test the fixed version of the client on Linux and check if the issue is fixed

[asheroto]

I haven't tested with #6084 yet.

Even when I disable chunking, the issue still occurs.

Whether I do this...

sudo -u www-data php occ config:app:set files max_chunk_size --value 2097152000

or this...

sudo -u www-data php occ config:app:set files max_chunk_size --value 0

the error message still comes up.

I might also mention that I am not using the virtual files but am syncing through "Choose what to sync". Not sure if that matters.

debug.zip is the debug archive. Example file is Google Chrome.msi. I changed nothing in the archive except that I redacted the server URL for privacy. Using a user named "Test-User" on a new installation of Nextcloud client 3.10.0 running on a VM, so the logs should be pretty small.

[asheroto]

Awesome, thanks for your quick response on this!

[ffenable]

Hi @claucambra

I tried mirall/3.10.506084 (build 13347) for linux from here

nothing changed, have the same issue

[claucambra]

@ffenable could you post both your desktop client's debug archive as well as your server's nextcloud.log?

[ffenable]

@claucambra
I would not like to publish clear logs and dumps. I guess these pieces of logs will be enough to understand the issue is.

server.log
client.log

I don't see anything interesting there anymore

I can say that on version mirall/3.7.50git (build 14472) everything works fine (nothing personal, it was just at hand)

[NathanHbt]

Same issue here with Nextcloud 3.10.0 on MacOs and S3 storage.

[bitfl0wer]

I still have the same issue as well. Using S3 Storage as my primary Storage. It's only the client too - Uploading via the Webinterface works fine.

[asheroto]

I am also using Windows FYI

[ffenable]

Hi, @claucambra, i am sorry, but do you have any thoughts on this? can we expect anything in the near future?

[asheroto]

@ffenable not sure if you saw this, but if you downgrade to version 3.9.4 that's a temporary workaround.

Sounds like the AppImage file with the fix did not resolve it for you, right? This issue should probably be reopened.

[ffenable]

@asheroto
yes, I understand that the old versions are in working order, but there is no easy way, implemented in the client, to roll back to them. Unfortunately, I can't control users' devices
Moreover, there are some oddities in the delivery channels(on mac, not sure how it is on other platforms):
stable channel: Nextcloud 3.10.0 is currently the newest version available.
beta channel: Nextcloud 3.9.83 is currently the newest version available.
looks bad

[dsonck92]

I can also confirm that #6084 nor #6086 fixed the issue. I also run my storage on S3 provided by MinIO.

[giacomoarru]

Hi, I am also affected by the issue. Why is it closed?

[origintopleft]

Downloaded the daily client build from the official NextCloud daily builds URL, still receiving the same problem. NextCloud server 27.1.1 (it's the latest on Docker Hub). I don't have access to a 3.9 client, so I'm uninstalling and reverting to Windows' built in WebDAV client until this is fixed.

To use Windows' client as a workaround:

• Open "This PC", and in the meatball (three-dot) menu, press "add a network location".
• Paste in your WebDAV URL. You can find it in the Files app on the web client, under "Files settings".
• You will be prompted for credentials. Enter them. (Note: If you use 2FA or SSO, you'll need to use an app password.)
• Finally, enter a name that makes the most sense for your NextCloud storage (I only have the one personal account, so I just went with "NextCloud").

NextCloud will appear in "This PC" in your network locations, next to any mapped Samba drives you may have, like below:

You will now be able to upload your large files to NextCloud from the desktop again, using standard file copy operations.

[asheroto]

That's a good alternate workaround to downgrading the client.

As far as I can tell, they have not released a new version of the client that includes this fix, but @claucambra may be able to confirm. I would think that the daily build would have included the fix. So if it should, then this bug is not yet resolved.

[wmeneses]

Please can someone explain to me how I can patch it myself to publish it to my clients?

[asheroto]

The daily builds are located here:
https://github.com/nextcloud/desktop/wiki/Daily-Builds
https://download.nextcloud.com/desktop/daily/windows/

Unfortunately just as @origintopleft has stated, the daily build, even as of today, 2023-10-10, does not fix the issue. I just tried it, version 3.10.50.

For now the only workarounds right now:
1.) Fix the bug and compile the code (this Issue has not determined the root cause to my knowledge)
2.) Downgrade to an earlier version (easiest and fastest)
3.) Use the WebDAV mapped drive (seen above)

@claucambra merged a commit 3 weeks ago with the fix, so it should have been resolved by now with a daily build, but unfortunately the issue is still present.

This issue should be probably be reopened. 😊

[claucambra]

There are enough reports if this not working that I'm reopening this issue

We are considering whether the issue lies at the client end or server end. Will update with more info when we have it

[claucambra]

We have been able to reproduce 412: Precondition failed on a test instance and have pushed another PR (#6133) that fixes this issue. We are preparing test builds of the client with the patch applied so that you may all test it to see if it resolves the issue

Thanks all for your patience, and apologies for introducing this bug

[claucambra]

Hey all, we can now provide patched clients containing the aforementioned fix. You may download said client from the links below:

Linux: https://cloud.nextcloud.com/s/xndfF9CWPk3jpXW
macOS: https://cloud.nextcloud.com/s/2dsCkoA5GzbbFXD
Windows: https://cloud.nextcloud.com/s/kbfFJcHC4Ms2kyy

Please let us know if these clients work correctly for you now

[wmeneses]

Works for me, thank you very much

[niallguerin]

Your patch build provided today for macOS Desktop Client resolves desktop client sync issues for me - we use NextCloud server 26.0.6 and AWS S3 storage. Tested file uploads from 20MB up to 1GB+. All upload fine now using patched client.

[asheroto]

I can confirm from my end that using the patched Windows version does resolve the issue.

After updating to the patched version, I saw the same error message again on first sync, but after that it began to work. So maybe it had to clear out the failure first, not sure. I then tried another different file that was 120 MB and it uploaded perfectly fine as well.

Much appreciated @claucambra!

Not sure if you want to close now or wait until it's been released in the daily builds, but looks like the issue is resolved with this patch.

[librecloudhost]

We are also impacted by this issue. We would be extremely grateful to see it released without any delay.

[HavokInspiration]

TL;DR

If you are using the desktop client (I only tested it on Windows) and still have the problem after increasing upload_max_filesize and post_max_size directives and this suggestion, try creating a environment variable named OWNCLOUD_CHUNK_SIZE on your client (Windows in my case) and defining a value that would match the size of the biggest file you try to upload (the value must be in bytes).
It will basically prevent nextcloud client to sent the file in chunks.
It's a bit of a dirty workaround but it's the only thing that worked for me after hours of searching for a solution.

Full reference

Full message

Hello,

If like me, you stuble upon this issue with the "Missing metadata for chunked upload" using the desktop client (Windows for my case) and you also tried suggestions from this issue like this one or increasing the php.ini upload_max_filesize and post_max_size directives and you still have the problem, I have another suggestion for you to test.

While looking for a solution to my problem, I stumbled upon this documentation page

Which led me to this one and more importantly this section

This page states that you can create a environment variable named OWNCLOUD_CHUNK_SIZE on your client (Windows in my case) which takes a value in bytes. All files below this value will be sent without being chunked (which was causing the issue from what I gathered). So try putting a value that matches the size of the biggest file you try to upload.
It's a bit of a dirty workaround but it's the only thing that worked for me after hours of searching for a solution.

Hope this helps.

[asheroto]

The issue I originally created was resolved in later desktop clients.