[Bug]: VFS is not working when Folder contains huge amount of files #5500

wagnbeu0 · 2023-03-07T13:56:55Z

⚠️ This issue respects the following points: ⚠️

This is a bug, not a question or a configuration/webserver/proxy issue.
This issue is not already reported on Github (I've searched it).
Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
Nextcloud Server is running on 64bit capable CPU, PHP and OS.
I agree to follow Nextcloud's Code of Conduct.

Bug description

The behaviour is already stated here: https://help.nextcloud.com/t/new-client-with-virtual-filesystem-cannot-sync-huge-folders/113917/30

When changing to VFS, and on the server or on the client a folder with a huge amount of files exist, syncing will not be finished. The client will hang, report "sync pending" or restart scan again and again.
this behaviour is reported by several users which have folders containing 50k - 100k files in a folder or a folder structure.

Steps to reproduce

Create a folder with ~70.000 files /on the link above scripts are available to create testdata)
Change client to VFS
Scanning the local directory will not complete. Workaround is to add small portions of files until they are synced (we did it ba adding 5000 files at a atime)

Expected behavior

VFS is able to sync local folder completely without interruption

Installation method

Community Manual installation with Archive

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.0

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

Default user-backend (database)
LDAP/ Active Directory
SSO - SAML
Other

Configuration report

{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "intranet.ben-wagner.de"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/intranet.ben-wagner.de:443\/nextcloud",
        "dbtype": "mysql",
        "version": "25.0.4.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",
        "installed": true,
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "default_phone_region": "DE",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "updater.release.channel": "stable",
        "maintenance": false,
        "loglevel": 0,
        "preview_libreoffice_path": "\/usr\/bin\/libreoffice",
        "theme": "",
        "filelocking.enabled": "true",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0
        },
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "mysql.utf8mb4": true,
        "app_install_overwrite": [
            "files_downloadactivity",
            "bookmarks_fulltextsearch",
            "admin_notifications",
            "keeweb",
            "files_reader",
            "social",
            "drawio",
            "files_mindmap",
            "mindmap_app",
            "fulltextsearch",
            "files_fulltextsearch_tesseract",
            "files_fulltextsearch",
            "fulltextsearch_elasticsearch",
            "ransomware_detection",
            "whiteboard",
            "files_trackdownloads",
            "bruteforcesettings",
            "richdocumentscode",
            "polls",
            "forms",
            "groupfolders",
            "tasks",
            "files_retention",
            "quota_warning",
            "appointments",
            "richdocuments",
            "workflow_pdf_converter",
            "files_texteditor",
            "files_markdown"
        ],
        "encryption.legacy_format_support": false,
        "encryption.key_storage_migrated": false,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "skeletondirectory": "\/var\/www\/intranet.ben-wagner.de\/nextcloud\/core\/skeleton_empty",
        "trashbin_retention_obligation": "auto, 90"
    }
}

List of activated Apps

Enabled:
  - activity: 2.17.0
  - admin_audit: 1.15.0
  - admin_notifications: 1.0.2
  - appointments: 1.14.11
  - bruteforcesettings: 2.5.0
  - calendar: 4.2.4
  - calendar_resource_management: 0.4.0
  - circles: 25.0.0
  - cloud_federation_api: 1.8.0
  - comments: 1.15.0
  - contacts: 5.1.0
  - contactsinteraction: 1.6.0
  - dav: 1.24.0
  - deck: 1.8.3
  - drawio: 2.1.0
  - federatedfilesharing: 1.15.0
  - federation: 1.15.0
  - files: 1.20.1
  - files_accesscontrol: 1.15.1
  - files_external: 1.17.0
  - files_fulltextsearch: 25.0.0
  - files_markdown: 2.3.6
  - files_pdfviewer: 2.6.0
  - files_retention: 1.14.1
  - files_rightclick: 1.4.0
  - files_sharing: 1.17.0
  - files_texteditor: 2.15.0
  - files_trashbin: 1.15.0
  - files_versions: 1.18.0
  - files_zip: 1.1.2
  - firstrunwizard: 2.14.0
  - forms: 3.2.0
  - fulltextsearch: 25.0.0
  - fulltextsearch_elasticsearch: 25.0.0
  - groupfolders: 13.1.1
  - keeweb: 0.6.10
  - logreader: 2.10.0
  - lookup_server_connector: 1.13.0
  - mail: 2.2.4
  - metadata: 0.17.0
  - music: 1.8.1
  - nextcloud_announcements: 1.14.0
  - notes: 4.6.0
  - notifications: 2.13.1
  - oauth2: 1.13.0
  - password_policy: 1.15.0
  - photos: 2.0.1
  - polls: 4.1.8
  - privacy: 1.9.0
  - provisioning_api: 1.15.0
  - quota_warning: 1.15.0
  - recommendations: 1.4.0
  - registration: 2.1.0
  - related_resources: 1.0.4
  - richdocuments: 7.1.1
  - richdocumentscode: 22.5.802
  - serverinfo: 1.15.0
  - settings: 1.7.0
  - sharebymail: 1.15.0
  - sharerenamer: 3.1.0
  - support: 1.8.0
  - survey_client: 1.13.0
  - systemtags: 1.15.0
  - tasks: 0.14.5
  - text: 3.6.0
  - theming: 2.0.1
  - twofactor_backupcodes: 1.14.0
  - updatenotification: 1.15.0
  - user_status: 1.5.0
  - viewer: 1.9.0
  - weather_status: 1.5.0
  - workflow_media_converter: 1.7.0
  - workflow_pdf_converter: 1.10.0
  - workflowengine: 2.7.0
Disabled:
  - dashboard: 7.1.0
  - encryption: 2.5.0
  - files_downloadactivity: 1.15.0
  - files_trackdownloads: 1.11.0
  - suspicious_login: 4.3.0
  - twofactor_totp
  - user_ldap

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

n/a

Additional info

n/a

PhilippSchlesinger · 2023-03-07T15:16:20Z

What is the version of Nextcloud Client?

wagnbeu0 · 2023-03-07T15:47:02Z

Latest available version for windows Am 07.03.2023 um 16:16 schrieb PhilippSchlesinger ***@***.***>: What is the version of Nextcloud Client? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

matzze2000 · 2023-03-17T07:11:14Z

I also had the same problem in the last few days, but with significantly fewer files. I had set up a new nextcloud installation (Ubuntu snap) and tried to sync my private data (approx. 1.5GB, approx. 6000 files) with the current Windows desktop client. At first it worked without any problems and then the sync was permanently stuck on individual files. After restarting the client, a few files were synced again and then it got stuck again (with the same PDF files). After I deactivated VFS, the sync ran without problems after a restart.

github-actions · 2023-04-14T08:31:09Z

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

github-actions · 2023-04-28T16:32:27Z

This bug report is getting automatically closed due to no answer since the issue has been staled. Thank you!

wagnbeu0 · 2023-10-17T12:05:45Z

Who can take care for this bug?

wagnbeu0 added 0. Needs triage bug labels Mar 7, 2023

szaimen transferred this issue from nextcloud/server Mar 7, 2023

github-actions bot added the stale label Apr 14, 2023

github-actions bot closed this as completed Apr 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: VFS is not working when Folder contains huge amount of files #5500

[Bug]: VFS is not working when Folder contains huge amount of files #5500

wagnbeu0 commented Mar 7, 2023

PhilippSchlesinger commented Mar 7, 2023

wagnbeu0 commented Mar 7, 2023 via email

matzze2000 commented Mar 17, 2023

github-actions bot commented Apr 14, 2023

github-actions bot commented Apr 28, 2023

wagnbeu0 commented Oct 17, 2023

[Bug]: VFS is not working when Folder contains huge amount of files #5500

[Bug]: VFS is not working when Folder contains huge amount of files #5500

Comments

wagnbeu0 commented Mar 7, 2023

⚠️ This issue respects the following points: ⚠️

Bug description

Steps to reproduce

Expected behavior

Installation method

Operating system

PHP engine version

Web server

Database engine version

Is this bug present after an update or on a fresh install?

Are you using the Nextcloud Server Encryption module?

What user-backends are you using?

Configuration report

List of activated Apps

Nextcloud Signing status

Nextcloud Logs

Additional info

PhilippSchlesinger commented Mar 7, 2023

wagnbeu0 commented Mar 7, 2023 via email

matzze2000 commented Mar 17, 2023

github-actions bot commented Apr 14, 2023

github-actions bot commented Apr 28, 2023

wagnbeu0 commented Oct 17, 2023