Errors with ADP/MFA and one-hour ADP limit

[pudge]

Describe the issue

When running with ADP and MFA on, it works for one hour, but then fails, and it will not re-start the sync correctly. It removes almost all of the local assets, then continues, but fails quickly again, and loops a few times before giving up.

How to reproduce the behavior?

docker run -it -v "/home/pudge/files/PhotosSync:/opt/icloud-photos-library" steilerdev/icloud-photos-sync:latest-ci-build -u "XXX@XXX" --enable-crash-reporting sync

With large size of photos, about 800GB, and 35,000 files.

Output below is for one of the four times I tried to run this.

Error Code

3f0944de-925b-4bc0-bd15-35b1df47e0f5

Relevant log output

----------------------------------------------------------------------------------------------------------------------------------------------------------------
Welcome to icloud-photos-sync, v.1.4.0-beta.1!
Made with <3 by steilerDev
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Authenticating user...
Device trusted
Advanced Data Protection requires additional cookies, acquiring...
Advanced Data Protection request not confirmed yet, retrying...
Advanced Data Protection requires additional cookies, acquiring...
Advanced Data Protection request not confirmed yet, retrying...
Advanced Data Protection requires additional cookies, acquiring...
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Starting sync at 8/26/2024, 1:36:53 PM
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Loaded local state: 441 assets & 0 albums
Fetched remote state: 35083 assets & 25 albums
Warning: Detected 5 albums, where asset counts don't match, please check the logs for more details (and see https://icps.steiler.dev/warnings/ for context)
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 30 and removing 411 local assets, as well as adding 35053 remote assets...
 ■■■■■                                    12% | Elapsed: 59m17s | 4333/35053 assets downloaded
Detected error during sync: SYNC_NETWORK: Network error during sync caused by Request failed with status code 410
Refreshing iCloud connection & retrying (attempt #2)...
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Loaded local state: 4355 assets & 0 albums
Fetched remote state: 35083 assets & 25 albums
Warning: Detected 5 albums, where asset counts don't match, please check the logs for more details (and see https://icps.steiler.dev/warnings/ for context)
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 56 and removing 4299 local assets, as well as adding 35027 remote assets...
                                          1% | Elapsed: 2m1s | 427/35027 assets downloaded
Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by ENOENT: no such file or directory, open '/opt/icloud-photos-library/_All-Photos/ARaa5uL3X35jUMyT5vN2HFOj2Sys.heic'
Refreshing iCloud connection & retrying (attempt #3)...
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Loaded local state: 503 assets & 0 albums
Fetched remote state: 35083 assets & 25 albums
Warning: Detected 5 albums, where asset counts don't match, please check the logs for more details (and see https://icps.steiler.dev/warnings/ for context)
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 57 and removing 446 local assets, as well as adding 35026 remote assets...
                                          1% | Elapsed: 1m5s | 428/35026 assets downloaded
Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by ENOENT: no such file or directory, open '/opt/icloud-photos-library/_All-Photos/ARaa5uL3X35jUMyT5vN2HFOj2Sys.heic'
Refreshing iCloud connection & retrying (attempt #4)...
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Loaded local state: 504 assets & 0 albums
Fetched remote state: 35083 assets & 25 albums
Warning: Detected 5 albums, where asset counts don't match, please check the logs for more details (and see https://icps.steiler.dev/warnings/ for context)
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 57 and removing 447 local assets, as well as adding 35026 remote assets...
                                          1% | Elapsed: 1m1s | 428/35026 assets downloaded
Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by ENOENT: no such file or directory, open '/opt/icloud-photos-library/_All-Photos/ARaa5uL3X35jUMyT5vN2HFOj2Sys.heic'
Refreshing iCloud connection & retrying (attempt #5)...
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Loaded local state: 504 assets & 0 albums
Fetched remote state: 35083 assets & 25 albums
Warning: Detected 5 albums, where asset counts don't match, please check the logs for more details (and see https://icps.steiler.dev/warnings/ for context)
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 57 and removing 447 local assets, as well as adding 35026 remote assets...
                                          1% | Elapsed: 1m11s | 428/35026 assets downloaded
Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by ENOENT: no such file or directory, open '/opt/icloud-photos-library/_All-Photos/ARaa5uL3X35jUMyT5vN2HFOj2Sys.heic'
Refreshing iCloud connection & retrying (attempt #6)...
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Loaded local state: 504 assets & 0 albums
Fetched remote state: 35083 assets & 25 albums
Warning: Detected 5 albums, where asset counts don't match, please check the logs for more details (and see https://icps.steiler.dev/warnings/ for context)
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 57 and removing 447 local assets, as well as adding 35026 remote assets...
                                          1% | Elapsed: 60s | 428/35026 assets downloaded
Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by ENOENT: no such file or directory, open '/opt/icloud-photos-library/_All-Photos/ARaa5uL3X35jUMyT5vN2HFOj2Sys.heic'
Refreshing iCloud connection & retrying (attempt #7)...
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Loaded local state: 504 assets & 0 albums
Fetched remote state: 35083 assets & 25 albums
Warning: Detected 5 albums, where asset counts don't match, please check the logs for more details (and see https://icps.steiler.dev/warnings/ for context)
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 57 and removing 447 local assets, as well as adding 35026 remote assets...
                                          1% | Elapsed: 1m2s | 426/35026 assets downloaded
Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by ENOENT: no such file or directory, open '/opt/icloud-photos-library/_All-Photos/ARaa5uL3X35jUMyT5vN2HFOj2Sys.heic'
Refreshing iCloud connection & retrying (attempt #8)...
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Loaded local state: 502 assets & 0 albums
Fetched remote state: 35083 assets & 25 albums
Warning: Detected 5 albums, where asset counts don't match, please check the logs for more details (and see https://icps.steiler.dev/warnings/ for context)
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 57 and removing 445 local assets, as well as adding 35026 remote assets...
                                          1% | Elapsed: 59s | 427/35026 assets downloaded
Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by ENOENT: no such file or directory, open '/opt/icloud-photos-library/_All-Photos/ARaa5uL3X35jUMyT5vN2HFOj2Sys.heic'
Refreshing iCloud connection & retrying (attempt #9)...
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Loaded local state: 503 assets & 0 albums
Fetched remote state: 35083 assets & 25 albums
Warning: Detected 5 albums, where asset counts don't match, please check the logs for more details (and see https://icps.steiler.dev/warnings/ for context)
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 57 and removing 446 local assets, as well as adding 35026 remote assets...
                                          1% | Elapsed: 60s | 426/35026 assets downloaded
Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by ENOENT: no such file or directory, open '/opt/icloud-photos-library/_All-Photos/ARaa5uL3X35jUMyT5vN2HFOj2Sys.heic'
Refreshing iCloud connection & retrying (attempt #10)...
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Loaded local state: 502 assets & 0 albums
Fetched remote state: 35083 assets & 25 albums
Warning: Detected 5 albums, where asset counts don't match, please check the logs for more details (and see https://icps.steiler.dev/warnings/ for context)
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 58 and removing 444 local assets, as well as adding 35025 remote assets...
                                          1% | Elapsed: 1m2s | 426/35025 assets downloaded
Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by ENOENT: no such file or directory, open '/opt/icloud-photos-library/_All-Photos/ARaa5uL3X35jUMyT5vN2HFOj2Sys.heic'
Refreshing iCloud connection & retrying (attempt #11)...
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
Error: APP_SYNC: Sync failed caused by SYNC_MAX_RETRY: Sync did not complete successfully within expected amount of tries (11) (error code: 3f0944de-925b-4bc0-bd15-35b1df47e0f5)

Operating system

Ubuntu 22.04

Execution environment

docker

icloud-photos-sync version

1.4.0-beta.1

Checklist

• This is an issue and not a question or feature request
• The sync was performed a couple of times and the issue is persisting and consistent
• Checked out open and known issues
• Enabled crash and error reporting