Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement exponential backoff for HTTP PUT #1095

Open
mariocynicys opened this issue Sep 7, 2022 · 2 comments · May be fixed by #1098
Open

Implement exponential backoff for HTTP PUT #1095

mariocynicys opened this issue Sep 7, 2022 · 2 comments · May be fixed by #1098
Labels
priority: P1 Big impact or workaround impractical; resolve before feature release type: enhancement New feature or request
Milestone
Backlog

Comments

@mariocynicys
Copy link

A feature request to track adding an exponential backoff strategy for HTTP PUTs.

shaka-packager/packager/file/http_file.cc

Lines 212 to 213 in 31129ee

// TODO: Implement retrying with exponential backoff, see
// "widevine_key_source.cc"

Not backing is sometimes problematic, as seen in here: shaka-project/shaka-streamer#103 (comment)
@joeyparrish
Copy link
Member

Totally on board with this.

@joeyparrish joeyparrish added type: enhancement New feature or request priority: P1 Big impact or workaround impractical; resolve before feature release labels Sep 7, 2022
@github-actions github-actions bot added this to the Backlog milestone Sep 7, 2022
@mariocynicys
Copy link
Author

mariocynicys commented Sep 8, 2022 •
edited
Loading

I am trying to compile shaka-packager using these instructions, but getting the following error. I assume it has to do with the branch changes since the output logs mentions things are pulled from master and not main.

@meryacine ➜ /workspaces/pkger $ gclient config https://github.com/shaka-project/shaka-packager.git --name=src --unmanaged && gclient sync --no-history
Warning: Running gclient on Python 3. 
If you encounter any issues, please file a bug on crbug.com under the Infra>SDK component.
Warning: Running gclient on Python 3. 
If you encounter any issues, please file a bug on crbug.com under the Infra>SDK component.

________ running 'git -c core.deltaBaseCacheLimit=2g clone --no-checkout --progress https://github.com/shaka-project/shaka-packager.git --depth=1 /workspaces/pkger/_gclient_src_fg6pfb8i' in '/workspaces/pkger'
Cloning into '/workspaces/pkger/_gclient_src_fg6pfb8i'...
remote: Enumerating objects: 4095, done.        
remote: Counting objects: 100% (4095/4095), done.        
remote: Compressing objects: 100% (2698/2698), done.        
remote: Total 4095 (delta 1388), reused 2827 (delta 1210), pack-reused 0        
Receiving objects: 100% (4095/4095), 29.32 MiB | 2.67 MiB/s, done.
Resolving deltas: 100% (1388/1388), done.
1>WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "fetch" "origin" "refs/heads/master:refs/remotes/origin/master" "--no-tags"' in /workspaces/pkger/src failed; will retry after a short nap...
1>WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "fetch" "origin" "refs/heads/master:refs/remotes/origin/master" "--no-tags"' in /workspaces/pkger/src failed; will retry after a short nap...
1>WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "fetch" "origin" "refs/heads/master:refs/remotes/origin/master" "--no-tags"' in /workspaces/pkger/src failed; will retry after a short nap...
1>WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "fetch" "origin" "refs/heads/master:refs/remotes/origin/master" "--no-tags"' in /workspaces/pkger/src failed; will retry after a short nap...

________ running 'git -c core.deltaBaseCacheLimit=2g clone --no-checkout --progress https://github.com/shaka-project/shaka-packager.git --depth=1 /workspaces/pkger/_gclient_src_m4qopnbt' in '/workspaces/pkger'
Cloning into '/workspaces/pkger/_gclient_src_m4qopnbt'...
remote: Enumerating objects: 4095, done.        
remote: Counting objects: 100% (4095/4095), done.        
remote: Compressing objects: 100% (2698/2698), done.        
remote: Total 4095 (delta 1388), reused 2827 (delta 1210), pack-reused 0        
Receiving objects: 100% (4095/4095), 29.32 MiB | 3.61 MiB/s, done.
Resolving deltas: 100% (1388/1388), done.
1>WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "fetch" "origin" "refs/heads/master:refs/remotes/origin/master" "--no-tags"' in /workspaces/pkger/src failed; will retry after a short nap...
1>WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "fetch" "origin" "refs/heads/master:refs/remotes/origin/master" "--no-tags"' in /workspaces/pkger/src failed; will retry after a short nap...
1>WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "fetch" "origin" "refs/heads/master:refs/remotes/origin/master" "--no-tags"' in /workspaces/pkger/src failed; will retry after a short nap...
1>WARNING: subprocess '"git" "-c" "core.deltaBaseCacheLimit=2g" "fetch" "origin" "refs/heads/master:refs/remotes/origin/master" "--no-tags"' in /workspaces/pkger/src failed; will retry after a short nap...
Syncing projects: 100% ( 1/ 1) src
src (ERROR)
----------------------------------------
[0:00:00] Started.
[0:00:00] 
[0:00:14] Finished running: git rev-parse --verify FETCH_HEAD
fatal: couldn't find remote ref refs/heads/master
[0:00:15] fatal: couldn't find remote ref refs/heads/master
[0:00:16] fatal: couldn't find remote ref refs/heads/master
[0:00:19] fatal: couldn't find remote ref refs/heads/master
[0:00:23] _____ Conflicting directory found in /workspaces/pkger/src. Moving to /workspaces/pkger/_bad_scm/srcrg6lpwam.
[0:00:23] 
[0:00:34] Finished running: git rev-parse --verify FETCH_HEAD
[0:00:34] fatal: couldn't find remote ref refs/heads/master
[0:00:35] fatal: couldn't find remote ref refs/heads/master
[0:00:36] fatal: couldn't find remote ref refs/heads/master
[0:00:39] fatal: couldn't find remote ref refs/heads/master
----------------------------------------
Error: Command 'git -c core.deltaBaseCacheLimit=2g fetch origin refs/heads/master:refs/remotes/origin/master --no-tags' returned non-zero exit status 128 in /workspaces/pkger/src


Warnings:
Conflicting directory /workspaces/pkger/src moved to /workspaces/pkger/_bad_scm/srcrg6lpwam.

@meryacine ➜ /workspaces/pkger $  ls
_bad_scm  src

Update: Solved this by editing .gclient file like so:

solutions = [
  { "name"        : 'src',
    # @main here to tell gclient to fetch main branch and not master
    "url"         : 'https://github.com/shaka-project/shaka-packager.git@main',
    "deps_file"   : 'DEPS',
    "managed"     : False,
    "custom_deps" : {
    },
    "custom_vars": {},
  },
]

@mariocynicys mariocynicys linked a pull request Sep 9, 2022 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority: P1 Big impact or workaround impractical; resolve before feature release type: enhancement New feature or request
Projects
None yet
Milestone
Backlog
Development

Successfully merging a pull request may close this issue.

feat(HTTP): Retry failed HTTP requests with exponential backoff mariocynicys/shaka-packager
2 participants
@joeyparrish @mariocynicys