[YouTube Change] Throttling at 1MB/s

[TheFrenchGhosty]

Copied from yt-dlp/yt-dlp#985 since it surely/will surely affect Invidious instances.

---

Description

YouTube seems to have implemented a new throttling method, an IP will get flagged and throttled consistently (and from the look of it, permanently) at ~1MB/s.

The test bellow is this video: https://www.youtube.com/watch?v=p2vOKrHzqcg downloaded with my Watch on PC Script slightly modified, on a server with a 1Gbit/s connection.

Bumping the --throttled-rate to 1500 doesn't change anything, it always extract something that will get downloaded at ~1MB/s

If I run the same script, on the same server, but in a VM that is using a VPN it downloads at a normal speed.

I asked @pukkandan and he seems to agrees that it looks like throttling.

Note: an Invidious instance is also running on the IP, albeit, not public (only semi-private with a dozen of users) it's an IP that is used more than a "normal" end user IP.

Verbose log

[debug] Command-line config: ['--throttled-rate', '100K', '--format', '(bestvideo[vcodec^=av01][height>=4320][fps>30]/bestvideo[vcodec^=vp9.2][height>=4320][fps>30]/bestvideo[vcodec^=vp9][height>=4320][fps>30]/bestvideo[vcodec^=avc1][height>=4320][fps>30]/bestvideo[height>=4320][fps>30]/bestvideo[vcodec^=av01][height>=4320]/bestvideo[vcodec^=vp9.2][height>=4320]/bestvideo[vcodec^=vp9][height>=4320]/bestvideo[vcodec^=avc1][height>=4320]/bestvideo[height>=4320]/bestvideo[vcodec^=av01][height>=2880][fps>30]/bestvideo[vcodec^=vp9.2][height>=2880][fps>30]/bestvideo[vcodec^=vp9][height>=2880][fps>30]/bestvideo[vcodec^=avc1][height>=2880][fps>30]/bestvideo[height>=2880][fps>30]/bestvideo[vcodec^=av01][height>=2880]/bestvideo[vcodec^=vp9.2][height>=2880]/bestvideo[vcodec^=vp9][height>=2880]/bestvideo[vcodec^=avc1][height>=2880]/bestvideo[height>=2880]/bestvideo[vcodec^=av01][height>=2160][fps>30]/bestvideo[vcodec^=vp9.2][height>=2160][fps>30]/bestvideo[vcodec^=vp9][height>=2160][fps>30]/bestvideo[vcodec^=avc1][height>=2160][fps>30]/bestvideo[height>=2160][fps>30]/bestvideo[vcodec^=av01][height>=2160]/bestvideo[vcodec^=vp9.2][height>=2160]/bestvideo[vcodec^=vp9][height>=2160]/bestvideo[vcodec^=avc1][height>=2160]/bestvideo[height>=2160]/bestvideo[vcodec^=av01][height>=1440][fps>30]/bestvideo[vcodec^=vp9.2][height>=1440][fps>30]/bestvideo[vcodec^=vp9][height>=1440][fps>30]/bestvideo[vcodec^=avc1][height>=1440][fps>30]/bestvideo[height>=1440][fps>30]/bestvideo[vcodec^=av01][height>=1440]/bestvideo[vcodec^=vp9.2][height>=1440]/bestvideo[vcodec^=vp9][height>=1440]/bestvideo[vcodec^=avc1][height>=1440]/bestvideo[height>=1440]/bestvideo[vcodec^=av01][height>=1080][fps>30]/bestvideo[vcodec^=vp9.2][height>=1080][fps>30]/bestvideo[vcodec^=vp9][height>=1080][fps>30]/bestvideo[vcodec^=avc1][height>=1080][fps>30]/bestvideo[height>=1080][fps>30]/bestvideo[vcodec^=av01][height>=1080]/bestvideo[vcodec^=vp9.2][height>=1080]/bestvideo[vcodec^=vp9][height>=1080]/bestvideo[vcodec^=avc1][height>=1080]/bestvideo[height>=1080]/bestvideo[vcodec^=av01][height>=720][fps>30]/bestvideo[vcodec^=vp9.2][height>=720][fps>30]/bestvideo[vcodec^=vp9][height>=720][fps>30]/bestvideo[vcodec^=avc1][height>=720][fps>30]/bestvideo[height>=720][fps>30]/bestvideo[vcodec^=av01][height>=720]/bestvideo[vcodec^=vp9.2][height>=720]/bestvideo[vcodec^=vp9][height>=720]/bestvideo[vcodec^=avc1][height>=720]/bestvideo[height>=720]/bestvideo[vcodec^=av01][height>=480][fps>30]/bestvideo[vcodec^=vp9.2][height>=480][fps>30]/bestvideo[vcodec^=vp9][height>=480][fps>30]/bestvideo[vcodec^=avc1][height>=480][fps>30]/bestvideo[height>=480][fps>30]/bestvideo[vcodec^=av01][height>=480]/bestvideo[vcodec^=vp9.2][height>=480]/bestvideo[vcodec^=vp9][height>=480]/bestvideo[vcodec^=avc1][height>=480]/bestvideo[height>=480]/bestvideo[vcodec^=av01][height>=360][fps>30]/bestvideo[vcodec^=vp9.2][height>=360][fps>30]/bestvideo[vcodec^=vp9][height>=360][fps>30]/bestvideo[vcodec^=avc1][height>=360][fps>30]/bestvideo[height>=360][fps>30]/bestvideo[vcodec^=av01][height>=360]/bestvideo[vcodec^=vp9.2][height>=360]/bestvideo[vcodec^=vp9][height>=360]/bestvideo[vcodec^=avc1][height>=360]/bestvideo[height>=360]/bestvideo[vcodec^=avc1][height>=240][fps>30]/bestvideo[vcodec^=av01][height>=240][fps>30]/bestvideo[vcodec^=vp9.2][height>=240][fps>30]/bestvideo[vcodec^=vp9][height>=240][fps>30]/bestvideo[height>=240][fps>30]/bestvideo[vcodec^=avc1][height>=240]/bestvideo[vcodec^=av01][height>=240]/bestvideo[vcodec^=vp9.2][height>=240]/bestvideo[vcodec^=vp9][height>=240]/bestvideo[height>=240]/bestvideo[vcodec^=avc1][height>=144][fps>30]/bestvideo[vcodec^=av01][height>=144][fps>30]/bestvideo[vcodec^=vp9.2][height>=144][fps>30]/bestvideo[vcodec^=vp9][height>=144][fps>30]/bestvideo[height>=144][fps>30]/bestvideo[vcodec^=avc1][height>=144]/bestvideo[vcodec^=av01][height>=144]/bestvideo[vcodec^=vp9.2][height>=144]/bestvideo[vcodec^=vp9][height>=144]/bestvideo[height>=144]/bestvideo)+(bestaudio[acodec^=opus]/bestaudio)/best', '--verbose', '--force-ipv4', '--ignore-errors', '--no-continue', '--no-overwrites', '--download-archive', 'archive.log', '--add-metadata', '--all-subs', '--embed-subs', '--output', '%(uploader)s - %(upload_date)s - %(title)s [%(id)s].%(ext)s', '--merge-output-format', 'mkv', '--batch-file', 'Source - Watch on PC.txt']
[debug] Batch file urls: ['https://www.youtube.com/watch?v=p2vOKrHzqcg']
[debug] Loading archive file 'archive.log'

[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] yt-dlp version 2021.09.02 (zip)
[debug] Python version 3.9.6 (CPython 64bit) - Linux-[REDACTED]
[debug] exe versions: ffmpeg 4.4, ffprobe 4.4, rtmpdump 2.4
[debug] Optional libraries: mutagen, pycryptodome, sqlite, websockets
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=p2vOKrHzqcg
[youtube] p2vOKrHzqcg: Downloading webpage
[youtube] p2vOKrHzqcg: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, source, codec:vp9.2, lang
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, source, vcodec:vp9.2(10), acodec, lang, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, id
[debug] Testing thumbnail 41
[info] p2vOKrHzqcg: Downloading 1 format(s): 313+251
[debug] Invoking downloader on "https://r3---sn-4g5lznes.googlevideo.com/videoplayback?expire=1631756162&ei=IktCYYLTDLTBmwfol4_IDA&ip=[REDACTED]&id=o-AOeE0qYG8e_tJyePCiIVGxWNXiihYgeCypFNzA9v3z7t&itag=313&source=youtube&requiressl=yes&mh=cg&mm=31%2C29&mn=sn-4g5lznes%2Csn-4g5ednsy&ms=au%2Crdu&mv=m&mvi=3&pl=21&initcwndbps=355000&vprv=1&mime=video%2Fwebm&gir=yes&clen=450924474&dur=524.457&lmt=1631710765060421&mt=1631733894&fvip=3&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=5516222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhANhsvJ-MuJhAfS9w2LZzXZenxsNM6qbPWnLM60oyL_4_AiEA8E4-dYlz6wkxf_uel8RtZ4LOQRar4U--xlI0-pJW4Cg%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAMvt-N7oHycbCc8xyMTa4MuIUwz_y6xCS2-mhfSQ5hm4AiEArqoIDW5ueVf2jbH_HK3UJ4FmypstNy8foaw8ZfLJF9s%3D"
[download] Destination: Linus Tech Tips - 20210914 - iPhone 13 - What Went Wrong [p2vOKrHzqcg].f313.webm
[download] 100% of 430.04MiB in 06:3507MiB/s ETA 00:001nown ETA
[debug] Invoking downloader on "https://r3---sn-4g5lznes.googlevideo.com/videoplayback?expire=1631756162&ei=IktCYYLTDLTBmwfol4_IDA&ip=[REDACTED]&id=o-AOeE0qYG8e_tJyePCiIVGxWNXiihYgeCypFNzA9v3z7t&itag=251&source=youtube&requiressl=yes&mh=cg&mm=31%2C29&mn=sn-4g5lznes%2Csn-4g5ednsy&ms=au%2Crdu&mv=m&mvi=3&pl=21&initcwndbps=355000&vprv=1&mime=audio%2Fwebm&gir=yes&clen=8177826&dur=524.481&lmt=1631710769717891&mt=1631733894&fvip=3&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=5511222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAOr9icFiAbTeeRWu34rTdDRQCwC6xX9lO7GghyXzQd0qAiEA8qm-6EI0KMN_faBrcp930r9KLB2AOSwGxRhptY7uBbM%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAMvt-N7oHycbCc8xyMTa4MuIUwz_y6xCS2-mhfSQ5hm4AiEArqoIDW5ueVf2jbH_HK3UJ4FmypstNy8foaw8ZfLJF9s%3D"
[download] Destination: Linus Tech Tips - 20210914 - iPhone 13 - What Went Wrong [p2vOKrHzqcg].f251.webm
[download] 100% of 7.80MiB in 00:0700MiB/s ETA 00:009nown ETA
[Merger] Merging formats into "Linus Tech Tips - 20210914 - iPhone 13 - What Went Wrong [p2vOKrHzqcg].mkv"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:Linus Tech Tips - 20210914 - iPhone 13 - What Went Wrong [p2vOKrHzqcg].f313.webm' -i 'file:Linus Tech Tips - 20210914 - iPhone 13 - What Went Wrong [p2vOKrHzqcg].f251.webm' -c copy -map 0:v:0 -map 1:a:0 'file:Linus Tech Tips - 20210914 - iPhone 13 - What Went Wrong [p2vOKrHzqcg].temp.mkv'
Deleting original file Linus Tech Tips - 20210914 - iPhone 13 - What Went Wrong [p2vOKrHzqcg].f251.webm (pass -k to keep)
Deleting original file Linus Tech Tips - 20210914 - iPhone 13 - What Went Wrong [p2vOKrHzqcg].f313.webm (pass -k to keep)
[EmbedSubtitle] There aren't any subtitles to embed
[Metadata] Adding metadata to "Linus Tech Tips - 20210914 - iPhone 13 - What Went Wrong [p2vOKrHzqcg].mkv"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:Linus Tech Tips - 20210914 - iPhone 13 - What Went Wrong [p2vOKrHzqcg].mkv' -i 'file:Linus Tech Tips - 20210914 - iPhone 13 - What Went Wrong [p2vOKrHzqcg].meta' -map 0 -dn -c copy -map_metadata 1 -metadata 'title=iPhone 13 - What Went Wrong?' -metadata date=20210914 -metadata 'description=[REDACTED] -metadata 'artist=Linus Tech Tips' 'file:Linus Tech Tips - 20210914 - iPhone 13 - What Went Wrong [p2vOKrHzqcg].temp.mkv'

[unixfox]

I'm for closing this issue until we can clearly replicate this issue on invidious. I personally haven't been able to replicate the issue on yewtu.be.

yt-dlp and invidious are quite different, we are always using the videoplayback URLs from the ANDROID client, and we use QUIC for proxying the data (although on my instance it's HTTP2).

[pukkandan]

Note: an Invidious instance is also running on the IP, albeit, not public (only semi-private with a dozen of users) it's an IP that is used more than a "normal" end user IP.

I would suggest adding some mechanism for logging the download speed and then test with this instance. Then we can know for sure if it affects only yt-dlp or it affects invidious as well

I personally haven't been able to replicate the issue on yewtu.be.

We have also not been able to reproduce the issue on yt-dlp. But youtube is known to do slow rollouts for these kinds of changes. So it would be beneficial if we can start investigating as soon as possible before this becomes a more widespread issue

[stelioskat]

Definitely there is some kind of throttling on yewtu.be. Trying to watch at 720p or 1080p and after a few seconds/minutes the stream gets stuck. I checked my home connection and this is not the issue.

[unixfox]

Definitely there is some kind of throttling on yewtu.be. Trying to watch at 720p or 1080p and after a few seconds/minutes the stream gets stuck. I checked my home connection and this is not the issue.

That's true, I can replicate the issue. Throttling is back, unfortunately.

[unixfox]

Useful resource: TeamNewPipe/NewPipeExtractor#780

[github-actions]

This issue has been automatically marked as stale and will be closed in 30 days because it has not had recent activity and is much likely outdated. If you think this issue is still relevant and applicable, you just have to post a comment and it will be unmarked.

[unixfox]

Doesn't seem to be an issue anymore.