Segment downloads 403 after 30s, requiring frequent re-extraction #221
Comments
|
I knew this would start eventually. I don't have the drive the yt-dlp people do to fight it much either. I might try updating which clients are used for requests, but once they inevitably all get hit, I'll probably just stop bothering. |
|
It seems like an issue somewhere else — my application currently makes practically the same Android DASH manifest request and to my knowledge that's still working as it has ( Might be worth noting that I had to add some retry handling since I'll occasionally get failures in the Android player response with a missing |
|
@fren-archive Just to confirm, are you using 0.5.0? That's the first tagged release with d6a4be6 and 1671114; anything prior to those will only pull the web formats. I'm not sure how ytarchive handles streams that require authentication since I rarely ever use that. If it only pulls the web format then you would need a valid POToken for sure, but beyond that it should work as it did before (i.e. failure on next player fetch if the cookies were rotated but should still continue using the valid manifest until it expires). |
|
I updated to 0.5.0 a couple days after posting this and added cookies and the issue went away for a while (forgot to update here). However, as of today it is happening even on the most recent version with valid cookies. |
|
I've added support for the po token in commit 99a510d It's been working for me. I only just started getting hit with the 30 second 403s around 24 hours ago, so finally had reason to implement it (and a way to verify since without it affecting me I'd have no idea if it was working). |
|
Yeah; the Android client that was used no longer returns streaming data. Not sure which clients are currently working if any. Can confirm that change works with a token and cookies on a fresh profile. In place of cookies for logged out users, you can pass the token's corresponding |
|
I now get 403's instantly even after extracting the poToken and the associated cookies from the same browser session. I've tried this multiple times, even attempting to extract cookies & the poToken on two different browsers. Also the browser was playing the stream normally just fine. I extracted the poToken from the YouTube embedded player. I also tried getting it from the main YouTube page, no dice. The browsers are logged out of YouTube. Also tried using yt-dlp for this, it doesn't seem to work either, I get 403's on it as well. Maybe passing through |
|
Magically, yt-dlp started working on like.. the 10th attempt? That was probably somehow my fault, though I have no idea what I had done wrong. Then I tried ytarchive as well, I also managed to make it work - you have to add "web+" before the actual poToken. However, I am once again getting 403's every 30 seconds: Maybe the Also the 403's do not seem to happen on yt-dlp with the poToken and cookies passed through, it doesn't pop up an error even with the debug flags set. |
I and several others on Discord are seeing frequent 403s while downloading segments, though some users reported they are not seeing this behavior. Specifically it happens 30s after each page extraction, meaning each instance of ytarchive needs to extract the URLs 120 times per hour instead of the 1 time that is intended. The output with
--debugset will look like this:With just 3-5 instances I was able to trigger bot detection measures, which caused the recordings to all fail nearly simultaneously. There are other effects, such as that content which is removed or set members-only will fail almost immediately instead of being able to continue for 6h. Passing cookies did not change the behavior.
This is, as far as I know, tied to the android client and potoken. When potoken is enabled (which it is on android client in most cases) URLs fail after 30s unless they have the proper value of
pot. This is similar to nsig calculation but substantially worse because it relies on fingerprinting methods rather than a simple javascript check, and probably cannot reasonably be emulated outside a browser.I have the same issue with yt-dlp if I try to use android or web clients, but ios and web_creator clients are not (yet) affected. web_creator also provides dash manifest urls which do not need nsig calculation so that seems like the easiest path to avoid the issue. Alternatively an option for the user to offload URL extraction to an external script (presumably yt-dlp) might be more robust long-term.
The text was updated successfully, but these errors were encountered: