[YouTube] Workaround Shorts UI for playlists by using a continuation for initial items #1104
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AudricV
added
enhancement
…for initial items YouTube don't return currently a continuation, if applicable, for the Shorts UI in playlists, restricting access to the 100th first items. The reel items returned don't give also upload date, uploader info and precise view count. Using a continuation which requests the first page of the playlist allows currently to get access to continuations, if applicable, and also standard video elements instead of Shorts ones, making extraction of upload date, uploader info and precise view count again possible. This continuation is only requested and used when the data structure should be the one of the Shorts UI. A fallback to the standard response is made in the cases where fetching or parsing the continuation response fails, or when no stream items are extracted. This change required to: - add a protocol buffer definition file for which its structure is based on reverse engineering of playlists continuations sent by WEB InnerTube client, received from InnerTube responses; - add a dependency on Protobuf Java Lite; - generate Java classes from the definition file with the protoc version corresponding to the one of the Protobuf Java Lite dependency used (see https://protobuf.dev/support/version-support/ for more details); they are excluded from Checkstyle checks as they do not follow our style rules at all; - add a dependency on Apache Commons Codec, as Java's Base64 class desugaring for Android is not released yet.
…dItems As continuations are now again available for playlists with Shorts UI, more items' tests can be enabled.
AudricV
force-pushed
the
yt_continuations-playlists-shorts-ui
branch
from
bccde47
to
2e3f6b4
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
YouTube don't return currently a continuation, if applicable, for the Shorts UI in playlists, restricting access to the 100th first items. The reel items returned don't give also upload date, uploader info and precise view count.
Using a continuation which requests the first page of the playlist allows currently to get access to continuations, if applicable, and also standard video elements instead of Shorts ones, making extraction of upload date, uploader info and precise view count again possible.
This continuation is only requested and used when the data structure should be the one of the Shorts UI.
A fallback to the standard response is made in the cases where fetching or parsing the continuation response fails, or when no stream items are extracted.
This change required to:
WEBInnerTube client, received from InnerTube responses;protocversion corresponding to the one of the Protobuf Java Lite dependency used (see https://protobuf.dev/support/version-support/ for more details); they are excluded from Checkstyle checks as they do not follow our style rules at all;Base64class desugaring for Android is not released yet.As a result of this change,
testMoreRelatedItemsmethod ofYoutubePlaylistExtractorTest.ShortsUItest class has been enabled, as we now get continuations for shorts playlists with more than 100 items and mocks of this test class had to be updated.