Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[YouTube] Add more parameters to InnerTube requests, use the iOS client for livestreams and fix extraction of embeddable age-restricted videos and contents with a warning before playback #780

Merged
merged 18 commits into from
Apr 9, 2022
Merged

[YouTube] Add more parameters to InnerTube requests, use the iOS client for livestreams and fix extraction of embeddable age-restricted videos and contents with a warning before playback #780

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
83f374b
[YouTube] Update client versions and fix a bug when using resetClient…
AudricV Dec 11, 2021
05b7fee
[YouTube] Add the cpn param to playback requests and try to spoof bet…
AudricV Dec 22, 2021
7d07924
[YouTube] Try to use lighter requests when extracting client version …
AudricV Jan 9, 2022
26f93f5
[YouTube] Extract streams of livestreams from the iOS client and disa…
AudricV Jan 15, 2022
b6bc521
[YouTube] Update client versions again
AudricV Jan 15, 2022
d0d91e6
Adress requested changes
AudricV Feb 5, 2022
5237694
Add setSeedForVideoTests method in YoutubeStreamExtractor tests
AudricV Feb 6, 2022
349ba8d
Improve tests and randomness
litetex Feb 7, 2022
3d38459
[YouTube] Reduce InnerTube response sizes by adding the prettyPrint p…
AudricV Mar 15, 2022
1dad3bf
[YouTube] Update again hardcoded client versions and update mobile us…
AudricV Mar 15, 2022
2e3da44
[YouTube] Add documentation about parameters added and clients versio…
AudricV Mar 26, 2022
6d27996
Improve code of getStringResultFromRegexArray methods in Utils
AudricV Mar 26, 2022
dfa4239
Fix missing imports and Checkstyle issues
AudricV Mar 27, 2022
9ca647a
Update mocks
AudricV Mar 15, 2022
11b5a22
Deduplicate code of getStringResultFromRegexArray methods in Utils
AudricV Apr 2, 2022
67288a0
[YouTube] Fix extraction of embeddable age-restricted videos, fix ext…
AudricV Apr 2, 2022
70812fa
Update YouTube stream mocks and disable YoutubeStreamExtractorRelated…
AudricV Apr 2, 2022
b30e341
Add link to learn more about the issue which makes YoutubeStreamExtra…
AudricV Apr 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
622 changes: 422 additions & 200 deletions ...tor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion ...ava/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeChannelExtractor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package org.schabi.newpipe.extractor.services.youtube.extractors;

import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.DISABLE_PRETTY_PRINT_PARAMETER;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.YOUTUBEI_V1_URL;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.addClientInfoHeaders;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.fixThumbnailUrl;
Expand Down Expand Up @@ -395,7 +396,8 @@ private Page getNextPageFrom(final JsonObject continuations,
.done())
.getBytes(UTF_8);

return new Page(YOUTUBEI_V1_URL + "browse?key=" + getKey(), null, channelIds, null, body);
return new Page(YOUTUBEI_V1_URL + "browse?key=" + getKey()
+ DISABLE_PRETTY_PRINT_PARAMETER, null, channelIds, null, body);
}

/**
Expand Down
5 changes: 3 additions & 2 deletions ...org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package org.schabi.newpipe.extractor.services.youtube.extractors;

import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.DISABLE_PRETTY_PRINT_PARAMETER;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.YOUTUBEI_V1_URL;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.addClientInfoHeaders;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.extractCookieValue;
Expand Down Expand Up @@ -90,8 +91,8 @@ public void onFetchPage(@Nonnull final Downloader downloader)
final Map<String, List<String>> headers = new HashMap<>();
addClientInfoHeaders(headers);

final Response response = getDownloader().post(YOUTUBEI_V1_URL + "next?key=" + getKey(),
headers, body, localization);
final Response response = getDownloader().post(YOUTUBEI_V1_URL + "next?key=" + getKey()
+ DISABLE_PRETTY_PRINT_PARAMETER, headers, body, localization);

initialData = JsonUtils.toJsonObject(getValidJsonResponseBody(response));
playlistData = initialData.getObject("contents").getObject("twoColumnWatchNextResults")
Expand Down
3 changes: 2 additions & 1 deletion ...org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMusicSearchExtractor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package org.schabi.newpipe.extractor.services.youtube.extractors;

import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.DISABLE_PRETTY_PRINT_PARAMETER;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.fixThumbnailUrl;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getTextFromObject;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getUrlFromNavigationEndpoint;
Expand Down Expand Up @@ -60,7 +61,7 @@ public void onFetchPage(@Nonnull final Downloader downloader)
final String[] youtubeMusicKeys = YoutubeParsingHelper.getYoutubeMusicKey();

final String url = "https://music.youtube.com/youtubei/v1/search?alt=json&key="
+ youtubeMusicKeys[0];
+ youtubeMusicKeys[0] + DISABLE_PRETTY_PRINT_PARAMETER;

final String params;

Expand Down
4 changes: 3 additions & 1 deletion ...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubePlaylistExtractor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package org.schabi.newpipe.extractor.services.youtube.extractors;

import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.DISABLE_PRETTY_PRINT_PARAMETER;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.YOUTUBEI_V1_URL;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.addClientInfoHeaders;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.extractPlaylistTypeFromPlaylistUrl;
Expand Down Expand Up @@ -317,7 +318,8 @@ private Page getNextPageFrom(final JsonArray contents)
.done())
.getBytes(StandardCharsets.UTF_8);

return new Page(YOUTUBEI_V1_URL + "browse?key=" + getKey(), body);
return new Page(YOUTUBEI_V1_URL + "browse?key=" + getKey()
+ DISABLE_PRETTY_PRINT_PARAMETER, body);
} else {
return null;
}
Expand Down
4 changes: 3 additions & 1 deletion ...java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeSearchExtractor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package org.schabi.newpipe.extractor.services.youtube.extractors;

import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.DISABLE_PRETTY_PRINT_PARAMETER;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.YOUTUBEI_V1_URL;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getJsonPostResponse;
import static org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getKey;
Expand Down Expand Up @@ -239,7 +240,8 @@ private Page getNextPageFrom(final JsonObject continuationItemRenderer) throws I
final String token = continuationItemRenderer.getObject("continuationEndpoint")
.getObject("continuationCommand").getString("token");

final String url = YOUTUBEI_V1_URL + "search?key=" + getKey();
final String url = YOUTUBEI_V1_URL + "search?key=" + getKey()
+ DISABLE_PRETTY_PRINT_PARAMETER;

return new Page(url, token);
}
Expand Down
Loading