[YouTube] Add more parameters to InnerTube requests, use the iOS client for livestreams and fix extraction of embeddable age-restricted videos and contents with a warning before playback

[AudricV]

• I carefully read the contribution guidelines and agree to them.
• I have tested the API against NewPipe.

This PR adds more parameters to InnerTube requests:

• the playbackContext JSON object sent in player requests of the desktop web client, only for the web client (it seems it was needed to avoid some throttling);

• a parameter not sent by all clients on player request bodies (I saw it on the web client and I do not see it right now): cpn, aka contentPlaybackNonce. It seems it's like a sort of client authenticity sent by all official clients on videoplayback URLs: it uses strong random values to generate a 16-character-string. This PR adds it on both parts (request bodies of player requests and videoplayback URLs) for all clients;

• the id of the video as a URL parameter of the InnerTube requests and a t parameter, on which we need to do more researches (a 12-character string which is also unique to each player request), only sent for mobile clients (and only done by the YouTube apps);

• for player and next requests: racyCheckOk, for age-restricted contents (doesn't seem to do something when used anonymously), contentCheckOk, to allow playback of contents with a warning before playing them because of the sensitive topics they contain;

• a new query parameter, set to false: prettyPrint. YouTube was returning pretty printed responses before but that's not the case anymore, because they added this parameter which reduces a lot response sizes (but not really transfer size): take a look at the following screenshot:

  

  (Where Transfert means Transfer and Taille means Size in French.)

This PR also supersedes #732 and fixes most of its problems:

• the fetch of the iOS player is only enabled for livestreams;
• the fetch of the Android player is disabled only for livestreams;
• extractor clients can force the fetch of the iOS and Android clients, for every stream type, by using two static methods in YoutubeStreamExtractor;
• streams available are not only 30fps streams (the way to get 60fps streams was discovered very recently, see [Findings] YouTube and HLS manifests on Apple clients (iOS and MacOS) #680);
• the HLS manifest was only used if an HLS manifest cannot be found on both web and Android clients.

Like said in #732, fetching the iOS client (with a deviceModel field in the JSON payload which matches a recent Apple device model (see https://gist.github.com/adamawolf/3048717) to get 60 fps streams and an iOS user agent to get a single HLS manifest with a regular streamingData JSON object instead of an hlsFormats object for livestreams) allows to get an HLS manifest for regular videos and an HLS manifest with separated video and audio for livestreams.

This PR fixes some bugs with the extraction of the client version and the key and use a very lighter way than the current one, still used as a fallback, to find the client version and key of YouTube and YouTube Music, used by their service worker, by respectively fetching https://www.youtube.com/sw.js and https://music.youtube.com/sw.js. A new method in the Utils class has been added to decrease code duplication and increase readability of my changes.

Harcoded client versions and mocks have been also updated to a more recent version.

This PR finally fixes the extraction of embeddable age-restricted contents (the ones available before), by using the new way discovered to get streams of them, as written in TeamNewPipe/NewPipe#8102.

NewPipe Debug APK to test the changes (source code): app-debug.zip

Fixes TeamNewPipe/NewPipe#8102, fixes TeamNewPipe/NewPipe#8103, closes #680 (for the extractor implementation).

[Stypox]

Note that this cpn parameter blocks (at least makes harder)

You could just have something that seeds the random number generator used by that parameter before running the tests

[Stypox]

In Utils.java add this code:

// at the top
private static final SecureRandom random = new SecureRandom();

// ...

// then add these methods
/**
 * Generates a random string using the secure random device {@link #random}.
 * {@link #setRandomSeed(long)} might be useful when mocking tests.
 * @param alphabet which characters to use
 * @param the length of the returned string
 * @return a random string of the requested length made of only characters from the provided alphabet
 */
public static String randomStringFromAlphabet(final String alphabet, final int length) {
    final StringBuilder stringBuilder = new StringBuilder();
    for (int i = 0; i < length; ++i) {
        stringBuilder.append(alphabet.charAt(random.nextInt(alphabet.length())));
    }
    return stringBuilder.toString()
}

/**
 * Seeds the random device used for {@link #randomStringFromAlphabet(String, int)}. Use this in tests so that they can be mocked as the same random numbers are always generated. This is not intended to be used outside of tests
 * @param seed the seed to pass to {@link SecureRandom#setSeed(long)}
 */
public setRandomSeed(final long seed) {
    random.setSeed(seed);
}

[AudricV]

Tests runs fine with the mock downloader on my computer (I updated mocks) but not in the CI. I applied what Stypox said, but it seems to be not really working. Someone has an idea for that?

[XiangRongLin]

@TiA4f8R My guess it that setSeed(), does not produce the same effect as when it is passed in through the contstructor. See https://docs.oracle.com/javase/8/docs/api/java/security/SecureRandom.html#setSeed-byte:A-

Reseeds this random object. The given seed supplements, rather than replaces, the existing seed. Thus, repeated calls are guaranteed never to reduce randomness.

You could try it out locally if it always returns the same values

[Stypox]

Oh, I assumed setSeed just set the seed. Anyway, setNumberGenerator also looks good :-)

[Stypox]

Code looks good to me (except comments, but you have probably a reason for making them like that, so don't make changes, I just wanted to open some other discussions) :-)

[Stypox]

@litetex should we proceed with merging this? It is ready in my opinion, and related tests succeed.

[litetex]

Minor changes

[litetex]

Shouldn't this value be in a constant?

[AudricV]

Probably not right now, but we should probably refactor how clients are managed in the extractor later, to deduplicate code used.

[litetex]

Not finished yet, but no time left for today.

Here my current review:

[FireMasterK]

I had a quick peek :)

[FireMasterK]

Why not just use a regular Random instance? I don't think we're doing anything of cryptographic importance

[AudricV]

If you remember correctly, I said in several places the JavaScript clients are using the window.crypto.getRandomValues, so I used this to mickmick the best official clients.

By doing some basic reverse engineering, even if I am not sure, they are also using this in the Android app.

[FireMasterK]

Could we potentially introduce an object/class for this than using a String[] array? (To remove the need to guess what each index is)

[AudricV]

This should be done when we will refactor clients (with requests bodies buildings too, see #780 (comment)).

[litetex]

Code LGTM now

• Please create followup issues/PRs

APK seems to work fine :)

• Age-restricted videos work again (for now)
• You can rewind YT live streams now, nice 👍

Good work

PS: I think I have to rebase my invidious PR after this 😆