trying to only use next endpoint for the major data #5003

unixfox · 2024-10-13T17:47:01Z

Description

This PR aims to avoid throwing a raw error and instead display all the data that we can extract from the /next endpoint but without displaying the "player" (video.js) element.

This is one big step towards #4985. Invidious will have to be able to display the watch page even when YouTube is blocking the IP.

This will give the same result as on www.youtube.com:

This way the user can still view the title of the video, accessing the channel page, viewing the comments and much more which is not yet blocked.

What changed?

The watch page is displayed properly but without the player if an error is found.
All the important data like title, viewcount is fetched from /next endpoint is unable from the /player endpoint.
The error message now throws the reason AND the subreason. Makes it clearer to the user what's really happening.

TODO

Integrate the embed page with the changes
Display the subreason on the watch page when there is an error
Fix description
Add mocks for a video without videoDetails

Try these videos ID for testing - on both a blocked IP and a not blocked one

98TyXNPdMRo (private video)

7KSfDyainYU (georestricted video)

KIuDBxqz__k (familySafe no)

yqaS07znJWY (unlisted video)

aaaaaaaaaaaa (notfound video)

Fixes

Closes [Feature request / Enhancement] Allow viewing video info when Youtube blocks video access for signed-out users. #4744

ic-scm · 2024-10-17T08:27:57Z

Thank you very much for this. Looking at instance yewtu.be which uses invidious-custom, it seems to be working just as expected, exactly as described in my linked issue, and with a 100% success rate.

Thank you for this great work - this is a huge step towards making Invidious convenient to use again. For me personally, it solves the entire problem.

The only thing I can potentially suggest is for the video thumbnail to be loaded and displayed on the page, separately from the broken video player. But that would be just a nice-to-have feature, and I'm guessing it may not be trivial to implement, when the official Youtube website just shows a black error screen, with the thumbnail nowhere to be seen.

syeopite · 2024-10-18T19:11:07Z

src/invidious/videos.cr

-  if reason = info["reason"]?
+  if info["reason"]? && info["subreason"]?
+    reason = info["reason"].as_s
+    puts info


Please remove the debug puts here

syeopite · 2024-10-18T19:33:39Z

src/invidious/helpers/errors.cr

-  next_steps = error_redirect_helper(env)
+  error_message = <<-END_HTML
+    <div class="error_message">
+      <h2>#{translate(locale, "error_processing_data_youtube")}</h2>
+      <p>#{translate(locale, message)}</p>
+      #{error_redirect_helper(env)}
+    </div>
+  END_HTML


Your changes here will show the Error while processing the data sent by YouTube message on any InfoException while also removing the next steps message on all InfoExceptions

Example:

And InfoExceptions are raised in places unrelated to YouTube as well such as when a user inserts the wrong password.

Related: #4651

SamantazFox · 2024-10-19T14:41:32Z

src/invidious/views/error.ecr

@@ -4,5 +4,4 @@

 <div class="h-box">
    <%= error_message %>
-    <%= next_steps %>


The error page is used for everything and not just the watch page, so it must contain at least the "refresh" and "switch instance" buttons. Alternatively, we could split the error page logic for the watch page from the rest.

SamantazFox · 2024-10-19T14:45:02Z

src/invidious/views/watch.ecr

+            <%= translate(locale, "error_from_youtube_unplayable") %> <%= video.reason %>
+        </h3>
+        <h3>
+            <%= translate(locale, "next_steps_error_message") %>


You probably want to use error_redirect_helper(). This error message alone (without any actions listed underneath) will be very confusing.

SamantazFox · 2024-10-19T14:47:26Z

src/invidious/views/watch.ecr

+            <span id="refresh-page">
+                <a href="<% env.request.resource %>"><%= translate(locale, "refresh_page") %></a>
+            </span>


Why did you add this link here? All browsers allow to refresh the page natively, and the close procimity with the "watch on youtube" link is prone to miss-clicks.

SamantazFox · 2024-10-19T14:53:58Z

src/invidious/videos/parser.cr

+  if !(video_details = player_response.dig?("videoDetails"))
+    video_details = {} of String => JSON::Any
+  end


Here you can use the more compact form:

Suggested change

if !(video_details = player_response.dig?("videoDetails"))

video_details = {} of String => JSON::Any

end

video_details = player_response.dig?("videoDetails")

video_details ||= {} of String => JSON::Any

SamantazFox · 2024-10-19T15:06:28Z

src/invidious/videos/parser.cr

+    published_txt = video_primary_renderer
+      .try &.dig?("dateText", "simpleText")
+
+    if published_txt.try &.as_s.includes?("ago") && !published_txt.nil?
+      published = decode_date(published_txt.as_s.lchop("Started streaming "))
+    elsif published_txt && published_txt.try &.as_s.matches?(/(\w{3} \d{1,2}, \d{4})$/)
+      published = Time.parse(published_txt.as_s.match!(/(\w{3} \d{1,2}, \d{4})$/)[0], "%b %-d, %Y", Time::Location::UTC)
+    else
+      published = Time.utc
+    end


Here I'd recommend using "scheduledStartTime" (see below) in the player response, as it's more accurate and doesn't need complex parsing logic.

{ "playabilityStatus": { "status": "LIVE_STREAM_OFFLINE", "reason": "This live event will begin in 32 hours.", "playableInEmbed": true, "liveStreamability": { "liveStreamabilityRenderer": { "videoId": "hUoukmX_BRQ", "offlineSlate": { "liveStreamOfflineSlateRenderer": { "scheduledStartTime": "1729468800", "mainText": { "runs": [ {"text": "Live in "}, {"text": "32 hours"} ] }, "subtitleText": { "simpleText": "October 21 at 12:00 AM" } } } } } } }

Note: if you plan to fallback to the "subtitleText" value, there is a non-breaking space before AM/PM:

SamantazFox · 2024-10-19T15:29:59Z

src/invidious/views/watch.ecr

@@ -96,7 +98,10 @@ we're going to need to do it here in order to allow for translations.

    <% if video.reason %>
        <h3>
-            <%= video.reason %>
+            <%= translate(locale, "error_from_youtube_unplayable") %> <%= video.reason %>


Ditto: keeping the error message inside a player placeholder.

SamantazFox · 2024-10-19T15:37:34Z

src/invidious/videos.cr

-      # dont error when it's a premiere.
-      # we already parsed most of the data and display the premiere date
-      raise InfoException.new(reason.as_s || "")
+      raise NotFoundException.new(reason + ": Video not found" || "")


That will expand to "Video unavailable: Video not found", which is not very explicit, imo.

SamantazFox · 2024-10-19T15:41:06Z

src/invidious/videos.cr

+    if info.dig?("subreason").nil?
+      subreason = info["subreason"].as_s
+    else
+      subreason = "No additional reason"
+    end


This bloc of code can be summarized to the following, as you're already checking info["subreason"]? falsiness in the parent if

Suggested change

if info.dig?("subreason").nil?

subreason = info["subreason"].as_s

else

subreason = "No additional reason"

end

subreason = info["subreason"].as_s

SamantazFox · 2024-10-19T15:42:49Z

src/invidious/videos.cr

@@ -313,7 +313,7 @@ def get_video(id, refresh = true, region = nil, force_refresh = false)
    end
  else
    video = fetch_video(id, region)
-    Invidious::Database::Videos.insert(video) if !region
+    Invidious::Database::Videos.insert(video) if !region && !video.info.dig?("reason")


You should make a getter dunction inside the Video class for reason/subreason.

SamantazFox · 2024-10-19T15:43:51Z

src/invidious/videos/parser.cr

-        "reason"  => JSON::Any.new(reason),
+        "version"   => JSON::Any.new(Video::SCHEMA_VERSION.to_i64),
+        "reason"    => JSON::Any.new(reason),
+        "subreason" => JSON::Any.new(subreason),


Don't forget to bump SCHEMA_VERSION in the Video class!

unixfox requested review from SamantazFox and a team as code owners October 13, 2024 17:47

unixfox marked this pull request as draft October 13, 2024 17:47

unixfox force-pushed the use-only-next-for-extra-info branch 2 times, most recently from de76e24 to 250cffb Compare October 13, 2024 18:35

unixfox mentioned this pull request Oct 13, 2024

update the mocks to today iv-org/mocks#3

Merged

unixfox force-pushed the use-only-next-for-extra-info branch 2 times, most recently from da00958 to a340868 Compare October 13, 2024 19:53

unixfox requested a review from syeopite October 13, 2024 21:40

unixfox mentioned this pull request Oct 13, 2024

[Feature request / Enhancement] Allow viewing video info when Youtube blocks video access for signed-out users. #4744

Open

unixfox added 2 commits October 14, 2024 00:07

trying to only use next endpoint for the major data

2a40f3b

extract islisted info + throw error only on not found video

9438a6d

unixfox force-pushed the use-only-next-for-extra-info branch from 60a5c1c to 9438a6d Compare October 13, 2024 22:08

fix detecting live_now

126f8fd

syeopite reviewed Oct 18, 2024

View reviewed changes

SamantazFox requested changes Oct 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

trying to only use next endpoint for the major data #5003

trying to only use next endpoint for the major data #5003

unixfox commented Oct 13, 2024 •

edited

Loading

ic-scm commented Oct 17, 2024

syeopite Oct 18, 2024

syeopite Oct 18, 2024 •

edited

Loading

SamantazFox Oct 19, 2024

SamantazFox Oct 19, 2024

SamantazFox Oct 19, 2024

SamantazFox Oct 19, 2024

SamantazFox Oct 19, 2024

SamantazFox Oct 19, 2024

SamantazFox Oct 19, 2024

SamantazFox Oct 19, 2024

SamantazFox Oct 19, 2024

SamantazFox Oct 19, 2024

SamantazFox Oct 19, 2024

trying to only use next endpoint for the major data #5003

Are you sure you want to change the base?

trying to only use next endpoint for the major data #5003

Conversation

unixfox commented Oct 13, 2024 • edited Loading

Description

What changed?

TODO

Try these videos ID for testing - on both a blocked IP and a not blocked one

Fixes

ic-scm commented Oct 17, 2024

Choose a reason for hiding this comment

syeopite Oct 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

unixfox commented Oct 13, 2024 •

edited

Loading

syeopite Oct 18, 2024 •

edited

Loading