Low playback performance: Pause to fill buffer, switch bitrate or DP

[traumschule]

Atlas should buffer some seconds before (re)starting playback. If it fails to fill cache fast enough from one provider, it could

• try another DP
• request content from different providers at the same time
• switch to lower bitrate
• debug mode: inform user about bitrate and download speed

Originally reported by bedeho:

We need to fix this, oh my
https://www.loom.com/share/ce91e29b529e4cd69a36c165cabd2209
@tomato @l1dev ... who is distributor lead? alexznet ... looking forward to reverse lookups in Pioneer

Also:

@tomato @l1dev Feels to me like we need some sort of internal group or initiative in the DAO to get all of this performance stuff in order, working flawlessly. Its a complex problem, and the solution will not be a one time fix, but more importantly setting up ongoing monitoring and error handling routines for leads and council.
Who can take charge and fix this?

Lastly:

Lastly: I think distributors are basically not doing a good job, as evaluated by Atlas performance. I made a Loom about this. I really wisht he council would get more focused on this, even if metrics have not been updated to emphasise this yet, because its just such an obvious and costly problem.
Can someone take charge of this problem, starting to analyse what is causing it, what better monitoring is required, better hardware, etc.?

Observations

Some facts with improvement ideas, should be weighed based on relevance:

• Atlas makes 80 requests before the video starts playing. => light player mode Ideas: Start playback before loading other thumbnails, detect user location, offer download link #2282
• The player stops, continues playback, stops, this repeats in a loop. => fill cache before continuing playback
• The video has extremely high bitrate (170 mps = 17.5 MB/s) which is upper HD quality => offer lower bitrate
• The player Atlas uses does not buffer but processes split-seconds of data as they arrive (and forgets after playback). => smarter buffering
• Atlas is unable to load and cache the whole video. User has to "Save as" manually to watch with a local player. => prefetch whole video
• When atlas finished playback and user repeats playback, video data is not played from cache but loaded from the same provider again. => reuse cache for replays
• Atlas uses persistent asset resolution to avoid random swithching between providers when reloading the same asset.

This ensures that one asset will not be fetched from multiple storage providers if present in more than one location.

Due to (the less smart) random provider selection this might not always be the fasted / closest provider available. #2748

Solutions

This is a complex problem with multiple variables:

• bitrate: main (limiting) factor for clients to fetch a video fast Store videos in different qualities, audio-only version joystream#3806
• provider latency: most relevant for small assets like fetching lots of thumbnails, discussed in Question: Fetching video assets #2756
• provider speed: most relevant to stream a video (not an issue in this case, see below)

To determine for each case what the actual bottleneck is and finding the best solution all above need to be taken into account.

• lower bitrate: most effective but hardest to implement because it requires fundamental changes to the Joystream CDN architecture.
• switch DP: if one provider has low speed, switch to another one (would not help in this case because the bitrate is so high) Atlas does not use the closest / fastest provider available. #2553
• fetch from multiple sources: if all providers are too slow on their own (HD videos), fill buffer from multiple DP (assuming client bandwidth is high enough)
• buffer-ahead: fill cache of X seconds before playback starts (the user could configure that but it violates current atlas design philosophy to minimize available UI options)
• improve/switch video library: for more control over buffering status and targets

Basically player.buffer() returns an object with just two methods(start, end) and property length. Length tells how many buffered time ranges are present in the video timeline, I just need to find and take the one which is close to the current video time. source

Lower bitrate

There is no point to deliver HD quality to mobile devices. To offer multiple bitrates the Joystream CDN needs to be redesigned because unfortunately it was not created with this feature in mind.

This job needs to be done by SP or external services because client-side browser based transcoding is not feasable:
After a video is uploaded it would be reprocessed into multiple formats based on (platform wide or per video) target audiences from mobile (180kps) over desktop (720kps) to HD for everything above.

Similar was suggested in Joystream/joystream#2793 (comment)

The platform should have the capability to expose the format and other characteristics of the media via some metadata
To design some kind of mechanism where in the platform has the capability to allow and comprehend some kind of configurable features like:
i. Adaptive bitrates while uploading, and as well as streaming via Argus
ii. Allow application to dictate the streaming formats like HLS, HDS, MPEG etc.,
It is important to define where or who is expected to implement the media format conversions.
a. The applications that may be built on top of joystream platform may expect some kind of metadata information either from Colossus or Argus to enable it to convert the ingest stream of data to desired formatted output.

Provider performance

To test speed for the asset displayed in above loom video https://play.joystream.org/video/26682
My player chose to load from this provider: https://joystreamoxygen.com/distributor/api/v1/assets/14225
Performing speed tests from different locations:

wget https://joystreamoxygen.com/distributor/api/v1/assets/14225
--2022-05-28 12:41:15--  https://joystreamoxygen.com/distributor/api/v1/assets/14225
Resolving joystreamoxygen.com (joystreamoxygen.com)... 65.108.97.154
Connecting to joystreamoxygen.com (joystreamoxygen.com)|65.108.97.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1001281430 (955M) [video/mp4]
Saving to: '14225'

14225                                                       100%[===>] 954.90M   105MB/s    in 9.1s    

2022-05-28 12:41:24 (105 MB/s) - '14225' saved [1001281430/1001281430]

wget https://joystreamoxygen.com/distributor/api/v1/assets/14225
--2022-05-28 14:42:43--  https://joystreamoxygen.com/distributor/api/v1/assets/14225
Resolving joystreamoxygen.com (joystreamoxygen.com)... 65.108.97.154
Connecting to joystreamoxygen.com (joystreamoxygen.com)|65.108.97.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1001281430 (955M) [video/mp4]
Saving to: ‘14225’

14225                                                       100%[===>] 954.90M   110MB/s    in 8.8s    

2022-05-28 14:42:52 (109 MB/s) - ‘14225’ saved [1001281430/1001281430]

wget https://joystreamoxygen.com/distributor/api/v1/assets/14225
--2022-05-28 13:43:41--  https://joystreamoxygen.com/distributor/api/v1/assets/14225
Resolving joystreamoxygen.com (joystreamoxygen.com)... 65.108.97.154
Connecting to joystreamoxygen.com (joystreamoxygen.com)|65.108.97.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1001281430 (955M) [video/mp4]
Saving to: ‘14225’

14225                           4%[=>                                               ]  42,68M  1,67MB/s    eta 8m 25s

The first two are SP/DP themselves, the third is a home connection (without VPN or proxy) and the achieved download speed is lower than the bitrate.

As you see video size of 955M is enormous for one minute video: 130000 to 170000 kbps, that is avg. 140 mbps or 17.5 MB/s.
Cache: 150 MB (9.1 sec)

For comparison space used per 1 minute video

• 144 kps: 8 MB per minute
• 720 kps: 43 MB per minute
• 170 mps: 1 GB per minute

Conclusion

In contrast to the original assumption by the reporter current DWG is not to blame. The described playback issue is related to the video library atlas uses and the Joystream CDN architecture lacking bitrate handling based on target audiences / device.