Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempt to parallelize YouTube Stream extractor #751

Open
wants to merge 2 commits into
base: dev
Choose a base branch
from

Conversation

FireMasterK
Copy link
Member

  • I carefully read the contribution guidelines and agree to them.
  • I have tested the API against NewPipe.
  • I agree to create a pull request for NewPipe as soon as possible to make it compatible with the changed API.

@FireMasterK FireMasterK marked this pull request as ready for review November 17, 2021 12:33
@FireMasterK FireMasterK mentioned this pull request Nov 17, 2021
5 tasks
@XiangRongLin
Copy link
Collaborator

XiangRongLin commented Nov 17, 2021

  1. Are there currently some performance problems that would warrant such a change and its developer overhead?
  2. If there are, how does it look like with this change. Any statistics or the like?

ps: I did NOT take a real look at the code

@Stypox
Copy link
Member

Stypox commented Nov 21, 2021

Why use Java futures manually? I'd rely on RxJava instead (like we already do in the app)

@litetex
Copy link
Member

litetex commented Nov 21, 2021

Why use Java futures manually? I'd rely on RxJava instead (like we already do in the app)

I think there is no RXJava inside this project.

@Stypox
Copy link
Member

Stypox commented Nov 21, 2021

I think there is no RXJava inside this project.

Yeah, I know. I'd propose to use it, what do you think?

@litetex
Copy link
Member

litetex commented Nov 22, 2021

Yeah, I know. I'd propose to use it, what do you think?

I don't know.

  • RxJava was built for event based programming however there are no events here. Just asynch tasks.
  • It would add a new dependency here; (Larger) dependencies tend to introduce problems...
  • I might be biased because personally the most RxJava code I read so far in NewPipe was horribly complex or caused problems that have been different for debugging

I'm also fine with the current way of using the java native way.
However before we get technical 👉 #751 (comment)

@FireMasterK
Copy link
Member Author

Was there any conclusion to what library to use? Should we go with the default Java API? I'd like to rework on this, one last time.

Copy link
Member

@Stypox Stypox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FireMasterK yes, let's use the default Java API. The current code structure looks good :-)

@FireMasterK
Copy link
Member Author

Are there currently some performance problems that would warrant such a change and its developer overhead?

Yes, currently in NewPipe's case, a minimum of 2 requests are made regardless. We parallelize them, to reduce the time taken by half. Another additional request may be made - ios or android client, depending if it is a live stream or not.

In Piped's case, a minimum of 4 requests are made since I force the android and ios player fetch.

If there are, how does it look like with this change. Any statistics or the like?

Piped's response times have dropped by ~400-500ms on average (it depends on how fast youtube's server responds), this should benefit NewPipe too.

@FireMasterK FireMasterK force-pushed the extractor-parallelism branch 2 times, most recently from ce9f3cb to be69e39 Compare September 7, 2022 08:05
@FireMasterK
Copy link
Member Author

I tried updating the mocks, but it appears to be unable to find some requests, anyone have any idea why this might be happening?

@FireMasterK FireMasterK force-pushed the extractor-parallelism branch 10 times, most recently from b0a0049 to e579898 Compare September 10, 2022 21:34
@FireMasterK FireMasterK marked this pull request as draft November 24, 2022 18:42
Copy link
Member

@Stypox Stypox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some time has passed, sorry... On a second thought this implementation is fine. @AudricV do you have anything against merging this, after it has been rebased? ;-)

@FireMasterK
Copy link
Member Author

It's still a draft since it would have issues with mocks.

What I'm going to do to fix them is remove the cpn variable before checking if a mock request matches a response.

@FireMasterK FireMasterK added the youtube service, https://www.youtube.com/ label Apr 27, 2023
@FireMasterK FireMasterK marked this pull request as ready for review June 30, 2023 16:36
@FireMasterK FireMasterK force-pushed the extractor-parallelism branch 2 times, most recently from 1dd3faa to 35f40b7 Compare June 30, 2023 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
youtube service, https://www.youtube.com/
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants