refactor download manager

[theScrabi]

• [5] I carefully read the contribution guidelines and agree to them.
• [no] I checked if the issue/feature exists in the latest version.

Gigaget wase once a downloader distributed on fdroid. However it is discontinued, tho we copypasted their code into NewPipe, and made gigaget our downloader.
Now we have to get rid of it, since its causing a lot of trouble.
Related issues:
#961
#779
#355
#345
#315

Maybe we could replace it with a Download Manager available on Android Arsenal. Feel free to browse a little and find a downloader you think fits to NewPipe.

A new download manager must have these abilities:

• using SAF (for downloading to sdcard)
• Be singe threaded (one thread per download)

Nice to have:

• Prevent/restrict download on mobile connection
• download files with external download manager Option to download using external download manager #9142
• easy accessing videos form download manager
• make it a lib and put it back on android arsenal
• show download status in titlebar/notification

[ghost]

@theScrabi From your link, Fetch looks like the most active and up to date project.

[theScrabi]

Uuuh fetch also has a wifi only switch :)

Does it also support notification?

[ghost]

@theScrabi No idea. There's not a lot of open source download managers for Android out there. Gigaget and uGet are pretty much the most well known ones. The ones you listed are more like side projects that aren't ready to be posted on the Play Store or even F-Droid.

You may have to stick with Gigaget for a while longer or even find somebody to fork Gigaget and fix the issues you mentioned. Or perhaps uGet can be ported to Newpipe? Not sure how easy that would be compared to porting Gigaget.

[miloslavnosek]

uGet is just frontend for CLI downloader aria2. Perhaps aria2 can be used in newpipe?

[ghost]

@LosEagle Actually, its a frontend for curl and aria2, and only curl is supported by default. aria2 support is included in uGet but you have to enable it in the settings before you are able to use it.

However, aria2 on it's own might be a good option but I'm not sure how easy it will be to add to Newpipe.

[ghost]

Well, using a native library would add another level of hell to NewPipe. For example Android's PackageManager is totally not reliable with installing native libraries, which can be avoided with ReLinker, and some other stuff. I'd just write something using OkHttp.

[theScrabi]

Well, using a native library would add another level of hell to NewPipe.

Oh yes. Well it was nice if there is already some external downloader available.
@wb9688 could you please write the downloader as a module for for gradle, like the NewPipeExtractor.

[ghost]

could you please write the downloader as a module for for gradle, like the NewPipeExtractor.

@theScrabi: I don't think we should do that, since we'll need to have things specific for NewPipe (for example getting a new stream URL from NewPipeExtractor when the URL it's using to download expires). I'll write a new downloader for NewPipe though.

[karyogamy]

Has anyone looked at this? It looks quite mature and modular.

[ghost]

@wb9688 Thanks for your efforts and be sure to take your time on this downloader project. Please make sure the downloader makes multiple simultaneous connections while downloading like Gigaget does, though, or I predict many people will complain.

[theScrabi]

So I worked myself into this a bit further, and what i found out is, that accessing files via Unix file system (File class) is not possible any more. Instead you would want to use the Android Storrage Access Framework, which comes with its on classes and methods to handle files.

Uri instead of file names

So instead of filenames SAF uses Uri, to access files. What we would want to do in order to access SD card files is getting the Uri for the root directory, and cooncan the path to our file onto the end of it.

InputStream/OutputStream

Files can not be handled with File class anymore, however from android we can request InputStream/OutputStreams and create/delete fiels.

gigaget

Our current download manager is bad rubbish, it's old code someone wrote to test his andorid skills, and therefore its not quite reliable and up to date. Also it is based ontop of the idea that you can download one file with parallel threads.

What I've seen so far is that there is no real replacement for a download manager that uses SAF on android arsenal (notify me if you find something different).

So we have two options here:

1. Refactor gigaget, so it does not use threads, and supports SAF
2. Write a downloadmanager from scratch

File picker

Our filepicker is cool, but it does also not use SAF, and on top of that its sometimes a big buggy, here it would be cool to use the android SAF file picker, but that is not relay usable until android 6.0 ... so meh, we need to stick to it, and try to translate from filepath to uri.

[ghost]

Didn't @wb9688 already say he was writing a new downloader? I don't think Gigaget is salvagable. Fixing it will probably be more work than wring a barebones downloader from scratch, and all of the mature alternatives have already been mentioned in this issue.

[ghost]

Yeah, I'll write a new download manager. I'll throw away that GigaGet garbage and then write something new that uses OkHttp and SAF. I'll also make a better UI and integrate jCodec.

[placidcasual98]

So what would the time scale will I be before we can download to SD cards?

[theScrabi]

This is hard to tell. It's done when its done, however it got requested so often that I think things will get rolling soon :D

@wb9688 I don't want to sound unpolide, but you also said you where going to implement support for AndroidTV, and PipeCast. So far you started those projects than abandon them.
I think it's ok if you work on things, but notify us about them once you have a decent amount of code, because otherwise people might not consider giving it a shot as well, and when you abandon them again the project is basically blocked.

[ghost]

@theScrabi: I'm sorry, I just don't have as much time anymore. Others are free to do it too, since I don't have much implemented of the new download manager yet.

[mauriciocolli]

I'm experimenting with this library: okdownload, which is a rewrite of the popular one that @karyogamy posted previously (with improvements in documentation, testing and other aspects).

I may have a working proof-of-concept in the next weekend.

[snappyapple632]

So with this issue seemingly fixed in 0.17, can this issue be finally closed, that is, when @kapodamy claims his bounty money?

[PeterHindes]

would youtube-dl be a good candidate? It's a separate open source program primarily used from the command line. It could likely be embedded in NewPipe.

[Stypox]

We know about youtube-dl (we even take inspiration from there for the extractor). But youtube-dl uses Python, which can't be run on Android unless with a heavy and slow emulator

[wb9688]

@PeterHindes and @Stypox: Plus youtube-dl isn't even a download manager and won't solve any of the issues mentioned here.

[PeterHindes]

Fair enough. I don't know much about youtube-dl, but I believe it has a binary form, IDK if it still packages python there though.

[wb9688]

@PeterHindes: The issue here is not retrieving the URL of the file, which NewPipeExtractor does wunderfully. The problem is actually downloading the file from the URL that NewPipeExtractor provides. youtube-dl would be an alternative for NewPipeExtractor, which we don't need, besides that youtube-dl lacks a lot of features.

[PeterHindes]

Maybe I'm not understanding. Does NewPipe Extractor provide a URL that you could for example download in a browser? I'm not familiar with the details of the process of downloading a YouTube video.

[opusforlife2]

Yep. It's Newpipe code that downloads the video. Look at the source file names.

[KaKi87]

Hello, any news on this ? Thanks

[ghost]

deleted by author

i want to take on this bounty puck - make a user happy with downloading within NewPipe the solution will be a process - a library - in a separate repository, added as clean dependency download process will be implemented in clojure with core.async - vegan magic compiled into java loaded via JitPack from github - like with deps.edn or docker-compose - no maven middlemen, straight from Jesus NewPipe will see it just as another java jar library once working and used - repo can be moved/forked or code copied-included - as TeamNewPipe deems better

simply put
there will be a concise pull request adding a new dependency
resulting in downloading UX being what user expects and wants and fulfilling objectives set in this issue

if there are no objections to such a solution - green light it and i will science the Mark-Watney-potatoes-crap out of it

[opusforlife2]

@Sergei-Udris Have you taken a look at all the issues related to the download manager? If not, I recommend you do so at least once.

[ghost]

deleted by author

> @Sergei-Udris Have you taken a look at all the issues related to the download manager? If not, I recommend you do so at least once.

i did - took a look at all issues mentioned in this thread

i must self-decline though for a different reason - i will not take this issue on
i am looking for bounties - yes, but thinking thoroughly how to do it most mercifully for myself - with highest degree of freedom without creating new worker-programs
clojure jvm - yes, pull request a clean dependency - yes
but - i will go for docker-services, no jvm-libs, to be a "true foreigner" - no linting-style or choice of libraries quarrels with existing app
instead - for example, some golang or multilingual self-hosted docker app
clean dependecy as i described - is exactly a true foreigner, while committing in-place would be like someone stuck in their own country while they are already a foreigner inside
docker - is indeed going true foreigner but in an angrier country-culture one does not want - doing evil servers and services and not whole-foods-plant-based clojurescript and browser-npm fast-unhealthy-snacks world
reason i won't do it - program must be both desktop and mobile, not just one - simple as that

:David-Mitchell i'm not eating a sandwich in WHSmith, you barbarian!

[opusforlife2]

i did - took a look at all issues mentioned in this thread

🤣 Ah no, that's not nearly enough. I was talking about all the related issues in the repo, with the downloader label. Otherwise, what would be the point of refactoring?