GDown don't retrieve and don't download entire folder contents #346
Comments
ilentt
changed the title
|
It has a limit of downloading 50 files from a folder link |
Maybe, although I already considered 50 files limitation by below command but It still do not work as my expectation. Do you have any idea to solve this problem? |
|
|
|
Can we split the list file into several part, each part have maximum 50 files. By this ways we can download whole folder regardless how many file in folder. Why we can't do by this way? |
The reason only able to download 50 files is that the gdown library, by default, uses the "cookie" method to access Google Drive files, and Google Drive has a limitation of 50 for direct downloads using cookies. You can use the Google Drive API along to download >50 files with the google-api-python-client library. |
Once you make the python file, what do you do? |
|
Yes, 50 limit is hard set due to the behavior of Google Drive. It returns the first 50 files, and gdown can't know what's the rest. To resolve this, probably we need to use browser emulator like Selenium to let the Javascript to load the rest of files. |
|
Using the below URL format, it seems to be possible to list more than 50 files without using cookies or the API. The output is in plain HTML. I'm not sure whether this method is officially supported or deprecated. https://drive.google.com/embeddedfolderview?id=16bB1g_gq7KDiJX2j9H323LyTxXjbg1S2 The main limitation I can see is that it takes a while for the server to generate the list (it may show more entries when you reload the page), and the page can render well before the folder has been fully indexed. The folder linked above contains 5,000 files, and it took several minutes for them all to be listed correctly. I don't know if there is a maximum number of files that can be shown. Note that the order shown on the page also doesn't match the order in which the files were indexed, i.e. "empty_file_999" was indexed before "empty_file_1000", but the latter is shown at the top of the list. |
|
@overcast07 thank you so much for this. I used this to write a custom script to download every file in a folder, and it seems to work beautifully! Maybe something similar can be part of gdown in the future. |
|
@DaniDipp can this script recursively download all the files and subfolder in a specific folder? |
|
@hxhcreate not at the moment, but that should be relatively easy to implement. But this discussion isn't related to this ticket. The Gist has it's own comment section |
|
I added the support for utf-8 named folders at here |
|
We can list all the file IDs and corresponding file names in a Google Drive folder using Google Sheets's API script easily: I modified the JS script to list the file IDs and names into distinct columns. function myFunction() {
var ss=SpreadsheetApp.getActiveSpreadsheet();
var s=ss.getActiveSheet();
var c=s.getActiveCell();
var fldr=DriveApp.getFolderById("FOLDER_ID");
var files=fldr.getFiles();
var names=[];
var ids=[];
while (files.hasNext()) {
f=files.next();
names.push([f.getName()]);
ids.push([f.getId()])
}
s.getRange(c.getRow(),c.getColumn(),names.length).setValues(ids);
s.getRange(c.getRow(),c.getColumn() + 1,names.length).setValues(names);
s.getRange(1, 1, s.getLastRow(), s.getLastColumn()).sort({column: c.getColumn() + 1, ascending: true});;
}With all the file IDs and names, we can download them using gdown or Google developer OAuth 2.0(Use this when gdown doesn't work). |
|
@DaniDipp @hxhcreate @fish4terrisa-MSDSM Thank you guys! |
|
Wow this is amazing! Thank you @DaniDipp @hxhcreate @fish4terrisa-MSDSM |
Provide environment information
Environment information
What OS are you using?
Linux Mint 21.3
Describe the Bug
Execute
gdownfrom terminal sometime it do not retrive and do not download entire content in folderCommand
Log
Expected Behavior
Download completely entire content in the folder
To Reproduce
The text was updated successfully, but these errors were encountered: