Filter/generate Lists.txt

[mspykerez]

Hey you! It's me it's your boy! Hows it going ? Any progress report ? :)

I was looking around internet archive and found a what appears to be a great no-intro 3DS collection by archiver_2020 ... the problem is that its over 1TB in size so I was thinking of ways to trim the download and I wonder if it is possible to filter a simple given list.txt containing all the names of the games so I can manually download the names on the list ?

Another alternative would be to generate a list.txt of some sorts from a given DAT file and apply the scoring system to it so then I can manually select all the files on the list for download with tools like https://torrent-file-editor.github.io if I download via torrent or Jdownloader to download from internet archive (Jdownloader link-grabber supports filtering by regular expressions and whatnot).

If anyone has a better way on how to trim this download, plz let me know because I really do not want to download over 1TB in files where many of them are just going to trash.

[andrebrait]

Well, you can already do it by redirecting the output to a file if you don't set an input directory (not sure how to do it on non-UNIX systems, but on UNIX-like OSes (and possibly cmder for Windows) it's just adding > file.txt to the end of the command)

But that is only going to give you a list of names, not URLs

[mspykerez]

You mean like this ?
python generate.py -r EUR,USA -l en --all-regions-with-lang -d "Nintendo - Nintendo 3DS (Decrypted) (Parent-Clone) (20200422-094756).dat" > list.txt

Says: WARNING [z009 - Super Mario (USA) (Demo) (Kiosk, E3 2011)]: no ROMs found in the DAT file

edit: oh wait it indeed generated a list on the script directory!

[mspykerez]

Another thing, can you input 2 .dat files and set a preference of one over the other ? (like for when we have a .dat for digital and one for non-digital games of the same system, some will have been released in both formats but I would prefer giving priority to one over the other, this should also work for when we have the same game with different filetypes).

[mspykerez]

Now that I think more about it, it would be a good idea to 'compare' in some way different platforms where the same game is present and give it a preset priority to one platform over the others.
Possible or nay ?

[andrebrait]

Most likely nay. The thing is that there's no programmatically sure way to compare games across platforms (that is, between DAT files), unless I go for the name of the game and do some fancy, but possibly inaccurate anyway, text similarity comparison. That is, unfortunately, not easy to do

…

On Wed 13 May 2020, 00:23 mspykerez, ***@***.***> wrote: Now that I think more about it, it would be a good idea to 'compare' in some way different platforms where the same game is present and give it a preset priority to one platform over the others. Possible or nay ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#24 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLX5OQLHEANBHFBWARVFJ3RRHD7ZANCNFSM4M6HKCAQ> .

[mspykerez]

humm k was just a thought thou, keep it going 👍

[mspykerez]

OK I finish downloading this set and it appears to be encrypted Roms so I have to decrypt them in order to play. The thing is before you decrypt the hashes matches exactly the encrypted-numbered-no-intro DAT but after you decrypt them they appear to have a different hash than the ones on the decrypted-numbered-no-intro DAT making the 1g1r script not picking any roms even if I use --no-scan it still doesn't pick any Roms by name.

This is an example of how the name is on the set: "3DS1706 - Cooking Mama - Sweet Shop (Europe)(M5).3ds"

And this is how it is on the DAT file: "1706 - Cooking Mama - Sweet Shop (Europe) (En,Fr,De,Es,It)"

How do I make it pick decrypted Roms then?

[andrebrait]

Hm, if neither the name nor the hash match, it's tough to come up with something.

As far as the name goes, it seems you just have to remove the prefix "3DS" from the files, or add them to the DAT.

Good thing you bumped into this one.

Could you calculate the SHA-1 hash of the decrypted ROM and compare it to the entry in the decrypted DAT?

Are you able to check if decryption was done correctly too?

Also, is there a header file entry in the DAT's header? (You can attach it here if you don't know how to check that)

[mspykerez]

Maybe I'm doing something wrong but It's not selecting any Roms even if I copy-paste the name of the Rom on the DAT file to my rom.3ds be it with or without --no-scan, maybe it's checking for something else like the file size ?

This is the MD5 for the Rom I have decrypted: 9E4BC01C74DB4098286B2640B63CC483
And this is the one on the DAT file: 209f6b1cfe1822442ebe9cb469a7c888
And this is the encrypted one that matches the encrypted-DAT file and Rom file: 73A6E2B2DA9878CAA158150BC331EDFE

The decryption was done with this tool: https://gbatemp.net/threads/batch-cia-3ds-decryptor-a-simple-batch-file-to-decrypt-cia-3ds.512385/
I don't think there's any header.

ps. I notice that the tool I use to decrypt also trims the files, dunno if that's a problem or not...

[mspykerez]

Further testing concludes that --no-scan is not working at all even with the original encrypted DAT+Rom, without '--no-scan' the original encrypted DAT+Rom is detected and moved plus it appears that it automatically changes the name of the Rom to the one on the DAT file, any way to disable that ?