Possibilities for handling fake torrents

[gitrickm]

When closing issue #120 "Move to wanted earlier", junalmeida made this comment:

About the bogus torrent file, @gitrickm I'm interested at the effectiveness of your script, but we need to open another ticket to discuss this subject. If your script is really effective to detect a bogus file, I have some ideas to remove it from torrent downloader, set the episode again to wanted and avoid the bogus hash to be redownloaded.

[gitrickm]

Here's some background. Sorry it's long...

I've been manually downloading torrents for a few years, but just recently started using sickbeard with transmission (on a Linux host).

I have 16 active series at the moment. Half of these are also recorded on my cable TV PVR, but I added them to my sickbeard show list to see how well it works and to have a backup for the PVR recording.

I use public trackers, so I'm more exposed to fake torrent files.

I haven't kept track in previous weeks, but it seems the shows that air on Sunday night are more prone to bogus torrents. I have 6 shows download on Sunday. Yesterday, 3 shows downloaded those rar files that direct you to a web site for a password.

The two obvious signs in my case for bogus torrents:

1. the bad torrents are added for download between 3am and 12 noon, whereas the good ones don't appear until 5pm or later (I'm in the mountain time zone)
2. the bad torrents so far have all been rar files

When I started investigating the bad torrents in the afternoon, I tried to re-snatch them to test what could be done. By then, the bad torrents were gone from the tracker sites. I also noticed that SB had magically changed the episode status back to "unaired". Later in the day, the 3 torrents were re-snatched and they were good. I can only guess, but maybe SB changed the status to unaired because the post-processing didn't find any valid video files (only the rar and txt files)?

My current attempt is to run a script called by the transmission complete feature. It simply looks at the file names in the download directory and if any rar files are found, it deletes the entire directory. I have since found how to remove the torrents and data from transmission using transmission-remote. I'll re-write that part of the script but it might not get exercised until next Sunday.

Again, I'm not sure how or when SB changed the episode status back to unaired, and I don't know if it ever changed from snatched to downloaded before getting set to unaired. It would help me to understand how this mechanism works, if it's quick and easy for you to explain.

Please let me know if there is more I can do to help.

[gitrickm]

I had already added a subprocess.call to transmission.py to run a script when SB successfully adds a torrent to transmission (it notifies me via SMS). It occurs to me to move my rar-detecting code to this script, and delete the torrent and data files right away instead of waiting until transmission runs my complete-time script. Now I just have to wait until a bogus torrent comes along again.

Everything I'm doing is pretty specific to my case. I'm not sure what would be an appropriate thing to do generally, since of course there are legitimate rar file torrents. In my case, if I exclude all rar files I know that there are plenty of other mp4, avi etc downloads.

[junalmeida]

But not every. Rar file is bogus. Most of private trackers have Rar files (although I consider this stupid) and some public trackers also have valid Rar (without password)

[gitrickm]

Agreed. That's what I said above about what I'm doing not being generally applicable.

Does it make sense to you that the episode status was somehow changed back to unaried such that it downloaded a good torrent later in the evening?

[VeNoMouS]

Every provider I use has rar's only... call me an old schooler, but its scene... its basicly so if the part of the transfer "fails" you only need to replace that one rar file rather than say a 4.5gb file....

[junalmeida]

I can't see how a transfer of a torrent can fail corrupting all the giant file. Torrents are resumable in essence.

[gitrickm]

Clearly, rejecting rar files will not work for many people. If it becomes a problem for me I could just go by time, and reject all downloads until after, say, 8pm local time. I'm rarely in a hurry to watch a show the same day it airs. This isn't probably something many others would choose to do either.

I moved my rar-detecting code to the script that I call from SB after a torrent is added. Besides telling transmission to remove-and-delete the torrent, I use the SB API to set the episode back to "wanted" status. Of course, this causes SB to immediately search for the episode again, likely finding the same fake torrent. I set it up to delay a few hours before setting the status to wanted.

I've manually downloaded torrents for years, and since the torrent comments help verify which are good I don't have a lot of experience with fake ones. For my case, deciding based on rar files or time of day works. How would other people automate detecting fake torrents? Is "Torrent Guard" useful, and could it be integrated into sickbeard?

junalmeida, I'm not sure what you had in mind for this feature. If robust detecting ideas are found, would you add it to SB itself, or add the torrent to the downloader and call a script for the user to write and return you a good/bad result to act upon?

[VeNoMouS]

@junalmeida it's not necessarily the tracker / torrent but it can be from the originating ftp source etc...

@gitrickm, i personally haven't seen a fake torrent on my private trackers.... i guess thats the joy of using private trackers, you weed out the crap :P

[Animosity022]

@gitrickm - I would check out something like http://www.torrent-invites.com/ and get on a few private trackers. I found that site earlier this year and it is very easy to get to a few reputable private trackers. Majority of mine are not scene so I get just a single file from them. Most of the scene ones are rar files and you need some post processing. Just from a privacy perspective, I would stick to private trackers/seedbox as I like to be anonymous :)

[junalmeida]

I guess that we could implement some blacklist of hash, and force SB to ignore those. This blacklist will increase with hashs added by an effective bogus detector, and so, episode comes back to wanted and SB will try another torrent.

[gitrickm]

Suddenly, overnight I had 7 bogus downloads. They were all WMV files so I added wmv to my detector to trash them along with RAR files.

However, all of these wmv downloads were for episodes with air dates that are 1 to 10 days in the future. Doesn't SB check the air dates before snatching? I can understand when the bogus rar files are downloaded about 12 hours before the show airs, but 10 days?

[VeNoMouS]

I say close this ticket, as its not valid to SB... this is a problem on the provider websites... and tbh, if your using a source that posts fake torrents all the time, thats really your own fault for using said provider... change provider's if your not happy with fake torrents being posted...

[junalmeida]

The problem of SB trying to snatch episodes in the future must be an issue of the main project of midgetspy.

[VeNoMouS]

It's not really up to SB to determine if the episode is fake or not, the tvdb src may be wrong, or it would be posted via itunes first before tv air date etc... its up to the site provider to weed out fake torrents... I think leaving this in the hands of the "scraper" is wrong.

[VeNoMouS]

I dunno, I personally have never had a "fake" torrent uploaded in the past ~9 years on any of the sites I've used... I use scene only based sites, none of the public based sites..

[junalmeida]

I use some of the public and I don't have problem with fake torrents. PublicHD is a good one.

[gitrickm]

I'm the OP. I'm OK if you want to close this. The system I implemented above is working for me but certainly not for general use. I download about 15 shows each week, and it's settled down to only a few fake torrents per month.

I could probably improve it by monitoring which providers account for the fake torrents and removing them if there is a pattern. About half of my torrents come from PublicHD, but I also have Torrentz, Kickass, BTDigg and The PirateBay. I just blindly kept adding providers until all my weekly shows were being found. I restrict the downloads to SD since I seem to go over my traffic quota. That might make it harder to get by with just one provider, but I haven't bothered to investigate since it's been working perfectly.