Fix partially done downloads and choke groups

[chros73]

Choke groups

I. Summary
This is a powerful feature that should be used if you deal with group of torrents (e.g. you use throttle groups) like here.
It allows to set 3 major property per group (2 others (up, down) per property, except for tracker.mode):

• choke_group.tracker.mode : decides on how aggressive a tracker should be
• choke_group.up.heuristics : set the heuristics used when deciding on which peers to choke and unchoke
• choke_group.up.max : set the max total number of unchoked peers for all torrents in this choke group

The last property is the really interesting one! It restricts the number of unchoked peers in a group!

I just found it by accident (while I have been working on fixing partially done donwloads) that dealing with choke heuristics has been dramatically changed since libtorrent v0.13.0 but rtorrent (client) part left untouched since then (last good version is 0.8.9-0.12.9 in this regard).

• until 0.8.9 it worked as a per download setting
• since 0.9.0 it should work as a group setting
• choke groups feature is enabled and working (means it's actively used by resource_manager) in libtorrent since v0.13.0
  • only 1 group is created, called default
• rtorrent hasn't been modified (just a bit to don't get broken)
  • if you want to modify group for a download then it affects all the downloads
  • only 1 group is used (default) for all the torrents, means upload_leech heuristics is used for seeding as well
  • setting choke_group.up.max doesn't work

It works fine by default until somebody wants to change the config - there's a default choke group - (a hack was applied for integrating into the client).

II. Bugs

• 1. all the old choke_heuristics code is unnecessary in command_download.cc
• 2. there are unnecessary code in resume(), confirm_finished() part of core/download_list.cc
• 3. protocol.choke_heuristics.up.seed hasn't been updated to upload_seed in command_network.cc , but I think it's not needed anymore
• 4. the new d.group property isn't saved and not initialized with the download in core/download_factory.cc , old code can be removed
• 5. d.group.name command gives back index instead of the name of the group
• 6. UI only displays the index of the group not the name in ui/download.cc
• 7. all the old choke_heuristics code is unnecessary in libtorrent@torrent/download.h and libtorrent@torrent/download.cc

III. Fixes for mentioned bugs

• 1. removed old code from command_download.cc
• 2. added new logic and removed old code from core/download_list.cc@resume() and core/download_list.cc@confirm_finished()
• 3. removed old code from core/download_list.cc
• 4. removed old code from core/download_factory.cc
• 5. fixed d.group_name in command_download.cc
• 6. fixed and modified UI on Info screen in ui/download.cc
• 7. removed old code from libtorrent@torrent/download.*
• 8. enabled real choke group code in command_groups.cc and got rid of the hack code (also in core/download.h)

IV. Additions

• renamed default choke group default to default_leech in [email protected]
• added a new default choke group called default_seed in main.cc
• added new helper UI command convert.group has been created in command_ui.cc (similarly to convert.throttle)
• modify UI on Info screen to include all the statistics that belongs to a group
  • Choke group: default_seed [upload_seed, download_leech, aggressive] [Max --/--]
  • Choke group stat: [Size 149] [Unchoked 18/0] [Queued 0/0] [Total 18/0] [Rate 952.3/ 0.0 KB]

V. Other fixes

• fixed saving total_skipped between sessions in core/download_factory.cc and core/download_store.cc (used on Info screen)

VI. Choke groups in action
I'll update the wiki page if it will be merged, but until then here's a teaser:
let's extend the Favoring group of torrents example to include public (!!!) torrents as well! :)

• we add one more throttle group for public torrents, called tardyup, along with slowup
• let's create 4 more choke groups (slowup_leech, slowup_seed, tardyup_leech, tardyup_seed) along with the built-in 2 ones (default_leech, default_seed)
  • name them after the corresponding throttle groups!
• assign the different groups to the proper download with the corresponding throttle group all the time!
  • it's just 4 lines of code when the above naming convention is used! :)
• we can also modify the choke_group.up.max setting on the fly (!) with a help of an external script (like we did with the throttle speed)
  • external script mod isn't included, so this part work work in this way

# Define throttle groups
throttle.up = slowup,600
throttle.up = tardyup,300

# Setting up choke groups that restricts the number of unchoked peers in a group
# Modify default choke groups for specail group
choke_group.up.heuristics.set=default_leech,upload_leech_experimental
choke_group.tracker.mode.set=default_leech,aggressive
choke_group.tracker.mode.set=default_seed,aggressive
# Set up choke groups for slowup group
choke_group.insert=slowup_leech
choke_group.insert=slowup_seed
choke_group.up.heuristics.set=slowup_leech,upload_leech
choke_group.up.heuristics.set=slowup_seed,upload_seed
choke_group.down.max.set=slowup_leech,200
choke_group.up.max.set=slowup_leech,200
choke_group.up.max.set=slowup_seed,125
# Set up choke groups for tardy group
choke_group.insert=tardyup_leech
choke_group.insert=tardyup_seed
choke_group.up.heuristics.set=tardyup_leech,upload_leech
choke_group.up.heuristics.set=tardyup_seed,upload_seed
choke_group.down.max.set=tardyup_leech,150
choke_group.up.max.set=tardyup_leech,150
choke_group.up.max.set=tardyup_seed,75

# helper method: Sets choke group to one of the default ones if there's no throttle or throttle is special one (NULL) otherwise sets it to one of the throttle name ones
method.insert = d.modify_choke_group, simple, "branch=((and,((d.throttle_name)),((not,((equal,((d.throttle_name)),((cat,NULL)))))))),((d.group.set,(cat,(d.throttle_name),_,(d.connection_current)))),((d.group.set,(cat,default_,(d.connection_current))))"
# Modify choke group when a torrent is resumed (even after hashchecking or after rtorrent is restarted) or finished or partially restarted
method.set_key = event.download.resumed, resumed_choke_group, "d.modify_choke_group="
method.set_key = event.download.finished, finished_choke_group, "d.modify_choke_group="
method.set_key = event.download.partially_restarted, partially_restarted_choke_group, "d.modify_choke_group="

# helper method: gets one of the below info with the help of getLimits.sh script (Variables are inside the script, you have to edit those values there!)
method.insert = get_limit, simple|private, "execute.capture=\"$cat=$pyro.bin_dir=,getLimits,$cfg.postfix=,.sh\",$argument.0=,$argument.1=,$argument.2=,$argument.3="
# Adjust max setting on the fly
schedule2 = adjust_slots_slowup,  47, 100, "choke_group.up.max.set=slowup_seed,\"$get_limit=$cat=slslots,$convert.kb=$throttle.global_up.rate=,$convert.kb=$throttle.up.rate=slowup,$convert.kb=$throttle.up.rate=tardyup\""
schedule2 = adjust_slots_tardyup, 48, 100, "choke_group.up.max.set=tardyup_seed,\"$get_limit=$cat=taslots,$convert.kb=$throttle.global_up.rate=,$convert.kb=$throttle.up.rate=slowup,$convert.kb=$throttle.up.rate=tardyup\""

[chros73]

Partially done downloads

Let's try to fit partially done downloads into the ecosystem of rtorrent:

• libtorrent already has some code related to this state, e.g. in torrent/data/download_data.h , but it hasn't been really used
• rtorrent already has commands for partially done downloads: d.is_partially_done, d.is_not_partially_done
• every finished download is also partially done!

I. Fixes for known bugs

• it will operate entirely based on selection of files (that's the only thing that makes sense from the user point of view)
• let's use partially done state of a download most of the time instead of finished (when it makes sense)
  • release seeds when a download is partially done in libtorrent:
    • torrent/peer/connection_list.cc , protocol/peer_connection_leech.cc, protocol/peer_connection_base.cc, protocol/handshake_manager.cc
  • let's trigger event.download.finished in rtorrent when a download is partially finished
  • define a new slot and a corresponding method in core/download_list.cc slot_partially_restarted() that will handle client part (also into core/download_list.h)
    • reset choke groups, d.complete, d.connection_current, d.peers_min, d.peers_max
    • it will trigger a new event.download.partially_restarted event
  • don't send complete info to trackers only if a download is truely finished
  • create a new event.download.partially_restarted event that is triggered when a partial download has been restarted
    • e.g. it can be used to assign custom choke groups to downloads
    • add this new event to relevant view filters in main.cc
      • complete, incomplete, seeding, leeching
  • modify d.timestamp.finished.set method in main.cc to be able to be overridden
  • create a new porperty and setter method in libtorrent@torrent/data/file_list.cc for selected_size_bytes
    • will be equal to size_bytes if a download is finished
    • will be equal to completed_bytes if a download is partyally done (and not the slected size of files, since they can be turnd off later!)
    • otherwise calculate the partial size based on the selected chunks of the selected files
    • it's a property in File_list: to don't fire up this expensive calculation all the time
    • save it into session and reload it during restart (similarly to chunks_wanted):
      • required by stopped torrents (update_priorities() method isn't triggered for them upon start)
• it should not break remusing downloads
• it should not break fast resume data
• it should not break hash-checking of a download

II. New commands in rtorrent

• d.is_done : to be able to distinguish between finished and partially done downloads
• d.selected_size_bytes : gets the correct size of a torrent

III. UI changes in rtorrent

• don't change the % , displayed total size (we want to differentiate between partially done and finished downloads), but display done in the place of remaining time if a download is partially done
  • 13.6G / 22.8G Rate: 0.0K / 0.0K Uploaded: 762.4M [59%] done [T R: 0.05 low tardyup]
• on the new compact view: don't change anything (there's no place for it), if a donwload isn't 100% and there's no remaining time that means it's partially finished
• display additional Size info on Info screen: downloaded / selected / total size:
  • Size: 450MB / 712MB / 970MB
• modify Chunk screen also to take into account partial downloads
• remaining time should display the correct amount all the time

[chros73]

Depends on:

• Fix partially done downloads and choke groups libtorrent#8 (Fix partially done downloads and choke groups)