Periodic peer disconnect and idling data transfer intervals

[kirillsc]

Hi All,

I am observing periodic interruptions in data transfer among peers that are using rtorrent. I am not sure if this is a bug, but I would appreciate if you could help me to understand the issue.

I am using rtorrent for file distribution within private network, specifically it is private subnets of a VPC on AWS. I run my own opensource bittorrent-tracker (https://github.com/webtorrent/bittorrent-tracker). My use case comprised of a single seeding server that needs to distribute a ~50GB folder among ~300 machines within the same private network. The seed server (1) creates the torrent file based on the local folder (2) shares this torrent file among all peers (3) each peer adds torrent file into the /rtorrent/watch/start/ folder and connects to the same tracker server (within the same network). All ~300 peers initiate downloading in about the same time.

The problem that I am repeatedly observing is prolonged periods of complete idling among all peers in the network. In other words, for the first ~5 minutes everything works as expected i.e., the seed server uploads at its maximum network bandwidth and all the peers receive pieces and also distribute chunks among themselves. Then after the initial ~5 minutes all peers disconnect from each other and idle for 5-10 minutes, eventually transfer resumes and lasts for another few minutes just to disconnect again. Two important observations: (1) if at the time of such idling, I manually restart the source seed server all transfer resumes for another few minutes, (2) the problem does not appear to be related just to the seed server only, because during the initial period of data transfer (before the first idling) there are many peers that manage to get the complete torrent file, however, none of them are sharing the data with the remaining peers during the idling period.

I am attaching log files from the seed server and one of the peer servers from one of my smaller scale experiments where I used only 20 peers.

In the log file I am seeing the following messages, but I am not sure about their relevance or how to debug them further. 



Handshake dropped: seeder rejected.
Received error: message:7 network error.
Upload unchoked slots adjust; currently:10 adjust:1

I am using rTorrent v0.9.8 and RHL8 OS.

I would appreciate any guidance on what could be an issue here.

Thank you.
server-log.log
client-log.log

[kannibalox]

Did everything work as expected in the smaller test? If not, would you happen to have a log from a peer that didn't successfully get past the stall? Can you share your config?

Just to break down the log messages you mentioned a bit:

• Handshake dropped: seeder rejected: This can happen two ways, one of which is only possible when using magnet links. The other is when rTorrent receives a connection from a seeder when it's also a seeder, so this message seems pretty harmless.
• Received error: message:7 network error.: Unfortunately this can refer to a couple different kinds of network errors, and the master branch has more specific strings. There's plenty of reasons this could happen during normal operation, so it's good to have a timeline but otherwise doesn't tell much on it's own.
• Upload unchoked slots adjust; currently:10 adjust:1: These are messages from the internal resource manager. By themselves, they're just informational messages telling you how many unchoked peer connections are active. Depending on your settings, it's unlikely but posssible rTorrent is clearing connections unecessarily

One funky thing I see in the logs that I don't think is normal is that within the space of second, rtorrent is starting an outgoing connection, receiving an incoming connection from the same host, then declaring that both connections received a network error. It's possible there's some weird race condition that happens in low latency networks. I assume all the clients are currently receiving the torrent at essentially the same time, would it be possible to try staggering the start across the servers?

[kirillsc]

Hi @kannibalox

Thanks for the quick response and explanation of the messages!

I was able to reproduce this issue using a single seeding server and a single client. I am attaching both logs and the configuration that was used. In this experiment the client experienced a stall in less than a minute after starting downloading the file.

To answer your last question, I am already spreading start up times across 20 seconds interval, however, the objective is to distribute files as fast as possible. I can artificially slow the process further (say by 1-2 minutes), but the issue is still present in the smallest scale tests.

Also, I don't want to diverge this conversation from the original topic, but I have also observed several times a case when a client shuts down half way through downloading a file. I have observed this when rtorrent client has been launched as a detached daemon process. I am attaching this log file as client_error2.log just in case it will make sense to you.

Let me know if I can provide any other debug information.

Thank you.
seed_server1.log
client1.log
config.log
client_error2.log

[kannibalox]

Hm, 20 seconds would be enough to prevent the behavior I was thinking of, and there's not anything else obvious in the 1-on-1 logs. My interest is sufficiently piqued that I may see if I can replicate it. Are there any other noteworthy details about your setup?

As for client_error2.log, that looks like a normal shutdown procedure. Those can be triggered by SIGINT or SIGHUP, or by RPC calls, see https://rtorrent-docs.readthedocs.io/en/latest/cmd-ref.html#term-system-shutdown-normal for more infortmation. If rtorrent encountered an error it couldn't handle or something, it would have just crashed hard instead.

[mysterfr]

Hi, just wanted to chime in as a I have a pretty similar issue, on a more classical setup I'd say (seedbox).
I run multiple rtorrent instances on the same box.
All instances have the same configuration file, except for watch/doing/seed/done folders.

For the last 10 days, one of those rtorrent instance behaves as the OP explained:

1. torrents start to be leetched/seeded, with decent bandwidth
2. all of a sudden, after some random time (I'd roughly say within 1 to 5 minutes at most), all network traffic suddently stops
3. after a (sometimes long) while, like 20-30 minutes, torrents start again to leetch/seed, and after a couple minutes all network traffic stops again.

I'm running a Debian 12 server, rtorrent v0.9.8 and librtorrent 0.13.8, from Debian repos.
I will increase log level to debugging for a while and come back with some logs.

In the meantime, if it's any use:

ldd /usr/bin/rtorrent
linux-vdso.so.1 (0x00007ffe57492000)
libncursesw.so.6 => /lib/x86_64-linux-gnu/libncursesw.so.6 (0x00007fca0bdc6000)
libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007fca0c005000)
libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007fca0bd17000)
libtorrent.so.21 => /lib/x86_64-linux-gnu/libtorrent.so.21 (0x00007fca0bc1b000)
libxmlrpc_server.so.3 => /lib/x86_64-linux-gnu/libxmlrpc_server.so.3 (0x00007fca0bc13000)
libxmlrpc.so.3 => /lib/x86_64-linux-gnu/libxmlrpc.so.3 (0x00007fca0bbfb000)
libxmlrpc_util.so.3 => /lib/x86_64-linux-gnu/libxmlrpc_util.so.3 (0x00007fca0bbf3000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fca0b800000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fca0bbd3000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fca0bbce000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fca0b61f000)
libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007fca0bb9f000)
libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007fca0bb6e000)
librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1 (0x00007fca0bb4f000)
libssh2.so.1 => /lib/x86_64-linux-gnu/libssh2.so.1 (0x00007fca0bb0e000)
libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x00007fca0bafa000)
libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x00007fca0ba51000)
libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007fca0b000000)
libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fca0b5cc000)
libldap-2.5.so.0 => /lib/x86_64-linux-gnu/libldap-2.5.so.0 (0x00007fca0b56d000)
liblber-2.5.so.0 => /lib/x86_64-linux-gnu/liblber-2.5.so.0 (0x00007fca0ba3f000)
libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007fca0b4b1000)
libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007fca0ba32000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fca0b492000)
libxmlrpc_xmlparse.so.3 => /lib/x86_64-linux-gnu/libxmlrpc_xmlparse.so.3 (0x00007fca0ba20000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fca0af21000)
/lib64/ld-linux-x86-64.so.2 (0x00007fca0c047000)
libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2 (0x00007fca0ad6b000)
libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fca0aa00000)
libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6 (0x00007fca0ad22000)
libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8 (0x00007fca0acd4000)
libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fca0ac53000)
libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fca0a926000)
libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fca0ac26000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fca0b48c000)
libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fca0a918000)
libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007fca0a8fb000)
libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007fca0a8d8000)
libxmlrpc_xmltok.so.3 => /lib/x86_64-linux-gnu/libxmlrpc_xmltok.so.3 (0x00007fca0a8bf000)
libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fca0a78b000)
libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fca0a776000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fca0ac1f000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fca0a765000)
libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8 (0x00007fca0a759000)

and

dpkg -l | grep torr
ii libtorrent-rasterbar2.0:amd64 2.0.8-1+b1 amd64 C++ bittorrent library by Rasterbar Software
ii libtorrent20:amd64 0.13.7-1 amd64 C++ BitTorrent library by Rakshasa
ii libtorrent21:amd64 0.13.8-2+b1 amd64 C++ BitTorrent library by Rakshasa
ii python3-libtorrent 2.0.8-1+b1 amd64 Python bindings for libtorrent-rasterbar (Python 3)
ii rtorrent 0.9.8-1 amd64 ncurses BitTorrent client based on LibTorrent from rakshasa

[mysterfr]

It seems it was probably related to some bogus torrent.
After removing quite a bunch of torrents (by batches of 20), rtorrent crashed, and upon restart started behaving normally again.