Send TCP packets on separate thread

[sardination]

Ticket(s) Closed

• Closes N/A

Description
There was a bug where TCP packets that were being read were occasionally not able to be decrypted, and this particularly happened for large TCP packets (file transfer). This is apparently due to the facts that TCP send is not atomic for large sizes and that we send TCP packets on multiple threads for a single socket, resulting in garbled contents when reading from the TCP socket.

Implementation
Introduce a "TCP send thread" for each socket that is dedicated to sending TCP packets (on both the client and server). The caller can send TCP packets the same way it has been doing. The implementation of tcp_send_constructed_packet has been modified to just add the packet onto a queue. Then, the TCP send thread checks the queue for contents and handles the sending of packets over the TCP socket.

Additionally, a potential problem with sending partial TCP packets has been resolved by continuing sends until the full packet has been sent instead of only sending the partial TCP packet.

This implementation also required the introduction of a blocking argument to the queue enqueue operation that allows blocking enqueues. This results in a few changes outside of the scope of this PR in order to match the new function signature.

Documentation & Tests Added
Queue tests for enqueue were updated to make use of the blocking argument.

Testing Instructions
Try connecting and dragging and dropping large files multiple times. Repeat this a few times with new connections. Files should send fully every time and there should be no server disconnects.

PR Checklist

• Did the PR author fully test this PR end-to-end?
• Did one PR reviewer fully test this PR end-to-end?
• Did one PR reviewer conduct a thorough code design review?

[codecov]

Codecov Report

Merging #7226 (d99a89b) into dev (fdcd360) will decrease coverage by 0.04%.
The diff coverage is 44.71%.

❗ Current head d99a89b differs from pull request most recent head 9e369bb. Consider uploading reports for the commit 9e369bb to get more accurate results

@@            Coverage Diff             @@
##              dev    #7226      +/-   ##
==========================================
- Coverage   54.70%   54.66%   -0.05%     
==========================================
  Files         160      160              
  Lines       32637    32727      +90     
==========================================
+ Hits        17855    17890      +35     
- Misses      14485    14540      +55     
  Partials      297      297              

Flag	Coverage Δ		*Carryforward flag
backend-services	47.44% <ø> (ø)		Carriedforward from fdcd360
protocol	56.24% <44.71%> (-0.06%)	⬇️

*This pull request uses carry forward flags. Click here to find out more.

Impacted Files	Coverage Δ
protocol/whist/network/tcp.c	0.00% <0.00%> (ø)
protocol/whist/utils/queue.c	83.82% <85.00%> (-6.18%)	⬇️
protocol/test/protocol_test.cpp	98.80% <100.00%> (ø)
protocol/whist/fec/wirehair/WirehairCodec.cpp	87.98% <0.00%> (-0.12%)	⬇️
protocol/whist/logging/logging.c	88.68% <0.00%> (+0.30%)	⬆️
protocol/whist/fec/wirehair/WirehairTools.cpp	77.54% <0.00%> (+0.70%)	⬆️
protocol/whist/utils/rwlock.c	96.66% <0.00%> (+1.66%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fdcd360...9e369bb. Read the comment docs.

[npip99]

Just a few comments, everything else looks great!

[npip99]

Yep!

[npip99]

Yep!

[github-actions]

Protocol End-to-End Streaming Test Results

Experiments Summary

Expand Summary

✅ Experiment 1 - Bandwidth: Unbounded, Delay: None, Packet Drops: None, Queue limit: default. Download logs:

aws s3 cp s3://whist-e2e-protocol-test-logs/suriya/tcp-send-thread/2022_09_14@20-42-16/ 2022_09_14@20-42-16/ --recursive

✅ Experiment 2 - Bandwidth: variable between 15Mbit and 30Mbit, Delay: 10 ms, Packet Drops: None, Queue Limit: None, Conditions change over time? Yes, frequency is variable between 1000 ms and 2000 ms. Download logs:

aws s3 cp s3://whist-e2e-protocol-test-logs/suriya/tcp-send-thread/2022_09_14@21-04-23/ 2022_09_14@21-04-23/ --recursive

✅ Experiment 3 - Bandwidth: variable between 10Mbit and 20Mbit, Delay: 10 ms, Packet Drops: None, Queue Limit: None, Conditions change over time? Yes, frequency is variable between 500 ms and 2000 ms. Download logs:

aws s3 cp s3://whist-e2e-protocol-test-logs/suriya/tcp-send-thread/2022_09_14@21-08-38/ 2022_09_14@21-08-38/ --recursive

✅ Experiment 4 - Bandwidth: 10Mbit, Delay: 10 ms, Packet Drops: None, Queue Limit: 100 packets, Conditions change over time? No.. Download logs:

aws s3 cp s3://whist-e2e-protocol-test-logs/suriya/tcp-send-thread/2022_09_14@21-12-57/ 2022_09_14@21-12-57/ --recursive

Full Results: link here