Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Fluvio client does not detect broken connection while consuming after machine suspend and resume #1514

Closed
ajhunyady opened this issue Aug 26, 2021 · 8 comments
Assignees
Labels
bug Something isn't working no-issue-activity

Comments

@ajhunyady
Copy link
Contributor

ajhunyady commented Aug 26, 2021

I started a client instance to listen on a topic/partition

fluvio consume fluvio-stars -k -B

[2021-08-26 19:12:15] 600
[2021-08-26 19:14:17] 600
[2021-08-26 19:16:18] 600
[2021-08-26 19:18:20] 600
[2021-08-26 19:20:22] 600
[2021-08-26 19:22:24] 600
[2021-08-26 19:24:25] 600
[2021-08-26 19:26:27] 600
[2021-08-26 19:28:29] 600

After a while I closed the laptop. When I returned, the connection looked alive... but soon realized that it does not receive traffic.

Expected behavior

(the minimum)

  • Exit out of the loop of connection error and print the error: Connection Error

(better)

  • Try to reconnect, and give up after a certain number of retries.
@ajhunyady ajhunyady added the bug Something isn't working label Aug 26, 2021
@nacardin nacardin added this to the 0.9.9 milestone Sep 23, 2021
@tjtelan
Copy link
Contributor

tjtelan commented Sep 23, 2021

Caused by #770

@nacardin nacardin modified the milestones: 0.9.9, 0.9.10 Sep 30, 2021
@nacardin nacardin changed the title [Bug]: Fluvio client does not detect broken connection [Bug]: Fluvio client does not detect broken connection while consuming Oct 4, 2021
@nacardin nacardin modified the milestones: 0.9.10, 0.9.11 Oct 7, 2021
@nacardin nacardin modified the milestones: 0.9.11, 0.9.12 Oct 21, 2021
@nacardin nacardin assigned tjtelan and unassigned morenol Oct 21, 2021
@tjtelan tjtelan modified the milestones: 0.9.12, 0.9.13 Oct 27, 2021
@nacardin nacardin modified the milestones: 0.9.13, 0.9.14 Nov 3, 2021
@tjtelan
Copy link
Contributor

tjtelan commented Nov 22, 2021

This should be approached as an automated test.

Start consume, simulate a network failure (e.g. delete the spu pod, and wait for pod to redeploy), verify reconnection and resuming consuming

@nacardin nacardin self-assigned this Nov 30, 2021
@nacardin nacardin changed the title [Bug]: Fluvio client does not detect broken connection while consuming [Bug]: Fluvio client does not detect broken connection while consuming after machine suspend and resume Nov 30, 2021
@nacardin
Copy link
Contributor

I'll check behavior on Mac laptop on 0.9.13

@nacardin
Copy link
Contributor

On MacOS, this behavior is still present as of 0.9.13. Closing the laptop for 1 minutes is fine, consume is still alive. If I close the laptop for about 10 minutes, the consume command is still running, but is not showing new records.

@nacardin nacardin removed their assignment Nov 30, 2021
@nacardin
Copy link
Contributor

nacardin commented Dec 2, 2021

Let's check netstat to see the state of the socket during this behavior

@morenol
Copy link
Contributor

morenol commented Dec 3, 2021

After suspend and resume in linux:

$ sudo netstat  -ntpe4     | grep 900
tcp        0      0 192.168.1.105:57768     3.211.184.49:9003       ESTABLISHED 1000       146968     15068/fluvio        
tcp        0      0 192.168.1.105:38622     3.211.184.49:9005       ESTABLISHED 1000       144072     15068/fluvio        

@sehz
Copy link
Contributor

sehz commented Dec 4, 2021

defering

@sehz sehz modified the milestones: 0.9.14, 0.10.0 Dec 4, 2021
@sehz sehz removed this from the 0.11.0 milestone Apr 18, 2023
@github-actions
Copy link

Stale issue message

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jul 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working no-issue-activity
Projects
None yet
Development

No branches or pull requests

5 participants