Serial link is closed after unsuccessful session #775
Labels
bug
Something isn't working
Comments
|
I created a backtrace when this happens in zenohd [2024-02-29T23:14:12Z TRACE zenoh_transport::unicast::universal::rx] Received: TransportMessage { body: KeepAlive(KeepAlive) }
[2024-02-29T23:14:14Z TRACE zenoh_transport::unicast::universal::rx] Received: TransportMessage { body: KeepAlive(KeepAlive) }
[2024-02-29T23:14:17Z TRACE zenoh_transport::unicast::universal::rx] Received: TransportMessage { body: KeepAlive(KeepAlive) }
[2024-02-29T23:14:19Z TRACE zenoh_transport::unicast::universal::rx] Received: TransportMessage { body: KeepAlive(KeepAlive) }
[2024-02-29T23:14:19Z DEBUG zenoh_transport::unicast::manager] future has timed out
thread 'async-std/runtime' panicked at 'explicit panic', io/zenoh-transport/src/unicast/manager.rs:751:17
stack backtrace:
0: rust_begin_unwind
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:593:5
1: core::panicking::panic_fmt
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:67:14
2: core::panicking::panic
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:117:5
3: zenoh_transport::unicast::manager::<impl zenoh_transport::manager::TransportManager>::handle_new_link_unicast::{{closure}}::{{closure}}
at ./io/zenoh-transport/src/unicast/manager.rs:751:17
4: <async_std::task::builder::SupportTaskLocals<F> as core::future::future::Future>::poll::{{closure}}
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-std-1.12.0/src/task/builder.rs:199:17
5: async_std::task::task_locals_wrapper::TaskLocalsWrapper::set_current::{{closure}}
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-std-1.12.0/src/task/task_locals_wrapper.rs:60:13
6: std::thread::local::LocalKey<T>::try_with
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/thread/local.rs:270:16
7: std::thread::local::LocalKey<T>::with
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/thread/local.rs:246:9
8: async_std::task::task_locals_wrapper::TaskLocalsWrapper::set_current
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-std-1.12.0/src/task/task_locals_wrapper.rs:55:9
9: <async_std::task::builder::SupportTaskLocals<F> as core::future::future::Future>::poll
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-std-1.12.0/src/task/builder.rs:197:13
10: async_executor::Executor::spawn::{{closure}}
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-executor-1.5.1/src/lib.rs:145:20
11: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/future/future.rs:125:9
12: async_task::raw::RawTask<F,T,S,M>::run
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-task-4.4.0/src/raw.rs:563:17
13: async_task::runnable::Runnable<M>::run
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-task-4.4.0/src/runnable.rs:782:18
14: async_executor::Executor::run::{{closure}}::{{closure}}
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-executor-1.5.1/src/lib.rs:236:21
15: <futures_lite::future::Or<F1,F2> as core::future::future::Future>::poll
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-lite-1.13.0/src/future.rs:529:33
16: async_executor::Executor::run::{{closure}}
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-executor-1.5.1/src/lib.rs:243:32
17: <futures_lite::future::Or<F1,F2> as core::future::future::Future>::poll
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-lite-1.13.0/src/future.rs:529:33
18: async_io::driver::block_on
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-io-1.13.0/src/driver.rs:146:33
19: async_global_executor::reactor::block_on::{{closure}}
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-global-executor-2.3.1/src/reactor.rs:3:18
20: async_global_executor::reactor::block_on
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-global-executor-2.3.1/src/reactor.rs:12:5
21: async_global_executor::threading::thread_main_loop::{{closure}}::{{closure}}
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-global-executor-2.3.1/src/threading.rs:95:17
22: std::thread::local::LocalKey<T>::try_with
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/thread/local.rs:270:16
23: std::thread::local::LocalKey<T>::with
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/thread/local.rs:246:9
24: async_global_executor::threading::thread_main_loop::{{closure}}
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-global-executor-2.3.1/src/threading.rs:89:13
25: std::panicking::try::do_call
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:500:40
26: __rust_try
27: std::panicking::try
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:464:19
28: std::panic::catch_unwind
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panic.rs:142:14
29: async_global_executor::threading::thread_main_loop
at /home/lieven/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-global-executor-2.3.1/src/threading.rs:88:12
30: core::ops::function::FnOnce::call_once
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[2024-02-29T23:14:22Z TRACE zenoh_transport::unicast::universal::rx] Received: TransportMessage { body: KeepAlive(KeepAlive) } |
|
I got the impression , this issue in tokio-serial is back : berkowski/tokio-serial#52 When the connection get lost and no recovery is possible : It could be that 80 bytes write blocked ( until the future timed out) or the serial read became impossible. |
|
Using strace to dig deeper, it looks like the code freezes for exactly 10 sec. |
|
I tweaked an option : transport.unicast.max_links in zenohd config zenohd logs for completeness : |
|
ESP32 crash details |
|
And another one after increasing ESP32 stack size |
|
To continue the story , the fact that the ESP32 crashes after some time looks to be linked to lack of heap space. Monitoring the heap shows that it continually decreases after every loop on open/close zenoh session with declare/undeclare publisher. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
See issue : eclipse-zenoh/zenoh-pico#357
It looks like the link is closed when a future times out and is never opened again for listening on data
The zenoh-pico code tries opening the session again, but from zenohd side, no more data is being read. It looks like connection/link was dropped completely.
Restarting zenohd , makes that everything works again for some time.
So it looks a bug at server side. I suspect that the serial is handled like a tcp connection, where you close the link.
To reproduce
Run zenoh-pico at client side via serial
System info
Platform Ubuntu 22
Zenoh Pico side on esp32
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
The text was updated successfully, but these errors were encountered: