Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Find out why we have so many warnings on missing RTPs on certain active subscriptions #121

Closed
daniel-abramov opened this issue Feb 1, 2023 · 3 comments
Closed

Find out why we have so many warnings on missing RTPs on certain active subscriptions #121

daniel-abramov opened this issue Feb 1, 2023 · 3 comments
Assignees
@daniel-abramov
Labels
T-Defect Something isn't working X-Needs-Investigation

Comments

@daniel-abramov
Copy link
Contributor

A log line like this one is not uncommon during normal sessions:

waterfall  | WARN[2023-02-01T18:19:37Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=RWSECBPMWU user_id="@copper-sunny-butterfly:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:38Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=AZGGKGVAFT user_id="@robintown:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:38Z] No RTP on subscription 6df7aea5-1bfa-45c6-91a6-87422782a541 (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=AZGGKGVAFT user_id="@robintown:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:38Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=YSMNDCBZWH user_id="@copper-resident-stingray:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:38Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=WDEKNIAADZ user_id="@cyan-patient-junglefowl:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:38Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=NUOTIRTNCN user_id="@simon.brandner:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:38Z] No RTP on subscription 6df7aea5-1bfa-45c6-91a6-87422782a541 (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=YSMNDCBZWH user_id="@copper-resident-stingray:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:38Z] No RTP on subscription 6df7aea5-1bfa-45c6-91a6-87422782a541 (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=DTFWXQLZHN user_id="@crimson-premier-grouse:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:38Z] No RTP on subscription 6df7aea5-1bfa-45c6-91a6-87422782a541 (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=WDEKNIAADZ user_id="@cyan-patient-junglefowl:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:38Z] No RTP on subscription 6df7aea5-1bfa-45c6-91a6-87422782a541 (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=NUOTIRTNCN user_id="@simon.brandner:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:39Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=RWSECBPMWU user_id="@copper-sunny-butterfly:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:40Z] No RTP on subscription 6df7aea5-1bfa-45c6-91a6-87422782a541 (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=AZGGKGVAFT user_id="@robintown:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:40Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=AZGGKGVAFT user_id="@robintown:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:40Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=YSMNDCBZWH user_id="@copper-resident-stingray:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:40Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=WDEKNIAADZ user_id="@cyan-patient-junglefowl:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:40Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=NUOTIRTNCN user_id="@simon.brandner:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:40Z] No RTP on subscription 6df7aea5-1bfa-45c6-91a6-87422782a541 (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=YSMNDCBZWH user_id="@copper-resident-stingray:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:40Z] No RTP on subscription 6df7aea5-1bfa-45c6-91a6-87422782a541 (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=DTFWXQLZHN user_id="@crimson-premier-grouse:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:40Z] No RTP on subscription 6df7aea5-1bfa-45c6-91a6-87422782a541 (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=WDEKNIAADZ user_id="@cyan-patient-junglefowl:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:40Z] No RTP on subscription 6df7aea5-1bfa-45c6-91a6-87422782a541 (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=NUOTIRTNCN user_id="@simon.brandner:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:41Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=RWSECBPMWU user_id="@copper-sunny-butterfly:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:42Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=AZGGKGVAFT user_id="@robintown:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:42Z] No RTP on subscription 6df7aea5-1bfa-45c6-91a6-87422782a541 (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=AZGGKGVAFT user_id="@robintown:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:42Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=YSMNDCBZWH user_id="@copper-resident-stingray:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:42Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=WDEKNIAADZ user_id="@cyan-patient-junglefowl:call.ems.host"
waterfall  | WARN[2023-02-01T18:19:42Z] No RTP on subscription 9db59e08-6390-44d5-a77a-11ced35eee2d (low)  conf_id=1675175890574TP04z8LAccuowlpZ device_id=NUOTIRTNCN user_id="@simon.brandner:call.ems.host"

It's not clear why there is an active subscription (with a low layer) that, despite being published, does not receive RTP frames with an intact publisher track. When the publisher track is gone, we remove all subscriptions associated with this track (check if we don't miss any edge cases).

It does not seem like this error has a big practical application (we did not observe any frozen subscription), yet it may point out to some bug in the SFU that must be resolved.
@daniel-abramov daniel-abramov self-assigned this Feb 8, 2023
@daniel-abramov daniel-abramov mentioned this issue Feb 23, 2023
Merged
@daniel-abramov
Copy link
Contributor Author

One of the reasons why this happens is if there is a video that is muted.

There are possible mitigations for this warning, see: #137 (comment)

@daniel-abramov
Copy link
Contributor Author

daniel-abramov commented Mar 21, 2023 •
edited
Loading

After running it with the OpenTelemetry, so far it looks like these happened in two cases for video tracks only:

  1. Video was muted.
  2. The given layer [to which we subscribed] has been published, but no RTP packets have been received on a layer. Switching to a lower layer solves the issue.

@SimonBrandner, do you remember if the Element Call sends the metadata update requests whenever the video gets muted/unmuted? - If it does, we could detect when the track is muted on the SFU side not to trigger false positives for the case (1). If we do so, we can use (2) as a hint that no RTP packets are received and automatically switch the quality to the lower layer preserving the video.

UPD: We receive the update of the metadata when muting/unmuting.

daniel-abramov added a commit that referenced this issue Mar 24, 2023
@daniel-abramov
subscription: don't assume muted tracks stalled
bcfebdb 
When the user mutes the video, we don't get any RTPs. We don't assume
this error anymore and don't change the status of the track and its
telemetry for that.

This removes false-positives related to the
#121
@daniel-abramov daniel-abramov mentioned this issue Mar 24, 2023
Merged
daniel-abramov added a commit that referenced this issue Mar 27, 2023
@daniel-abramov
subscription: don't assume muted tracks stalled
7c372e0 
When the user mutes the video, we don't get any RTPs. We don't assume
this error anymore and don't change the status of the track and its
telemetry for that.

This removes false-positives related to the
#121
@daniel-abramov
Copy link
Contributor Author

daniel-abramov commented Mar 29, 2023 •
edited
Loading

After several sessions with OpenTelemetry it has become clear that these are caused by the browser not publishing higher levels (1) and videos being muted (2).

(2) was addressed by a PR.
(1) can't be solved (Pion informs us about tracks, but there is no guarantee that we'll get RTP packets).

See #131 for the related issue of missing RTP packets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@daniel-abramov daniel-abramov

Labels
T-Defect Something isn't working X-Needs-Investigation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@daniel-abramov