SIGSEGV interrupt after TLS warning without having TLS enabled

[bartvp1]

Describe the bug
Even though I dont use TLS in my OvenMediaEngine configuration, it still occasionally warns about not being able to do something related to TLS, which often coincides with a crash of the Origin server.

I can't confirm whether this is related to the TLS warning, but since most crashes end with these same warnings I have my suspicions.

To Reproduce
No steps to reproduce, this seems to happen out of the blue

Expected behavior
I would expect there not to be any logs or crashes related to TLS if I'm not using any TLS for this service, because the load balancer terminates it.

Logs
Origin:

[2024-10-15 08:19:33.916] W [SPRTMP-t1935:11] OpenSSL | tls.cpp:328  | Tls::Read() returns 5 (errno: 11): [OpenSSL] error:00000000:lib(0)::reason(0) (0)
[2024-10-15 08:19:33.916] W [SPAPISvr-t8080:20] OpenSSL | tls.cpp:328  | Tls::Read() returns 5 (errno: 11): [OpenSSL] error:00000000:lib(0)::reason(0) (0)
[2024-10-15 08:19:33.916] C [SPRTMP-t1935:11] OvenMediaEngine | signals.cpp:124  | OME v0.16.8 ((From archive)) received signal 11 (SIGSEGV), interrupt.
[2024-10-15 08:19:33.916] C [SPAPISvr-t8080:20] OvenMediaEngine | signals.cpp:124  | OME v0.16.8 ((From archive)) received signal 11 (SIGSEGV), interrupt.

Server

• OS: Linux, using K8s containers
• OvenMediaEngine Version: v0.16.8
• Branch: release

[dimiden]

Since there is no need to use TLS in the SPRTMP thread, the related logs should not be output. It seems unusual that they are being printed.

It is likely that an issue such as a stack overflow may have caused incorrect execution. For a more detailed analysis, please share the full logs, Server.xml, and any dump files generated in the /usr/share/ovenmediaengine/dumps directory inside the Docker container.

[bartvp1]

Here's the server configuration for the origin:

<?xml version="1.0" encoding="UTF-8"?>

<Server version="8">
    <Name>OvenMediaEngine</Name>
    <Type>origin</Type>
    <IP>*</IP>
    <StunServer>stun.l.google.com:19302</StunServer>

    <Managers>
        <Host>
            <Names>
                <Name>*</Name>
            </Names>
        </Host>
        <API>
            <AccessToken>${env:OME_API_ACCESS_TOKEN}</AccessToken>
        </API>
    </Managers>

    <Bind>
        <Managers>
            <API>
                <Port>8080</Port>
            </API>
        </Managers>

        <Providers>
            <RTMP>
                <Port>1935</Port>
            </RTMP>
        </Providers>

        <Publishers>
            <OVT>
                <Port>9000</Port>
            </OVT>
        </Publishers>
    </Bind>

    <VirtualHosts>
        <VirtualHost include="VHost*.xml" />
        <VirtualHost>
            <Name>default</Name>
            <Host>
                <Names>
                    <Name>*</Name>
                </Names>
            </Host>

            <AdmissionWebhooks>
                <ControlServerUrl>${env:ADMISSION_ADDRESS}</ControlServerUrl>
                <SecretKey>${env:ADMISSION_SECRET_KEY}</SecretKey>
                <Timeout>3000</Timeout>
                <Enables>
                    <Providers>rtmp</Providers>
                </Enables>
            </AdmissionWebhooks>

            <OriginMapStore>
                <RedisServer>
                    <Host>${env:REDIS_HOST}:${env:REDIS_PORT}</Host>
                    <Auth>${env:REDIS_PASSWORD}</Auth>
                </RedisServer>

                <OriginHostName>${env:POD_IP}</OriginHostName>
            </OriginMapStore>

            <Applications>
                <Application>
                    <Name>app</Name>
                    <Type>live</Type>

                    <OutputProfiles>
                        <OutputProfile>
                            <Name>ABR</Name>
                            <OutputStreamName>${OriginStreamName}</OutputStreamName>
                            <Playlist>
                                <Name>abr</Name>
                                <FileName>abr</FileName>
                                <Options>
                                    <WebRtcAutoAbr>true</WebRtcAutoAbr>
                                </Options>

                                <Rendition>
                                    <Name>Full HD</Name>
                                    <Video>video_1080</Video>
                                    <Audio>cond_audio_opus</Audio>
                                </Rendition>
                                <Rendition>
                                    <Name>HD</Name>
                                    <Video>video_720</Video>
                                    <Audio>cond_audio_opus</Audio>
                                </Rendition>
                                <Rendition>
                                    <Name>SD</Name>
                                    <Video>video_360</Video>
                                    <Audio>cond_audio_opus</Audio>
                                </Rendition>
                            </Playlist>

                            <Encodes>
                                <Video>
                                    <Name>video_1080</Name>
                                    <Codec>h264</Codec>
                                    <Height>1080</Height>
                                    <Bitrate>2048000</Bitrate>
                                    <Framerate>24</Framerate>
                                    <ThreadCount>4</ThreadCount>
                                    <Preset>slower</Preset>
                                </Video>
                                <Video>
                                    <Name>video_720</Name>
                                    <Codec>h264</Codec>
                                    <Height>720</Height>
                                    <Bitrate>1024000</Bitrate>
                                    <Framerate>24</Framerate>
                                    <ThreadCount>4</ThreadCount>
                                    <Preset>slower</Preset>
                                </Video>
                                <Video>
                                    <Name>video_360</Name>
                                    <Codec>h264</Codec>
                                    <Height>360</Height>
                                    <Bitrate>700000</Bitrate>
                                    <Framerate>24</Framerate>
                                    <ThreadCount>4</ThreadCount>
                                    <Preset>slower</Preset>
                                </Video>

                                <Audio>
                                    <Name>bypass_audio</Name>
                                    <Bypass>true</Bypass>
                                </Audio>
                                <Audio>
                                    <Name>cond_audio_opus</Name>
                                    <Codec>opus</Codec>
                                    <Bitrate>128000</Bitrate>
                                    <Samplerate>48000</Samplerate>
                                    <Channel>2</Channel>
                                </Audio>
                            </Encodes>
                        </OutputProfile>
                    </OutputProfiles>

                    <Providers>
                        <RTMP>
                            <BlockDuplicateStreamName>true</BlockDuplicateStreamName>
                        </RTMP>
                    </Providers>

                    <Publishers>
                        <OVT />
                        <FILE>
                            <RootPath>/opt/ovenmediaengine/recordings</RootPath>
                            <FilePath>/${VirtualHost}/${Application}/${Stream}/manual-recording-${StartTime:YYYYMMDDhhmmss}.ts</FilePath>
                            <InfoPath>/${VirtualHost}/${Application}/${Stream}/manual-recording.xml</InfoPath>
                            <StreamMap>
                                <Enable>${env:ENABLE_AUTOMATED_RECORDING:false}</Enable>
                                <Path>./record.xml</Path>
                            </StreamMap>
                        </FILE>
                    </Publishers>
                </Application>
            </Applications>
        </VirtualHost>
    </VirtualHosts>
</Server>

I wasn't able to get any dumps from as the K8s container was deleted

[dimiden]

@bartvp1
Thank you for the configuration file. Could you also upload the logs?

[bartvp1]

These are the startup logs, I wasn't able to retrieve any other logs before the crash since they havent been persisted

I [OvenMediaEngine:1] Config | config_manager.cpp:239  | Trying to set logfile in directory... (/var/log/ovenmediaengine)
W [OvenMediaEngine:1] Socket | socket.cpp:1092 | [#5] [0x7f79ada341d0] Could not send data: -1 ([errno] Network is unreachable (101))
W [OvenMediaEngine:1] WebRTC Publisher | webrtc_publisher.cpp:105  | WebRTC Publisher is disabled by configuration
W [OvenMediaEngine:1] LLHLS Publisher | llhls_publisher.cpp:78   | LLHLS Publisher is disabled by configuration
W [OvenMediaEngine:1] Thumbnail | thumbnail_publisher.cpp:79   | ThumbnailPublisher is disabled by configuration
W [OvenMediaEngine:1] HLS Publisher | hls_publisher.cpp:78   | HLS Publisher is disabled by configuration
W [OvenMediaEngine:1] Transcoder | transcoder_gpu.cpp:54   | No supported NVIDIA Accelerator
W [OvenMediaEngine:1] Transcoder | transcoder_gpu.cpp:64   | No supported Xilinx Media Accelerator
W [OvenMediaEngine:1] Transcoder | transcoder_gpu.cpp:74   | No supported Intel QuickSync Accelerator
W [OvenMediaEngine:1] Transcoder | transcoder_gpu.cpp:84   | No supported Netint VPU Accelerator
W [OvenMediaEngine:1] WebRTC Provider | webrtc_provider.cpp:139  | WebRTCProvider is disabled by configuration
W [OvenMediaEngine:1] MpegtsProvider | mpegts_provider.cpp:147  | MPEGTSProvider is disabled by configuration
W [OvenMediaEngine:1] SrtProvider | srt_provider.cpp:50   | SrtProvider is disabled by configuration
W [OvenMediaEngine:1] Publisher | publisher.cpp:68   | Push publisher is disabled in #default#app application, so it was not created
W [OvenMediaEngine:1] Provider | provider.cpp:91   | OVT provider is disabled in #default#app application, so it was not created
W [OvenMediaEngine:1] Provider | provider.cpp:91   | RTSP Pull provider is disabled in #default#app application, so it was not created
W [OvenMediaEngine:1] Provider | provider.cpp:91   | Scheduled provider is disabled in #default#app application, so it was not created
W [OvenMediaEngine:1] Provider | provider.cpp:91   | Multiplex provider is disabled in #default#app application, so it was not created

[dimiden]

@bartvp1
Thank you for your response.
Since the code did not run as expected, we may need to pinpoint the error location through the crash dump or review the logs to identify the situation before the issue occurred.

If you could mount the logs and dump path to persistent storage and share the relevant details when the issue is reproduced, it would be greatly appreciated.
(Even if the logs seem unnecessary, such as the OME version and host information, please do not remove them and upload the full logs.)

[bartvp1]

Thank you, I'll post the logs here when this problem appears again!

[bartvp1]

This issue was caused by a 524 Timeout from the admission webhook (cloudflare, using https).

Another observation: Even though the timeout was set to 3000ms i the origin server config, it would sometimes take up to a minute for the SSL warning to show up and kill OME.

[dimiden]

@bartvp1
We are having difficulty analyzing the cause as it is not reproducible in our test environment.
Could you send us the admission webhook URL so that we can check it in our test environment?
If possible, please send it to [email protected]. Thank you!

[bartvp1]

Since the root cause has been resolved (timeout between Cloudflare and the admission webhook server), I can't reproduce this anymore either.

So the only issue remaining is the timeout (Server.VirtualHosts.VirtualHost.AdmissionWebhooks.Timeout) is not behaving like expected

[dimiden]

@bartvp1
Thank you for the confirmation. Based on what you mentioned, it seems like a timeout-related issue, and I will closely investigate this part.

Regarding the Server.VirtualHosts.VirtualHost.AdmissionWebhooks.Timeout setting you mentioned, since it is a connection timeout, it will not trigger a timeout if the connection is established and the admission webhook server responds slowly afterwards.