Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WARN: child terminated by signal: 11: Segmentation fault fault for ( C++ ) #1948

Open
m9rco opened this issue Aug 15, 2024 · 10 comments
Open

WARN: child terminated by signal: 11: Segmentation fault fault for ( C++ ) #1948

m9rco opened this issue Aug 15, 2024 · 10 comments
Labels
bug

Comments

@m9rco
Copy link

m9rco commented Aug 15, 2024 •
edited
Loading

step 0

env

$ clang --version
clang version 16.0.0 (Red Hat 16.0.0-2.module_el8+405+25122a8c)
Target: x86_64-redhat-linux-gnu
Thread model: posix
InstalledDir: /bin
uftrace detected system features:
...         prefix: /usr/local
...         libelf: [ on  ] - more flexible ELF data handling
...          libdw: [ on  ] - DWARF debug info support
...      libpython: [ on  ] - python tracing & scripting support
...      libluajit: [ on  ] - luajit scripting support
...    libncursesw: [ on  ] - TUI support
...   cxa_demangle: [ on  ] - full demangler support with libstdc++
...     perf_event: [ on  ] - perf (PMU) event support
...       schedule: [ on  ] - scheduler event support
...       capstone: [ on  ] - full dynamic tracing support
...  libtraceevent: [ OFF ] - kernel tracing support
...      libunwind: [ OFF ] - stacktrace support (optional for debugging)

source code

$ https://github.com/namhyung/uftrace@master

354c270f4e5319562a57a9c2711b88a760751a63

step 1

$ clang -pg ... app
$ readelf -s app |grep mcount

   405: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND mcount@GLIBC_2.2.5 (2)
432268: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND mcount
$ ldd app

linux-vdso.so.1 (0x00007ffca499e000)
/$LIB/libonion.so => /lib64/libonion.so (0x00007fdee9f13000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fdee9aec000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdee98cc000)
libanl.so.1 => /lib64/libanl.so.1 (0x00007fdee96c8000)
libm.so.6 => /lib64/libm.so.6 (0x00007fdee9346000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fdee8fb1000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fdee8d99000)
libc.so.6 => /lib64/libc.so.6 (0x00007fdee89d4000)
/lib64/ld-linux-x86-64.so.2 (0x00007fdee9cf0000)

step 2

 $  export ld_library_path=/data/home/lab/lib64/:/usr/local/lib && uftrace record --time -t 1s --no-libcall --no-event  ./app
#0  x86_64_fallback_frame_state (context=0xb191a630, context=0xb191a630, fs=0xb191a720) at ./md-unwind-support.h:63
63        if (*(unsigned char *)(pc+0) == 0x48
[Current thread is 1 (Thread 0x7f7e60882bc0 (LWP 3235876))]
(gdb) bt
#0  x86_64_fallback_frame_state (context=0xb191a630, context=0xb191a630, fs=0xb191a720) at ./md-unwind-support.h:63
#1  uw_frame_state_for (context=context@entry=0xb191a630, fs=fs@entry=0xb191a720) at ../../../libgcc/unwind-dw2.c:1265
#2  0x00007f7e5f4e8f2b in _Unwind_RaiseException (exc=exc@entry=0xbc2f7a40) at ../../../libgcc/unwind.inc:104
#3  0x00007f7e5f7867eb in __cxxabiv1::__cxa_throw (obj=<optimized out>,
    tinfo=0x7f7e5fa77ee8 <typeinfo for std::invalid_argument>,
    dest=0x7f7e5f79c0c0 <std::invalid_argument::~invalid_argument()>)
    at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90
#4  0x00007f7e60459089 in mcount () at /data/home/lab/uftrace/arch/x86_64/mcount.S:80
#5  0x0000000000000000 in ?? ()

0x0000000000000000

uftrace/arch/x86_64/mcount.S

Line 80 in c54660b

retq

@m9rco
Copy link
Author

m9rco commented Aug 15, 2024 •
edited
Loading

Additionally, and if started with the command

$ uftrace record --time -t 1s --no-libcall --no-event

it might take more than 10 minutes. Is there an attach method that can be supported? This way, the startup command won't need to be modified.

@m9rco
Copy link
Author

m9rco commented Aug 16, 2024

help me ~

@m9rco m9rco changed the title Segmentation fault for C++ WARN: child terminated by signal: 11: Segmentation fault fault for C++ Aug 16, 2024
@m9rco m9rco changed the title WARN: child terminated by signal: 11: Segmentation fault fault for C++ WARN: child terminated by signal: 11: Segmentation fault fault for ( C++ ) Aug 16, 2024
@namhyung
Copy link
Owner

Hi, it looks like mcount function is throwing an exception. Can you run uftrace with -vv option so that it can print some debug messages? I don't see anything specific to uftrace here.

@m9rco
Copy link
Author

m9rco commented Aug 17, 2024

okk，Let me fill in some details

@m9rco
Copy link
Author

m9rco commented Aug 17, 2024 •
edited
Loading

thx ~~ The following is my debug messages :)

uftrace: running uftrace v0.16-13-gc546 ( x86_64 dwarf python3 luajit tui perf sched dynamic kernel )
uftrace: checking binary ./app
uftrace: removing uftrace.data.old directory
uftrace: creating 8 thread(s) for recording
uftrace: start writer thread 0
uftrace: start writer thread 1
uftrace: start writer thread 3
uftrace: start writer thread 2
uftrace: start writer thread 4
uftrace: start writer thread 6
uftrace: start writer thread 7
uftrace: using /usr/local/lib/uftrace/libmcount.so library for tracing
uftrace: start writer thread 5
mcount: initializing mcount library
symbol: found rela.dyn section with 24 entry.
wrap: dlopen is called for 'libdebuginfod.so.1'
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
mcount: mcount setup done
mcount: new session started: 14d65ecaa04027b5: app
mcount: preparing shmem buffers: tid = 549552
uftrace: MSG SESSION: 549552: /data/home/app (14d65ecaa04027b5)
uftrace: MSG START: /uftrace-14d65ecaa04027b5-549552-000
uftrace: MSG TASK_START : 549552/549552
GameServer: 2264

..................................................... ( Wait about 10 minutes.. )

uftrace: MSG START: /uftrace-14d65ecaa04027b5-603412-000
uftrace: MSG TASK_START : 549552/603412
uftrace: MSG START: /uftrace-14d65ecaa04027b5-603413-000
uftrace: MSG TASK_START : 549552/603413
uftrace: MSG START: /uftrace-14d65ecaa04027b5-603414-000
uftrace: MSG TASK_START : 549552/603414
uftrace: MSG START: /uftrace-14d65ecaa04027b5-603423-000
uftrace: MSG TASK_START : 549552/603423
uftrace: MSG START: /uftrace-14d65ecaa04027b5-603530-000
uftrace: MSG TASK_START : 549552/603530
uftrace: MSG START: /uftrace-14d65ecaa04027b5-603531-000
uftrace: MSG TASK_START : 549552/603531
uftrace: MSG START: /uftrace-14d65ecaa04027b5-603593-000
uftrace: MSG TASK_START : 549552/603593
uftrace: MSG START: /uftrace-14d65ecaa04027b5-603594-000
uftrace: MSG TASK_START : 549552/603594
uftrace: MSG START: /uftrace-14d65ecaa04027b5-603533-000
uftrace: MSG TASK_START : 549552/603533
uftrace: MSG DLOPEN: 549552: 0x7ff3dc025000 /lib64/libdebuginfod.so.1
uftrace: MSG DLOPEN: 549552: 0x7ff3dbd96000 /lib64/libcurl.so.4
uftrace: MSG DLOPEN: 549552: 0x7ff3dbb6f000 /lib64/libnghttp2.so.14
uftrace: MSG DLOPEN: 549552: 0x7ff3db951000 /lib64/libidn2.so.0
uftrace: MSG DLOPEN: 549552: 0x7ff3db6e2000 /lib64/libssh.so.4
uftrace: MSG DLOPEN: 549552: 0x7ff3db4d1000 /lib64/libpsl.so.5
uftrace: MSG DLOPEN: 549552: 0x7ff3db235000 /lib64/libssl.so.1.1
uftrace: MSG DLOPEN: 549552: 0x7ff3dad44000 /lib64/libcrypto.so.1.1
uftrace: MSG DLOPEN: 549552: 0x7ff3daaef000 /lib64/libgssapi_krb5.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3da805000 /lib64/libkrb5.so.3
uftrace: MSG DLOPEN: 549552: 0x7ff3da5ee000 /lib64/libk5crypto.so.3
uftrace: MSG DLOPEN: 549552: 0x7ff3da3ea000 /lib64/libcom_err.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3df294000 /lib64/libldap-2.4.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3df282000 /lib64/liblber-2.4.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3df273000 /lib64/libbrotlidec.so.1
uftrace: MSG DLOPEN: 549552: 0x7ff3da069000 /lib64/libunistring.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3d9e58000 /lib64/libkrb5support.so.0
uftrace: MSG DLOPEN: 549552: 0x7ff3df26c000 /lib64/libkeyutils.so.1
uftrace: MSG DLOPEN: 549552: 0x7ff3d9c40000 /lib64/libresolv.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3df24c000 /lib64/libsasl2.so.3
uftrace: MSG DLOPEN: 549552: 0x7ff3df229000 /lib64/libbrotlicommon.so.1
uftrace: MSG DLOPEN: 549552: 0x7ff3d9a15000 /lib64/libselinux.so.1
uftrace: MSG DLOPEN: 549552: 0x7ff3df1fe000 /lib64/libcrypt.so.1
uftrace: MSG DLOPEN: 549552: 0x7ff3d9791000 /lib64/libpcre2-8.so.0
uftrace: MSG DLOPEN: 549552: 0x7ff3d5c5a000 /lib64/libnss_files.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3d5a53000 /lib64/libnss_dns.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3c79bc000 ../../../lib64//libtss_sdk.so
uftrace: MSG DLOPEN: 549552: 0x7ff3dc025000 /lib64/libdebuginfod.so.1
uftrace: MSG DLOPEN: 549552: 0x7ff3dbd96000 /lib64/libcurl.so.4
uftrace: MSG DLOPEN: 549552: 0x7ff3dbb6f000 /lib64/libnghttp2.so.14
uftrace: MSG DLOPEN: 549552: 0x7ff3db951000 /lib64/libidn2.so.0
uftrace: MSG DLOPEN: 549552: 0x7ff3db6e2000 /lib64/libssh.so.4
uftrace: MSG DLOPEN: 549552: 0x7ff3db4d1000 /lib64/libpsl.so.5
uftrace: MSG DLOPEN: 549552: 0x7ff3db235000 /lib64/libssl.so.1.1
uftrace: MSG DLOPEN: 549552: 0x7ff3dad44000 /lib64/libcrypto.so.1.1
uftrace: MSG DLOPEN: 549552: 0x7ff3daaef000 /lib64/libgssapi_krb5.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3da805000 /lib64/libkrb5.so.3
uftrace: MSG DLOPEN: 549552: 0x7ff3da5ee000 /lib64/libk5crypto.so.3
uftrace: MSG DLOPEN: 549552: 0x7ff3da3ea000 /lib64/libcom_err.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3df294000 /lib64/libldap-2.4.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3df282000 /lib64/liblber-2.4.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3df273000 /lib64/libbrotlidec.so.1
uftrace: MSG DLOPEN: 549552: 0x7ff3da069000 /lib64/libunistring.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3d9e58000 /lib64/libkrb5support.so.0
uftrace: MSG DLOPEN: 549552: 0x7ff3df26c000 /lib64/libkeyutils.so.1
uftrace: MSG DLOPEN: 549552: 0x7ff3d9c40000 /lib64/libresolv.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3df24c000 /lib64/libsasl2.so.3
uftrace: MSG DLOPEN: 549552: 0x7ff3df229000 /lib64/libbrotlicommon.so.1
uftrace: MSG DLOPEN: 549552: 0x7ff3d9a15000 /lib64/libselinux.so.1
uftrace: MSG DLOPEN: 549552: 0x7ff3df1fe000 /lib64/libcrypt.so.1
uftrace: MSG DLOPEN: 549552: 0x7ff3d9791000 /lib64/libpcre2-8.so.0
uftrace: MSG DLOPEN: 549552: 0x7ff3d5c5a000 /lib64/libnss_files.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3d5a53000 /lib64/libnss_dns.so.2
uftrace: MSG DLOPEN: 549552: 0x7ff3c79bc000 ../../../lib64//libtss_sdk.so
uftrace: MSG DLOPEN: 549552: 0x7ff3c7797000 /tmp/tsssdk_cfg/liblight_feature.so
uftrace: MSG START: /uftrace-14d65ecaa04027b5-603679-000
uftrace: MSG TASK_START : 549552/603679
uftrace: MSG START: /uftrace-14d65ecaa04027b5-603678-000
uftrace: MSG TASK_START : 549552/603678
uftrace: MSG START: /uftrace-14d65ecaa04027b5-603532-000
uftrace: MSG TASK_START : 549552/603532
uftrace: MSG START: /uftrace-14d65ecaa04027b5-603902-000
uftrace: MSG TASK_START : 549552/603902
uftrace: waiting for FORK2
uftrace: waiting for FORK2
uftrace: waiting for FORK2
uftrace: waiting for FORK2
uftrace: waiting for FORK2
uftrace: waiting for FORK2
..................................................... ( Very, very many of the same logs ... )
uftrace: waiting for FORK2
uftrace: waiting for FORK2
uftrace: waiting for FORK2
uftrace: waiting for FORK2
uftrace: all process/thread exited
WARN: child terminated by signal: 11: Segmentation fault
elapsed time:   1202.226988623 sec
 system time:      2.341284000 sec
   user time:   1201.049087000 sec
uftrace: stop writer thread 5
uftrace: stop writer thread 6
uftrace: stop writer thread 0
uftrace: stop writer thread 7
uftrace: stop writer thread 4
uftrace: stop writer thread 1
uftrace: stop writer thread 3
uftrace: stop writer thread 2
uftrace: reading uftrace.data/task.txt file
session: new session: pid = 549552, session = 14d65ecaa04027b5
session: task session: tid = 549552, session = 14d65ecaa04027b5
session: new task: tid = 549552 (app), session = 14d65ecaa04027b5
session: task session: tid = 603412, session = 14d65ecaa04027b5
session: new task: tid = 603412 (app), session = 14d65ecaa04027b5
session: task session: tid = 603413, session = 14d65ecaa04027b5
session: new task: tid = 603413 (app), session = 14d65ecaa04027b5
session: task session: tid = 603414, session = 14d65ecaa04027b5
session: new task: tid = 603414 (app), session = 14d65ecaa04027b5
session: task session: tid = 603423, session = 14d65ecaa04027b5
session: new task: tid = 603423 (app), session = 14d65ecaa04027b5
session: task session: tid = 603530, session = 14d65ecaa04027b5
session: new task: tid = 603530 (app), session = 14d65ecaa04027b5
session: task session: tid = 603531, session = 14d65ecaa04027b5
session: new task: tid = 603531 (app), session = 14d65ecaa04027b5
session: task session: tid = 603593, session = 14d65ecaa04027b5
session: new task: tid = 603593 (app), session = 14d65ecaa04027b5
session: task session: tid = 603594, session = 14d65ecaa04027b5
session: new task: tid = 603594 (app), session = 14d65ecaa04027b5
session: task session: tid = 603533, session = 14d65ecaa04027b5
session: new task: tid = 603533 (app), session = 14d65ecaa04027b5
session: task session: tid = 603679, session = 14d65ecaa04027b5
session: new task: tid = 603679 (app), session = 14d65ecaa04027b5
session: task session: tid = 603678, session = 14d65ecaa04027b5
session: new task: tid = 603678 (app), session = 14d65ecaa04027b5
session: task session: tid = 603532, session = 14d65ecaa04027b5
session: new task: tid = 603532 (app), session = 14d65ecaa04027b5
session: task session: tid = 603902, session = 14d65ecaa04027b5
session: new task: tid = 603902 (app), session = 14d65ecaa04027b5
uftrace: flushing /uftrace-14d65ecaa04027b5-549552-000
uftrace: flushing /uftrace-14d65ecaa04027b5-603412-000
uftrace: flushing /uftrace-14d65ecaa04027b5-603413-000
uftrace: flushing /uftrace-14d65ecaa04027b5-603414-000
uftrace: flushing /uftrace-14d65ecaa04027b5-603423-000
uftrace: flushing /uftrace-14d65ecaa04027b5-603530-000
uftrace: flushing /uftrace-14d65ecaa04027b5-603531-000
uftrace: flushing /uftrace-14d65ecaa04027b5-603593-000
uftrace: flushing /uftrace-14d65ecaa04027b5-603594-000
uftrace: flushing /uftrace-14d65ecaa04027b5-603533-000
uftrace: flushing /uftrace-14d65ecaa04027b5-603679-000
uftrace: flushing /uftrace-14d65ecaa04027b5-603678-000
uftrace: flushing /uftrace-14d65ecaa04027b5-603532-000
uftrace: flushing /uftrace-14d65ecaa04027b5-603902-000
uftrace: unlink for session: 14d65ecaa04027b5
uftrace: reading symbols for session 14d65ecaa04027b5
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: saving symbols to uftrace.data/libzstd.so.1.4.4.sym
symbol: saving symbols to uftrace.data/libz.so.1.2.11.sym
symbol: saving symbols to uftrace.data/libtraceevent.so.1.5.3.sym
symbol: saving symbols to uftrace.data/libstdc++.so.6.0.25.sym
symbol: saving symbols to uftrace.data/librt-2.28.so.sym
symbol: saving symbols to uftrace.data/libpthread-2.28.so.sym
symbol: saving symbols to uftrace.data/libonion_security.so.1.0.19.sym
symbol: saving symbols to uftrace.data/libm-2.28.so.sym
symbol: saving symbols to uftrace.data/liblzma.so.5.2.4.sym
symbol: saving symbols to uftrace.data/libgcc_s-8-20210514.so.1.sym
symbol: saving symbols to uftrace.data/libelf-0.190.so.sym
symbol: saving symbols to uftrace.data/libdw-0.190.so.sym
symbol: saving symbols to uftrace.data/libdl-2.28.so.sym
symbol: saving symbols to uftrace.data/libcapstone.so.4.sym
symbol: saving symbols to uftrace.data/libc-2.28.so.sym
symbol: saving symbols to uftrace.data/libbz2.so.1.0.6.sym
symbol: saving symbols to uftrace.data/libanl-2.28.so.sym
symbol: saving symbols to uftrace.data/ld-2.28.so.sym
symbol: saving symbols to uftrace.data/liblight_feature.so.sym
symbol: saving symbols to uftrace.data/libunistring.so.2.sym
symbol: saving symbols to uftrace.data/libssl.so.1.1.sym
symbol: saving symbols to uftrace.data/libssh.so.4.sym
symbol: saving symbols to uftrace.data/libselinux.so.1.sym
symbol: saving symbols to uftrace.data/libsasl2.so.3.sym
symbol: saving symbols to uftrace.data/libresolv.so.2.sym
symbol: saving symbols to uftrace.data/libpsl.so.5.sym
symbol: saving symbols to uftrace.data/libpcre2-8.so.0.sym
symbol: saving symbols to uftrace.data/libnss_files.so.2.sym
symbol: saving symbols to uftrace.data/libnss_dns.so.2.sym
symbol: saving symbols to uftrace.data/libnghttp2.so.14.sym
symbol: saving symbols to uftrace.data/libldap-2.4.so.2.sym
symbol: saving symbols to uftrace.data/liblber-2.4.so.2.sym
symbol: saving symbols to uftrace.data/libkrb5support.so.0.sym
symbol: saving symbols to uftrace.data/libkrb5.so.3.sym
symbol: saving symbols to uftrace.data/libkeyutils.so.1.sym
symbol: saving symbols to uftrace.data/libk5crypto.so.3.sym
symbol: saving symbols to uftrace.data/libidn2.so.0.sym
symbol: saving symbols to uftrace.data/libgssapi_krb5.so.2.sym
symbol: saving symbols to uftrace.data/libdebuginfod.so.1.sym
symbol: saving symbols to uftrace.data/libcurl.so.4.sym
symbol: saving symbols to uftrace.data/libcrypto.so.1.1.sym
symbol: saving symbols to uftrace.data/libcrypt.so.1.sym
symbol: saving symbols to uftrace.data/libcom_err.so.2.sym
symbol: saving symbols to uftrace.data/libbrotlidec.so.1.sym
symbol: saving symbols to uftrace.data/libbrotlicommon.so.1.sym
symbol: saving symbols to uftrace.data/app.sym
symbol: saving symbols to uftrace.data/libtss_sdk.so.sym

gdb debuger

(gdb) bt
#0  0x00007ff3de02bacf in uw_frame_state_for () from /lib64/libgcc_s.so.1
#1  0x00007ff3de02cf2b in _Unwind_RaiseException () from /lib64/libgcc_s.so.1
#2  0x00007ff3de2ca7eb in __cxa_throw () from /lib64/libstdc++.so.6
#3  0x00007ff3def9fdcf in mcount () at /root/uftrace/arch/x86_64/mcount.S:80
#4  0x0000000000000000 in ?? ()

@namhyung
Copy link
Owner

Is there any backtrace from uftrace (or libmcount) when it got the segfault?

Also I'm not sure why it takes too long.. what do you see when you use a different time filter like 10ms?

@honggyukim honggyukim added the bug label Aug 18, 2024
@honggyukim
Copy link
Collaborator

honggyukim commented Aug 18, 2024 •
edited
Loading

It would be a lot more helpful if you could share us what your app is so that we can reproduce your problem. If you cannot share your program, then the test program is better to be minimized rather than running a giant program for testing.

You can get some reference of the bug report from #1949.

@m9rco
Copy link
Author

m9rco commented Aug 19, 2024 •
edited
Loading

Thx @honggyukim , my program is a giant program, originating from business software. here is my test on ggerganov/llama.cpp, I found that it also encountered a segmentation fault. The specific test path is as follows ~ ( #1949

uftrace record --no-libcall ./llama-cli
Log start
main: build = 3601 (2339a0be)
main: built with cc (GCC) 8.5.0 20210514
main: seed  = 1724037308
WARN: Segmentation fault: address not mapped (addr: 0x14bc576c)
WARN:  if this happens only with uftrace, please consider -e/--estimate-return option.

WARN: Backtrace from uftrace v0.16-13-gc546 ( x86_64 dwarf python3 luajit tui perf sched dynamic kernel )
WARN: =====================================
WARN: [2] (llama_load_model_from_file[52a9e0] <= llama_init_from_gpt_params[5b9a9b])
WARN: [1] (llama_init_from_gpt_params[5b99fa] <= main[431e9a])
WARN: [0] (main[43170a] <= __libc_start_main[7fa15de29d85])

Please report this bug to https://github.com/namhyung/uftrace/issues.

WARN: child terminated by signal: 11: Segmentation fault

It still gets crashed even with -e option.

uftrace record --no-libcall -e ./llama-cli
Log start
main: build = 3601 (2339a0be)
main: built with cc (GCC) 8.5.0 20210514
main: seed  = 1724037362
WARN: Segmentation fault: address not mapped (addr: 0x14bdfaba)
WARN: Backtrace from uftrace v0.16-13-gc546 ( x86_64 dwarf python3 luajit tui perf sched dynamic kernel )
WARN: =====================================
WARN: [4] (llama_load_model_from_file[52a9e0] <= llama_init_from_gpt_params[5b9a9b])
WARN: [3] (llama_model_default_params[4d779a] <= llama_model_params_from_gpt_params[5b50a8])
WARN: [2] (llama_model_params_from_gpt_params[5b509d] <= llama_init_from_gpt_params[5b9a36])
WARN: [1] (llama_init_from_gpt_params[5b99fa] <= main[431e9a])
WARN: [0] (main[43170a] <= __libc_start_main[7f5343c97d85])

Please report this bug to https://github.com/namhyung/uftrace/issues.

WARN: child terminated by signal: 11: Segmentation fault

Here is the backtrace.

$ gdb -q --args uftrace record --no-libcall -e ./llama-cli
Reading symbols from uftrace...
(gdb) r
Starting program: /usr/local/bin/uftrace record --no-libcall -e ./llama-cli
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[Detaching after fork from child process 1137111]
[New Thread 0x7ffff4bce700 (LWP 1137112)]
[New Thread 0x7ffff43cd700 (LWP 1137113)]
[New Thread 0x7ffff3bcc700 (LWP 1137114)]
[New Thread 0x7ffff33cb700 (LWP 1137115)]
[New Thread 0x7ffff2bca700 (LWP 1137116)]
[New Thread 0x7ffff23c9700 (LWP 1137117)]
[New Thread 0x7ffff1bc8700 (LWP 1137118)]
[New Thread 0x7ffff13c7700 (LWP 1137119)]
Log start
main: build = 3601 (2339a0be)
main: built with cc (GCC) 8.5.0 20210514
main: seed  = 1724037407
WARN: Segmentation fault: address not mapped (addr: 0x14bf5cf4)
WARN: Backtrace from uftrace v0.16-13-gc546 ( x86_64 dwarf python3 luajit tui perf sched dynamic kernel )
WARN: =====================================
WARN: [4] (llama_load_model_from_file[52a9e0] <= llama_init_from_gpt_params[5b9a9b])
WARN: [3] (llama_model_default_params[4d779a] <= llama_model_params_from_gpt_params[5b50a8])
WARN: [2] (llama_model_params_from_gpt_params[5b509d] <= llama_init_from_gpt_params[5b9a36])
WARN: [1] (llama_init_from_gpt_params[5b99fa] <= main[431e9a])
WARN: [0] (main[43170a] <= __libc_start_main[7ffff6a79d85])

Please report this bug to https://github.com/namhyung/uftrace/issues.

WARN: child terminated by signal: 11: Segmentation fault
[Thread 0x7ffff13c7700 (LWP 1137119) exited]
[Thread 0x7ffff1bc8700 (LWP 1137118) exited]
[Thread 0x7ffff23c9700 (LWP 1137117) exited]
[Thread 0x7ffff2bca700 (LWP 1137116) exited]
[Thread 0x7ffff33cb700 (LWP 1137115) exited]
[Thread 0x7ffff3bcc700 (LWP 1137114) exited]
[Thread 0x7ffff43cd700 (LWP 1137113) exited]
[Thread 0x7ffff4bce700 (LWP 1137112) exited]
[Inferior 1 (process 1136592) exited with code 02]
Missing separate debuginfos, use: dnf debuginfo-install bash-4.4.20-4.tl3.tencentos.x86_64 brotli-1.0.6-3.tl3.x86_64 bzip2-libs-1.0.6-26.tl3.x86_64 capstone-4.0.2-5.el8.x86_64 cyrus-sasl-lib-2.1.27-6.tl3.x86_64 elfutils-debuginfod-client-0.190-2.tl3.x86_64 elfutils-libelf-0.190-2.tl3.x86_64 elfutils-libs-0.190-2.tl3.x86_64 glibc-2.28-225.tl3.x86_64 keyutils-libs-1.5.10-9.tl3.x86_64 krb5-libs-1.18.2-22.tl3.x86_64 libcom_err-1.45.6-5.tl3.x86_64 libcurl-7.61.1-30.tl3.2.x86_64 libgcc-8.5.0-18.tl3.x86_64 libidn2-2.2.0-1.tl3.x86_64 libnghttp2-1.33.0-3.tl3.1.x86_64 libpsl-0.20.2-6.tl3.x86_64 libselinux-2.9-8.tl3.x86_64 libssh-0.9.6-6.tl3.x86_64 libstdc++-8.5.0-18.tl3.x86_64 libtraceevent-1.5.3-1.tl3.x86_64 libxcrypt-4.1.1-6.tl3.x86_64 libzstd-1.4.4-1.tl3.x86_64 ncurses-libs-6.1-10.20180224.tl3.x86_64 openssl-libs-1.1.1k-9.tl3.3.x86_64 pcre2-10.32-2.tl3.x86_64 xz-libs-5.2.4-4.tl3.x86_64 zlib-1.2.11-21.tl3.x86_64
(gdb) bt
No stack.
gdb ./llama-cli ~/corefile/core-llama-cli-11-1000-1000-1133916-1724037308
GNU gdb (GDB) Red Hat Enterprise Linux 9.2-4.tl3
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./llama-cli...

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing
[New LWP 1133916]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

warning: the debug information found in "/usr/lib/debug//lib64/libbz2.so.1.0.6-1.0.6-26.tl3.x86_64.debug" does not match "/lib64/libbz2.so.1" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug//usr/lib64/libbz2.so.1.0.6-1.0.6-26.tl3.x86_64.debug" does not match "/lib64/libbz2.so.1" (CRC mismatch).

Missing separate debuginfo for /lib64/libbz2.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/fe/194562c2e19c235fb8c13b3ec7931029ed2f7f.debug
Core was generated by `./llama-cli'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000052a9e0 in llama_load_model_from_file (path_model=0x1e1d510 "models/7B/ggml-model-f16.gguf",
    params=...) at src/llama.cpp:16936
16936           struct llama_model_params   params) {
Missing separate debuginfos, use: dnf debuginfo-install libgcc-8.5.0-18.tl3.x86_64 libgomp-8.5.0-18.tl3.x86_64 libstdc++-8.5.0-18.tl3.x86_64 libtraceevent-1.5.3-1.tl3.x86_64 libzstd-1.4.4-1.tl3.x86_64 xz-libs-5.2.4-4.tl3.x86_64 zlib-1.2.11-21.tl3.x86_64
(gdb) bt
#0  0x000000000052a9e0 in llama_load_model_from_file (path_model=0x1e1d510 "models/7B/ggml-model-f16.gguf",
    params=...) at src/llama.cpp:16936
#1  0x00000000005b9a9b in llama_init_from_gpt_params (params=...) at common/common.cpp:2107
#2  0x0000000000431e9a in main (argc=<optimized out>, argv=<optimized out>) at examples/main/main.cpp:210
(gdb)

I tried debugging using lldb ~

$ lldb                                 
(lldb) file /usr/local/bin/uftrace
Current executable set to '/usr/local/bin/uftrace' (x86_64).
(lldb) settings set -- target.run-args record --no-libcall -e ./llama-cli
(lldb) run
Process 1521648 launched: '/usr/local/bin/uftrace' (x86_64)
warning: (x86_64) /lib64/libonion.so unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libdl.so.2 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/librt.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libstdc++.so.6 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libelf.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libdw.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libcapstone.so.4 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libm.so.6 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libncursesw.so.6 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libtinfo.so.6 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libpthread.so.0 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libc.so.6 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libgcc_s.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libzstd.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
Log start
main: build = 3601 (2339a0be)
main: built with cc (GCC) 8.5.0 20210514 (TencentOS 8.5.0-18) for x86_64-redhat-linux
main: seed  = 1724050099
WARN: Segmentation fault: address not mapped (addr: 0x16425d50)
WARN: Backtrace from uftrace v0.16-13-gc546 ( x86_64 dwarf python3 luajit tui perf sched dynamic kernel )
WARN: =====================================
WARN: [4] (llama_load_model_from_file[52a9e0] <= llama_init_from_gpt_params[5b9a9b])
WARN: [3] (llama_model_default_params[4d779a] <= llama_model_params_from_gpt_params[5b50a8])
WARN: [2] (llama_model_params_from_gpt_params[5b509d] <= llama_init_from_gpt_params[5b9a36])
WARN: [1] (llama_init_from_gpt_params[5b99fa] <= main[431e9a])
WARN: [0] (main[43170a] <= __libc_start_main[7ffff6a79d85])

Please report this bug to https://github.com/namhyung/uftrace/issues.

Process 1521648 stopped and restarted: thread 1 received signal: SIGCHLD
WARN: child terminated by signal: 11: Segmentation fault
warning: (x86_64) /lib64/libdebuginfod.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libcurl.so.4 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libpsl.so.5 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libssl.so.1.1 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libk5crypto.so.3 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libcom_err.so.2 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libldap-2.4.so.2 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/liblber-2.4.so.2 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libbrotlidec.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libkrb5support.so.0 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libkeyutils.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libresolv.so.2 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libsasl2.so.3 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libbrotlicommon.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libselinux.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libpcre2-8.so.0 unsupported DW_FORM values: 0x1f20 0x1f21
Process 1521648 exited with status = 2 (0x00000002)
(lldb) tb
No breakpoints currently set.

I'm not sure if these warnings make a difference

 lldb -c core-llama-cli-11-1000-1000-1521657-1724050099 -- ~/t/lab/llama.cpp/llama-cli
(lldb) target create "/data/home/user00/t/lab/llama.cpp/llama-cli" --core "core-llama-cli-11-1000-1000-1521657-1724050099"
warning: (x86_64) /lib64/libonion.so unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libstdc++.so.6 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libm.so.6 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libgcc_s.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libpthread.so.0 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libc.so.6 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libdl.so.2 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/librt.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libelf.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libdw.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libcapstone.so.4 unsupported DW_FORM values: 0x1f20 0x1f21
warning: (x86_64) /lib64/libzstd.so.1 unsupported DW_FORM values: 0x1f20 0x1f21
Core file '/data/home/user00/corefile/core-llama-cli-11-1000-1000-1521657-1724050099' (x86_64) was loaded.
(lldb) bt
* thread #1, name = 'llama-cli', stop reason = signal SIGSEGV
  * frame #0: 0x000000000052a9e0 llama-cli`llama_load_model_from_file(path_model="models/7B/ggml-model-f16.gguf", params=llama_model_params @ 0x00007fffffffbf70) at llama.cpp:16936:45
    frame #1: 0x00000000005b9a9b llama-cli`llama_init_from_gpt_params(params=0x00007fffffffc7b0) at common.cpp:2107:43
    frame #2: 0x0000000000431e9a llama-cli`main(argc=<unavailable>, argv=<unavailable>) at main.cpp:210:69
    frame #3: 0x00007ffff6a79d85 libc.so.6`__libc_start_main + 229
    frame #4: 0x000000000043896e llama-cli`_start + 46

@m9rco
Copy link
Author

m9rco commented Aug 19, 2024 •
edited
Loading

Thx @namhyung , I will collect the backtrace information and also change the collection frequency (if I use --nop, it doesn't seem to cause a segmentation fault).

$ gdb -q --args uftrace record --time -t 10ms -vv --no-libcall --no-event ./app server-bin:app_dev_ 

Reading symbols from uftrace...
(gdb) r
Starting program: /usr/local/bin/uftrace record --time -t 10ms -vv --no-libcall --no-event ./app server-bin:app_dev_ --bus-key=1681 --id=1.0.13.1 --conf-path=../conf/ --pid-file=server.pid --log-path=/data/home/user00/log/app_1.0.13.1 --log-basename=dev_marcopu
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: the debug information found in "/usr/lib/debug//lib64/libbz2.so.1.0.6-1.0.6-26.tl3.x86_64.debug" does not match "/lib64/libbz2.so.1" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug//usr/lib64/libbz2.so.1.0.6-1.0.6-26.tl3.x86_64.debug" does not match "/lib64/libbz2.so.1" (CRC mismatch).

Missing separate debuginfo for /lib64/libbz2.so.1
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/fe/194562c2e19c235fb8c13b3ec7931029ed2f7f.debug
uftrace: running uftrace v0.16-13-gc546 ( x86_64 dwarf python3 luajit tui perf sched dynamic kernel )
uftrace: checking binary ./app
uftrace: removing uftrace.data.old directory
[Detaching after fork from child process 1282147]
uftrace: creating 8 thread(s) for recording
uftrace: using /usr/local/lib/uftrace/libmcount.so library for tracing
[New Thread 0x7ffff4e20700 (LWP 1282148)]
uftrace: start writer thread 0
[New Thread 0x7fffeffff700 (LWP 1282149)]
uftrace: start writer thread 1
[New Thread 0x7fffef7fe700 (LWP 1282150)]
uftrace: start writer thread 2
[New Thread 0x7fffeeffd700 (LWP 1282151)]
uftrace: start writer thread 3
[New Thread 0x7fffee7fc700 (LWP 1282152)]
uftrace: start writer thread 4
[New Thread 0x7fffedffb700 (LWP 1282153)]
uftrace: start writer thread 5
[New Thread 0x7fffed7fa700 (LWP 1282154)]
uftrace: start writer thread 6
[New Thread 0x7fffecff9700 (LWP 1282155)]
uftrace: start writer thread 7
mcount: initializing mcount library
symbol: found rela.dyn section with 24 entry.
wrap: dlopen is called for 'libdebuginfod.so.1'
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
mcount: mcount setup done
mcount: new session started: dfb32f90c3d6f87b: app
mcount: preparing shmem buffers: tid = 1282147
uftrace: MSG SESSION: 1282147: /data/home/user00/wedo/bin/app (dfb32f90c3d6f87b)
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1282147-000
uftrace: MSG TASK_START : 1282147/1282147
GameServer: 2264
uftrace: MSG  END : /uftrace-dfb32f90c3d6f87b-1282147-000
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1282147-001
GameServer: 2264
uftrace: MSG  END : /uftrace-dfb32f90c3d6f87b-1282147-000
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1282147-001
uftrace: MSG  END : /uftrace-dfb32f90c3d6f87b-1282147-001
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1282147-000
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1316324-000
uftrace: MSG TASK_START : 1282147/1316324
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1316325-000
uftrace: MSG TASK_START : 1282147/1316325
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1316345-000
uftrace: MSG TASK_START : 1282147/1316345
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1316346-000
uftrace: MSG TASK_START : 1282147/1316346
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1316484-000
uftrace: MSG TASK_START : 1282147/1316484
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1316485-000
uftrace: MSG TASK_START : 1282147/1316485
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1316547-000
uftrace: MSG TASK_START : 1282147/1316547
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1316548-000
uftrace: MSG TASK_START : 1282147/1316548
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1316487-000
uftrace: MSG TASK_START : 1282147/1316487
uftrace: MSG DLOPEN: 1282147: 0x7ffff4c39000 /lib64/libdebuginfod.so.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff49aa000 /lib64/libcurl.so.4
uftrace: MSG DLOPEN: 1282147: 0x7ffff4783000 /lib64/libnghttp2.so.14
uftrace: MSG DLOPEN: 1282147: 0x7ffff4565000 /lib64/libidn2.so.0
uftrace: MSG DLOPEN: 1282147: 0x7ffff42f6000 /lib64/libssh.so.4
uftrace: MSG DLOPEN: 1282147: 0x7ffff40e5000 /lib64/libpsl.so.5
uftrace: MSG DLOPEN: 1282147: 0x7ffff3e49000 /lib64/libssl.so.1.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff3958000 /lib64/libcrypto.so.1.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff3703000 /lib64/libgssapi_krb5.so.2
uftrace: MSG DLOPEN: 1282147: 0x7ffff3419000 /lib64/libkrb5.so.3
uftrace: MSG DLOPEN: 1282147: 0x7ffff3202000 /lib64/libk5crypto.so.3
uftrace: MSG DLOPEN: 1282147: 0x7ffff2ffe000 /lib64/libcom_err.so.2
uftrace: MSG DLOPEN: 1282147: 0x7ffff7ea2000 /lib64/libldap-2.4.so.2
uftrace: MSG DLOPEN: 1282147: 0x7ffff7e90000 /lib64/liblber-2.4.so.2
uftrace: MSG DLOPEN: 1282147: 0x7ffff7e81000 /lib64/libbrotlidec.so.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff2c7d000 /lib64/libunistring.so.2
uftrace: MSG DLOPEN: 1282147: 0x7ffff2a6c000 /lib64/libkrb5support.so.0
uftrace: MSG DLOPEN: 1282147: 0x7ffff7e7a000 /lib64/libkeyutils.so.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff2854000 /lib64/libresolv.so.2
uftrace: MSG DLOPEN: 1282147: 0x7ffff7e5a000 /lib64/libsasl2.so.3
uftrace: MSG DLOPEN: 1282147: 0x7ffff7e37000 /lib64/libbrotlicommon.so.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff2629000 /lib64/libselinux.so.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff7e0c000 /lib64/libcrypt.so.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff23a5000 /lib64/libpcre2-8.so.0
uftrace: MSG DLOPEN: 1282147: 0x7fffee86e000 /lib64/libnss_files.so.2
uftrace: MSG DLOPEN: 1282147: 0x7fffee667000 /lib64/libnss_dns.so.2
uftrace: MSG DLOPEN: 1282147: 0x7fffe05d0000 ../../../lib64//libtss_sdk.so
uftrace: MSG DLOPEN: 1282147: 0x7ffff4c39000 /lib64/libdebuginfod.so.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff49aa000 /lib64/libcurl.so.4
uftrace: MSG DLOPEN: 1282147: 0x7ffff4783000 /lib64/libnghttp2.so.14
uftrace: MSG DLOPEN: 1282147: 0x7ffff4565000 /lib64/libidn2.so.0
uftrace: MSG DLOPEN: 1282147: 0x7ffff42f6000 /lib64/libssh.so.4
uftrace: MSG DLOPEN: 1282147: 0x7ffff40e5000 /lib64/libpsl.so.5
uftrace: MSG DLOPEN: 1282147: 0x7ffff3e49000 /lib64/libssl.so.1.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff3958000 /lib64/libcrypto.so.1.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff3703000 /lib64/libgssapi_krb5.so.2
uftrace: MSG DLOPEN: 1282147: 0x7ffff3419000 /lib64/libkrb5.so.3
uftrace: MSG DLOPEN: 1282147: 0x7ffff3202000 /lib64/libk5crypto.so.3
uftrace: MSG DLOPEN: 1282147: 0x7ffff2ffe000 /lib64/libcom_err.so.2
uftrace: MSG DLOPEN: 1282147: 0x7ffff7ea2000 /lib64/libldap-2.4.so.2
uftrace: MSG DLOPEN: 1282147: 0x7ffff7e90000 /lib64/liblber-2.4.so.2
uftrace: MSG DLOPEN: 1282147: 0x7ffff7e81000 /lib64/libbrotlidec.so.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff2c7d000 /lib64/libunistring.so.2
uftrace: MSG DLOPEN: 1282147: 0x7ffff2a6c000 /lib64/libkrb5support.so.0
uftrace: MSG DLOPEN: 1282147: 0x7ffff7e7a000 /lib64/libkeyutils.so.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff2854000 /lib64/libresolv.so.2
uftrace: MSG DLOPEN: 1282147: 0x7ffff7e5a000 /lib64/libsasl2.so.3
uftrace: MSG DLOPEN: 1282147: 0x7ffff7e37000 /lib64/libbrotlicommon.so.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff2629000 /lib64/libselinux.so.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff7e0c000 /lib64/libcrypt.so.1
uftrace: MSG DLOPEN: 1282147: 0x7ffff23a5000 /lib64/libpcre2-8.so.0
uftrace: MSG DLOPEN: 1282147: 0x7fffee86e000 /lib64/libnss_files.so.2
uftrace: MSG DLOPEN: 1282147: 0x7fffee667000 /lib64/libnss_dns.so.2
uftrace: MSG DLOPEN: 1282147: 0x7fffe05d0000 ../../../lib64//libtss_sdk.so
uftrace: MSG DLOPEN: 1282147: 0x7fffe03ab000 /tmp/tsssdk_cfg/liblight_feature.so
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1316626-000
uftrace: MSG TASK_START : 1282147/1316626
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1316625-000
uftrace: MSG TASK_START : 1282147/1316625
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1316486-000
uftrace: MSG TASK_START : 1282147/1316486
uftrace: MSG START: /uftrace-dfb32f90c3d6f87b-1316862-000
uftrace: MSG TASK_START : 1282147/1316862
uftrace: waiting for FORK2
uftrace: waiting for FORK2
uftrace: waiting for FORK2
uftrace: waiting for FORK2
...
uftrace: waiting for FORK2
uftrace: waiting for FORK2
uftrace: waiting for FORK2
uftrace: waiting for FORK2
uftrace: all process/thread exited
WARN: child terminated by signal: 11: Segmentation fault
elapsed time:   1197.231679194 sec
 system time:      2.339842000 sec
   user time:   1196.674537000 sec
uftrace: stop writer thread 3
uftrace: stop writer thread 7
uftrace: stop writer thread 1
[Thread 0x7fffeeffd700 (LWP 1282151) exited]
[Thread 0x7fffecff9700 (LWP 1282155) exited]
uftrace: stop writer thread 4
[Thread 0x7fffeffff700 (LWP 1282149) exited]
uftrace: stop writer thread 6
[Thread 0x7fffee7fc700 (LWP 1282152) exited]
uftrace: stop writer thread 5
[Thread 0x7fffed7fa700 (LWP 1282154) exited]
uftrace: stop writer thread 0
[Thread 0x7fffedffb700 (LWP 1282153) exited]
[Thread 0x7ffff4e20700 (LWP 1282148) exited]
uftrace: reading uftrace.data/task.txt file
session: new session: pid = 1282147, session = dfb32f90c3d6f87b
session: task session: tid = 1282147, session = dfb32f90c3d6f87b
session: new task: tid = 1282147 (app), session = dfb32f90c3d6f87b
session: task session: tid = 1316324, session = dfb32f90c3d6f87b
session: new task: tid = 1316324 (app), session = dfb32f90c3d6f87b
session: task session: tid = 1316325, session = dfb32f90c3d6f87b
session: new task: tid = 1316325 (app), session = dfb32f90c3d6f87b
session: task session: tid = 1316345, session = dfb32f90c3d6f87b
session: new task: tid = 1316345 (app), session = dfb32f90c3d6f87b
session: task session: tid = 1316346, session = dfb32f90c3d6f87b
session: new task: tid = 1316346 (app), session = dfb32f90c3d6f87b
session: task session: tid = 1316484, session = dfb32f90c3d6f87b
session: new task: tid = 1316484 (app), session = dfb32f90c3d6f87b
session: task session: tid = 1316485, session = dfb32f90c3d6f87b
session: new task: tid = 1316485 (app), session = dfb32f90c3d6f87b
session: task session: tid = 1316547, session = dfb32f90c3d6f87b
session: new task: tid = 1316547 (app), session = dfb32f90c3d6f87b
session: task session: tid = 1316548, session = dfb32f90c3d6f87b
session: new task: tid = 1316548 (app), session = dfb32f90c3d6f87b
session: task session: tid = 1316487, session = dfb32f90c3d6f87b
session: new task: tid = 1316487 (app), session = dfb32f90c3d6f87b
session: task session: tid = 1316626, session = dfb32f90c3d6f87b
session: new task: tid = 1316626 (app), session = dfb32f90c3d6f87b
session: task session: tid = 1316625, session = dfb32f90c3d6f87b
session: new task: tid = 1316625 (app), session = dfb32f90c3d6f87b
session: task session: tid = 1316486, session = dfb32f90c3d6f87b
session: new task: tid = 1316486 (app), session = dfb32f90c3d6f87b
session: task session: tid = 1316862, session = dfb32f90c3d6f87b
session: new task: tid = 1316862 (app), session = dfb32f90c3d6f87b
uftrace: stop writer thread 2
[Thread 0x7fffef7fe700 (LWP 1282150) exited]
uftrace: flushing /uftrace-dfb32f90c3d6f87b-1282147-000
uftrace: flushing /uftrace-dfb32f90c3d6f87b-1316324-000
uftrace: flushing /uftrace-dfb32f90c3d6f87b-1316325-000
uftrace: flushing /uftrace-dfb32f90c3d6f87b-1316345-000
uftrace: flushing /uftrace-dfb32f90c3d6f87b-1316346-000
uftrace: flushing /uftrace-dfb32f90c3d6f87b-1316484-000
uftrace: flushing /uftrace-dfb32f90c3d6f87b-1316485-000
uftrace: flushing /uftrace-dfb32f90c3d6f87b-1316547-000
uftrace: flushing /uftrace-dfb32f90c3d6f87b-1316548-000
uftrace: flushing /uftrace-dfb32f90c3d6f87b-1316487-000
uftrace: flushing /uftrace-dfb32f90c3d6f87b-1316626-000
uftrace: flushing /uftrace-dfb32f90c3d6f87b-1316625-000
uftrace: flushing /uftrace-dfb32f90c3d6f87b-1316486-000
uftrace: flushing /uftrace-dfb32f90c3d6f87b-1316862-000
uftrace: unlink for session: dfb32f90c3d6f87b
uftrace: reading symbols for session dfb32f90c3d6f87b
symbol: found rela.dyn section with 24 entry.
[New Thread 0x7fffecff9700 (LWP 1319887)]
[Thread 0x7fffecff9700 (LWP 1319887) exited]
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
[New Thread 0x7fffecff9700 (LWP 1319899)]
[Thread 0x7fffecff9700 (LWP 1319899) exited]
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
[New Thread 0x7fffecff9700 (LWP 1319919)]
[Thread 0x7fffecff9700 (LWP 1319919) exited]
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
[New Thread 0x7fffecff9700 (LWP 1319963)]
[Thread 0x7fffecff9700 (LWP 1319963) exited]
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
[New Thread 0x7fffecff9700 (LWP 1319987)]
[Thread 0x7fffecff9700 (LWP 1319987) exited]
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
[New Thread 0x7fffecff9700 (LWP 1320003)]
[Thread 0x7fffecff9700 (LWP 1320003) exited]
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
[New Thread 0x7fffecff9700 (LWP 1320048)]
[Thread 0x7fffecff9700 (LWP 1320048) exited]
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: cannot find debug file: No DWARF information found
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: found rela.dyn section with 24 entry.
symbol: saving symbols to uftrace.data/libzstd.so.1.4.4.sym
symbol: saving symbols to uftrace.data/libz.so.1.2.11.sym
symbol: saving symbols to uftrace.data/libtraceevent.so.1.5.3.sym
symbol: saving symbols to uftrace.data/libstdc++.so.6.0.25.sym
symbol: saving symbols to uftrace.data/librt-2.28.so.sym
symbol: saving symbols to uftrace.data/libpthread-2.28.so.sym
symbol: saving symbols to uftrace.data/libonion_security.so.1.0.19.sym
symbol: saving symbols to uftrace.data/libm-2.28.so.sym
symbol: saving symbols to uftrace.data/liblzma.so.5.2.4.sym
symbol: saving symbols to uftrace.data/libgcc_s-8-20210514.so.1.sym
symbol: saving symbols to uftrace.data/libelf-0.190.so.sym
symbol: saving symbols to uftrace.data/libdw-0.190.so.sym
symbol: saving symbols to uftrace.data/libdl-2.28.so.sym
symbol: saving symbols to uftrace.data/libcapstone.so.4.sym
symbol: saving symbols to uftrace.data/libc-2.28.so.sym
symbol: saving symbols to uftrace.data/libbz2.so.1.0.6.sym
symbol: saving symbols to uftrace.data/libanl-2.28.so.sym
symbol: saving symbols to uftrace.data/ld-2.28.so.sym
symbol: saving symbols to uftrace.data/liblight_feature.so.sym
symbol: saving symbols to uftrace.data/libunistring.so.2.sym
symbol: saving symbols to uftrace.data/libssl.so.1.1.sym
symbol: saving symbols to uftrace.data/libssh.so.4.sym
symbol: saving symbols to uftrace.data/libselinux.so.1.sym
symbol: saving symbols to uftrace.data/libsasl2.so.3.sym
symbol: saving symbols to uftrace.data/libresolv.so.2.sym
symbol: saving symbols to uftrace.data/libpsl.so.5.sym
symbol: saving symbols to uftrace.data/libpcre2-8.so.0.sym
symbol: saving symbols to uftrace.data/libnss_files.so.2.sym
symbol: saving symbols to uftrace.data/libnss_dns.so.2.sym
symbol: saving symbols to uftrace.data/libnghttp2.so.14.sym
symbol: saving symbols to uftrace.data/libldap-2.4.so.2.sym
symbol: saving symbols to uftrace.data/liblber-2.4.so.2.sym
symbol: saving symbols to uftrace.data/libkrb5support.so.0.sym
symbol: saving symbols to uftrace.data/libkrb5.so.3.sym
symbol: saving symbols to uftrace.data/libkeyutils.so.1.sym
symbol: saving symbols to uftrace.data/libk5crypto.so.3.sym
symbol: saving symbols to uftrace.data/libidn2.so.0.sym
symbol: saving symbols to uftrace.data/libgssapi_krb5.so.2.sym
symbol: saving symbols to uftrace.data/libdebuginfod.so.1.sym
symbol: saving symbols to uftrace.data/libcurl.so.4.sym
symbol: saving symbols to uftrace.data/libcrypto.so.1.1.sym
symbol: saving symbols to uftrace.data/libcrypt.so.1.sym
symbol: saving symbols to uftrace.data/libcom_err.so.2.sym
symbol: saving symbols to uftrace.data/libbrotlidec.so.1.sym
symbol: saving symbols to uftrace.data/libbrotlicommon.so.1.sym
symbol: saving symbols to uftrace.data/app.sym
symbol: saving symbols to uftrace.data/libtss_sdk.so.sym
[Inferior 1 (process 1281382) exited with code 02]
Missing separate debuginfos, use: dnf debuginfo-install libgcc-8.5.0-18.tl3.x86_64 libidn2-2.2.0-1.tl3.x86_64 libnghttp2-1.33.0-3.tl3.1.x86_64 libpsl-0.20.2-6.tl3.x86_64 libselinux-2.9-8.tl3.x86_64 libssh-0.9.6-6.tl3.x86_64 libstdc++-8.5.0-18.tl3.x86_64 libtraceevent-1.5.3-1.tl3.x86_64 libxcrypt-4.1.1-6.tl3.x86_64 libzstd-1.4.4-1.tl3.x86_64 ncurses-libs-6.1-10.20180224.tl3.x86_64 openssl-libs-1.1.1k-9.tl3.3.x86_64 pcre2-10.32-2.tl3.x86_64 xz-libs-5.2.4-4.tl3.x86_64 zlib-1.2.11-21.tl3.x86_64
(gdb)
(gdb) bt
No stack.
(gdb)

Program terminated with signal SIGSEGV, Segmentation fault.

#0  x86_64_fallback_frame_state (context=0xba61a630, context=0xba61a630, fs=0xba61a720)
    at ./md-unwind-support.h:63
63        if (*(unsigned char *)(pc+0) == 0x48
[Current thread is 1 (Thread 0x7ffff7fd8c00 (LWP 1516324))]

(gdb)
(gdb)
(gdb) bt
#0  x86_64_fallback_frame_state (context=0xba61a630, context=0xba61a630, fs=0xba61a720)
    at ./md-unwind-support.h:63
#1  uw_frame_state_for (context=context@entry=0xba61a630, fs=fs@entry=0xba61a720)
    at ../../../libgcc/unwind-dw2.c:1265
#2  0x00007ffff6c40f2b in _Unwind_RaiseException (exc=exc@entry=0xc570d020)
    at ../../../libgcc/unwind.inc:104
#3  0x00007ffff6ede7eb in __cxxabiv1::__cxa_throw (obj=<optimized out>,
    tinfo=0x7ffff71cfee8 <typeinfo for std::invalid_argument>,
    dest=0x7ffff6ef40c0 <std::invalid_argument::~invalid_argument()>)
    at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90
#4  0x00007ffff7bb3dcf in mcount () at /root/uftrace/arch/x86_64/mcount.S:80
#5  0x0000000000000000 in ?? ()

@namhyung
Copy link
Owner

Looks like the stack memory was overwritten by something. Does your program reads return address from stack or calculate something from it? But it doesn't explain the -e case still..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@namhyung @m9rco @honggyukim