Enabling --debug-dump leads to infinite loop in simulation

[quasiquote1]

Hi,

I'm facing a weird situation where on an assertion failure the simulation hangs for a while until Sparta emits a message saying

terminate called after throwing an instance of 'sparta::SpartaException'
what(): Infinite loop was detected during simulation!

This problem goes away if we enable --debug-dump never on the commandline. Do you know what is going on?

I tried commenting out all of the code in our dumpDebugContent_() callbacks, and it still has the same infinite loop issue.

Thanks for any help on this

[klingaard]

I haven't seen that issue, but I have seen the backtracer taking an exorbitant amount of time on a simulation crash. Basically, the framework calls addr2line for the backtrace, but for some reason addr2line (later versions) are taking a really long time to provide a response.

When the simulator "sits there" in gdb, can you ctrl-c it and see where it is?

[quasiquote1]

Here's my backtrace when I hit ctrl-c

[klingaard]

Yep, it's the addr2line issue. In your build of sparta, add a && false to this line:

map/sparta/src/Backtrace.cpp

Line 271 in 64e119f

if(line_info){

and try again. You won't get a backtrace anymore, but you'll still get an error dump.

[quasiquote1]

Thanks, it's work for me.

[VictorYing]

Hi Knute, to solve these sorts of hangs, would you be open to modifying Sparta to not call addr2line to get line info in backtraces, or to introduce an option for Sparta to do so (i.e., enable SpartaException::backtrace() to pass line_info = false to BacktraceData::render())?

[klingaard]

Yes. This is actually causing me a headache! 😆

Adding support to the framework to disable a backtrace dump now. Just an FYI -- this only seems to be an issue with clang-based builds.

[VictorYing]

Thanks!

For whatever reason, we're hitting this with GCC 11-based builds in Ubuntu 22.04, but I don't know if it might be related to any compiler or linker options we chose in our project. The typical pattern I observe is that the child addr2line process actually exited but the parent process somehow remains stuck in the fread() loop.

[klingaard]

Actually, while implementing this, I just learned that the framework already supports it!

In your main function or wherever your sparta::app::CommandLineSimulator instance is created, add this line:

        // Disable backtrace
        cls.getSimulationConfiguration().debug_dump_options =
            sparta::app::SimulationConfiguration::PostRunDebugDumpOptions::DEBUG_DUMP_NOTHING;

Can you give this a try? I just did and it appears to do what we want. 😉

[VictorYing]

Yes, we already have been using this option on our end: this is --debug-dump never that Luke mentioned above. I meant to suggest it would be valuable to keep debug dumps (which, especially if there are dumpDebugContent_() override methods defined for various units, can give a lot of helpful info on the state of the simulation when an error is encountered) but just skip the use of addr2line in the backtrace. It seems if you just set line_info = false then you still get backtraces with function names, while avoiding the use of addr2line which just means giving up line numbers within those functions.

[klingaard]

Actually, the command line flag --debug-dump never is not the same as the override as I mention.

There are two things:

1. Turn off the debug dump file generation completely (--debug-dump never)
2. Turn off backtrace only (sparta::app::SimulationConfiguration::PostRunDebugDumpOptions::DEBUG_DUMP_NOTHING)

[VictorYing]

Ah, thanks for clarifying.

[klingaard]

The code I provided prevents the call to backtrace collection completely, while still giving you an error dump file.

[klingaard]

Can you let me know if that was a good solution for you?

[VictorYing]

I was checking back in on this thread and it looks like I failed to post a reply to your question: yes, --debug-dump-options asserts_only (a.k.a. PostRunDebugDumpOptions::DEBUG_DUMP_NOTHING) works. thank you for suggesting it.

My reason for checking back in on this thread today though is that I encountered the addr2line invocation hanging occurring upon a segfault which triggered sparta::app::crit_err_hdlr, which tries to get a backtrace. Of course I would prefer to get backtraces to work properly instead of disable them.

[timsnyder]

I probably should have filed this in another discussion but I didn't think it was going to be this long... 😞

I'm also looking at improving the backtrace support because the addr2line stuff has been busted for me for awhile. Not having a proper backtrace with line numbers is highly annoying and I'd like it to be fixed and be more robust than the piping to addr2line solution we came up with long ago.

I'm going to look at why we didn't just use Boost::stacktrace. At first glance https://www.boost.org/doc/libs/1_65_1/doc/html/stacktrace/getting_started.html#stacktrace.getting_started.exceptions_with_stacktrace looks like it will provide something that looks much better than what I'm getting now:

Backtrace:
================================================================================
  (1) simulator(+0x16f096d) [0x55d5a571896d]
      ??:0
  (2) simulator(+0x15c7d41) [0x55d5a55efd41]
      ??:0
  (3) simulator(+0x15c8008) [0x55d5a55f0008]
      ??:0
  (4) simulator(+0xa67223) [0x55d5a4a8f223]
      ??:0
  (5) simulator(+0xa5ba96) [0x55d5a4a83a96]
      ??:0
  (6) simulator(+0xa34b58) [0x55d5a4a5cb58]
      ??:0
  (7) simulator(+0x1183e90) [0x55d5a51abe90]
      ??:0
  (8) simulator(+0x3941b8) [0x55d5a43bc1b8]
      ??:0
  (9) simulator(+0x15cedc9) [0x55d5a55f6dc9]
      ??:0
 (10) simulator(+0x15e7ae4) [0x55d5a560fae4]
      ??:0
 (11) simulator(+0x15e7a5c) [0x55d5a560fa5c]
      ??:0
 (12) simulator(+0x15e7a8e) [0x55d5a560fa8e]
      ??:0
 (13) simulator(+0x15e6ded) [0x55d5a560eded]
      ??:0
 (14) simulator(+0x142fa3a) [0x55d5a5457a3a]
      ??:0
 (15) simulator(+0x142f76d) [0x55d5a545776d]
      ??:0
 (16) simulator(+0x28b6c7) [0x55d5a42b36c7]
      ??:0
 (17) simulator(+0x368916) [0x55d5a4390916]
      ??:0
 (18) simulator(+0x36a7bd) [0x55d5a43927bd]
      ??:0
 (19) /lib64/libc.so.6(__libc_start_main+0xf3) [0x14a285e3d493]
      ??:0
 (20) simulator(+0x1ea079) [0x55d5a4212079]
      ??:0

In my setup, I see that calling addr2line with the address in square brackets (like sparta does)

addr2line 0x55d5a571896d -e simulator

yields the super helpful ??:0 result, but if I pass it the other address

addr2line +0x16f096d -e simulator

it gives me an actual file and line number that make sense. I haven't figured out why this is happening yet.

I have been seeing simulations just hang when they hit exceptions but haven't had time to dig into why. I'll try just avoiding the backtrace to see if that signature falls out of my regressions.

@klingaard let me know if you have thoughts about adopting Boost::stacktrace or any other 3rd party, portable backtrace renderer.

[VictorYing]

Regarding the two addresses, my guess is that 0x55d5a4a83a96 is the virtual address of the instruction in the process's address space, and the +0x28b6c7 is the offset of the instruction from the beginning of the chunk of memory where the binary was loaded into that address space (which may change from run to run due to ASLR). While a process is running, you could confirm my hypothesis by examining /proc/[pid]/maps to see where various binaries have been loaded into the virtual address space. Because addr2line only examines the binary and not the running process, it would make sense that it needs the latter address.

[klingaard]

That makes a lot of sense. And @timsnyder I'd love to move to something other than addr2line. addr2line is mostly "hanging" on clang built versions of the simulator. If you build with gcc it goes pretty quickly.

@timsnyder do you have an implementation using the Boost::stacktrace?

[timsnyder]

No, I got busy with travel and dealing with other shenanigan's this summer and didn't implement anything.