Allocated pointers not fully freed by ~SpartaSharedPointerAllocator() #364

yokushida · 2022-09-19T06:47:46Z

yokushida
Sep 19, 2022

Hi,

I added trace parser to the feed instruction from trace file to sparta_core_example's fetch stage. The ex_inst allocation in Fetch::fetchInstruction_() and inst vector emplace_back and sending retain the original version while assignments of some additional fields of ExampleInst are added. Running with some program trace with -r (--run-time) larger then the program's length (e.g. -r 10000000000 for a trace with around 1,400,000 insts), I bump into the following issue:
...
Running...
Retired <top.cpu.core0.rob.stats.total_number_retired val:1000000 normal vis:1000000000000> instructions
Received Signal 11 (Segmentation fault), address is (nil) from (nil)
( 1) ./sparta_core_example(
map/sparta/src/Backtrace.cpp:322
( 2) /lib64/libpthread.so.0(+0xf630) [0x7ffff68ad630]
??:0
( 3) /lib64/libc.so.6(fread+0x20) [0x7ffff495f180]
??:0
< ... backtrace to Fetch.cpp and the way up to main.cpp ... >
Exiting
Some ManagedTriggers were not destroyed before static uninitialization. This is probably a mistake and a memory leak
WARNING: Seems that not all of the blocks made it back.
'sparta::SpartaSharedPointerAllocator::~SpartaSharedPointerAllocator() [with PointerT = core_example::ExampleInst]'
Allocated: 40
Returned: 0

Printing allocated_ and free_idx_ in allocate_() of SpartaSharedPointerAllocator() shows that before allocated__ is incremented to 40, there are only 4 times where free_idx_ is incremented by 1 and consumed at that allocation attempt, with the sequence of printing like:
Before allocate_(): allocated_ = 0, free_idx_=0; After allocate(): allocated_ = 0, free_idx_=0
Before allocate_(): allocated_ = 0, free_idx_=0; After allocate(): allocated_ = 1, free_idx_=0
Before allocate_(): allocated_ = 0, free_idx_=0; After allocate(): allocated_ = 2, free_idx_=0
...
Before allocate_(): allocated_ = 28, free_idx_=1; After allocate(): allocated_ = 28, free_idx_=0
...
Before allocate_(): allocated_ = 31, free_idx_=1; After allocate(): allocated_ = 31, free_idx_=0
...
Before allocate_(): allocated_ = 34, free_idx_=1; After allocate(): allocated_ = 34, free_idx_=0
...
Before allocate_(): allocated_ = 37, free_idx_=1; After allocate(): allocated_ = 37, free_idx_=0
...
Before allocate_(): allocated_ = 40, free_idx_=1; After allocate(): allocated_ = 40, free_idx_=0
Before allocate_(): allocated_ = 40, free_idx_=1; After allocate(): allocated_ = 40, free_idx_=0
Before allocate_(): allocated_ = 40, free_idx_=1; After allocate(): allocated_ = 40, free_idx_=0
=>the allocated_ is not incremented anymore to the end of sim.

Another observation is if -r is specified with a number less than the trace inst count (or say #cycles), meaning the sim run is terminated before the trace feeding ends, this error will not be encountered.

For sparta::allocate_sparta_shared_pointer allocation in fetch stage, everything remains as the original version. I also track the constructor's and destructor's unlink_() calls, but still not finding a way to identify the root cause more efficiently yet. Is there some suggestion on debugging this issue?

Thanks!

Answered by yokushida

Sep 22, 2022

The root cause issue is not related to the allocator in the error message of the original question but is due to my bug in trace ending parsing. Marked as answered.

View full answer

ghost · 2022-09-19T15:37:13Z

ghost
Sep 19, 2022

So I'm assuming you know the cause of the seg fault; that's not your question. 😄 But rather, you're wondering why the error message from the Allocator?

First of all, I'm pretty sure the Allocator isn't the cause of the seg fault, but without a full stack, I can't be certain. Are you observing the seg fault in a destruction of one of your data types?

During a seg fault, there's the possibility that destruction of certain objects (like the seg faulting class) are not destroyed and any objects allocated will not be returned to the Allocator prior to the Allocator's destruction.

Secondly, the variables allocated_ and free_idx_ represent two "pointers" in the Allocator (perhaps I could have named them better) that are "chasing" each other.

allocated_ represents the number of allocations requested -- for example the number of ExampleInsts allocated over time.
free_idx_ represents the number of returned objects back to the Allocator

So if free_idx_ == allocated_ this means all pointers have been returned to the Allocator -- or all SpartaSharedPointers have been destroyed (and returned their memory back to the Allocator).

If the simulator is properly torn down by the framework (on a successful run) and the modeler did not leave resources on the heap that use the Allocator, then you will not see any error messages. The main reason this works: the Allocator is always destroyed last in simulation. This gives all resources an opportunity to release pointers held in buffers, queues, vectors, lists, etc back to the Allocator. Anytime this order is disturbed (like in a seg fault situation) the Allocator will complain if not all objects are return or a double-free situation might occur (seg fault) since the Allocator has deleted ALL backend memory AND THEN an allocated object from that Allocator is deleted.

Is there some suggestion on debugging this issue?

I think in your case, you're either dealing with a nullptr situation or a double free. For the nullptr, add an sparta_assert before the use of the pointer returned from your trace parser/injector.

Another idea for the double-free situation -- augment the SpartaSharedPointerAllocator adding a static bool is_allocator_deleted_ = false. Set that true in SpartaSharedPointerAllocator::~SpartaSharedPointerAllocator() and check for it being false in releaseObject_:

        void releaseObject_(void * block) const noexcept override {
            sparta_assert(false == is_allocator_deleted_, "Allocator was already destroyed and an object is being returned");
            static_cast<MemBlock *>(block)->ref_count->p->~PointerT();
        }

I might add this to the allocator, but I fear cutting into its performance.

3 replies

yokushida Sep 21, 2022
Author

Thanks for the guidelines! I've tried the suggestion and more cross checking, with the following reply and observations:

Seg fault I observe should be related to data type allocated by allocator. The only thing allocated by allocator is the "ExampleInstPtr ex_inst;" as the original (master branch) version. For trace fields storage, a struct type is added, but not allocated by the allocator. The allocated ex_inst only capture some fields from the struct entries into its member fields. The full back trace are listed below:

Running...
Retired <top.cpu.core0.rob.stats.total_number_retired val:1000000 normal vis:1000000000000> instructions
Received Signal 11 (Segmentation fault), address is (nil) from (nil)
( 1) ./sparta_core_example(
     /dev_root/cluster01/home/yokushida/map/sparta/src/Backtrace.cpp:322
( 2) /lib64/libpthread.so.0(+0xf630) [0x7ffff68ad630]
     ??:0
( 3) /lib64/libc.so.6(fread+0x20) [0x7ffff495f180]
     ??:0
( 4) ./sparta_core_example(
     /dev_root/cluster01/home/yokushida/map/sparta/example/CoreModel/src/Fetch.cpp:289
( 5) ./sparta_core_example(
     /dev_root/cluster01/home/yokushida/map/sparta/example/CoreModel/src/Fetch.cpp:91
( 6) ./sparta_core_example(
     /dev_root/cluster01/home/yokushida/map/sparta/src/Scheduler.cpp:531
( 7) ./sparta_core_example(
     /dev_root/cluster01/home/yokushida/map/sparta/src/Simulation.cpp:1060
( 8) ./sparta_core_example(
     /dev_root/Linux/RHEL6/gcc-7.4.0/include/c++/7.4.0/bits/unique_ptr.h:147
( 9) ./sparta_core_example(
     /dev_root/cluster01/home/yokushida/map/sparta/src/CommandLineSimulator.cpp:2247
(10) ./sparta_core_example(
     /dev_root/cluster01/home/yokushida/map/sparta/example/CoreModel/src/main.cpp:127
(11) /lib64/libc.so.6(__libc_start_main+0xf5) [0x7ffff4912555]
     ??:0
(12) ./sparta_core_example(
     ??:?
Exiting
Some ManagedTriggers were not destroyed before static uninitialization. This is probably a mistake and a memory leak
WARNING: Seems that not all of the blocks made it back.  
'sparta::SpartaSharedPointerAllocator<PointerT>::~SpartaSharedPointerAllocator() [with PointerT = core_example::ExampleInst]'
Allocated: 40
Returned: 0

Along with the allocated_ and free_idx_ value printing in allocate_() function listed previously, I think the seg fault should be of the "ExampleInstPtr ex_inst;" and ex_inst s being sended(moved) across the stages. I might go for gdb/debugging if further needed. (the root cause may still be what I added).

A question here might be: Will the things not allocated by the allocator trigger the seg fault shown from the message above ?

Thanks for the suggestions! I added the sparta_assert into releaseObject_(), and assertion holds always, there's no such case that "Allocator was already destroyed and an object is being returned!".

I did a cross check with the original master branch version, with the following observations:

Original version | --run-time=1000000 | allocate ExampleInst w/ dummy_opcodes[rand() % dummy_opcodes.size()] in Fetch::fetchInstruction_ :
=> No seg fault triggered - with allocated_ and free_idx_ printing in allocate_() of sparta/sparta/utils/SpartaSharedPointerAllocator.hpp:

<!!! Only displaying sim ending lines!!!>
...
Before allocate_()'s if: allocated_: 66, free_idx_: 14; After allocate_()'s if: allocated_: 66, free_idx_: 13
Before allocate_()'s if: allocated_: 66, free_idx_: 13; After allocate_()'s if: allocated_: 66, free_idx_: 12
Before allocate_()'s if: allocated_: 66, free_idx_: 13; After allocate_()'s if: allocated_: 66, free_idx_: 12
Before allocate_()'s if: allocated_: 66, free_idx_: 13; After allocate_()'s if: allocated_: 66, free_idx_: 12
Before allocate_()'s if: allocated_: 66, free_idx_: 12; After allocate_()'s if: allocated_: 66, free_idx_: 11
Running Complete
<with ending metrics, summary infos printed> ...

Original version | --run-time=1000000 | allocate ExampleInst w/ dummy_opcodes[1] in Fetch::fetchInstruction_ :
=> No seg fault triggered - with allocated_ and free_idx_ printing in allocate_():

Before allocate_()'s if: allocated_: 0, free_idx_: 0; After allocate_()'s if: allocated_: 1, free_idx_: 0
Before allocate_()'s if: allocated_: 1, free_idx_: 0; After allocate_()'s if: allocated_: 2, free_idx_: 0
Before allocate_()'s if: allocated_: 2, free_idx_: 0; After allocate_()'s if: allocated_: 3, free_idx_: 0
...
< free_idx_ incremented to 1 and consumed when allocated_ = 28, 31, 34 >
...
Before allocate_()'s if: allocated_: 36, free_idx_: 0; After allocate_()'s if: allocated_: 37, free_idx_: 0
Before allocate_()'s if: allocated_: 37, free_idx_: 1; After allocate_()'s if: allocated_: 37, free_idx_: 0
Before allocate_()'s if: allocated_: 37, free_idx_: 0; After allocate_()'s if: allocated_: 38, free_idx_: 0
Before allocate_()'s if: allocated_: 38, free_idx_: 0; After allocate_()'s if: allocated_: 39, free_idx_: 0
Before allocate_()'s if: allocated_: 39, free_idx_: 0; After allocate_()'s if: allocated_: 40, free_idx_: 0
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
...
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
Running Complete
<with ending metrics, summary infos printed> ...

My version | --run-time=10000000000 (> input trace's #inst_cycle, to let program's trace fully execute) | allocate ExampleInst w/ dummy_opcodes[rand() % dummy_opcodes.size()] in Fetch::fetchInstruction_ :
=> Seg fault triggered - with allocated_ and free_idx_ printing in allocate_():

<!!! Only displaying sim ending lines!!!>
Before allocate_()'s if: allocated_: 66, free_idx_: 18; After allocate_()'s if: allocated_: 66, free_idx_: 17
Before allocate_()'s if: allocated_: 66, free_idx_: 17; After allocate_()'s if: allocated_: 66, free_idx_: 16
Before allocate_()'s if: allocated_: 66, free_idx_: 18; After allocate_()'s if: allocated_: 66, free_idx_: 17
Before allocate_()'s if: allocated_: 66, free_idx_: 18; After allocate_()'s if: allocated_: 66, free_idx_: 17
Before allocate_()'s if: allocated_: 66, free_idx_: 17; After allocate_()'s if: allocated_: 66, free_idx_: 16
Received Signal 11 (Segmentation fault), address is (nil) from (nil)
( 1) ./sparta_core_example(
...
<with ending metrics, summary infos printed> ...

My version | --run-time=10000000000 (> input trace's #inst_cycle, to let program's trace fully execute)| allocate ExampleInst w/ dummy_opcodes[1] in Fetch::fetchInstruction_ :
=> Seg fault triggered - with allocated_ and free_idx_ printing in allocate_():

<!!! Only displaying sim ending lines!!!>
Before allocate_()'s if: allocated_: 0, free_idx_: 0; After allocate_()'s if: allocated_: 1, free_idx_: 0
Before allocate_()'s if: allocated_: 1, free_idx_: 0; After allocate_()'s if: allocated_: 2, free_idx_: 0
Before allocate_()'s if: allocated_: 2, free_idx_: 0; After allocate_()'s if: allocated_: 3, free_idx_: 0
...
< free_idx_ incremented to 1 and consumed when allocated_ = 28, 31, 34 >
...
Before allocate_()'s if: allocated_: 36, free_idx_: 0; After allocate_()'s if: allocated_: 37, free_idx_: 0
Before allocate_()'s if: allocated_: 37, free_idx_: 1; After allocate_()'s if: allocated_: 37, free_idx_: 0
Before allocate_()'s if: allocated_: 37, free_idx_: 0; After allocate_()'s if: allocated_: 38, free_idx_: 0
Before allocate_()'s if: allocated_: 38, free_idx_: 0; After allocate_()'s if: allocated_: 39, free_idx_: 0
Before allocate_()'s if: allocated_: 39, free_idx_: 0; After allocate_()'s if: allocated_: 40, free_idx_: 0
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
...
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
Received Signal 11 (Segmentation fault), address is (nil) from (nil)
( 1) ./sparta_core_example(
...
<with ending metrics, summary infos printed> ...

My version | --run-time=100000 (< input trace's #inst_cycle, sim terminated(capped) before program's trace inst seq finishes) | allocate ExampleInst w/ dummy_opcodes[1] in Fetch::fetchInstruction_ :
=> No seg fault triggered - with allocated_ and free_idx_ printing in allocate_():

<!!! Only displaying sim ending lines!!!>
...
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
Before allocate_()'s if: allocated_: 40, free_idx_: 1; After allocate_()'s if: allocated_: 40, free_idx_: 0
Running Complete
  Simulation Performance      : ...
...
<with ending metrics, summary infos printed> ...

For instruction feeding (random gen) of the original master branch, the simulation's end limit is triggered by --run-time (-r), while for my version, simulation could be terminated as the original version by setting -r less than the trace's #inst, or let the trace fully executed in MAP with -r set > #inst_cycle. of trace file.

As shown in the case 1 and 2 for Original version, at simulation ending, the allocated_ and free_idx_'s value seem to indicate "Seems that not all of the blocks made it back", but not triggering seg fault. For case 5 my version, I capped the simulation before trace's insts are fully exercised, and no seg fault either for this case. So I am wondering if the original version has this underlying seg fault issue as well, but due to "capped", the seg fault is not triggered. Another thing to be identified is why "capped" would not trigger seg fault for my version (and may also applies to the original version) ?
Or maybe the seg fault has nothing to do with the allocator operations, but caused by other self-added memory leakage ? If only I'm hitting this, it might still be leakage in my dev.

It would be helpful for developers to provide some design guidance on the free_idx_, releaseBlock_(), unlink_() ... calling chain and the multiple constructor versions of SpartaSharedPointer, to help user debug issues around them.

Thanks! Appreciate your assistance!

ghost Sep 21, 2022

I think I understand your use case. Let me reiterate it and see if you agree:

You've modified Fetch to pull instructions from a custom trace
If the runtime is greater than the number of instructions fetched, you'll experience a seg fault
If the runtime is less than the trace size, no seg fault

Curious as to what happens if you do not supply the --run-time option. I guess it's the second bullet above.

I have a hunch on what might be happening and I can push a fix to the core example to prevent this from happening. In the core example, the Allocator is a singleton defined here and it's random on when it will be destructed with respect to OTHER singletons in simulation. Is your trace injector a singleton as well or defined in Fetch? My fix is to make the Allocators an actual TreeNode in the hierarchy that is allocated first and deleted last no matter what's going on simulation.

I've augmented my version of the core example with an "early out" in fetch to stop generating more instructions (simulating an end-of-trace scenario). I can't get it to seg fault like what you're seeing. Can you share a simple diff of what you did in Fetch?

It would be helpful for developers to provide some design guidance on the free_idx_, releaseBlock_(), unlink_() ... calling chain and the multiple constructor versions of SpartaSharedPointer, to help user debug issues around them.

Sure, I can add more information.

yokushida Sep 22, 2022
Author

I've fixed the bug, and the bug is in the code I added. The root cause is due to not processing safely during trace ending.

Yes, you understand my case correctly. When no -r provided, seg fault is hit (now it's clear that I am the blame ><).

The printed message after I fix the bug is below, and it shows allocated_ and free_idx_ are matching towards equal at the sim ending:

Before allocate_()'s if: allocated_: 66, free_idx_: 17; After allocate_()'s if: allocated_: 66, free_idx_: 16
Before allocate_()'s if: allocated_: 66, free_idx_: 16; After allocate_()'s if: allocated_: 66, free_idx_: 15
Before allocate_()'s if: allocated_: 66, free_idx_: 15; After allocate_()'s if: allocated_: 66, free_idx_: 14
Before allocate_()'s if: allocated_: 66, free_idx_: 14; After allocate_()'s if: allocated_: 66, free_idx_: 13
Before allocate_()'s if: allocated_: 66, free_idx_: 13; After allocate_()'s if: allocated_: 66, free_idx_: 12
[fetch] In fetch for(upper) loop: getAllTraceInstSize() is 0
getAllTraceInstSize() is 0 - No instr to fetch
getAllTraceInstSize() is 0 - No instr to fetch
getAllTraceInstSize() is 0 - No instr to fetch
getAllTraceInstSize() is 0 - No instr to fetch
getAllTraceInstSize() is 0 - No instr to fetch
getAllTraceInstSize() is 0 - No instr to fetch
Before allocate_()'s if: allocated_: 10, free_idx_: 6; After allocate_()'s if: allocated_: 10, free_idx_: 5
Before allocate_()'s if: allocated_: 8, free_idx_: 8; After allocate_()'s if: allocated_: 8, free_idx_: 7
getAllTraceInstSize() is 0 - No instr to fetch
Running Complete
  Simulation Performance ...
...

Thanks again for your assistance and effort!

yokushida · 2022-09-22T07:37:29Z

yokushida
Sep 22, 2022
Author

The root cause issue is not related to the allocator in the error message of the original question but is due to my bug in trace ending parsing. Marked as answered.

1 reply

klingaard Sep 23, 2022
Maintainer

Good to hear!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allocated pointers not fully freed by ~SpartaSharedPointerAllocator() #364

{{title}}

Replies: 2 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Allocated pointers not fully freed by ~SpartaSharedPointerAllocator() #364

yokushida Sep 19, 2022

Replies: 2 comments · 4 replies

ghost Sep 19, 2022

yokushida Sep 21, 2022 Author

ghost Sep 21, 2022

yokushida Sep 22, 2022 Author

yokushida Sep 22, 2022 Author

klingaard Sep 23, 2022 Maintainer

yokushida
Sep 19, 2022

Replies: 2 comments 4 replies

ghost
Sep 19, 2022

yokushida Sep 21, 2022
Author

yokushida Sep 22, 2022
Author

yokushida
Sep 22, 2022
Author

klingaard Sep 23, 2022
Maintainer