Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log hpx threads on forced shutdown #6329

Closed
Pansysk75 opened this issue Aug 24, 2023 · 2 comments
Closed

Log hpx threads on forced shutdown #6329

Pansysk75 opened this issue Aug 24, 2023 · 2 comments

Comments

@Pansysk75
Copy link
Member

Pansysk75 commented Aug 24, 2023

When an HPX process hangs (usually because some hpx thread hasn't terminated properly), the process is often killed with a termination signal.
In such cases, logging information about any alive hpx threads would really help me in debugging, especially when an issue is hard to reproduce locally.

@hkaiser I can certainly add this if you also think it makes sense, only could you please point me to where exit signals are handled? Thanks!

@hkaiser
Copy link
Member

hkaiser commented Aug 24, 2023

@Pansysk75 Thanks! The termination handlers are here:

namespace hpx {
///////////////////////////////////////////////////////////////////////////
void handle_termination(char const* reason)
{
if (hpx::threads::coroutines::attach_debugger_on_sigv)
{
util::attach_debugger();
}
if (hpx::threads::coroutines::diagnostics_on_terminate)
{
int const verbosity = hpx::threads::coroutines::exception_verbosity;
if (verbosity >= 2)
{
std::cerr << full_build_string() << "\n";
}
#if defined(HPX_HAVE_STACKTRACES)
if (verbosity >= 1)
{
std::size_t const trace_depth =
util::from_string<std::size_t>(get_config_entry(
"hpx.trace_depth", HPX_HAVE_THREAD_BACKTRACE_DEPTH));
std::cerr << "{stack-trace}: " << hpx::util::trace(trace_depth)
<< "\n";
}
#endif
std::cerr << "{what}: " << (reason ? reason : "Unknown reason")
<< "\n";
}
}
HPX_CORE_EXPORT BOOL WINAPI termination_handler(DWORD ctrl_type)
{
switch (ctrl_type)
{
case CTRL_C_EVENT:
handle_termination("Ctrl-C");
break;
case CTRL_BREAK_EVENT:
handle_termination("Ctrl-Break");
break;
case CTRL_CLOSE_EVENT:
handle_termination("Ctrl-Close");
break;
case CTRL_LOGOFF_EVENT:
handle_termination("Logoff");
break;
case CTRL_SHUTDOWN_EVENT:
handle_termination("Shutdown");
break;
default:
break;
}
return FALSE;
}
} // namespace hpx
#else
#include <signal.h>
#include <stdlib.h>
#include <string.h>
namespace hpx {
///////////////////////////////////////////////////////////////////////////
[[noreturn]] HPX_CORE_EXPORT void termination_handler(int signum)
{
if (signum != SIGINT &&
hpx::threads::coroutines::attach_debugger_on_sigv)
{
util::attach_debugger();
}
if (hpx::threads::coroutines::diagnostics_on_terminate)
{
int const verbosity = hpx::threads::coroutines::exception_verbosity;
char* reason = strsignal(signum);
if (verbosity >= 2)
{
std::cerr << full_build_string() << "\n";
}
#if defined(HPX_HAVE_STACKTRACES)
if (verbosity >= 1)
{
std::size_t const trace_depth =
util::from_string<std::size_t>(get_config_entry(
"hpx.trace_depth", HPX_HAVE_THREAD_BACKTRACE_DEPTH));
std::cerr << "{stack-trace}: " << hpx::util::trace(trace_depth)
<< "\n";
}
#endif
std::cerr << "{what}: " << (reason ? reason : "Unknown reason")
<< "\n";
}
std::abort();
}
} // namespace hpx
#endif

bors bot pushed a commit that referenced this issue Sep 10, 2023
6330: Adding basic logging to collective operations r=hkaiser a=hkaiser

- flyby: fixing JeMalloc integration for dependent projects


6340: Log alive hpx threads on exit r=hkaiser a=Pansysk75

Adds logging when an HPX program is terminated suddenly (ie by a termination signal, addresses issue #6329).
I'll leave it as draft until I make up my mind on whether I think it is actually a good idea




Co-authored-by: Hartmut Kaiser <[email protected]>
Co-authored-by: Panos Syskakis <[email protected]>
@Pansysk75
Copy link
Member Author

Pansysk75 commented Sep 10, 2023

Issue resolved by PR #6340

@hkaiser hkaiser added this to the 1.10.0 milestone Jan 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants