Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Destroy the ComputationClient when the program exits #5750

Merged
merged 2 commits into from
Oct 31, 2023

Conversation

will-cromar
Copy link
Collaborator

  • Remove ComputationClient::PrepareToExit(). Any non-trivial destruction can be added to the destructor. @jonb377 found that in practice, any attempt to use PrepareToExit resulted in a segfault, likely due to the sequencing of Python's atexit with other teardown.
  • The static variable initialization inside of GetComputationClient is thread safe after C++11: https://codereview.stackexchange.com/a/197494
  • Confirmed that ComputationClient is destroyed by adding a print to the destructor.

@will-cromar will-cromar marked this pull request as ready for review October 30, 2023 23:52
Copy link
Collaborator

@jonb377 jonb377 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks Will!

@jonb377
Copy link
Collaborator

jonb377 commented Oct 30, 2023

Let's wait for the GPU tests to finish before merging, the two left running are what segfaulted for me.

std::call_once(g_computation_client_once,
[&]() { g_computation_client = std::move(CreateClient()); });
return g_computation_client.load();
static auto client = std::unique_ptr<ComputationClient>(CreateClient());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the static variable is guaranteed to be initialized once, do you still need the function GetComputationClientIfInitialized and the variable std::atomic<bool> g_computation_client_initialized(false);?

Copy link
Collaborator Author

@will-cromar will-cromar Oct 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The client is guaranteed to be initialized once on the first time we call into GetComputationClient. There are a couple of calls that want to tell if client has been initialized (ie comparing GetComputationClientIfInitialized to nullptr), but there's not a way I can see to tell if client has been initialized. As soon as you call GetComputationClient, it gets created. That's why I added the extra flag outside of GetComputationClient.

On a second look, I think it would actually be clearer to eliminate ComputationClient* GetComputationClientIfInitialized and replace it with bool ComputationClientIsInitialzed().

@will-cromar will-cromar merged commit 83778f0 into master Oct 31, 2023
17 checks passed
mbzomowski pushed a commit to mbzomowski-test-org/xla that referenced this pull request Nov 16, 2023
* Destroy the ComputationClient when the program exits

* Fix extra error when PJRT_DEVICE is not set
ManfeiBai pushed a commit that referenced this pull request Nov 29, 2023
* Destroy the ComputationClient when the program exits

* Fix extra error when PJRT_DEVICE is not set
ManfeiBai pushed a commit that referenced this pull request Nov 29, 2023
* Destroy the ComputationClient when the program exits

* Fix extra error when PJRT_DEVICE is not set
chunnienc pushed a commit to chunnienc/xla that referenced this pull request Dec 14, 2023
* Destroy the ComputationClient when the program exits

* Fix extra error when PJRT_DEVICE is not set
golechwierowicz pushed a commit that referenced this pull request Jan 12, 2024
* Destroy the ComputationClient when the program exits

* Fix extra error when PJRT_DEVICE is not set
bhavya01 pushed a commit that referenced this pull request Apr 22, 2024
* Destroy the ComputationClient when the program exits

* Fix extra error when PJRT_DEVICE is not set
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants