PS-9384: Sporadic crashes in Jenkins on start up due to race betweet … #5417
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…dict_stats_thread and cost model initialization https://perconadev.atlassian.net/browse/PS-9384 Problem: -------- Both debug and release version of server crash sporadically while running different tests in Jenkins with stacktraces referencing to Cost_model_server::init() being called from InnoDB's dict_stats_thread(). Analysis: --------- Investigation has shown that there is a race condition between code handling auto-updating of histograms from InnoDB background thread and the main thread performing server start-up. The code responsible for updating histogram, which was introduced by Upstream in 8.4.0, initializes LEX structure to perform its duties and tries to use global Optimizer cost model object as part of this. OTOH the main thread performing server start-up concurrently initializes and destroys this global object several times after this background thread has been started and sets it to the final working state much later in the process of start-up, before we start accepting user queries. Not surprisingly concurrent usage of this global object and its init/deinit cause crashes. In theory, the problem exists in Upstream but probably is normally invisible there, as to trigger it, some updates to tables are needed, so persistent stats recalculation and histogram update are requested. And in the Upstream this probably can normally happen only after user requests start being processed (by which time global cost model object has proper stable state). While in Percona Server, we have telemetry component enabled by default, and code which on first start up of server updates mysql.component table, which triggers stats/histogram update request. As result this race becomes visible. OTOH this specific scenario should only affect the first start of the server for installation, and not later restarts. But if there are other components which update tables during initialization/start up time the issue might become more prominent. Solution: --------- Delay processing of requests to update stats/histograms in background thread until server is fully operational (and thus global optimizer cost model is fully initialized and stable).
satya-bodapati
approved these changes
Sep 11, 2024
|
Jenkins results on different platforms for patch which only differs from the above commit in comments seems to confirm that problem is solved by it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…dict_stats_thread and cost model initialization
https://perconadev.atlassian.net/browse/PS-9384
Problem:
Both debug and release version of server crash sporadically while running different tests in Jenkins with stacktraces referencing to Cost_model_server::init() being called from InnoDB's dict_stats_thread().
Analysis:
Investigation has shown that there is a race condition between code handling auto-updating of histograms from InnoDB background thread and the main thread performing server start-up. The code responsible for updating histogram, which was introduced by Upstream in 8.4.0, initializes LEX structure to perform its duties and tries to use global Optimizer cost model object as part of this. OTOH the main thread performing server start-up concurrently initializes and destroys this global object several times after this background thread has been started and sets it to the final working state much later in the process of start-up, before we start accepting user queries. Not surprisingly concurrent usage of this global object and its init/deinit cause crashes.
In theory, the problem exists in Upstream but probably is normally invisible there, as to trigger it, some updates to tables are needed, so persistent stats recalculation and histogram update are requested. And in the Upstream this probably can normally happen only after user requests start being processed (by which time global cost model object has proper stable state).
While in Percona Server, we have telemetry component enabled by default, and code which on first start up of server updates mysql.component table, which triggers stats/histogram update request. As result this race becomes visible. OTOH this specific scenario should only affect the first start of the server for installation, and not later restarts. But if there are other components which update tables during initialization/start up time the issue might become more prominent.
Solution:
Delay processing of requests to update stats/histograms in background thread until server is fully operational (and thus global optimizer cost model is fully initialized and stable).