HPCC-32789 Fix potential race deadlock in Thor splitter #19235
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Jira Issue: https://hpccsystems.atlassian.net//browse/HPCC-32789 Jirabot Action Result: |
jakesmith
force-pushed
the
HPCC-32789-newsplitter-deadlock
branch
from
d77e68b to
cbe2b51
Compare
jakesmith
changed the title
jakesmith
force-pushed
the
HPCC-32789-newsplitter-deadlock
branch
from
cbe2b51 to
0b2b661
Compare
jakesmith
commented
Oct 25, 2024
thorlcr/thorutil/thbuf.cpp
Outdated
| @@ -1464,6 +1469,15 @@ class CSharedWriteAheadBase : public CSimpleInterface, implements ISharedSmartBu | |||
| } | |||
| rowsRead++; | |||
| const void *retrow = rowSet->getRow(row++); | |||
| if (lastWasNull) | |||
There was a problem hiding this comment.
@ghalliday - views on leaving this sanity checking in?
ghalliday
requested changes
Oct 25, 2024
thorlcr/thorutil/thbuf.cpp
Outdated
| @@ -1395,6 +1399,7 @@ class CSharedWriteAheadBase : public CSimpleInterface, implements ISharedSmartBu | |||
| Owned<CRowSet> outputOwnedRows; | |||
| CRowSet *rowSet; | |||
| unsigned row, rowsInRowSet; | |||
| bool lastWasNull=false; // for sanity check only (there should never be two consequetive nulls) | |||
There was a problem hiding this comment.
This should be set back to false in reset() - otherwise a splitter in a child query that has no records will probably trigger the failure. Alternatively we could remove this code as discussed.
If the splitter reading arms (COutput) were reading from the same page (CRowSet chunk) as the write ahead was writing to, then the write ahead could expand that row set and cause the reader to read unexpected row data (e.g. null rows). This caused the splitter arm to premuturely finish, leaving the splitter unbalanced and stalling as the writeahead blocked soon afterwards since the finished arm was too far behind. The bug was that the row set should never expand. It is pre-sized to avoid that. However, the condition of 'fullness' was incorrect, relying only on a dynamic calculation of total row size. The fix is to also check that the number of rows does not exceed the capacity. NB: The bug predates the new splitter code, however the new splitter implementation also changed the way the splitter arms interacted with writeahead. Previously the arm would call writeahead once it hit max explicitly, rather than blocking in the underlying ISharedSmartBuffer implementation. But it would still be possible I think to hit this bug (albeit less likely). Signed-off-by: Jake Smith <[email protected]>
jakesmith
force-pushed
the
HPCC-32789-newsplitter-deadlock
branch
from
0b2b661 to
35a34fd
Compare
ghalliday
approved these changes
Oct 25, 2024
|
Jirabot Action Result: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If the splitter reading arms (COutput) were reading from the same page (CRowSet chunk) as the write ahead was writing to, then the write ahead could expand that row set and cause the reader to read unexpected row data (e.g. null rows).
This caused the splitter arm to premuturely finish, leaving the splitter unbalanced and stalling as the writeahead blocked soon afterwards since the finished arm was too far behind.
The bug was that the row set should never expand. It is pre-sized to avoid that. However, the condition of 'fullness' was incorrect, relying only on a dynamic calculation of total row size. The fix is to also check that the number of rows does not exceed the capacity.
NB: The bug predates the new splitter code, however the new splitter implementation also changed the way the splitter arms interacted with writeahead. Previously the arm would call writeahead once it hit max explicitly, rather than blocking in the underlying ISharedSmartBuffer implementation.
But it would still be possible I think to hit this bug (albeit less likely).
Type of change:
Checklist:
Smoketest:
Testing: