[BugFix] With a high streamload frequency and too many tablets involved, TabletCommitInfos will take up too much memory. #53329

zhangheihei · 2024-11-28T09:04:21Z

Why I'm doing:

Upgrading from 3.1.2 to 3.1.15, the memory grows too fast.
Version 3.1.2 memory didn't grow too fast because of 35115 refactoring here.
dump info:

What I'm doing:

Fixes #53330

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

fe/fe-core/src/main/java/com/starrocks/transaction/DatabaseTransactionMgr.java

mergify · 2024-11-29T03:05:04Z

rebbase

❌ Sorry but I didn't understand the command. Please consult the commands documentation 📚.

gengjun-git · 2024-11-29T03:05:21Z

@Mergifyio rebase

…ed, TabletCommitInfos will take up too much memory. Signed-off-by: edwinhzhang <[email protected]>

Signed-off-by: edwinhzhang <[email protected]>

mergify · 2024-11-29T03:06:03Z

rebase

✅ Branch has been successfully rebased

sonarcloud · 2024-11-29T03:11:58Z

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2024-11-29T07:50:36Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-11-29T07:51:22Z

[FE Incremental Coverage Report]

✅ pass : 8 / 8 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/transaction/DatabaseTransactionMgr.java	6	6	100.00%	[]
🔵	com/starrocks/transaction/TransactionState.java	2	2	100.00%	[]

github-actions · 2024-11-29T07:51:37Z

@Mergifyio backport branch-3.4

github-actions · 2024-11-29T07:51:39Z

@Mergifyio backport branch-3.3

github-actions · 2024-11-29T07:51:40Z

@Mergifyio backport branch-3.2

github-actions · 2024-11-29T07:51:42Z

@Mergifyio backport branch-3.1

mergify · 2024-11-29T07:51:46Z

backport branch-3.4

✅ Backports have been created

#53371 [BugFix] With a high streamload frequency and too many tablets involved, TabletCommitInfos will take up too much memory. (backport #53329) has been created for branch branch-3.4

mergify · 2024-11-29T07:51:47Z

backport branch-3.3

✅ Backports have been created

#53372 [BugFix] With a high streamload frequency and too many tablets involved, TabletCommitInfos will take up too much memory. (backport #53329) has been created for branch branch-3.3

mergify · 2024-11-29T07:51:48Z

backport branch-3.2

✅ Backports have been created

#53373 [BugFix] With a high streamload frequency and too many tablets involved, TabletCommitInfos will take up too much memory. (backport #53329) has been created for branch branch-3.2 but encountered conflicts

mergify · 2024-11-29T07:51:50Z

backport branch-3.1

✅ Backports have been created

#53374 [BugFix] With a high streamload frequency and too many tablets involved, TabletCommitInfos will take up too much memory. (backport #53329) has been created for branch branch-3.1 but encountered conflicts

…ed, TabletCommitInfos will take up too much memory. (#53329) Signed-off-by: edwinhzhang <[email protected]> (cherry picked from commit 86476df)

…ed, TabletCommitInfos will take up too much memory. (#53329) Signed-off-by: edwinhzhang <[email protected]> (cherry picked from commit 86476df) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/transaction/DatabaseTransactionMgr.java

github-actions · 2024-11-29T07:54:48Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

…ed, TabletCommitInfos will take up too much memory. (StarRocks#53329) Signed-off-by: edwinhzhang <[email protected]>

zhangheihei requested a review from a team as a code owner November 28, 2024 09:04

mergify bot assigned zhangheihei Nov 28, 2024

zhangheihei force-pushed the bug-fix-oom branch from d744b1b to 37e768e Compare November 28, 2024 09:18

github-actions bot added behavior_changed 3.4 3.3 3.2 3.1 labels Nov 28, 2024

trueeyu previously approved these changes Nov 28, 2024

View reviewed changes

luohaha reviewed Nov 28, 2024

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/transaction/DatabaseTransactionMgr.java Outdated Show resolved Hide resolved

zhangheihei dismissed trueeyu’s stale review via d35464f November 28, 2024 13:50

zhangheihei force-pushed the bug-fix-oom branch 2 times, most recently from 0b8627a to 07ca645 Compare November 28, 2024 14:11

luohaha approved these changes Nov 28, 2024

View reviewed changes

trueeyu approved these changes Nov 29, 2024

View reviewed changes

github-actions bot removed the behavior_changed label Nov 29, 2024

gengjun-git self-assigned this Nov 29, 2024

zhangheihei added 3 commits November 29, 2024 03:06

[BugFix] With a high streamload frequency and too many tablets involv…

646d537

…ed, TabletCommitInfos will take up too much memory. Signed-off-by: edwinhzhang <[email protected]>

fix bug

fac2230

Signed-off-by: edwinhzhang <[email protected]>

Optimization Logic

dcf2f0c

Signed-off-by: edwinhzhang <[email protected]>

gengjun-git force-pushed the bug-fix-oom branch from 07ca645 to dcf2f0c Compare November 29, 2024 03:06

gengjun-git approved these changes Nov 29, 2024

View reviewed changes

luohaha enabled auto-merge (squash) November 29, 2024 03:08

luohaha merged commit 86476df into StarRocks:main Nov 29, 2024
45 checks passed

github-actions bot removed the 3.4 label Nov 29, 2024

github-actions bot removed the 3.3 label Nov 29, 2024

github-actions bot removed the 3.2 label Nov 29, 2024

github-actions bot removed the 3.1 label Nov 29, 2024

mergify bot mentioned this pull request Nov 29, 2024

[BugFix] With a high streamload frequency and too many tablets involved, TabletCommitInfos will take up too much memory. (backport #53329) #53371

Open

42 tasks

mergify bot mentioned this pull request Nov 29, 2024

[BugFix] With a high streamload frequency and too many tablets involved, TabletCommitInfos will take up too much memory. (backport #53329) #53372

Open

42 tasks

mergify bot mentioned this pull request Nov 29, 2024

[BugFix] With a high streamload frequency and too many tablets involved, TabletCommitInfos will take up too much memory. (backport #53329) #53373

Closed

42 tasks

mergify bot mentioned this pull request Nov 29, 2024

[BugFix] With a high streamload frequency and too many tablets involved, TabletCommitInfos will take up too much memory. (backport #53329) #53374

Closed

42 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BugFix] With a high streamload frequency and too many tablets involved, TabletCommitInfos will take up too much memory. #53329

[BugFix] With a high streamload frequency and too many tablets involved, TabletCommitInfos will take up too much memory. #53329

zhangheihei commented Nov 28, 2024 •

edited by trueeyu

Loading

mergify bot commented Nov 29, 2024

gengjun-git commented Nov 29, 2024

mergify bot commented Nov 29, 2024

sonarcloud bot commented Nov 29, 2024

github-actions bot commented Nov 29, 2024

github-actions bot commented Nov 29, 2024

github-actions bot commented Nov 29, 2024

github-actions bot commented Nov 29, 2024

github-actions bot commented Nov 29, 2024

github-actions bot commented Nov 29, 2024

mergify bot commented Nov 29, 2024 •

edited

Loading

mergify bot commented Nov 29, 2024 •

edited

Loading

mergify bot commented Nov 29, 2024 •

edited

Loading

mergify bot commented Nov 29, 2024 •

edited

Loading

github-actions bot commented Nov 29, 2024

[BugFix] With a high streamload frequency and too many tablets involved, TabletCommitInfos will take up too much memory. #53329

[BugFix] With a high streamload frequency and too many tablets involved, TabletCommitInfos will take up too much memory. #53329

Conversation

zhangheihei commented Nov 28, 2024 • edited by trueeyu Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

mergify bot commented Nov 29, 2024

❌ Sorry but I didn't understand the command. Please consult the commands documentation 📚.

gengjun-git commented Nov 29, 2024

mergify bot commented Nov 29, 2024

✅ Branch has been successfully rebased

sonarcloud bot commented Nov 29, 2024

Quality Gate passed

github-actions bot commented Nov 29, 2024

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Nov 29, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented Nov 29, 2024

github-actions bot commented Nov 29, 2024

github-actions bot commented Nov 29, 2024

github-actions bot commented Nov 29, 2024

mergify bot commented Nov 29, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Nov 29, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Nov 29, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Nov 29, 2024 • edited Loading

✅ Backports have been created

github-actions bot commented Nov 29, 2024

[BE Incremental Coverage Report]

zhangheihei commented Nov 28, 2024 •

edited by trueeyu

Loading

mergify bot commented Nov 29, 2024 •

edited

Loading

mergify bot commented Nov 29, 2024 •

edited

Loading

mergify bot commented Nov 29, 2024 •

edited

Loading

mergify bot commented Nov 29, 2024 •

edited

Loading