feat(stream): support row count for arrangement backfill

[kwannoel]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Supports recoverable row count for arrangement backfill. We will persist and recover it.

It's to ensure that arrangement backfill remains compatible with background ddl. We must persist the row count so that after recovery, we can recover the estimated progress for the materialized view in show jobs.

The meta-side logic is the same as with NoShuffleBackfill, after recovery, meta will receive updates from NoShuffleBackfill executors to reconstruct the progress bar.

Progress is calculated with: total_snapshot_row_count / total_upstream_row_count in meta.

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• I have added test labels as necessary. See details.
• I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
• My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
• All checks passed in ./risedev check (or alias, ./risedev c)
• My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

• My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

[yezizp2012]

Rest LGTM

[BugenZhao]

Can you elaborate more on the motivation?

[kwannoel]

Can you elaborate more on the motivation?

Updated

[kwannoel]

Only e2e test (release) failed as expected. It failed in today's main-cron as well.

[kwannoel]

Any comments @BugenZhao @chenzl25 ?

[BugenZhao]

Seems I missed something. Prior to this PR, how do we recover the progress (bar) for background no-shuffle backfill?

[kwannoel]

Seems I missed something. Prior to this PR, how do we recover the progress (bar) for background no-shuffle backfill?

There are several PRs which would commit the row count state to the state table for background no shuffle:

• feat(streaming): persist row_count in backfill state_table #12415 (persists the row count)
• feat(streaming): recover no_shuffle_backfill #12493 (recovers the row count)

[BugenZhao]

There are several PRs which would commit the row count state to the state table for background no shuffle:

I see. I once thought the changes in utils would be shared by no-shuffle backfill and arrangement backfill, but it seems not.

[kwannoel]

There are several PRs which would commit the row count state to the state table for background no shuffle:

I see. I once thought the changes in utils would be shared by no-shuffle backfill and arrangement backfill, but it seems not.

Perhaps it is possible. I chose not to do so for now, because arrangement backfill's logic is per vnode in comparison the no shuffle backfill.

We can refactor it in the future.