Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable compaction ITs on MSQ engine #16778

Merged
merged 7 commits into from
Jul 30, 2024
Merged

Enable compaction ITs on MSQ engine #16778

merged 7 commits into from
Jul 30, 2024

Conversation

gargvishesh
Copy link
Contributor

@gargvishesh gargvishesh commented Jul 23, 2024 •
edited
Loading

Description

Follow-up to #16291, this PR enables a subset of existing native compaction ITs on the MSQ engine. In the process, the following changes are introduced in the MSQ compaction flow:

  • Populates metricsSpec in CompactionState from querySpec in MSQControllerTask instead of dataSchema
  • Adds check for pre-rolled-up segments having AggregatorFactory with different input and output column names
  • Fixes passing missing cluster-by clause in scan queries
  • Adds annotation of CompactionState to tombstone segments

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@github-actions github-actions bot added Area - Batch Ingestion Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Jul 23, 2024
@gargvishesh
Merge branch 'refs/heads/master' into enable-its-on-msq
509949b 
# Conflicts:
#	extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java
kfaraz
kfaraz reviewed Jul 23, 2024
View reviewed changes
Copy link
Contributor

@kfaraz kfaraz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The actual IT changes look good.
Have some queries around the other fixes.

server/src/main/java/org/apache/druid/segment/indexing/DataSchema.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/druid/segment/indexing/DataSchema.java Outdated Show resolved Hide resolved
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java Outdated Show resolved Hide resolved
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java Outdated Show resolved Hide resolved
...-core/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java Show resolved Hide resolved
...-core/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java Outdated Show resolved Hide resolved
...-core/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java Outdated Show resolved Hide resolved
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java Show resolved Hide resolved
...ration-tests/src/test/java/org/apache/druid/tests/coordinator/duty/ITAutoCompactionTest.java Outdated Show resolved Hide resolved
@gargvishesh gargvishesh requested a review from kfaraz July 24, 2024 11:50
kfaraz
kfaraz reviewed Jul 25, 2024
View reviewed changes
Copy link
Contributor

@kfaraz kfaraz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor comments.

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/druid/segment/indexing/CombinedDataSchema.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/druid/segment/indexing/CombinedDataSchema.java Show resolved Hide resolved
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java Outdated Show resolved Hide resolved
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java Show resolved Hide resolved
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java Outdated Show resolved Hide resolved
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java Show resolved Hide resolved
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java Show resolved Hide resolved
server/src/main/java/org/apache/druid/segment/indexing/CombinedDataSchema.java Outdated Show resolved Hide resolved
...-core/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java Outdated Show resolved Hide resolved
@gargvishesh gargvishesh requested a review from kfaraz July 26, 2024 11:47
kfaraz
kfaraz approved these changes Jul 29, 2024
View reviewed changes
Copy link
Contributor

@kfaraz kfaraz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 after merge conflicts are resolved and CI is green.

Thanks for the tests, @gargvishesh !

...-core/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQCompactionRunner.java Outdated Show resolved Hide resolved
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java Show resolved Hide resolved
indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java Outdated Show resolved Hide resolved
@gargvishesh
Merge branch 'refs/heads/master' into enable-its-on-msq
d676ca6 
# Conflicts:
#	extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/indexing/MSQCompactionRunnerTest.java
#	server/src/main/java/org/apache/druid/client/indexing/ClientCompactionRunnerInfo.java
#	server/src/test/java/org/apache/druid/client/indexing/ClientCompactionRunnerInfoTest.java
@gargvishesh
Add coverage test
496db40
@gargvishesh
Fix test
dbc3629
@kfaraz kfaraz merged commit e9ea243 into apache:master Jul 30, 2024
88 checks passed
sreemanamala pushed a commit to sreemanamala/druid that referenced this pull request Aug 6, 2024
@gargvishesh @sreemanamala
Enable compaction ITs on MSQ engine (apache#16778)
d4e603d 
Follow-up to apache#16291, this commit enables a subset of existing native compaction ITs on the MSQ engine.

In the process, the following changes have been introduced in the MSQ compaction flow:
- Populate `metricsSpec` in `CompactionState` from `querySpec` in `MSQControllerTask` instead of `dataSchema`
- Add check for pre-rolled-up segments having `AggregatorFactory` with different input and output column names
- Fix passing missing cluster-by clause in scan queries
- Add annotation of `CompactionState` to tombstone segments
@kfaraz kfaraz added this to the 31.0.0 milestone Oct 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kfaraz kfaraz kfaraz approved these changes

Assignees
No one assigned
Labels
Area - Batch Ingestion Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262
Projects
None yet
Milestone
31.0.0
Development

Successfully merging this pull request may close these issues.
2 participants
@gargvishesh @kfaraz