Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable SortProjectTransposeRule #15002

Merged
merged 14 commits into from
Sep 29, 2023
Merged

Enable SortProjectTransposeRule #15002

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
af9e3ed
Enable already passing tests in DecoupledPlanningCalciteQueryTest
kgyrtkirk Sep 15, 2023
b22bd26
empty
kgyrtkirk Sep 15, 2023
1644321
add transposerule; fix prefix
kgyrtkirk Sep 18, 2023
38bfec3
accept a few more
kgyrtkirk Sep 18, 2023
6bb32b3
fix postagg
kgyrtkirk Sep 18, 2023
51e5cd0
Merge remote-tracking branch 'apache/master' into decouple-test-fixes-1
kgyrtkirk Sep 21, 2023
0610299
new-ignores
kgyrtkirk Sep 21, 2023
10743ec
col renames
kgyrtkirk Sep 26, 2023
4604ec9
fix ref
kgyrtkirk Sep 26, 2023
cc63c99
fix ref
kgyrtkirk Sep 26, 2023
ba65bd9
remove change of alias prefix for alternate postagg call
kgyrtkirk Sep 27, 2023
7224c2c
Merge remote-tracking branch 'apache/master' into decouple-test-fixes-1
kgyrtkirk Sep 27, 2023
f8a7c56
Merge remote-tracking branch 'apache/master' into decouple-test-fixes-1
kgyrtkirk Sep 27, 2023
1c61435
Merge remote-tracking branch 'apache/master' into decouple-test-fixes-1
kgyrtkirk Sep 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ...a/org/apache/druid/query/aggregation/datasketches/hll/sql/HllSketchSqlAggregatorTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -824,8 +824,8 @@ public void testtHllSketchPostAggsPostSort()
.postAggregators(
ImmutableList.of(
new HllSketchToEstimatePostAggregator("p1", new FieldAccessPostAggregator("p0", "a0"), false),
new HllSketchToEstimatePostAggregator("s1", new FieldAccessPostAggregator("s0", "a0"), false),
new HllSketchToStringPostAggregator("s3", new FieldAccessPostAggregator("s2", "a0"))
new HllSketchToEstimatePostAggregator("_p1", new FieldAccessPostAggregator("_p0", "a0"), false),
new HllSketchToStringPostAggregator("_p3", new FieldAccessPostAggregator("_p2", "a0"))
)
)
.context(QUERY_CONTEXT_DEFAULT)
Expand Down
8 changes: 4 additions & 4 deletions ...he/druid/query/aggregation/datasketches/quantiles/sql/DoublesSketchSqlAggregatorTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -692,13 +692,13 @@ public void testDoublesSketchPostAggsPostSort()
0.5
),
new DoublesSketchToQuantilePostAggregator(
"s1",
new FieldAccessPostAggregator("s0", "a0:agg"),
"_p1",
new FieldAccessPostAggregator("_p0", "a0:agg"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did these change?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are 2 callsites to create the postaggregates - one of them uses p and the other s as prefix; I was getting almost 100% matching plans with column alias mistmatches because of the different prefixes - so I've changed that; as the main importance of them is to have them unique.

I've today tried to remove the change of that prefix thing - and I think that change might not be needed in this patch.

I've update the PR

0.5
),
new DoublesSketchToQuantilePostAggregator(
"s3",
new FieldAccessPostAggregator("s2", "a0:agg"),
"_p3",
new FieldAccessPostAggregator("_p2", "a0:agg"),
0.9800000190734863
)
)
Expand Down
8 changes: 4 additions & 4 deletions ...g/apache/druid/query/aggregation/datasketches/theta/sql/ThetaSketchSqlAggregatorTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -747,8 +747,8 @@ public void testThetaSketchPostAggsPostSort()
null
),
new SketchEstimatePostAggregator(
"s1",
new FieldAccessPostAggregator("s0", "a0"),
"_p1",
new FieldAccessPostAggregator("_p0", "a0"),
null
)
)
Expand Down Expand Up @@ -803,8 +803,8 @@ public void testThetaSketchPostAggsPostSortFinalizeOuterSketches()
null
),
new SketchEstimatePostAggregator(
"s1",
new FieldAccessPostAggregator("s0", "a0"),
"_p1",
new FieldAccessPostAggregator("_p0", "a0"),
null
)
)
Expand Down
1 change: 1 addition & 0 deletions sql/src/main/java/org/apache/druid/sql/calcite/planner/CalciteRulesManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@ public class CalciteRulesManager
CoreRules.FILTER_VALUES_MERGE,
CoreRules.PROJECT_FILTER_VALUES_MERGE,
CoreRules.PROJECT_VALUES_MERGE,
CoreRules.SORT_PROJECT_TRANSPOSE,
CoreRules.AGGREGATE_VALUES
);

Expand Down
2 changes: 1 addition & 1 deletion sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidQuery.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -650,7 +650,7 @@ private static Sorting computeSorting(

projection = Projection.preAggregation(sortProject, plannerContext, rowSignature, virtualColumnRegistry);
} else {
projection = Projection.postAggregation(sortProject, plannerContext, rowSignature, "s");
projection = Projection.postAggregation(sortProject, plannerContext, rowSignature, "p");
}

return Sorting.create(orderBys, offsetLimit, projection);
Expand Down
2 changes: 1 addition & 1 deletion sql/src/test/java/org/apache/druid/sql/calcite/CalciteArraysQueryTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2206,7 +2206,7 @@ public void testArrayAggGroupByArrayAggFromSubquery()
.intervals(querySegmentSpec(Filtration.eternity()))
.granularity(Granularities.ALL)
.context(QUERY_CONTEXT_NO_STRINGIFY_ARRAY)
.postAggregators(new ExpressionPostAggregator("s0", "1", null, ExprMacroTable.nil()))
.postAggregators(new ExpressionPostAggregator("p0", "1", null, ExprMacroTable.nil()))
.build()
),
useDefault ?
Expand Down
3 changes: 0 additions & 3 deletions sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2733,7 +2733,6 @@ public void testGroupByWithSelectAndOrderByProjections()
);
}

@DecoupledIgnore(mode = Modes.PLAN_MISMATCH)
@Test
public void testTopNWithSelectProjections()
{
Expand Down Expand Up @@ -11620,7 +11619,6 @@ public void testSortProjectAfterNestedGroupBy()
);
}

@DecoupledIgnore(mode = Modes.CANNOT_CONVERT)
@Test
public void testPostAggWithTimeseries()
{
Expand Down Expand Up @@ -11664,7 +11662,6 @@ public void testPostAggWithTimeseries()
);
}

@DecoupledIgnore(mode = Modes.PLAN_MISMATCH)
@Test
public void testPostAggWithTopN()
{
Expand Down