Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Choose cheapest aggregated path as base for chunk-wise aggregation #7025

Merged
merged 68 commits into from
Dec 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
20ff42b
Choose cheapest aggregated path as base for chunk-wise aggregation
akuzm Jun 12, 2024
f9508a3
test refs
akuzm Jun 12, 2024
cca447a
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Jun 12, 2024
769a346
reference REL_16_3 transparent_decompression-* partitionwise-*
akuzm Jun 17, 2024
5fe7c52
reference REL_14_11 transparent_decompression-* partitionwise-*
akuzm Jun 17, 2024
20b6940
parallel plans
akuzm Jun 18, 2024
e017290
remove debug
akuzm Jun 18, 2024
b0d9c44
easier diff
akuzm Jun 18, 2024
c66df25
cleanup
akuzm Jun 18, 2024
3b9e42b
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Jun 18, 2024
06a2606
fixes for projection
akuzm Jun 18, 2024
9a0a988
stabilize the cagg tests
akuzm Jun 19, 2024
21fb3ae
Support chunkwise aggregation with projection
akuzm Jun 19, 2024
00288ac
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Jun 19, 2024
4b0a1be
tmp debug -- cost based partial aggregation paths
akuzm Jun 20, 2024
59ec486
fix for custom agg paths
akuzm Jun 20, 2024
bab2db4
remove debug
akuzm Jun 20, 2024
7b0a442
fix the tests
akuzm Jun 21, 2024
e785032
stabilize the telemetry_stats
akuzm Jun 22, 2024
55fedd2
partitionwise test
akuzm Jun 22, 2024
b23a737
delete old alternative ref
akuzm Jun 22, 2024
63c7ab4
add vacuum analyze to watermark update test
akuzm Jun 22, 2024
7b795b5
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Jun 26, 2024
9115937
try order by in caggs
akuzm Jun 27, 2024
392e342
reference REL_16_3 append-* parallel-* partitionwise-* query-* cagg_u…
akuzm Jun 27, 2024
8f23e8b
reference REL_14_11 append-* parallel-* partitionwise-* query-* cagg_…
akuzm Jun 27, 2024
a9168f6
try another way
akuzm Jun 27, 2024
fb9e1ac
unify test
akuzm Jun 27, 2024
bda8e9b
benchmark partitionwise projection (2024-06-27 no. 1)
akuzm Jun 27, 2024
b856e45
ref?
akuzm Jun 27, 2024
9239b8c
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Jul 7, 2024
52825e9
refs
akuzm Jul 7, 2024
e7d187d
disable on v14
akuzm Jul 16, 2024
be2821f
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Jul 16, 2024
e87fb2a
test
akuzm Jul 23, 2024
af7f8e1
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Jul 23, 2024
19a156f
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Aug 21, 2024
bb37597
oops
akuzm Aug 21, 2024
144b89e
refs
akuzm Aug 21, 2024
bc72b3d
lost update test changes
akuzm Aug 21, 2024
5020c49
remove projection
akuzm Aug 21, 2024
de95e08
typo
akuzm Aug 21, 2024
8b1b711
not sure
akuzm Aug 21, 2024
be5ec30
benchmark partitionwise projection (2024-08-21 no. 3)
akuzm Aug 21, 2024
8a75199
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Aug 26, 2024
b5b607c
Merge remote-tracking branch 'akuzm/chunkwise-projection' into HEAD
akuzm Aug 26, 2024
1eae5b5
reference REL_16_3 transparent_decompression-*
akuzm Aug 26, 2024
20c81f9
Merge origin/main into tmp (using imerge)
akuzm Sep 10, 2024
8009a76
fixes after merge
akuzm Sep 10, 2024
cfff28a
Merge origin/main into heads/tmp (using imerge)
akuzm Dec 5, 2024
f1c5782
fix the reference after merge
akuzm Dec 5, 2024
4820a50
hypercore test
akuzm Dec 5, 2024
10518fd
reference REL_17_0-80-gb7467ab71c transparent_decompression-*
akuzm Dec 5, 2024
2fa3344
reference REL_14_11 transparent_decompression-*
akuzm Dec 5, 2024
2385f1d
reference REL_16_4-111-g925b3aa857 transparent_decompression-*
akuzm Dec 5, 2024
eb0cfe4
partialize the entire pathlist
akuzm Dec 5, 2024
518132b
fix another hypercore test
akuzm Dec 5, 2024
7af9df6
reference REL_17_0-80-gb7467ab71c cagg_union_view-*
akuzm Dec 5, 2024
d786a65
fix the debug guc
akuzm Dec 5, 2024
12e5dd4
reference REL_17_0-80-gb7467ab71c cagg_union_view-* continuous_aggs-*…
akuzm Dec 5, 2024
11b1db5
reference REL_14_11 cagg_union_view-* continuous_aggs-* transparent_d…
akuzm Dec 5, 2024
f9186ea
reference REL_16_4-111-g925b3aa857 cagg_union_view-* continuous_aggs-…
akuzm Dec 5, 2024
455e92b
benchmark chunkwise aggregation path changes (2024-12-09 no. 1)
akuzm Dec 9, 2024
5f5879a
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Dec 9, 2024
c7c8f60
reference REL_15_7 transparent_decompression-*
akuzm Dec 9, 2024
5ab2a7e
reference REL_17_0-80-gb7467ab71c transparent_decompression-*
akuzm Dec 9, 2024
b77fbfb
reference REL_14_11 transparent_decompression-*
akuzm Dec 9, 2024
3ba8a8c
reference REL_16_4-111-g925b3aa857 transparent_decompression-*
akuzm Dec 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
362 changes: 177 additions & 185 deletions tsl/src/chunkwise_agg.c

Large diffs are not rendered by default.

8 changes: 8 additions & 0 deletions tsl/src/nodes/vector_agg/plan.c
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,10 @@ has_vector_agg_node(Plan *plan, bool *has_normal_agg)
{
append_plans = castNode(Append, plan)->appendplans;
}
if (IsA(plan, MergeAppend))
{
append_plans = castNode(MergeAppend, plan)->mergeplans;
}
else if (IsA(plan, CustomScan))
{
custom = castNode(CustomScan, plan);
Expand Down Expand Up @@ -471,6 +475,10 @@ try_insert_vector_agg_node(Plan *plan)
{
append_plans = castNode(Append, plan)->appendplans;
}
else if (IsA(plan, MergeAppend))
{
append_plans = castNode(MergeAppend, plan)->mergeplans;
}
else if (IsA(plan, CustomScan))
{
CustomScan *custom = castNode(CustomScan, plan);
Expand Down
110 changes: 52 additions & 58 deletions tsl/test/expected/cagg_union_view-14.out
Original file line number Diff line number Diff line change
Expand Up @@ -355,31 +355,29 @@ SELECT _timescaledb_functions.cagg_watermark(:boundary_view_id);

-- first UNION child should have no rows because no materialization has happened yet and 2nd child should have 4 rows
:PREFIX SELECT * FROM boundary_view;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------
Finalize GroupAggregate (actual rows=4 loops=1)
Group Key: (time_bucket(10, _hyper_5_5_chunk."time"))
-> Sort (actual rows=4 loops=1)
-> Merge Append (actual rows=4 loops=1)
Sort Key: (time_bucket(10, _hyper_5_5_chunk."time"))
Sort Method: quicksort
-> Append (actual rows=4 loops=1)
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(10, _hyper_5_5_chunk."time")
-> Index Scan Backward using _hyper_5_5_chunk_boundary_test_time_idx on _hyper_5_5_chunk (actual rows=1 loops=1)
Index Cond: ("time" >= '-2147483648'::integer)
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(10, _hyper_5_6_chunk."time")
-> Index Scan Backward using _hyper_5_6_chunk_boundary_test_time_idx on _hyper_5_6_chunk (actual rows=1 loops=1)
Index Cond: ("time" >= '-2147483648'::integer)
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(10, _hyper_5_7_chunk."time")
-> Index Scan Backward using _hyper_5_7_chunk_boundary_test_time_idx on _hyper_5_7_chunk (actual rows=1 loops=1)
Index Cond: ("time" >= '-2147483648'::integer)
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(10, _hyper_5_8_chunk."time")
-> Index Scan Backward using _hyper_5_8_chunk_boundary_test_time_idx on _hyper_5_8_chunk (actual rows=1 loops=1)
Index Cond: ("time" >= '-2147483648'::integer)
(22 rows)
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(10, _hyper_5_5_chunk."time")
-> Index Scan Backward using _hyper_5_5_chunk_boundary_test_time_idx on _hyper_5_5_chunk (actual rows=1 loops=1)
Index Cond: ("time" >= '-2147483648'::integer)
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(10, _hyper_5_6_chunk."time")
-> Index Scan Backward using _hyper_5_6_chunk_boundary_test_time_idx on _hyper_5_6_chunk (actual rows=1 loops=1)
Index Cond: ("time" >= '-2147483648'::integer)
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(10, _hyper_5_7_chunk."time")
-> Index Scan Backward using _hyper_5_7_chunk_boundary_test_time_idx on _hyper_5_7_chunk (actual rows=1 loops=1)
Index Cond: ("time" >= '-2147483648'::integer)
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(10, _hyper_5_8_chunk."time")
-> Index Scan Backward using _hyper_5_8_chunk_boundary_test_time_idx on _hyper_5_8_chunk (actual rows=1 loops=1)
Index Cond: ("time" >= '-2147483648'::integer)
(20 rows)

-- result should have 4 rows
SELECT * FROM boundary_view ORDER BY time_bucket;
Expand All @@ -403,26 +401,24 @@ SELECT _timescaledb_functions.cagg_watermark(:boundary_view_id);

-- both sides of the UNION should return 2 rows
:PREFIX SELECT * FROM boundary_view;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------
Append (actual rows=4 loops=1)
-> Index Scan using _hyper_6_9_chunk__materialized_hypertable_6_time_bucket_idx on _hyper_6_9_chunk (actual rows=2 loops=1)
Index Cond: (time_bucket < 30)
-> Finalize GroupAggregate (actual rows=2 loops=1)
Group Key: (time_bucket(10, _hyper_5_7_chunk."time"))
-> Sort (actual rows=2 loops=1)
-> Merge Append (actual rows=2 loops=1)
Sort Key: (time_bucket(10, _hyper_5_7_chunk."time"))
Sort Method: quicksort
-> Append (actual rows=2 loops=1)
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(10, _hyper_5_7_chunk."time")
-> Index Scan Backward using _hyper_5_7_chunk_boundary_test_time_idx on _hyper_5_7_chunk (actual rows=1 loops=1)
Index Cond: ("time" >= 30)
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(10, _hyper_5_8_chunk."time")
-> Index Scan Backward using _hyper_5_8_chunk_boundary_test_time_idx on _hyper_5_8_chunk (actual rows=1 loops=1)
Index Cond: ("time" >= 30)
(17 rows)
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(10, _hyper_5_7_chunk."time")
-> Index Scan Backward using _hyper_5_7_chunk_boundary_test_time_idx on _hyper_5_7_chunk (actual rows=1 loops=1)
Index Cond: ("time" >= 30)
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(10, _hyper_5_8_chunk."time")
-> Index Scan Backward using _hyper_5_8_chunk_boundary_test_time_idx on _hyper_5_8_chunk (actual rows=1 loops=1)
Index Cond: ("time" >= 30)
(15 rows)

-- result should have 4 rows
SELECT * FROM boundary_view ORDER BY time_bucket;
Expand Down Expand Up @@ -592,8 +588,8 @@ ORDER by 1;

-- plan output
:PREFIX SELECT * FROM mat_m1 ORDER BY 1;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------
Sort (actual rows=3 loops=1)
Sort Key: _hyper_9_15_chunk.time_bucket
Sort Method: quicksort
Expand All @@ -604,27 +600,25 @@ ORDER by 1;
Group Key: (time_bucket(5, _hyper_7_11_chunk.a))
Filter: ((sum(_hyper_7_11_chunk.c) > 50) AND ((avg(_hyper_7_11_chunk.b))::integer > 12))
Rows Removed by Filter: 1
-> Sort (actual rows=3 loops=1)
-> Merge Append (actual rows=3 loops=1)
Sort Key: (time_bucket(5, _hyper_7_11_chunk.a))
Sort Method: quicksort
-> Append (actual rows=3 loops=1)
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(5, _hyper_7_11_chunk.a)
-> Index Scan Backward using _hyper_7_11_chunk_ht_intdata_a_idx on _hyper_7_11_chunk (actual rows=2 loops=1)
Index Cond: (a >= 25)
Filter: ((b < 16) AND (c > 20))
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(5, _hyper_7_13_chunk.a)
-> Index Scan Backward using _hyper_7_13_chunk_ht_intdata_a_idx on _hyper_7_13_chunk (actual rows=3 loops=1)
Index Cond: (a >= 25)
Filter: ((b < 16) AND (c > 20))
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(5, _hyper_7_14_chunk.a)
-> Index Scan Backward using _hyper_7_14_chunk_ht_intdata_a_idx on _hyper_7_14_chunk (actual rows=1 loops=1)
Index Cond: (a >= 25)
Filter: ((b < 16) AND (c > 20))
Rows Removed by Filter: 2
(30 rows)
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(5, _hyper_7_11_chunk.a)
-> Index Scan Backward using _hyper_7_11_chunk_ht_intdata_a_idx on _hyper_7_11_chunk (actual rows=2 loops=1)
Index Cond: (a >= 25)
Filter: ((b < 16) AND (c > 20))
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(5, _hyper_7_13_chunk.a)
-> Index Scan Backward using _hyper_7_13_chunk_ht_intdata_a_idx on _hyper_7_13_chunk (actual rows=3 loops=1)
Index Cond: (a >= 25)
Filter: ((b < 16) AND (c > 20))
-> Partial GroupAggregate (actual rows=1 loops=1)
Group Key: time_bucket(5, _hyper_7_14_chunk.a)
-> Index Scan Backward using _hyper_7_14_chunk_ht_intdata_a_idx on _hyper_7_14_chunk (actual rows=1 loops=1)
Index Cond: (a >= 25)
Filter: ((b < 16) AND (c > 20))
Rows Removed by Filter: 2
(28 rows)

-- Test caggs with different time types
CREATE TABLE smallint_table (time smallint, value int);
Expand Down
Loading
Loading