Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

experiments with vectorized aggregation #6793

Closed
wants to merge 108 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
e64faf7
vectorized aggregation as separate plan node
akuzm Mar 25, 2024
ef4cd87
fixes
akuzm Mar 25, 2024
93b4d0f
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Mar 25, 2024
8cb97e0
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Mar 27, 2024
b27d2b1
something works
akuzm Mar 27, 2024
13ba173
full switch
akuzm Mar 28, 2024
be203fd
fix the build
akuzm Mar 28, 2024
ee8b1f4
remove the old planning approach
akuzm Mar 28, 2024
e146937
remove more of old planning
akuzm Mar 28, 2024
753bf0d
typos
akuzm Mar 28, 2024
4a4f20b
use enum indexes for settings
akuzm Mar 28, 2024
beba737
cleanup
akuzm Mar 28, 2024
2dbda15
benchmark separate vectorized agg (2024-03-28 no. 1)
akuzm Mar 28, 2024
175cbf2
split out common code
akuzm Mar 28, 2024
21faf6e
show costs in explain
akuzm Mar 28, 2024
fa2fb4d
wrong prefix
akuzm Mar 28, 2024
0ed166f
Merge remote-tracking branch 'akuzm/vector-separate' into HEAD
akuzm Mar 29, 2024
30a6069
Remove temporary data from DecompressChunkPath
akuzm Mar 29, 2024
e25267d
rename
akuzm Mar 29, 2024
5e6221d
typo
akuzm Mar 29, 2024
5c4af48
benchmark separate vectorized agg (2024-03-29 no. 2)
akuzm Mar 29, 2024
4130683
produce partials for each batch
akuzm Mar 29, 2024
e7f01ab
benchmark separate vectorized agg (2024-03-29 no. 3)
akuzm Mar 29, 2024
cff844d
more generic interface
akuzm Mar 29, 2024
209838e
fix outer_var resolution
akuzm Mar 29, 2024
dfb92af
Revert "disable filters"
akuzm Mar 29, 2024
6ef84c1
support filters?
akuzm Mar 29, 2024
4db7cea
fix outer_var resolution
akuzm Mar 29, 2024
287f3b4
fix ref
akuzm Mar 29, 2024
a50069f
Merge remote-tracking branch 'akuzm/vector-separate' into HEAD
akuzm Mar 29, 2024
9bdae30
fix for filtered out batches
akuzm Mar 29, 2024
398f317
benchmark vectorized agg with filter (2024-03-29 no. 4)
akuzm Mar 29, 2024
eaca282
fix build on windows
akuzm Mar 29, 2024
fdca7a7
something that doesn't work
akuzm Apr 2, 2024
60c6eab
fix
akuzm Apr 2, 2024
7887ebd
fix
akuzm Apr 2, 2024
84b93b7
separate file for functions
akuzm Apr 3, 2024
cdf0b58
adjust chunk sizes to avoid sorting in parallel append
akuzm Apr 3, 2024
bb3a2e2
compat
akuzm Apr 4, 2024
f5bbfb0
benchmark vectorized agg in separate node (2024-04-04 no. 1)
akuzm Apr 4, 2024
3b6d263
benchmark vectorized agg in separate node (2024-04-04 no. 2)
akuzm Apr 4, 2024
43401ed
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Apr 4, 2024
d93ef39
benchmark vectorized agg in separate node (2024-04-04 no. 3)
akuzm Apr 4, 2024
76dc6dc
fix runtime chunk exclusion with vectorized aggregation
akuzm Apr 4, 2024
cd3e862
Don't copy compressed slot to compressed batch struct
akuzm Apr 4, 2024
6693d9f
Always copy out when detoasting as well
akuzm Apr 4, 2024
991711a
cache the byvalue flag
akuzm Apr 4, 2024
f9b7ff8
Merge remote-tracking branch 'akuzm/no-compressed-copy' into HEAD
akuzm Apr 4, 2024
cad424e
benchmark vectorized agg all together (2024-04-04 no. 4)
akuzm Apr 4, 2024
e34ad75
fixed-length by-reference types
akuzm Apr 4, 2024
fe85559
benchmark no compressed copy (2024-04-04 no. 5)
akuzm Apr 4, 2024
119220c
cleanup
akuzm Apr 4, 2024
b695ca3
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Apr 5, 2024
16e5390
boop the CI
akuzm Apr 5, 2024
ae4151a
review fixes
akuzm Apr 5, 2024
f27c162
some coverage
akuzm Apr 5, 2024
8f00045
assert
akuzm Apr 5, 2024
0695543
copy more fields from agg plan
akuzm Apr 5, 2024
0b9709f
fix build on 13
akuzm Apr 5, 2024
b12b6e5
Merge remote-tracking branch 'akuzm/vector-separate' into HEAD
akuzm Apr 5, 2024
d0da278
Merge remote-tracking branch 'akuzm/no-compressed-copy' into HEAD
akuzm Apr 5, 2024
0646f6c
drafts
akuzm Apr 5, 2024
fa0c3b2
use dedicated function state
akuzm Apr 6, 2024
9ffc75c
use the agg definition
akuzm Apr 6, 2024
13ad5f7
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Apr 9, 2024
d4716c0
remove debug prints
akuzm Apr 9, 2024
ab82a22
Merge remote-tracking branch 'akuzm/no-compressed-copy' into HEAD
akuzm Apr 9, 2024
406f085
Merge remote-tracking branch 'akuzm/no-compressed-copy' into HEAD
akuzm Apr 9, 2024
be91171
rename table
akuzm Apr 9, 2024
895a96a
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Apr 9, 2024
aaa3837
Merge remote-tracking branch 'akuzm/vector-separate' into HEAD
akuzm Apr 9, 2024
6da0650
return the inner loop
akuzm Apr 10, 2024
f7d70a7
vectorized aggregation for count(*)
akuzm Apr 10, 2024
f5aa906
Add segmentby columns to the list of data columns as well
akuzm Apr 10, 2024
76d6b59
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Apr 10, 2024
4568d62
scalar
akuzm Apr 10, 2024
f37fa93
fixes
akuzm Apr 10, 2024
4ec877c
ref
akuzm Apr 10, 2024
c9a3f80
fix for i386
akuzm Apr 10, 2024
8d14a12
Treat segmentby columns same as compressed columns with default value
akuzm Apr 10, 2024
d450c5b
Merge remote-tracking branch 'akuzm/scalar' into HEAD
akuzm Apr 10, 2024
e9e7bf9
spelling
akuzm Apr 10, 2024
7f0bd22
make it even more uniform
akuzm Apr 10, 2024
0ec8c15
simplify scalar columns in vector agg
akuzm Apr 10, 2024
eca08e7
make it even more uniform
akuzm Apr 10, 2024
22ba22d
Merge remote-tracking branch 'akuzm/scalar' into HEAD
akuzm Apr 10, 2024
dfa1953
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Apr 11, 2024
5f88a23
cleanup after merge
akuzm Apr 11, 2024
faf8130
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Apr 11, 2024
6ed5614
Merge remote-tracking branch 'akuzm/scalar' into HEAD
akuzm Apr 11, 2024
3e4fde8
Merge remote-tracking branch 'akuzm/vector-separate' into HEAD
akuzm Apr 11, 2024
f97f2b0
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Apr 11, 2024
a1128dc
Merge remote-tracking branch 'akuzm/scalar' into HEAD
akuzm Apr 11, 2024
e78efb7
prepare for multiple aggregates
akuzm Apr 11, 2024
178b359
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Apr 12, 2024
c019642
multiple aggregate functions
akuzm Apr 12, 2024
25cb1ae
grouping policy
akuzm Apr 12, 2024
32f88ed
tmp
akuzm Apr 12, 2024
f551c1d
more checks?
akuzm Apr 17, 2024
dd65d7c
grouping on segmentby
akuzm Apr 17, 2024
dd9875c
test ref
akuzm Apr 17, 2024
8ae7e37
remove debug
akuzm Apr 17, 2024
5f46bc2
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Apr 22, 2024
37f13fa
add a test
akuzm Apr 22, 2024
362eaf4
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Apr 22, 2024
2cf656c
more tests
akuzm Apr 23, 2024
8e1651c
typo
akuzm Apr 24, 2024
8ad6967
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Apr 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/compat/compat.h
Original file line number Diff line number Diff line change
Expand Up @@ -974,6 +974,7 @@ object_ownercheck(Oid classid, Oid objectid, Oid roleid)

#if PG14_LT
#define F_SUM_INT4 2108
#define F_COUNT_ 2803
#endif

/*
Expand Down
25 changes: 25 additions & 0 deletions src/guc.c
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,18 @@ static const struct config_enum_entry require_vector_qual_options[] = {
#endif

DebugRequireVectorQual ts_guc_debug_require_vector_qual = RVQ_Allow;

#ifdef TS_DEBUG
static const struct config_enum_entry require_vector_agg_options[] = {
{ "allow", RVA_Allow, false },
{ "forbid", RVA_Forbid, false },
{ "require", RVA_Require, false },
{ NULL, 0, false }
};
#endif

DebugRequireVectorAgg ts_guc_debug_require_vector_agg = RVA_Allow;

bool ts_guc_debug_compression_path_info = false;

static bool ts_guc_enable_hypertable_create = true;
Expand Down Expand Up @@ -820,6 +832,19 @@ _guc_init(void)
/* assign_hook= */ NULL,
/* show_hook= */ NULL);

DefineCustomEnumVariable(/* name= */ MAKE_EXTOPTION("debug_require_vector_agg"),
/* short_desc= */
"ensure that vectorized aggregation is used or not",
/* long_desc= */ "this is for debugging purposes",
/* valueAddr= */ (int *) &ts_guc_debug_require_vector_agg,
/* bootValue= */ RVQ_Allow,
/* options = */ require_vector_agg_options,
/* context= */ PGC_USERSET,
/* flags= */ 0,
/* check_hook= */ NULL,
/* assign_hook= */ NULL,
/* show_hook= */ NULL);

DefineCustomEnumVariable(/* name= */ MAKE_EXTOPTION("debug_require_vector_qual"),
/* short_desc= */
"ensure that non-vectorized or vectorized filters are used in "
Expand Down
9 changes: 9 additions & 0 deletions src/guc.h
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,15 @@ extern TSDLLEXPORT bool ts_guc_debug_require_batch_sorted_merge;

extern TSDLLEXPORT bool ts_guc_debug_allow_cagg_with_deprecated_funcs;

typedef enum DebugRequireVectorAgg
{
RVA_Allow = 0,
RVA_Forbid,
RVA_Require,
} DebugRequireVectorAgg;

extern TSDLLEXPORT DebugRequireVectorAgg ts_guc_debug_require_vector_agg;

void _guc_init(void);

typedef enum
Expand Down
30 changes: 20 additions & 10 deletions src/planner/partialize.c
Original file line number Diff line number Diff line change
Expand Up @@ -446,12 +446,18 @@ generate_agg_pushdown_path(PlannerInfo *root, Path *cheapest_total_path, RelOptI
PathTarget *partial_grouping_target, bool can_sort, bool can_hash,
double d_num_groups, GroupPathExtraData *extra_data)
{
// mybt();

/* Get subpaths */
List *subpaths = get_subpaths_from_append_path(cheapest_total_path, false);

/* No subpaths available or unsupported append node */
if (subpaths == NIL)
{
// fprintf(stderr, "no subpaths in append path:\n");
// my_print(cheapest_total_path);
return;
}

if (list_length(subpaths) < 2)
{
Expand Down Expand Up @@ -538,20 +544,24 @@ generate_agg_pushdown_path(PlannerInfo *root, Path *cheapest_total_path, RelOptI
/* Create new append paths */
if (sorted_subpaths != NIL)
{
add_path(partially_grouped_rel,
copy_append_like_path(root,
cheapest_total_path,
sorted_subpaths,
partial_grouping_target));
Path *sorted = copy_append_like_path(root,
cheapest_total_path,
sorted_subpaths,
partial_grouping_target);
// fprintf(stderr, "sorted!!!\n");
// my_print(sorted);
add_path(partially_grouped_rel, sorted);
}

if (hashed_subpaths != NIL)
{
add_path(partially_grouped_rel,
copy_append_like_path(root,
cheapest_total_path,
hashed_subpaths,
partial_grouping_target));
Path *hashed = copy_append_like_path(root,
cheapest_total_path,
hashed_subpaths,
partial_grouping_target);
// fprintf(stderr, "hashed!!!\n");
// my_print(hashed);
add_path(partially_grouped_rel, hashed);
}
}

Expand Down
7 changes: 5 additions & 2 deletions tsl/src/nodes/vector_agg/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
set(SOURCES
${CMAKE_CURRENT_SOURCE_DIR}/exec.c ${CMAKE_CURRENT_SOURCE_DIR}/functions.c
${CMAKE_CURRENT_SOURCE_DIR}/plan.c)
${CMAKE_CURRENT_SOURCE_DIR}/exec.c
${CMAKE_CURRENT_SOURCE_DIR}/functions.c
${CMAKE_CURRENT_SOURCE_DIR}/plan.c
${CMAKE_CURRENT_SOURCE_DIR}/grouping_policy_all.c
${CMAKE_CURRENT_SOURCE_DIR}/grouping_policy_segmentby.c)
target_sources(${TSL_LIBRARY_NAME} PRIVATE ${SOURCES})
Loading
Loading