Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rework cursor creation #16533

Merged
merged 89 commits into from
Aug 16, 2024
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
9fcc1b0
rework cursor creation
clintropolis Jun 1, 2024
ff3999c
add dump segment test, fixes
clintropolis Jun 3, 2024
dc4a6b4
final
clintropolis Jun 3, 2024
dca2650
adjustments
clintropolis Jun 5, 2024
7263807
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Jun 5, 2024
5f60f65
oops, missing unstaged from merge
clintropolis Jun 5, 2024
3858f36
adjust
clintropolis Jun 12, 2024
6718f2b
adjust
clintropolis Jun 12, 2024
0b5d399
missed a spot
clintropolis Jun 12, 2024
f693e3e
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Jul 24, 2024
8184213
changes:
clintropolis Jul 26, 2024
3824754
leaky leaky
clintropolis Jul 26, 2024
d22e1cf
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Jul 26, 2024
6c32775
split out CursorMakerFactory interface, adjustments
clintropolis Jul 26, 2024
04e5e47
fixup inspections
clintropolis Jul 27, 2024
bde77ba
fix test, style
clintropolis Jul 27, 2024
920a86e
add mark/resetMark to cursor to improve topN performance
clintropolis Jul 29, 2024
414a420
VectorCursor doesnt need to be Closeable anymore
clintropolis Jul 30, 2024
38df2aa
add mark/resetMark to Offset
clintropolis Jul 30, 2024
d6130e0
remove unused closer
clintropolis Jul 30, 2024
460f6df
fix offset mark/resetMark
clintropolis Jul 30, 2024
d225b16
make IndexTaskTest.testNumShardsAndPartitionDimensionsProvided actual…
clintropolis Jul 30, 2024
815f0bc
remove unused
clintropolis Jul 30, 2024
546ab37
style
clintropolis Jul 30, 2024
cdf3c39
rename resetMark to resetToMark, javadocs
clintropolis Jul 30, 2024
4746212
update benchmarks
clintropolis Jul 31, 2024
39eee30
group by benchmark force instead of true
clintropolis Jul 31, 2024
246dd70
cleanup from self review
clintropolis Jul 31, 2024
ab10354
fixup
clintropolis Jul 31, 2024
d4509b9
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Aug 1, 2024
02a3f28
fix leaky leak in test method
clintropolis Aug 1, 2024
1760e67
cleanup
clintropolis Aug 1, 2024
3339311
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Aug 1, 2024
a612955
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Aug 2, 2024
5d95ae7
partial review stuff
clintropolis Aug 3, 2024
296fd80
javadocs
clintropolis Aug 3, 2024
e567c16
remove Granularity from CursorBuildSpec, add granularity virtual colu…
clintropolis Aug 3, 2024
9a2033f
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Aug 3, 2024
ae40775
fix build
clintropolis Aug 3, 2024
d2828fb
fix static checks
clintropolis Aug 3, 2024
db91f8a
switch descending to preferredOrdering
clintropolis Aug 5, 2024
2782511
javadoc
clintropolis Aug 5, 2024
e678ab4
fixes
clintropolis Aug 5, 2024
e4f2598
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Aug 5, 2024
45e1cc4
fix
clintropolis Aug 5, 2024
4c2d976
adjust
clintropolis Aug 5, 2024
c1f6ca5
adjust
clintropolis Aug 5, 2024
6532b13
oops
clintropolis Aug 5, 2024
4b20621
fix
clintropolis Aug 5, 2024
c42be76
simplify
clintropolis Aug 6, 2024
a995140
fixes and test
clintropolis Aug 6, 2024
dc62f10
more test more fix
clintropolis Aug 6, 2024
5867742
more test
clintropolis Aug 6, 2024
4a55d86
fix test
clintropolis Aug 6, 2024
20b01f6
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Aug 6, 2024
dfba713
fix compile
clintropolis Aug 6, 2024
e9a64dc
nicer
clintropolis Aug 6, 2024
54fce97
fixup merge style
clintropolis Aug 6, 2024
397e1f5
add test comments
clintropolis Aug 7, 2024
d58b778
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Aug 7, 2024
05217df
cleanup, fix segment analyzer bug
clintropolis Aug 7, 2024
c8545ea
fix build
clintropolis Aug 7, 2024
94c7658
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Aug 7, 2024
44523cd
add intervals back to windowed storage adapter for DruidSegmentReader…
clintropolis Aug 8, 2024
99af30c
adjustments
clintropolis Aug 9, 2024
c474638
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Aug 9, 2024
6e2ed96
fix mistake
clintropolis Aug 9, 2024
b4a48a6
loosen contract of mark resetToMark, add tests
clintropolis Aug 10, 2024
fd4232c
a bit more test
clintropolis Aug 10, 2024
dae82e2
fix
clintropolis Aug 10, 2024
f41a80a
style
clintropolis Aug 10, 2024
da683af
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Aug 12, 2024
444c5b1
handle virtual gran name conflicts
clintropolis Aug 12, 2024
65f2375
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Aug 12, 2024
212af82
oops, fix test
clintropolis Aug 13, 2024
3a78558
topN with better reset to not be wasteful, add getOrdering to CursorH…
clintropolis Aug 14, 2024
a04c17e
oops
clintropolis Aug 14, 2024
242c918
remove Cursor.mark, Cursor.resetToMark, force TopN algorithm to use h…
clintropolis Aug 14, 2024
ec3e7a1
remove Offset.mark Offset.resetToMark
clintropolis Aug 14, 2024
37250fd
Merge remote-tracking branch 'upstream/master' into cursor-rework
clintropolis Aug 14, 2024
287f7e2
remove stale unused
clintropolis Aug 14, 2024
19acd70
unused
clintropolis Aug 14, 2024
d066760
more unused cleanup
clintropolis Aug 14, 2024
176c3a5
misc cleanup
clintropolis Aug 14, 2024
514f031
adjust
clintropolis Aug 14, 2024
1594f73
safer timeseries, minor adjustments
clintropolis Aug 15, 2024
9b37926
oops
clintropolis Aug 15, 2024
24406df
nicer cursor build spec granularity decoration
clintropolis Aug 15, 2024
3200af9
missed a spot
clintropolis Aug 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,9 @@
package org.apache.druid.benchmark;

import com.google.common.collect.ImmutableList;
import com.google.common.collect.Iterables;
import org.apache.druid.common.config.NullHandling;
import org.apache.druid.java.util.common.Intervals;
import org.apache.druid.java.util.common.granularity.Granularities;
import org.apache.druid.java.util.common.guava.Sequence;
import org.apache.druid.java.util.common.io.Closer;
import org.apache.druid.js.JavaScriptConfig;
import org.apache.druid.query.aggregation.BufferAggregator;
Expand All @@ -35,9 +33,10 @@
import org.apache.druid.segment.BaseFloatColumnValueSelector;
import org.apache.druid.segment.ColumnSelectorFactory;
import org.apache.druid.segment.Cursor;
import org.apache.druid.segment.CursorBuildSpec;
import org.apache.druid.segment.CursorMaker;
import org.apache.druid.segment.QueryableIndex;
import org.apache.druid.segment.QueryableIndexStorageAdapter;
import org.apache.druid.segment.VirtualColumns;
import org.apache.druid.segment.column.ValueType;
import org.apache.druid.segment.generator.GeneratorColumnSchema;
import org.apache.druid.segment.generator.GeneratorSchemaInfo;
Expand All @@ -60,7 +59,6 @@
import org.openjdk.jmh.infra.Blackhole;

import java.nio.ByteBuffer;
import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.function.Function;

Expand Down Expand Up @@ -165,32 +163,26 @@ private double compute(final Function<ColumnSelectorFactory, BufferAggregator> a
{
final QueryableIndexStorageAdapter adapter = new QueryableIndexStorageAdapter(index);

final Sequence<Cursor> cursors = adapter.makeCursors(
null,
index.getDataInterval(),
VirtualColumns.EMPTY,
Granularities.ALL,
false,
null
);
final CursorBuildSpec buildSpec = CursorBuildSpec.builder()
.setInterval(index.getDataInterval())
.setGranularity(Granularities.ALL)
.build();
try (final CursorMaker maker = adapter.asCursorMaker(buildSpec)) {
final Cursor cursor = maker.makeCursor();

final List<Double> results = cursors
.map(cursor -> {
final BufferAggregator bufferAggregator = aggregatorFactory.apply(cursor.getColumnSelectorFactory());
bufferAggregator.init(aggregationBuffer, 0);
final BufferAggregator bufferAggregator = aggregatorFactory.apply(cursor.getColumnSelectorFactory());
bufferAggregator.init(aggregationBuffer, 0);

while (!cursor.isDone()) {
bufferAggregator.aggregate(aggregationBuffer, 0);
cursor.advance();
}
while (!cursor.isDone()) {
bufferAggregator.aggregate(aggregationBuffer, 0);
cursor.advance();
}

final Double dbl = (Double) bufferAggregator.get(aggregationBuffer, 0);
bufferAggregator.close();
return dbl;
})
.toList();
final Double dbl = (Double) bufferAggregator.get(aggregationBuffer, 0);
bufferAggregator.close();

return Iterables.getOnlyElement(results);
return dbl;
}
}

private static class NativeBufferAggregator implements BufferAggregator
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
import org.apache.druid.common.config.NullHandling;
import org.apache.druid.java.util.common.Intervals;
import org.apache.druid.java.util.common.granularity.Granularities;
import org.apache.druid.java.util.common.guava.Sequence;
import org.apache.druid.java.util.common.io.Closer;
import org.apache.druid.math.expr.ExpressionProcessing;
import org.apache.druid.query.expression.TestExprMacroTable;
Expand All @@ -33,9 +32,10 @@
import org.apache.druid.query.filter.SelectorDimFilter;
import org.apache.druid.segment.ColumnValueSelector;
import org.apache.druid.segment.Cursor;
import org.apache.druid.segment.CursorBuildSpec;
import org.apache.druid.segment.CursorMaker;
import org.apache.druid.segment.QueryableIndex;
import org.apache.druid.segment.QueryableIndexStorageAdapter;
import org.apache.druid.segment.VirtualColumns;
import org.apache.druid.segment.column.ValueType;
import org.apache.druid.segment.generator.GeneratorColumnSchema;
import org.apache.druid.segment.generator.GeneratorSchemaInfo;
Expand All @@ -58,7 +58,6 @@
import org.openjdk.jmh.infra.Blackhole;

import java.util.Arrays;
import java.util.List;
import java.util.concurrent.TimeUnit;

@State(Scope.Benchmark)
Expand Down Expand Up @@ -146,52 +145,38 @@ public void tearDown() throws Exception
@Benchmark
public void expressionFilter(Blackhole blackhole)
{
final Sequence<Cursor> cursors = new QueryableIndexStorageAdapter(index).makeCursors(
expressionFilter.toFilter(),
index.getDataInterval(),
VirtualColumns.EMPTY,
Granularities.ALL,
false,
null
);
final List<?> results = cursors
.map(cursor -> {
final ColumnValueSelector selector = cursor.getColumnSelectorFactory().makeColumnValueSelector("x");
consumeString(cursor, selector, blackhole);
return null;
})
.toList();

blackhole.consume(results);
final CursorBuildSpec buildSpec = CursorBuildSpec.builder()
.setFilter(expressionFilter.toFilter())
.setInterval(index.getDataInterval())
.setGranularity(Granularities.ALL)
.build();
try (final CursorMaker maker = new QueryableIndexStorageAdapter(index).asCursorMaker(buildSpec)) {
final Cursor cursor = maker.makeCursor();


final ColumnValueSelector selector = cursor.getColumnSelectorFactory().makeColumnValueSelector("x");
while (!cursor.isDone()) {
blackhole.consume(selector.getObject());
cursor.advance();
}
}
}

@Benchmark
public void nativeFilter(Blackhole blackhole)
{
final Sequence<Cursor> cursors = new QueryableIndexStorageAdapter(index).makeCursors(
nativeFilter.toFilter(),
index.getDataInterval(),
VirtualColumns.EMPTY,
Granularities.ALL,
false,
null
);
final List<?> results = cursors
.map(cursor -> {
final ColumnValueSelector selector = cursor.getColumnSelectorFactory().makeColumnValueSelector("x");
consumeString(cursor, selector, blackhole);
return null;
})
.toList();

blackhole.consume(results);
}

private void consumeString(final Cursor cursor, final ColumnValueSelector selector, final Blackhole blackhole)
{
while (!cursor.isDone()) {
blackhole.consume(selector.getLong());
cursor.advance();
final CursorBuildSpec buildSpec = CursorBuildSpec.builder()
.setFilter(nativeFilter.toFilter())
.setInterval(index.getDataInterval())
.setGranularity(Granularities.ALL)
.build();
try (final CursorMaker maker = new QueryableIndexStorageAdapter(index).asCursorMaker(buildSpec)) {
final Cursor cursor = maker.makeCursor();
final ColumnValueSelector selector = cursor.getColumnSelectorFactory().makeColumnValueSelector("x");
while (!cursor.isDone()) {
blackhole.consume(selector.getObject());
cursor.advance();
}
}
}
}
Loading
Loading