Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add flatmapConcat with parallelism support #1702

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

He-Pin
Copy link
Member

@He-Pin He-Pin commented Jan 10, 2025

Motivation:
Add support for #1672
Based on: #1701

Modification:

  1. Add flatmapConcat with parallelism
  2. add optimization to avoid materialization at the same time.

Result:

  1. flatmapConcat with parallelism supported

Benchmark:

[info] Benchmark                                       Mode  Cnt         Score         Error  Units
[info] FlatMapConcatBenchmark.completedFuture         thrpt    5   1045035.962 ±   95438.439  ops/s
[info] FlatMapConcatBenchmark.completedFutureP1       thrpt    5  10200822.173 ±  506488.622  ops/s
[info] FlatMapConcatBenchmark.internalSingleSource    thrpt    5  17561424.472 ± 2469846.548  ops/s
[info] FlatMapConcatBenchmark.internalSingleSourceP1  thrpt    5  17434317.937 ± 2230811.585  ops/s
[info] FlatMapConcatBenchmark.mapBaseline             thrpt    5  41100524.324 ± 5565726.687  ops/s
[info] FlatMapConcatBenchmark.normalFuture            thrpt    5    312508.444 ±   45271.476  ops/s
[info] FlatMapConcatBenchmark.normalFutureP1          thrpt    5    327743.914 ±   29201.871  ops/s
[info] FlatMapConcatBenchmark.oneElementList          thrpt    5   1162074.042 ±   78000.108  ops/s
[info] FlatMapConcatBenchmark.oneElementListP1        thrpt    5  10988702.693 ±  812309.621  ops/s
[info] FlatMapConcatBenchmark.sourceDotSingle         thrpt    5  11460931.634 ± 1107362.582  ops/s
[info] FlatMapConcatBenchmark.sourceDotSingleP1       thrpt    5  11313667.518 ± 1521344.870  ops/s

jmh:run -i 5 -wi 5 -f1 -t1  org.apache.pekko.stream.FlatMapConcatBenchmark

@He-Pin He-Pin added the t:stream Pekko Streams label Jan 10, 2025
@He-Pin He-Pin added this to the 1.2.0 milestone Jan 10, 2025
@He-Pin He-Pin requested review from pjfanning, mdedetrich, raboof, Roiocam, jrudolph and nvollmar and removed request for pjfanning January 10, 2025 13:16
@He-Pin He-Pin changed the title Add flatmapConcat with parallelism support feat: Add flatmapConcat with parallelism support Jan 10, 2025
@He-Pin He-Pin force-pushed the flatMapConcat branch 2 times, most recently from 5c723a0 to 77ac9a1 Compare January 10, 2025 13:32
@He-Pin
Copy link
Member Author

He-Pin commented Jan 10, 2025

I hate 2.12.x

@He-Pin He-Pin added the performance Related to performance label Jan 10, 2025
@jrudolph
Copy link
Contributor

I wonder if it is really necessary, wouldn't it be easier to just prematerialize to get the same effect?

@He-Pin
Copy link
Member Author

He-Pin commented Jan 10, 2025

Yes, but I think it would be nice to have this out of box. otherwise will need a mapAsync and prematerialize and then flatmapConcat, and I think the current implementation will have better performance than that too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Related to performance t:stream Pekko Streams
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants