Skip to content

Commit

Permalink
initial release general stream slicing with count-based windows
Browse files Browse the repository at this point in the history
  • Loading branch information
PhilippGrulich authored Mar 27, 2019
1 parent 2d43dc6 commit 10f340e
Showing 1 changed file with 10 additions and 6 deletions.
16 changes: 10 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ This repository provides Scotty, a framework for efficient window aggregations f
- High performance window aggregation with stream slicing.
- Scales to thousands of concurrent windows.
- Support for Tumbling, Sliding, and Session Windows.
- Initial support for Count based windows.
- Out-of-order processing.
- Aggregate Sharing among all concurrent windows.
- Connector for [Apache Flink](https://flink.apache.org/).
Expand All @@ -24,10 +25,13 @@ KeyedScottyWindowOperator<Tuple, Tuple2<Integer, Integer>, Tuple2<Integer, Integ
new KeyedScottyWindowOperator<>(new SumWindowFunction());

// Add multiple windows to the same operator
windowOperator.addWindow(new TumblingWindow(1000));
windowOperator.addWindow(new SlidingWindow(1000,5000));
windowOperator.addWindow(new TumblingWindow(WindowMeasure.Time, 1000));
windowOperator.addWindow(new SlidingWindow(WindowMeasure.Time, 1000, 5000));
windowOperator.addWindow(new SessionWindow(1000));

// Add count based window
windowOperator.addWindow(new TumblingWindow(WindowMeasure.Count, 1000))

// Add operator to Flink job
stream.keyBy(0)
.process(windowOperator)
Expand All @@ -36,7 +40,7 @@ stream.keyBy(0)
### Benchmark:


Throughput in comparison to the Flink standard window operator (Window Buckets) for Sliding Windows:
Throughput in comparison to the Flink standard window operator (Window Buckets) for Sliding Event-Time Windows:
We fix the window size to 60 seconds and modify the slide size.
If the slide size gets smaller, Flink has to maintain a higher number of overlapping (concurrent) windows.

Expand All @@ -50,7 +54,7 @@ Throughput in comparison to Flink for concurrent Tumbling Windows:
We plan to extend our framework with the following features:

- Support for User-Defined windows
- Support for count-based windows and other window measures
- User-defined window measures
- Support for Refinements
- Connector for [Apache Beam](https://beam.apache.org/)
- Support of Flink Checkpoints and State Backends
Expand Down Expand Up @@ -90,7 +94,7 @@ In this paper, we present the first general stream slicing technique for window
```
@inproceedings{traub2019efficient,
title={Efficient Window Aggregation with General Stream Slicing},
author={Traub, Jonas and Grulich, Philipp and Cu{\'e}llar, Alejandro Rodr{\'\i}guez and Bre{\ss}, Sebastian and Katsifodimos, Asterios and Rabl, Tilmann and Markl, Volker},
author={Traub, Jonas and Grulich, Philipp M. and Cu{\'e}llar, Alejandro Rodr{\'\i}guez and Bre{\ss}, Sebastian and Katsifodimos, Asterios and Rabl, Tilmann and Markl, Volker},
booktitle={22th International Conference on Extending Database Technology (EDBT)},
year={2019}
}
Expand Down Expand Up @@ -133,7 +137,7 @@ solutions.
```
@inproceedings{traub2018scotty,
title={Scotty: Efficient Window Aggregation for out-of-order Stream Processing},
author={Traub, Jonas and Grulich, Philipp Marian and Cuellar, Alejandro Rodríguez and Breß, Sebastian and Katsifodimos, Asterios and Rabl, Tilmann and Markl, Volker},
author={Traub, Jonas and Grulich, Philipp M. and Cuellar, Alejandro Rodríguez and Breß, Sebastian and Katsifodimos, Asterios and Rabl, Tilmann and Markl, Volker},
booktitle={34th IEEE International Conference on Data Engineering (ICDE)},
year={2018}
}
Expand Down

0 comments on commit 10f340e

Please sign in to comment.