Skip to content

Commit

Permalink
GitBook: [#1415] add V2 engine page
Browse files Browse the repository at this point in the history
  • Loading branch information
sajjad-moradi authored and gitbook-bot committed Aug 31, 2022
1 parent c3111a2 commit 05afa4b
Show file tree
Hide file tree
Showing 5 changed files with 40 additions and 1 deletion.
Binary file added .gitbook/assets/image (51) (1).png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified .gitbook/assets/image (51).png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@
* [Ingestion Aggregations](developers/advanced/ingestion-level-aggregations.md)
* [Ingestion Transformations](developers/advanced/ingestion-level-transformations.md)
* [Null Value Support](developers/advanced/null-value-support.md)
* [V2 Multi-Stage Query Engine](developers/advanced/v2-multi-stage-query-engine.md)
* [Advanced Pinot Setup](developers/advanced/advanced-pinot-setup.md)
* [Plugins](developers/plugin-architecture/README.md)
* [Write Custom Plugins](developers/plugin-architecture/write-custom-plugins/README.md)
Expand Down
38 changes: 38 additions & 0 deletions developers/advanced/v2-multi-stage-query-engine.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# V2 Multi-Stage Query Engine

## Overview

The new multi-stage query engine (a.k.a V2 query engine) is designed to support more complex SQL semantics such as `JOIN`, `OVER` window, `MATCH_RECOGNIZE` and eventually, make Pinot support closer to full ANSI SQL semantics. 

![](https://www.datocms-assets.com/75153/1661707313-query-execution-model.png)-->![](https://www.datocms-assets.com/75153/1661533401-screenshot\_1.png)

It also resolves the bottleneck effect for the broker reduce stage where only a single machine is dedicated to perform heavy lifting such as high cardinality `GROUP BY` result merging; `ORDER BY` sorting, etc.

## How to use the V2 query engine

To enable the V2 engine, 

1. please make sure to either 
* [Building Apache Pinot](https://github.com/apache/pinot#building-pinot) using the latest master commit.
* Download the latest Apache Pinot docker image using the [official guide](https://docs.pinot.apache.org/basics/getting-started/running-pinot-in-docker).
2. Please add the following configurations to your cluster config:
* ```
"pinot.multistage.engine.enabled": "true",
"pinot.server.instance.currentDataTableVersion": "4",
"pinot.query.server.port": "8421",
"pinot.query.runner.port": "8442"
```
3. Start the cluster normally, you should see the following window in the controller query page:
* <img src="../../.gitbook/assets/image (51).png" alt="" data-size="original">
## Design Details
The overall PEP design doc and discussion can be found in the following links
* [PEP discussion Github Issue](https://github.com/apache/pinot/issues/8260) and
* [PEP design doc](https://docs.google.com/document/d/10-vL\_bUrI-Pi2oYudWyUlQl9Kf0cLrW-Z8hGczkCPik/edit)
2 changes: 1 addition & 1 deletion operators/operating-pinot/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,5 +33,5 @@ Here are some related blog posts from the Apache Pinot community. You can find a
<img src="../../.gitbook/assets/image (19).png" alt="" data-size="original"> [\
**Monitoring Apache Pinot with JMX, Prometheus and Grafana**](https://medium.com/apache-pinot-developer-blog/monitoring-apache-pinot-99034050c1a5)

<img src="../../.gitbook/assets/image (51).png" alt="" data-size="original"> **** \
<img src="../../.gitbook/assets/image (51) (1).png" alt="" data-size="original"> **** \
****[**Achieving 99th percentile latency SLA using Apache Pinot**](https://medium.com/apache-pinot-developer-blog/achieving-99th-percentile-latency-sla-using-apache-pinot-2ba4ce1d9eff)****

0 comments on commit 05afa4b

Please sign in to comment.