Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add concurent compaction docs #15218

Merged
merged 16 commits into from
Oct 27, 2023
Merged

docs: add concurent compaction docs #15218

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
2e43bc9
split up compaction page into general info and a separate page for ma…
317brian Oct 3, 2023
09ce393
add concurrent compaction info
317brian Oct 19, 2023
95dce16
Merge branch 'master' into concurrent-compaction
317brian Oct 19, 2023
8168f7e
Merge branch 'master' into concurrent-compaction
317brian Oct 19, 2023
f4cf2fc
delete empty heading
317brian Oct 19, 2023
fabee9e
clean up manual compact page
317brian Oct 20, 2023
387fd2b
add apache license
317brian Oct 20, 2023
3caf1ae
fix typo
317brian Oct 20, 2023
3bf8ab2
change UI text
317brian Oct 23, 2023
a217887
Merge branch 'master' into concurrent-compaction
317brian Oct 23, 2023
b803299
Merge branch 'master' into concurrent-compaction
317brian Oct 25, 2023
293d12b
change naming to concurrent append and replace
317brian Oct 25, 2023
a60d3fe
words
317brian Oct 25, 2023
73a1255
remove context section
317brian Oct 25, 2023
94483cc
address comments from reviewers
317brian Oct 25, 2023
14c8c95
Update docs/data-management/automatic-compaction.md
317brian Oct 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 80 additions & 1 deletion docs/data-management/automatic-compaction.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ To get statistics by API, send a [`GET` request](../api-reference/automatic-comp

## Examples

The following examples demonstrate potential use cases in which auto-compaction may improve your Druid performance. See more details in [Compaction strategies](../data-management/compaction.md#compaction-strategies). The examples in this section do not change the underlying data.
The following examples demonstrate potential use cases in which auto-compaction may improve your Druid performance. See more details in [Compaction strategies](../data-management/compaction.md#compaction-guidelines). The examples in this section do not change the underlying data.

### Change segment granularity

Expand Down Expand Up @@ -203,6 +203,85 @@ The following auto-compaction configuration compacts updates the `wikipedia` seg
}
```

## Concurrent append and replace

:::info
Concurrent append and replace is an [experimental feature](../development/experimental.md) and is not currently available for SQL-based ingestion.
:::

If you enable automatic compaction, you can use concurrent append and replace to concurrently compact data as you ingest it for streaming and legacy JSON-based batch ingestion.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If you enable automatic compaction, you can use concurrent append and replace to concurrently compact data as you ingest it for streaming and legacy JSON-based batch ingestion.
This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress.


Setting up concurrent append and replace is a two-step process. The first is to update your datasource and the second is to update your ingestion job.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not exactly correct. It doesn't make a lot of sense to "update a datasource" unless you mean adding data to a datasource.

Moreover, we shouldn't even look at this as a two step process, rather as an opt-in behaviour. Any ingestion job that wants to run concurrently with other ingestion jobs needs to use the correct lock types.

Please see the other suggestion.


Using concurrent append and replace in the following scenarios can be beneficial:

- If the job with an `APPEND` task and the job with a `REPLACE` task have the same segment granularity. For example, when a datasource and its streaming ingestion job have the same granularity.
- If the job with an `APPEND` task has a finer segment granularity than the replacing job.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Setting up concurrent append and replace is a two-step process. The first is to update your datasource and the second is to update your ingestion job.
Using concurrent append and replace in the following scenarios can be beneficial:
- If the job with an `APPEND` task and the job with a `REPLACE` task have the same segment granularity. For example, when a datasource and its streaming ingestion job have the same granularity.
- If the job with an `APPEND` task has a finer segment granularity than the replacing job.
You can enable concurrent append and replace by ensuring the following:
- The append task (with `appendToExisting` set to `true`) has `taskLockType` set to `APPEND` in the task context.
- The replace task (with `appendToExisting` set to `false`) has `taskLockType` set to `REPLACE` in the task context.
- The segment granularity of the append task is equal to or finer than the segment granularity of the replace task.


We do not recommend using concurrent append and replace when the job with an `APPEND` task has a coarser granularity than the job with a `REPLACE` task. For example, if the `APPEND` job has a yearly granularity and the `REPLACE` job has a monthly granularity. The job that finishes second will fail.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This point should be in a note or warning block.

Two more points to call out are that:

At any point in time
- There can only be a single task that holds a `REPLACE` lock on a given interval of a datasource.
- There may be multiple tasks that hold `APPEND` locks on a given interval of a datasource and append data to that interval simultaneously.


### Configure concurrent append and replace

##### Update the compaction settings with the API

First, prepare your datasource for concurrent append and replace by setting its task lock type to `REPLACE`.
Add the `taskContext` like you would any other auto-compaction setting through the API:

```shell
curl --location --request POST 'http://localhost:8081/druid/coordinator/v1/config/compaction' \
--header 'Content-Type: application/json' \
--data-raw '{
"dataSource": "YOUR_DATASOURCE",
"taskContext": {
"taskLockType": "REPLACE"
}
}'
```

##### Update the compaction settings with the UI

In the **Compaction config** for a datasource, set **Allow concurrent compactions (experimental)** to **True**.

#### Add a task lock type to your ingestion job

Next, you need to configure the task lock type for your ingestion job. For streaming jobs, the context parameter goes in your supervisor spec. For legacy JSON-based batch ingestion, the context parameter goes in your ingestion spec. You can provide the context parameter through the API like any other parameter for a streaming ingestion or JSON-based batch ingestion or UI.

##### Add the task lock type through the API
Copy link
Contributor

@AmatyaAvadhanula AmatyaAvadhanula Oct 25, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we please explicitly add that a streaming supervisor spec must always have an APPEND lock when using concurrent append and replace?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added below in the append section that talks about lock type


Add the following JSON snippet to your supervisor or ingestion spec if you're using the API:

```json
"context": {
"taskLockType": LOCK_TYPE
}
```

The `LOCK_TYPE` depends on what you're trying to accomplish.

Set `taskLockType` to `REPLACE` if you're replacing data. For example, if you use any of the following partitioning types, use `REPLACE`:

- hash partitioning
- range partitioning
- dynamic partitioning with append to existing set to `false`

Set `taskLockType` to `APPEND` if dynamic partitioning with append to existing is set to `true`.

If you have multiple append jobs all targeting the same datasource and want them to run simultaneously, you need to also include the following context parameter:

```json
"useSharedLock": "true"
```

Keep in mind that `taskLockType` takes precedence over `useSharedLock`. Do not use it with `REPLACE` task locks.

##### Add a task lock using the Druid console

As part of the **Load data** wizard for classic batch (JSON-based ingestion) and streaming ingestion, you can configure the task lock type for the ingestion during the **Publish** step:

- If you set **Append to existing** to **True**, you can then set **Allow concurrent append tasks (experimental)** to **True**.
- If you set **Append to existing** to **False**, you can then set **Allow concurrent replace tasks (experimental)** to **True**.


## Learn more

See the following topics for more information:
Expand Down
146 changes: 9 additions & 137 deletions docs/data-management/compaction.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,10 @@ description: "Defines compaction and automatic compaction (auto-compaction or au
~ specific language governing permissions and limitations
~ under the License.
-->

Query performance in Apache Druid depends on optimally sized segments. Compaction is one strategy you can use to optimize segment size for your Druid database. Compaction tasks read an existing set of segments for a given time interval and combine the data into a new "compacted" set of segments. In some cases the compacted segments are larger, but there are fewer of them. In other cases the compacted segments may be smaller. Compaction tends to increase performance because optimized segments require less per-segment processing and less memory overhead for ingestion and for querying paths.

## Compaction strategies
## Compaction guidelines

There are several cases to consider compaction for segment optimization:

Expand All @@ -43,18 +44,20 @@ By default, compaction does not modify the underlying data of the segments. Howe

Compaction does not improve performance in all situations. For example, if you rewrite your data with each ingestion task, you don't need to use compaction. See [Segment optimization](../operations/segment-optimization.md) for additional guidance to determine if compaction will help in your environment.

## Types of compaction
## Choose your compaction type
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this heading aligns with the rest of headings.

Also, the type of compaction is not really much of a choice as say how partioning type is a choice (range or hashed or dynamic, where we are choosing three different paths that give you 3 different results).

We should just call this Ways to run compaction or something in a similar vein.


You can configure the Druid Coordinator to perform automatic compaction, also called auto-compaction, for a datasource. Using its [segment search policy](../design/coordinator.md#segment-search-policy-in-automatic-compaction), the Coordinator periodically identifies segments for compaction starting from newest to oldest. When the Coordinator discovers segments that have not been compacted or segments that were compacted with a different or changed spec, it submits compaction tasks for the time interval covering those segments.
Automatic compaction, also called auto-compaction, works in most use cases and should be your first option.

Automatic compaction works in most use cases and should be your first option. To learn more, see [Automatic compaction](../data-management/automatic-compaction.md).
The Coordinator uses its [segment search policy](../design/coordinator.md#segment-search-policy-in-automatic-compaction) to periodically identify segments for compaction starting from newest to oldest. When the Coordinator discovers segments that have not been compacted or segments that were compacted with a different or changed spec, it submits compaction tasks for the time interval covering those segments.

To learn more, see [Automatic compaction](../data-management/automatic-compaction.md).

In cases where you require more control over compaction, you can manually submit compaction tasks. For example:

- Automatic compaction is running into the limit of task slots available to it, so tasks are waiting for previous automatic compaction tasks to complete. Manual compaction can use all available task slots, therefore you can complete compaction more quickly by submitting more concurrent tasks for more intervals.
- You want to force compaction for a specific time range or you want to compact data out of chronological order.

See [Setting up a manual compaction task](#setting-up-manual-compaction) for more about manual compaction tasks.
See [Setting up a manual compaction task](./manual-compaction.md#setting-up-manual-compaction) for more about manual compaction tasks.

## Data handling with compaction

Expand Down Expand Up @@ -101,141 +104,10 @@ Druid only rolls up the output segment when `rollup` is set for all input segmen
See [Roll-up](../ingestion/rollup.md) for more details.
You can check that your segments are rolled up or not by using [Segment Metadata Queries](../querying/segmentmetadataquery.md#analysistypes).

## Setting up manual compaction

To perform a manual compaction, you submit a compaction task. Compaction tasks merge all segments for the defined interval according to the following syntax:

```json
{
"type": "compact",
"id": <task_id>,
"dataSource": <task_datasource>,
"ioConfig": <IO config>,
"dimensionsSpec": <custom dimensionsSpec>,
"transformSpec": <custom transformSpec>,
"metricsSpec": <custom metricsSpec>,
"tuningConfig": <parallel indexing task tuningConfig>,
"granularitySpec": <compaction task granularitySpec>,
"context": <task context>
}
```

|Field|Description|Required|
|-----|-----------|--------|
|`type`|Task type. Set the value to `compact`.|Yes|
|`id`|Task ID|No|
|`dataSource`|Data source name to compact|Yes|
|`ioConfig`|I/O configuration for compaction task. See [Compaction I/O configuration](#compaction-io-configuration) for details.|Yes|
|`dimensionsSpec`|When set, the compaction task uses the specified `dimensionsSpec` rather than generating one from existing segments. See [Compaction dimensionsSpec](#compaction-dimensions-spec) for details.|No|
|`transformSpec`|When set, the compaction task uses the specified `transformSpec` rather than using `null`. See [Compaction transformSpec](#compaction-transform-spec) for details.|No|
|`metricsSpec`|When set, the compaction task uses the specified `metricsSpec` rather than generating one from existing segments.|No|
|`segmentGranularity`|Deprecated. Use `granularitySpec`.|No|
|`tuningConfig`|[Tuning configuration](../ingestion/native-batch.md#tuningconfig) for parallel indexing. `awaitSegmentAvailabilityTimeoutMillis` value is not supported for compaction tasks. Leave this parameter at the default value, 0.|No|
|`granularitySpec`|When set, the compaction task uses the specified `granularitySpec` rather than generating one from existing segments. See [Compaction `granularitySpec`](#compaction-granularity-spec) for details.|No|
|`context`|[Task context](../ingestion/tasks.md#context)|No|

:::info
Note: Use `granularitySpec` over `segmentGranularity` and only set one of these values. If you specify different values for these in the same compaction spec, the task fails.
:::

To control the number of result segments per time chunk, you can set [`maxRowsPerSegment`](../ingestion/native-batch.md#partitionsspec) or [`numShards`](../ingestion/../ingestion/native-batch.md#tuningconfig).

:::info
You can run multiple compaction tasks in parallel. For example, if you want to compact the data for a year, you are not limited to running a single task for the entire year. You can run 12 compaction tasks with month-long intervals.
:::

A compaction task internally generates an `index` or `index_parallel` task spec for performing compaction work with some fixed parameters. For example, its `inputSource` is always the [`druid` input source](../ingestion/input-sources.md), and `dimensionsSpec` and `metricsSpec` include all dimensions and metrics of the input segments by default.

Compaction tasks typically fetch all [relevant segments](#compaction-io-configuration) prior to launching any subtasks, _unless_ the following properties are all set to non-null values. It is strongly recommended to set them to non-null values to maximize performance and minimize disk usage of the `compact` task:

- [`granularitySpec`](#compaction-granularity-spec), with non-null values for each of `segmentGranularity`, `queryGranularity`, and `rollup`
- [`dimensionsSpec`](#compaction-dimensions-spec)
- `metricsSpec`

Compaction tasks exit without doing anything and issue a failure status code in either of the following cases:

- If the interval you specify has no data segments loaded.
- If the interval you specify is empty.

Note that the metadata between input segments and the resulting compacted segments may differ if the metadata among the input segments differs as well. If all input segments have the same metadata, however, the resulting output segment will have the same metadata as all input segments.


### Example compaction task

The following JSON illustrates a compaction task to compact _all segments_ within the interval `2020-01-01/2021-01-01` and create new segments:

```json
{
"type": "compact",
"dataSource": "wikipedia",
"ioConfig": {
"type": "compact",
"inputSpec": {
"type": "interval",
"interval": "2020-01-01/2021-01-01"
}
},
"granularitySpec": {
"segmentGranularity": "day",
"queryGranularity": "hour"
}
}
```

`granularitySpec` is an optional field.
If you don't specify `granularitySpec`, Druid retains the original segment and query granularities when compaction is complete.

### Compaction I/O configuration

The compaction `ioConfig` requires specifying `inputSpec` as follows:

|Field|Description|Default|Required|
|-----|-----------|-------|--------|
|`type`|Task type. Set the value to `compact`.|none|Yes|
|`inputSpec`|Specification of the target [interval](#interval-inputspec) or [segments](#segments-inputspec).|none|Yes|
|`dropExisting`|If `true`, the task replaces all existing segments fully contained by either of the following:<br />- the `interval` in the `interval` type `inputSpec`.<br />- the umbrella interval of the `segments` in the `segment` type `inputSpec`.<br />If compaction fails, Druid does not change any of the existing segments.<br />**WARNING**: `dropExisting` in `ioConfig` is a beta feature. |false|No|
|`allowNonAlignedInterval`|If `true`, the task allows an explicit [`segmentGranularity`](#compaction-granularity-spec) that is not aligned with the provided [interval](#interval-inputspec) or [segments](#segments-inputspec). This parameter is only used if [`segmentGranularity`](#compaction-granularity-spec) is explicitly provided.<br /><br />This parameter is provided for backwards compatibility. In most scenarios it should not be set, as it can lead to data being accidentally overshadowed. This parameter may be removed in a future release.|false|No|

The compaction task has two kinds of `inputSpec`:

#### Interval `inputSpec`

|Field|Description|Required|
|-----|-----------|--------|
|`type`|Task type. Set the value to `interval`.|Yes|
|`interval`|Interval to compact.|Yes|

#### Segments `inputSpec`

|Field|Description|Required|
|-----|-----------|--------|
|`type`|Task type. Set the value to `segments`.|Yes|
|`segments`|A list of segment IDs.|Yes|

### Compaction dimensions spec

|Field|Description|Required|
|-----|-----------|--------|
|`dimensions`| A list of dimension names or objects. Cannot have the same column in both `dimensions` and `dimensionExclusions`. Defaults to `null`, which preserves the original dimensions.|No|
|`dimensionExclusions`| The names of dimensions to exclude from compaction. Only names are supported here, not objects. This list is only used if the dimensions list is null or empty; otherwise it is ignored. Defaults to `[]`.|No|

### Compaction transform spec

|Field|Description|Required|
|-----|-----------|--------|
|`filter`| The `filter` conditionally filters input rows during compaction. Only rows that pass the filter will be included in the compacted segments. Any of Druid's standard [query filters](../querying/filters.md) can be used. Defaults to 'null', which will not filter any row. |No|

### Compaction granularity spec

|Field|Description|Required|
|-----|-----------|--------|
|`segmentGranularity`|Time chunking period for the segment granularity. Defaults to 'null', which preserves the original segment granularity. Accepts all [Query granularity](../querying/granularities.md) values.|No|
|`queryGranularity`|The resolution of timestamp storage within each segment. Defaults to 'null', which preserves the original query granularity. Accepts all [Query granularity](../querying/granularities.md) values.|No|
|`rollup`|Enables compaction-time rollup. To preserve the original setting, keep the default value. To enable compaction-time rollup, set the value to `true`. Once the data is rolled up, you can no longer recover individual records.|No|

## Learn more

See the following topics for more information:
- [Segment optimization](../operations/segment-optimization.md) for guidance to determine if compaction will help in your case.
- [Manual compaction](./manual-compaction.md) for how to run a one-time compaction task
- [Automatic compaction](automatic-compaction.md) for how to enable and configure automatic compaction.

Loading