Skip to content

Commit

Permalink
adjust config, docs
Browse files Browse the repository at this point in the history
  • Loading branch information
clintropolis committed Oct 10, 2023
1 parent 785f9cc commit 692e3d3
Show file tree
Hide file tree
Showing 5 changed files with 20 additions and 16 deletions.
4 changes: 2 additions & 2 deletions docs/configuration/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -798,9 +798,9 @@ Support for 64-bit floating point columns was released in Druid 0.11.0, so if yo
Prior to version 0.13.0, Druid string columns treated `''` and `null` values as interchangeable, and numeric columns were unable to represent `null` values, coercing `null` to `0`. Druid 0.13.0 introduced a mode which enabled SQL compatible null handling, allowing string columns to distinguish empty strings from nulls, and numeric columns to contain null rows.

|Property|Description|Default|
|---|---|---|
|--------|-----------|-------|
|`druid.generic.useDefaultValueForNull`|Set to `false` to store and query data in SQL compatible mode. When set to `true` (legacy mode), `null` values will be stored as `''` for string columns and `0` for numeric columns.|`false`|
|`druid.generic.useThreeValueLogic`|Set to `true` to use SQL compatible three-value logic when processing Druid filters when `druid.generic.useDefaultValueForNull` is `false` and `druid.expressions.useStrictBooleans` is `true`. When set to `false` Druid uses 2 value logic for filter processing, even when `druid.generic.useDefaultValueForNull=false` and `druid.expressions.useStrictBooleans` is true.|`true`|
|`druid.generic.useThreeValueLogicForNativeFilters`|Set to `true` to use SQL compatible three-value logic when processing native Druid filters when `druid.generic.useDefaultValueForNull=false` and `druid.expressions.useStrictBooleans=true`. When set to `false` Druid uses 2 value logic for filter processing, even when `druid.generic.useDefaultValueForNull=false` and `druid.expressions.useStrictBooleans=true`. See [boolean handling](../querying/sql-data-types.md#boolean-logic) for more details|`true`|
|`druid.generic.ignoreNullsForStringCardinality`|When set to `true`, `null` values will be ignored for the built-in cardinality aggregator over string columns. Set to `false` to include `null` values while estimating cardinality of only string columns using the built-in cardinality aggregator. This setting takes effect only when `druid.generic.useDefaultValueForNull` is set to `true` and is ignored in SQL compatibility mode. Additionally, empty strings (equivalent to null) are not counted when this is set to `true`. |`false`|
This mode does have a storage size and query performance cost, see [segment documentation](../design/segments.md#handling-null-values) for more details.

Expand Down
2 changes: 2 additions & 0 deletions docs/querying/filters.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ sidebar_label: "Filters"
A filter is a JSON object indicating which rows of data should be included in the computation for a query. It’s essentially the equivalent of the WHERE clause in SQL.
Filters are commonly applied on dimensions, but can be applied on aggregated metrics, for example, see [Filtered aggregator](./aggregations.md#filtered-aggregator) and [Having filters](./having.md).

By default, Druid uses SQL compatible three-value logic when filters, see [boolean logic](./sql-data-types.md#boolean-logic) for more details.

Apache Druid supports the following types of filters.

## Selector filter
Expand Down
6 changes: 3 additions & 3 deletions docs/querying/sql-data-types.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,9 +155,9 @@ values are treated as zeroes. This was the default prior to Druid 28.0.0.
By default, Druid uses [SQL three-valued logic](https://en.wikipedia.org/wiki/Three-valued_logic#SQL) for filter processing
and boolean expression evaluation. This behavior relies on three settings:

* [`druid.generic.useDefaultValueForNull`](../configuration/index.md#sql-compatible-null-handling) set to false (default), a runtime property which allows NULL values to exist in numeric columns and expressions, and string typed columns to distinguish between NULL and the empty string
* [`druid.expressions.useStrictBooleans`](../configuration/index.md#expression-processing-configurations) set to true (default), a runtime property controls Druid's boolean logic mode for expressions, as well as coercing all expression boolean values to be represented with a 1 for true and 0 for false
* [`druid.generic.useThreeValueLogic`](../configuration/index.md#sql-compatible-null-handling) set to true (default), a runtime property which decouples three-value logic handling from `druid.generic.useDefaultValueForNull` and `druid.expressions.useStrictBooleans` for backwards compatibility with older versions of Druid that did not fully support SQL compatible null value logic handling
* [`druid.generic.useDefaultValueForNull`](../configuration/index.md#sql-compatible-null-handling) must be set to false (default), a runtime property which allows NULL values to exist in numeric columns and expressions, and string typed columns to distinguish between NULL and the empty string
* [`druid.expressions.useStrictBooleans`](../configuration/index.md#expression-processing-configurations) must be set to true (default), a runtime property controls Druid's boolean logic mode for expressions, as well as coercing all expression boolean values to be represented with a 1 for true and 0 for false
* [`druid.generic.useThreeValueLogicForNativeFilters`](../configuration/index.md#sql-compatible-null-handling) must be set to true (default), a runtime property which decouples three-value logic handling from `druid.generic.useDefaultValueForNull` and `druid.expressions.useStrictBooleans` for backwards compatibility with older versions of Druid that did not fully support SQL compatible null value logic handling

If any of these settings is configured with a non-default value, Druid will use two-valued logic for non-expression based filters. Expression based filters are controlled independently with `druid.expressions.useStrictBooleans`, which if set to false Druid will use two-valued logic for expressions.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ public static boolean sqlCompatible()
public static boolean useThreeValueLogic()
{
return NullHandling.sqlCompatible() &&
INSTANCE.isUseThreeValueLogic() &&
INSTANCE.isUseThreeValueLogicForNativeFilters() &&
ExpressionProcessing.useStrictBooleans();
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ public class NullValueHandlingConfig
{
private static final Logger LOG = new Logger(NullValueHandlingConfig.class);
public static final String NULL_HANDLING_CONFIG_STRING = "druid.generic.useDefaultValueForNull";
public static final String THREE_VALUE_LOGIC_CONFIG_STRING = "druid.generic.useThreeValueLogic";
public static final String THREE_VALUE_LOGIC_CONFIG_STRING = "druid.generic.useThreeValueLogicForNativeFilters";

//added to preserve backward compatibility
//and not count nulls during cardinality aggrgation over strings
Expand All @@ -38,16 +38,16 @@ public class NullValueHandlingConfig
@JsonProperty("useDefaultValueForNull")
private final boolean useDefaultValuesForNull;

@JsonProperty("useThreeValueLogic")
private final boolean useThreeValueLogic;
@JsonProperty("useThreeValueLogicForNativeFilters")
private final boolean useThreeValueLogicForNativeFilters;

@JsonProperty("ignoreNullsForStringCardinality")
private final boolean ignoreNullsForStringCardinality;

@JsonCreator
public NullValueHandlingConfig(
@JsonProperty("useDefaultValueForNull") Boolean useDefaultValuesForNull,
@JsonProperty("useThreeValueLogic") Boolean useThreeValueLogic,
@JsonProperty("useThreeValueLogicForNativeFilters") Boolean useThreeValueLogicForNativeFilters,
@JsonProperty("ignoreNullsForStringCardinality") Boolean ignoreNullsForStringCardinality
)
{
Expand All @@ -56,10 +56,12 @@ public NullValueHandlingConfig(
} else {
this.useDefaultValuesForNull = useDefaultValuesForNull;
}
if (useThreeValueLogic == null) {
this.useThreeValueLogic = Boolean.valueOf(System.getProperty(THREE_VALUE_LOGIC_CONFIG_STRING, "true"));
if (useThreeValueLogicForNativeFilters == null) {
this.useThreeValueLogicForNativeFilters = Boolean.valueOf(
System.getProperty(THREE_VALUE_LOGIC_CONFIG_STRING, "true")
);
} else {
this.useThreeValueLogic = useThreeValueLogic;
this.useThreeValueLogicForNativeFilters = useThreeValueLogicForNativeFilters;
}
if (ignoreNullsForStringCardinality == null) {
this.ignoreNullsForStringCardinality = Boolean.valueOf(System.getProperty(
Expand All @@ -85,7 +87,7 @@ public NullValueHandlingConfig(
StringUtils.format(docsBaseFormat, version, "null-values")
);
}
if (!this.useThreeValueLogic) {
if (!this.useThreeValueLogicForNativeFilters) {
LOG.warn(
"druid.generic.useThreeValueLogic set to 'false', we recommend using 'true' if using SQL to query Druid for the most SQL compliant behavior, see %s for details",
StringUtils.format(docsBaseFormat, version, "boolean-logic")
Expand All @@ -103,8 +105,8 @@ public boolean isUseDefaultValuesForNull()
return useDefaultValuesForNull;
}

public boolean isUseThreeValueLogic()
public boolean isUseThreeValueLogicForNativeFilters()
{
return useThreeValueLogic;
return useThreeValueLogicForNativeFilters;
}
}

0 comments on commit 692e3d3

Please sign in to comment.