Skip to content

Commit

Permalink
1
Browse files Browse the repository at this point in the history
  • Loading branch information
@zxealous @liugddx
zxealous authored and liugddx committed Feb 19, 2024
1 parent b0d724d commit 826b118
Show file tree
Hide file tree
Showing 6 changed files with 120 additions and 86 deletions.
4 changes: 3 additions & 1 deletion docker/thirdparties/docker-compose/elasticsearch/scripts/data/data1.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,7 @@
"c_person": [
{"name": "Andy", "age": 18},
{"name": "Tim", "age": 28}
]
],
"my_wildcard": "This string can be quite lengthy",
"level": "debug"
}
7 changes: 7 additions & 0 deletions docker/thirdparties/docker-compose/elasticsearch/scripts/index/es7_test1.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,13 @@
"ignore_above": 256
}
}
},
"my_wildcard": {
"type": "wildcard"
},
"level": {
"type": "constant_keyword",
"value": "debug"
}
}
}
Expand Down
66 changes: 34 additions & 32 deletions docs/en/docs/lakehouse/multi-catalog/es.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,45 +50,47 @@ After switching to the ES Catalog, you will be in the `dafault_db` so you don't

### Parameter Description

| Parameter | Required or Not | Default Value | Description |
| ---------------------- | --------------- | ------------- | ------------------------------------------------------------ |
| `hosts` | Yes | | ES address, can be one or multiple addresses, or the load balancer address of ES |
| `user` | No | Empty | ES username |
| `password` | No | Empty | Password of the corresponding user |
| `doc_value_scan` | No | true | Whether to obtain value of the target field by ES/Lucene columnar storage |
| `keyword_sniff` | No | true | Whether to sniff the text.fields in ES based on keyword; If this is set to false, the system will perform matching after tokenization. |
| Parameter | Required or Not | Default Value | Description |
|------------------------|-----------------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------|
| `hosts` | Yes | | ES address, can be one or multiple addresses, or the load balancer address of ES |
| `user` | No | Empty | ES username |
| `password` | No | Empty | Password of the corresponding user |
| `doc_value_scan` | No | true | Whether to obtain value of the target field by ES/Lucene columnar storage |
| `keyword_sniff` | No | true | Whether to sniff the text.fields in ES based on keyword; If this is set to false, the system will perform matching after tokenization. |
| `nodes_discovery` | No | true | Whether to enable ES node discovery, set to true by default; set to false in network isolation environments and only connected to specified nodes |
| `ssl` | No | false | Whether to enable HTTPS access mode for ES, currently follows a "Trust All" method in FE/BE |
| `mapping_es_id` | No | false | Whether to map the `_id` field in the ES index |
| `like_push_down` | No | true | Whether to transform like to wildcard push down to es, this increases the cpu consumption of the es. |
| `include_hidden_index` | No | false | Whether to include hidden index, default to false. |
| `ssl` | No | false | Whether to enable HTTPS access mode for ES, currently follows a "Trust All" method in FE/BE |
| `mapping_es_id` | No | false | Whether to map the `_id` field in the ES index |
| `like_push_down` | No | true | Whether to transform like to wildcard push down to es, this increases the cpu consumption of the es. |
| `include_hidden_index` | No | false | Whether to include hidden index, default to false. |

> 1. In terms of authentication, only HTTP Basic authentication is supported and it requires the user to have read privilege for the index and paths including `/_cluster/state/` and `_nodes/http` ; if you have not enabled security authentication for the cluster, you don't need to set the `user` and `password`.
>
> 2. If there are multiple types in the index in 5.x and 6.x, the first type is taken by default.
## Column Type Mapping

| ES Type | Doris Type | Comment |
| ------------- | ----------- |-------------------------------------------------------------------------|
| null | null | |
| boolean | boolean | |
| byte | tinyint | |
| short | smallint | |
| integer | int | |
| long | bigint | |
| unsigned_long | largeint | |
| float | float | |
| half_float | float | |
| double | double | |
| scaled_float | double | |
| date | date | Only support default/yyyy-MM-dd HH:mm:ss/yyyy-MM-dd/epoch_millis format |
| keyword | string | |
| text | string | |
| ip | string | |
| nested | string | |
| object | string | |
| other | unsupported | |
| ES Type | Doris Type | Comment |
|------------------|-------------|-------------------------------------------------------------------------|
| null | null | |
| boolean | boolean | |
| byte | tinyint | |
| short | smallint | |
| integer | int | |
| long | bigint | |
| unsigned_long | largeint | |
| float | float | |
| half_float | float | |
| double | double | |
| scaled_float | double | |
| date | date | Only support default/yyyy-MM-dd HH:mm:ss/yyyy-MM-dd/epoch_millis format |
| keyword | string | |
| text | string | |
| ip | string | |
| constant_keyword | string | |
| wildcard | string | |
| nested | string | |
| object | string | |
| other | unsupported | |

<version since="dev">

Expand Down Expand Up @@ -167,7 +169,7 @@ ES Catalogs support predicate pushdown to ES, which means only the filtered data
For the sake of optimization, operators will be converted into the following ES queries:

| SQL syntax | ES 5.x+ syntax |
| -------------- | :--------------------------: |
|----------------|:----------------------------:|
| = | term query |
| in | terms query |
| > , < , >= , ⇐ | range query |
Expand Down
66 changes: 34 additions & 32 deletions docs/zh-CN/docs/lakehouse/multi-catalog/es.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,26 +69,28 @@ CREATE CATALOG es PROPERTIES (
## 列类型映射

| ES Type | Doris Type | Comment |
|---|---|------------------------------------------------------------|
|null| null||
| boolean | boolean | |
| byte| tinyint| |
| short| smallint| |
| integer| int| |
| long| bigint| |
| unsigned_long| largeint | |
| float| float| |
| half_float| float| |
| double | double | |
| scaled_float| double | |
| date | date | 仅支持 default/yyyy-MM-dd HH:mm:ss/yyyy-MM-dd/epoch_millis 格式 |
| keyword | string | |
| text |string | |
| ip |string | |
| nested |string | |
| object |string | |
|other| unsupported ||
| ES Type | Doris Type | Comment |
|------------------|-------------|------------------------------------------------------------|
| null | null | |
| boolean | boolean | |
| byte | tinyint | |
| short | smallint | |
| integer | int | |
| long | bigint | |
| unsigned_long | largeint | |
| float | float | |
| half_float | float | |
| double | double | |
| scaled_float | double | |
| date | date | 仅支持 default/yyyy-MM-dd HH:mm:ss/yyyy-MM-dd/epoch_millis 格式 |
| keyword | string | |
| text | string | |
| ip | string | |
| constant_keyword | string | |
| wildcard | string | |
| nested | string | |
| object | string | |
| other | unsupported | |

<version since="dev">

Expand Down Expand Up @@ -161,18 +163,18 @@ ES Catalog 支持过滤条件的下推: 过滤条件下推给ES，这样只有
下面的操作符(Operators)会被优化成如下ES Query:
| SQL syntax | ES 5.x+ syntax |
|-------|:---:|
| = | term query|
| in | terms query |
| > , < , >= , ⇐ | range query |
| and | bool.filter |
| or | bool.should |
| not | bool.must_not |
| not in | bool.must_not + terms query |
| is\_not\_null | exists query |
| is\_null | bool.must_not + exists query |
| esquery | ES原生json形式的QueryDSL |
| SQL syntax | ES 5.x+ syntax |
|----------------|:----------------------------:|
| = | term query |
| in | terms query |
| > , < , >= , ⇐ | range query |
| and | bool.filter |
| or | bool.should |
| not | bool.must_not |
| not in | bool.must_not + terms query |
| is\_not\_null | exists query |
| is\_null | bool.must_not + exists query |
| esquery | ES原生json形式的QueryDSL |
### 启用列式扫描优化查询速度(enable\_docvalue\_scan=true)
Expand Down
Loading

0 comments on commit 826b118

Please sign in to comment.