Supported format in QUICKWIT

[HeenaBansal2009]

I have a question
What all result formats quickwit supports. I read somewhere that quickwit supports only JSON as of now.
I would really appreciate if some can confirm the supported query results formats as well as the ingested data format for quickwit .

[fulmicoton]

Hello @HeenaBansal2009 . Right now the input has to be json, although we plan to support opentelemetry rapidly.

The query on the other hand is a simple search query language that allows boolean expressions.

[HeenaBansal2009]

@fulmicoton , Thanks for your quick response . However , I am curious to know if Quickwit FT search works with regexexp like
^clickhouse
clickhouse*
clickhouse%
clickhouse?
%clickhouse%

[fulmicoton]

This exists in tantivy but not in quickiwt yet.

[HeenaBansal2009]

Thank you @fulmicoton for confirmation.
I am able to ingest data in quickwit and search . However when I search using curl command , I am getting read async error.
What could go wrong here.
heena@Clickhouse1:~/quickwit-v0.2.1$ ./quickwit index search --index hackernews_5 --query Ambulance 2022-05-04T13:25:27.169Z ERROR quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads" 2022-05-04T13:25:27.171Z ERROR quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads" { "numHits": 1, "hits": [ { "by": [ "sgk284" ], "id": [ 2923885 ], "kids": [ 2923989, 2925247, 2924320, 2925442, 2924224, 2923994, 2924209, 2924702, 2925235, 2925010, 2924319, 2924638, 2925781, 2923943, 2924298 ], "score": [ 622 ], "text": [ "" ], "time": [ 1314251037 ], "title": [ "Icon Ambulance" ], "type": [ "story" ], "url": [ "https://plus.google.com/107117483540235115863/posts/gcSStkKxXTw" ] } ], "elapsedTimeMicros": 77324, "errors": [ "SplitSearchError { error: \"Internal error: An IO error occurred: 'Unsupported operation. StorageDirectory only supports async reads: \"ccf34dbac4614904b1124b751756dab8.term\"'.\", split_id: \"01G26NHMCV1BAP61AS006H7A75\", retryable_error: true }", "SplitSearchError { error: \"Internal error: An IO error occurred: 'Unsupported operation. StorageDirectory only supports async reads: \"6dc68fd1122c44a985ccf5348907c5f8.term\"'.\", split_id: \"01G26NK8YX0DM4YSVH6J9YD1GN\", retryable_error: true }" ] }
The output with curl command to search the same keyword.
heena@Clickhouse1:~/quickwit-v0.2.1$ curl "http://0.0.0.0:7280/api/v1/hackernews_5/search/stream?query=Ambulance&outputFormat=csv&fastField=id" curl: (18) transfer closed with outstanding read data remaining heena@Clickhouse1:~/quickwit-v0.2.1$

Attached the console logs when queried the commands ,This might be helpful

2022-05-04T13:24:03.927Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T13:24:13.927Z  INFO quickwit_serve::rest: search_stream index_id=hackernews_5 request=SearchStreamRequestQueryString { query: "google", search_fields: None, start_timestamp: None, end_timestamp: None, fast_field: "id", output_format: ClickHouseRowBinary, partition_by_field: None }
2022-05-04T13:24:13.927Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start: 2678183120, split_footer_end: 2678792526 }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T13:24:13.968Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:13.969Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:13.970Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start: 2678183120, split_footer_end: 2678792526 }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T13:24:13.972Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T13:24:14.006Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:14.006Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:14.007Z ERROR quickwit_serve::rest: Error when streaming search results. error=Internal error: `Internal error: `An IO error occurred: 'Unsupported operation. StorageDirectory only supports async reads: "ccf34dbac4614904b1124b751756dab8.term"'`.`.
2022-05-04T13:24:14.009Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T13:24:49.399Z  INFO quickwit_serve::rest: search_stream index_id=hackernews_5 request=SearchStreamRequestQueryString { query: "google.com", search_fields: None, start_timestamp: None, end_timestamp: None, fast_field: "id", output_format: Csv, partition_by_field: None }
2022-05-04T13:24:49.400Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start: 2678183120, split_footer_end: 2678792526 }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T13:24:49.442Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:49.442Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:49.443Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start: 2678183120, split_footer_end: 2678792526 }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T13:24:49.452Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T13:24:49.494Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:49.495Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:24:49.496Z ERROR quickwit_serve::rest: Error when streaming search results. error=Internal error: `Internal error: `An IO error occurred: 'Unsupported operation. StorageDirectory only supports async reads: "ccf34dbac4614904b1124b751756dab8.term"'`.`.
2022-05-04T13:24:49.503Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T13:26:29.659Z  INFO quickwit_serve::rest: search_stream index_id=hackernews_5 request=SearchStreamRequestQueryString { query: "Ambulance", search_fields: None, start_timestamp: None, end_timestamp: None, fast_field: "id", output_format: Csv, partition_by_field: None }
2022-05-04T13:26:29.661Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start: 2678183120, split_footer_end: 2678792526 }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T13:26:29.705Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:26:29.706Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:26:29.707Z  INFO search_adapter:leaf_search_stream: quickwit_search::service: leaf_search index="hackernews_5" splits=[SplitIdAndFooterOffsets { split_id: "01G26NHEB10T2DX37288EKX0SJ", split_footer_start: 270323695, split_footer_end: 278910648 }, SplitIdAndFooterOffsets { split_id: "01G26NHMCV1BAP61AS006H7A75", split_footer_start: 2678183120, split_footer_end: 2678792526 }, SplitIdAndFooterOffsets { split_id: "01G26NK8YX0DM4YSVH6J9YD1GN", split_footer_start: 349970236, split_footer_end: 350048435 }]
2022-05-04T13:26:29.713Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed
2022-05-04T13:26:29.756Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NHMCV1BAP61AS006H7A75}:warmup: quickwit_directories::storage_directory: path="ccf34dbac4614904b1124b751756dab8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:26:29.757Z ERROR search_adapter:leaf_search_stream:leaf_search_stream:leaf_search_stream_single_split{split_id=01G26NK8YX0DM4YSVH6J9YD1GN}:warmup: quickwit_directories::storage_directory: path="6dc68fd1122c44a985ccf5348907c5f8.term" msg="Unsupported operation. StorageDirectory only supports async reads"
2022-05-04T13:26:29.757Z ERROR quickwit_serve::rest: Error when streaming search results. error=Internal error: `Internal error: `An IO error occurred: 'Unsupported operation. StorageDirectory only supports async reads: "ccf34dbac4614904b1124b751756dab8.term"'`.`.
2022-05-04T13:26:29.761Z ERROR search_adapter:leaf_search_stream:leaf_search_stream: quickwit_search::search_stream::leaf: Failed to send leaf search stream result. Stop sending. Cause: channel closed

[fulmicoton]

I opened a bug in #1366.

Let's move the discussion there.

[HeenaBansal2009]

Thanks @fulmicoton . I really appreciate your response .

[HeenaBansal2009]

meta.json

{ "index_settings": { "docstore_compression": "lz4" }, "segments": [ { "segment_id": "6dc68fd1-122c-44a9-85cc-f5348907c5f8", "max_doc": 421980, "deletes": null } ], "schema": [ { "name": "id", "type": "u64", "options": { "indexed": true, "fieldnorms": false, "fast": "single", "stored": true } }, { "name": "type", "type": "text", "options": { "indexing": { "record": "basic", "fieldnorms": true, "tokenizer": "raw" }, "stored": true } }, { "name": "by", "type": "text", "options": { "indexing": { "record": "position", "fieldnorms": true, "tokenizer": "default" }, "stored": true } }, { "name": "time", "type": "i64", "options": { "indexed": true, "fieldnorms": false, "fast": "single", "stored": true } }, { "name": "text", "type": "text", "options": { "indexing": { "record": "position", "fieldnorms": true, "tokenizer": "raw" }, "stored": true } }, { "name": "kids", "type": "i64", "options": { "indexed": true, "fieldnorms": false, "fast": "multi", "stored": true } }, { "name": "url", "type": "text", "options": { "indexing": { "record": "position", "fieldnorms": true, "tokenizer": "default" }, "stored": true } }, { "name": "score", "type": "i64", "options": { "indexed": true, "fieldnorms": false, "fast": "single", "stored": true } }, { "name": "title", "type": "text", "options": { "indexing": { "record": "position", "fieldnorms": true, "tokenizer": "default" }, "stored": true } } ], "opstamp": 421981 }
metastore.json
`
{
"version": "0",
"index": {
"version": "1",
"index_id": "hackernews_5",
"index_uri": "file:///home/heena/quickwit-v0.2.1/qwdata/indexes/hackernews_5",
"checkpoint": {
".cli-ingest-source": {}
},
"doc_mapping": {
"field_mappings": [
{
"name": "id",
"type": "u64",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "type",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "raw",
"record": "basic"
},
{
"name": "by",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "default",
"record": "position"
},
{
"name": "time",
"type": "i64",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "text",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "raw",
"record": "position"
},
{
"name": "kids",
"type": "array",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "url",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "default",
"record": "position"
},
{
"name": "score",
"type": "i64",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "title",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "default",
"record": "position"
}
],
"tag_fields": [],
"store_source": false
},
"indexing_settings": {
"timestamp_field": "time",
"commit_timeout_secs": 60,
"split_num_docs_target": 10000000,
"merge_enabled": true,
"merge_policy": {
"demux_factor": 8,
"merge_factor": 10,
"max_merge_factor": 12
},
"resources": {
"num_threads": 1,
"heap_size": 2000000000
}
},
"search_settings": {
"default_search_fields": [
"title",
"by",
"text",
"url"
]
},
"create_timestamp": 1651637961,
"update_timestamp": 1651638973
},
"splits": [
{
"split_state": "Published",
"update_timestamp": 1651638912,
"version": "1",
"split_id": "01G26NHEB10T2DX37288EKX0SJ",
"num_docs": 327347,
"size_in_bytes": 432879779,
"time_range": {
"start": 1314110908,
"end": 1593532186
},
"create_timestamp": 1651638908,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 270323695,
"end": 278910648
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26N674ASKNNVG8DQKDYKRJM",
"num_docs": 387410,
"size_in_bytes": 502431085,
"time_range": {
"start": 1253159097,
"end": 1584476765
},
"create_timestamp": 1651638541,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 312636135,
"end": 322041208
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26MQHW2BPQGPJJK3V44TJVJ",
"num_docs": 568590,
"size_in_bytes": 719620809,
"time_range": {
"start": 1160418111,
"end": 1510673702
},
"create_timestamp": 1651638064,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 438230039,
"end": 438328080
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26MYWBR1YCXASA89KZ8F8GW",
"num_docs": 411217,
"size_in_bytes": 519948004,
"time_range": {
"start": 1160418111,
"end": 1509492103
},
"create_timestamp": 1651638300,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 317238052,
"end": 326780516
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26N81RVV4BNM0C2MWER0Z78",
"num_docs": 478959,
"size_in_bytes": 625017918,
"time_range": {
"start": 1292822357,
"end": 1587103597
},
"create_timestamp": 1651638603,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 388813589,
"end": 388900337
}
},
{
"split_state": "Published",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26NHMCV1BAP61AS006H7A75",
"num_docs": 3678305,
"size_in_bytes": 4736677883,
"time_range": {
"start": 1160418111,
"end": 1591921222
},
"create_timestamp": 1651638906,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 2678183120,
"end": 2678792526
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26N0Q0X5PDV52YA6EKKBGFF",
"num_docs": 271265,
"size_in_bytes": 344296679,
"time_range": {
"start": 1216780920,
"end": 1511405528
},
"create_timestamp": 1651638358,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 212666409,
"end": 218978775
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26N2HPB28PTYA8TDTKRP8D6",
"num_docs": 279867,
"size_in_bytes": 361323563,
"time_range": {
"start": 1230934379,
"end": 1513359201
},
"create_timestamp": 1651638419,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 225477171,
"end": 232727176
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26N4CBC75E4TY1HNZMS7J9C",
"num_docs": 396428,
"size_in_bytes": 511474431,
"time_range": {
"start": 1241269791,
"end": 1516223095
},
"create_timestamp": 1651638481,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 321116808,
"end": 331051249
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26N9WDRQMEM9FQ68ZH04Y4G",
"num_docs": 307345,
"size_in_bytes": 400861275,
"time_range": {
"start": 1299809877,
"end": 1588657284
},
"create_timestamp": 1651638660,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 250380683,
"end": 257950016
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26NDS1DD9ZRN5GQB1F8HKPK",
"num_docs": 331782,
"size_in_bytes": 429910307,
"time_range": {
"start": 1306025856,
"end": 1590737498
},
"create_timestamp": 1651638788,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 270652419,
"end": 278863516
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26NFKNBXHCCJXPZSTNHBBY8",
"num_docs": 245442,
"size_in_bytes": 321793812,
"time_range": {
"start": 1310760863,
"end": 1591921222
},
"create_timestamp": 1651638846,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 200656781,
"end": 206628263
}
},
{
"split_state": "Published",
"update_timestamp": 1651638973,
"version": "1",
"split_id": "01G26NK8YX0DM4YSVH6J9YD1GN",
"num_docs": 421980,
"size_in_bytes": 559498837,
"time_range": {
"start": 1318868981,
"end": 1595630349
},
"create_timestamp": 1651638968,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 349970236,
"end": 350048435
}
}
]
}

{
"version": "0",
"index": {
"version": "1",
"index_id": "hackernews_5",
"index_uri": "file:///home/heena/quickwit-v0.2.1/qwdata/indexes/hackernews_5",
"checkpoint": {
".cli-ingest-source": {}
},
"doc_mapping": {
"field_mappings": [
{
"name": "id",
"type": "u64",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "type",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "raw",
"record": "basic"
},
{
"name": "by",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "default",
"record": "position"
},
{
"name": "time",
"type": "i64",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "text",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "raw",
"record": "position"
},
{
"name": "kids",
"type": "array",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "url",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "default",
"record": "position"
},
{
"name": "score",
"type": "i64",
"stored": true,
"fast": true,
"indexed": true
},
{
"name": "title",
"type": "text",
"stored": true,
"fast": false,
"tokenizer": "default",
"record": "position"
}
],
"tag_fields": [],
"store_source": false
},
"indexing_settings": {
"timestamp_field": "time",
"commit_timeout_secs": 60,
"split_num_docs_target": 10000000,
"merge_enabled": true,
"merge_policy": {
"demux_factor": 8,
"merge_factor": 10,
"max_merge_factor": 12
},
"resources": {
"num_threads": 1,
"heap_size": 2000000000
}
},
"search_settings": {
"default_search_fields": [
"title",
"by",
"text",
"url"
]
},
"create_timestamp": 1651637961,
"update_timestamp": 1651638973
},
"splits": [
{
"split_state": "Published",
"update_timestamp": 1651638912,
"version": "1",
"split_id": "01G26NHEB10T2DX37288EKX0SJ",
"num_docs": 327347,
"size_in_bytes": 432879779,
"time_range": {
"start": 1314110908,
"end": 1593532186
},
"create_timestamp": 1651638908,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 270323695,
"end": 278910648
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26N674ASKNNVG8DQKDYKRJM",
"num_docs": 387410,
"size_in_bytes": 502431085,
"time_range": {
"start": 1253159097,
"end": 1584476765
},
"create_timestamp": 1651638541,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 312636135,
"end": 322041208
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26MQHW2BPQGPJJK3V44TJVJ",
"num_docs": 568590,
"size_in_bytes": 719620809,
"time_range": {
"start": 1160418111,
"end": 1510673702
},
"create_timestamp": 1651638064,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 438230039,
"end": 438328080
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26MYWBR1YCXASA89KZ8F8GW",
"num_docs": 411217,
"size_in_bytes": 519948004,
"time_range": {
"start": 1160418111,
"end": 1509492103
},
"create_timestamp": 1651638300,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 317238052,
"end": 326780516
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26N81RVV4BNM0C2MWER0Z78",
"num_docs": 478959,
"size_in_bytes": 625017918,
"time_range": {
"start": 1292822357,
"end": 1587103597
},
"create_timestamp": 1651638603,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 388813589,
"end": 388900337
}
},
{
"split_state": "Published",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26NHMCV1BAP61AS006H7A75",
"num_docs": 3678305,
"size_in_bytes": 4736677883,
"time_range": {
"start": 1160418111,
"end": 1591921222
},
"create_timestamp": 1651638906,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 2678183120,
"end": 2678792526
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26N0Q0X5PDV52YA6EKKBGFF",
"num_docs": 271265,
"size_in_bytes": 344296679,
"time_range": {
"start": 1216780920,
"end": 1511405528
},
"create_timestamp": 1651638358,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 212666409,
"end": 218978775
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26N2HPB28PTYA8TDTKRP8D6",
"num_docs": 279867,
"size_in_bytes": 361323563,
"time_range": {
"start": 1230934379,
"end": 1513359201
},
"create_timestamp": 1651638419,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 225477171,
"end": 232727176
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26N4CBC75E4TY1HNZMS7J9C",
"num_docs": 396428,
"size_in_bytes": 511474431,
"time_range": {
"start": 1241269791,
"end": 1516223095
},
"create_timestamp": 1651638481,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 321116808,
"end": 331051249
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26N9WDRQMEM9FQ68ZH04Y4G",
"num_docs": 307345,
"size_in_bytes": 400861275,
"time_range": {
"start": 1299809877,
"end": 1588657284
},
"create_timestamp": 1651638660,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 250380683,
"end": 257950016
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26NDS1DD9ZRN5GQB1F8HKPK",
"num_docs": 331782,
"size_in_bytes": 429910307,
"time_range": {
"start": 1306025856,
"end": 1590737498
},
"create_timestamp": 1651638788,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 270652419,
"end": 278863516
}
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1651638945,
"version": "1",
"split_id": "01G26NFKNBXHCCJXPZSTNHBBY8",
"num_docs": 245442,
"size_in_bytes": 321793812,
"time_range": {
"start": 1310760863,
"end": 1591921222
},
"create_timestamp": 1651638846,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 200656781,
"end": 206628263
}
},
{
"split_state": "Published",
"update_timestamp": 1651638973,
"version": "1",
"split_id": "01G26NK8YX0DM4YSVH6J9YD1GN",
"num_docs": 421980,
"size_in_bytes": 559498837,
"time_range": {
"start": 1318868981,
"end": 1595630349
},
"create_timestamp": 1651638968,
"tags": [],
"demux_num_ops": 0,
"footer_offsets": {
"start": 349970236,
"end": 350048435
}
}
]
}
`