Skip to content

Commit

Permalink
Merge pull request #122 from yrodiere/connection-pool-size
Browse files Browse the repository at this point in the history
Adjust indexing parallelism (DB connection pool size, indexing queue count)
  • Loading branch information
yrodiere authored Jan 8, 2024
2 parents 3cd1b19 + 4447235 commit b052cbd
Showing 1 changed file with 8 additions and 4 deletions.
12 changes: 8 additions & 4 deletions src/main/resources/application.properties
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,18 @@ quarkus.resteasy-reactive.path=/api
## after some inactivity; see https://github.com/quarkusio/search.quarkus.io/issues/100
quarkus.datasource.jdbc.url=jdbc:h2:mem:searchquarkusio;DB_CLOSE_DELAY=-1
quarkus.hibernate-orm.database.generation=drop-and-create
## We need a large enough connection pool to accommodate highly-parallel indexing.
quarkus.datasource.jdbc.max-size=100
## But we don't need it to be that large after indexing.
quarkus.datasource.jdbc.min-size=0
## Hibernate Search
quarkus.hibernate-search-orm.elasticsearch.version=opensearch:2.11
quarkus.elasticsearch.devservices.image-name=opensearch-custom-plugin:2.11.0
## Limit parallelism of indexing, because OpenSearch can only handle so many pending indexing requests.
## This leads to at most 6*40=240 documents being indexed in parallel, which should be plenty
## Limit parallelism of indexing, because OpenSearch can only handle so many documents in its buffers.
## This leads to at most 12*20=240 documents being indexed in parallel, which should be plenty
## given how large our documents can be.
quarkus.hibernate-search-orm.elasticsearch.indexing.queue-count=6
quarkus.hibernate-search-orm.elasticsearch.indexing.max-bulk-size=40
quarkus.hibernate-search-orm.elasticsearch.indexing.queue-count=12
quarkus.hibernate-search-orm.elasticsearch.indexing.max-bulk-size=20
## We need to apply a custom OpenSearch mapping to exclude very large fields from the _source
quarkus.hibernate-search-orm.elasticsearch.schema-management.mapping-file=indexes/mapping-template.json
quarkus.hibernate-search-orm.elasticsearch.schema-management.settings-file=indexes/settings-template.json
Expand Down

0 comments on commit b052cbd

Please sign in to comment.