Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

introduce common operations directory for OSB workloads #514

Merged
merged 5 commits into from
Dec 12, 2024
Merged

introduce common operations directory for OSB workloads #514

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7fe6c56
restructure common operations folder based on discussion with ian
OVI3D0 Nov 22, 2024
e735d34
refactor common ops changes based on discussion with ian
OVI3D0 Dec 2, 2024
e6ed9ec
rework common ops configuration
OVI3D0 Dec 4, 2024
81b43e1
add missing params and update geonames workload
OVI3D0 Dec 9, 2024
aefcaab
parameterize request-timeout and warmup-time-period
OVI3D0 Dec 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions common_operations/check_cluster_health.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"name": "check-cluster-health",
"operation": {
"operation-type": "cluster-health",
"index": "{{ index_name }}",
"request-params": {
"wait_for_status": "{{cluster_health | default('green')}}",
"wait_for_no_relocating_shards": "true"
},
"retry-until-success": true
}
}
5 changes: 5 additions & 0 deletions common_operations/common_workload_setup.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{{ benchmark.collect(parts="delete_index.json") }},
{{ benchmark.collect(parts="create_index.json") }},
{{ benchmark.collect(parts="check_cluster_health.json") }},
{{ benchmark.collect(parts="index_append.json") }},
{{ benchmark.collect(parts="force_merge.json") }}
OVI3D0 marked this conversation as resolved.
Show resolved Hide resolved
6 changes: 6 additions & 0 deletions common_operations/create_index.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"operation": {
"operation-type": "create-index",
"settings": {{index_settings | default(default_index_settings | default({})) | tojson}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, is default({})) necessary if the workload has nothing specified?

{% with default_index_settings={} %}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found that when I removed that final default({}) I was getting jinja2 templating errors, even if I specified default_settings={} in the geonames workload

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's necessary because when default_index_settings is not specified, the {} is needed for the JSON to be valid.

}
}
3 changes: 3 additions & 0 deletions common_operations/delete_index.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"operation": "delete-index"
}
25 changes: 25 additions & 0 deletions common_operations/force_merge.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
{
"operation": {
"operation-type": "force-merge",
"request-timeout": 7200{%- if max_num_segments is defined %},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you feel request_timeout should be parameterized here, so someone who wants to override it can do it? It's unlikely, but you never know.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm of the opinion that there's no harm in parameterizing most of the workload values. I added a param for request-timeout in the latest revision. Let me know what you think!

"max-num-segments": {{ max_num_segments | tojson }}
{%- endif %}
}
},
{
"name": "refresh-after-force-merge",
"operation": "refresh"
},
{
"name": "wait-until-merges-finish",
"operation": {
"operation-type": "index-stats",
"index": "_all",
"condition": {
"path": "_all.total.merges.current",
"expected-value": 0
},
"retry-until-success": true,
"include-in-reporting": false
}
}
10 changes: 10 additions & 0 deletions common_operations/index_append.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"operation": "index-append",
"warmup-time-period": 120,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar suggestion as the force merge request timeout.. If you do add parameters, the README's will need to be updated.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to the request-timeout comment, I added a param for this as well.

"clients": {{bulk_indexing_clients | default(8)}},
"ignore-response-error-level": "{{error_level | default('non-fatal')}}"
},
{
"name": "refresh-after-index",
"operation": "refresh"
}
266 changes: 22 additions & 244 deletions geonames/test_procedures/default.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,60 +3,9 @@
"description": "Indexes the whole document corpus using OpenSearch default settings. We only adjust the number of replicas as we benchmark a single node cluster and Benchmark will only start the benchmark if the cluster turns green. Document ids are unique so all index operations are append only. After that a couple of queries are run.",
"default": true,
"schedule": [
{
"operation": "delete-index"
},
{
"operation": {
"operation-type": "create-index",
"settings": {{index_settings | default({}) | tojson}}
}
},
{
"name": "check-cluster-health",
"operation": {
"operation-type": "cluster-health",
"index": "geonames",
"request-params": {
"wait_for_status": "{{cluster_health | default('green')}}",
"wait_for_no_relocating_shards": "true"
},
"retry-until-success": true
}
},
{
"operation": "index-append",
"warmup-time-period": 120,
"clients": {{bulk_indexing_clients | default(8)}},
"ignore-response-error-level": "{{error_level | default('non-fatal')}}"
},
{
"name": "refresh-after-index",
"operation": "refresh"
},
{
"operation": {
"operation-type": "force-merge",
"request-timeout": 7200
}
},
{
"name": "refresh-after-force-merge",
"operation": "refresh"
},
{
"name": "wait-until-merges-finish",
"operation": {
"operation-type": "index-stats",
"index": "_all",
"condition": {
"path": "_all.total.merges.current",
"expected-value": 0
},
"retry-until-success": true,
"include-in-reporting": false
}
},
{% with default_index_settings={} %}
{{ benchmark.collect(parts="../../common_operations/common_workload_setup.json") }},
{% endwith %}
{
"operation": "index-stats",
"warmup-iterations": {{ index_stats_warmup_iterations or warmup_iterations | default(500) | tojson }},
Expand Down Expand Up @@ -239,214 +188,43 @@
"name": "append-no-conflicts-index-only",
"description": "Indexes the whole document corpus using OpenSearch default settings. We only adjust the number of replicas as we benchmark a single node cluster and Benchmark will only start the benchmark if the cluster turns green. Document ids are unique so all index operations are append only.",
"schedule": [
{
"operation": "delete-index"
},
{
"operation": {
"operation-type": "create-index",
"settings": {{index_settings | default({}) | tojson}}
}
},
{
"name": "check-cluster-health",
"operation": {
"operation-type": "cluster-health",
"index": "geonames",
"request-params": {
"wait_for_status": "{{cluster_health | default('green')}}",
"wait_for_no_relocating_shards": "true"
},
"retry-until-success": true
}
},
{
"operation": "index-append",
"warmup-time-period": 120,
"clients": {{bulk_indexing_clients | default(8)}},
"ignore-response-error-level": "{{error_level | default('non-fatal')}}"
},
{
"operation": {
"operation-type": "force-merge",
"request-timeout": 7200
}
},
{
"name": "wait-until-merges-finish",
"operation": {
"operation-type": "index-stats",
"index": "_all",
"condition": {
"path": "_all.total.merges.current",
"expected-value": 0
},
"retry-until-success": true,
"include-in-reporting": false
}
}
{% with default_index_settings={} %}
{{ benchmark.collect(parts="../../common_operations/common_workload_setup.json") }}
{% endwith %}
]
},
{
"name": "append-sorted-no-conflicts",
"description": "Indexes the whole document corpus in an index sorted by country_code field in ascending order. Document ids are unique so all index operations are append only.",
"schedule": [
{
"operation": "delete-index"
},
{
"operation": {
"operation-type": "create-index",
"settings": {%- if index_settings is defined %} {{index_settings | tojson}} {%- else %} {
"index.sort.field": ["country_code.raw", "admin1_code.raw"],
"index.sort.order": ["asc", "asc"]
}{%- endif %}
}
},
{
"name": "check-cluster-health",
"operation": {
"operation-type": "cluster-health",
"index": "geonames",
"request-params": {
"wait_for_status": "{{cluster_health | default('green')}}",
"wait_for_no_relocating_shards": "true"
},
"retry-until-success": true
}
},
{
"operation": "index-append",
"warmup-time-period": 120,
"clients": {{bulk_indexing_clients | default(8)}},
"ignore-response-error-level": "{{error_level | default('non-fatal')}}"
},
{
"operation": {
"operation-type": "force-merge",
"request-timeout": 7200
}
},
{
"name": "wait-until-merges-finish",
"operation": {
"operation-type": "index-stats",
"index": "_all",
"condition": {
"path": "_all.total.merges.current",
"expected-value": 0
},
"retry-until-success": true,
"include-in-reporting": false
}
}
{% with default_index_settings={
"index.sort.field": ["country_code.raw", "admin1_code.raw"],
"index.sort.order": ["asc", "asc"]
} %}
{{ benchmark.collect(parts="../../common_operations/common_workload_setup.json") }}
{% endwith %}
]
},
{
"name": "append-fast-with-conflicts",
"description": "Indexes the whole document corpus using a setup that will lead to a larger indexing throughput than the default settings. Benchmark will produce duplicate ids in 25% of all documents (not configurable) so we can simulate a scenario with appends most of the time and some updates in between.",
"schedule": [
{
"operation": "delete-index"
},
{
"operation": {
"operation-type": "create-index",
"settings": {%- if index_settings is defined %} {{index_settings | tojson}} {%- else %} {
"index.refresh_interval": "30s",
"index.number_of_shards": {{number_of_shards | default(6)}},
"index.translog.flush_threshold_size": "4g"
}{%- endif %}
}
},
{
"name": "check-cluster-health",
"operation": {
"operation-type": "cluster-health",
"index": "geonames",
"request-params": {
"wait_for_status": "{{cluster_health | default('green')}}",
"wait_for_no_relocating_shards": "true"
},
"retry-until-success": true
}
},
{
"operation": "index-update",
"warmup-time-period": 45,
"clients": {{bulk_indexing_clients | default(8)}},
"ignore-response-error-level": "{{error_level | default('non-fatal')}}"
},
{
"operation": {
"operation-type": "force-merge",
"request-timeout": 7200
}
},
{
"name": "wait-until-merges-finish",
"operation": {
"operation-type": "index-stats",
"index": "_all",
"condition": {
"path": "_all.total.merges.current",
"expected-value": 0
},
"retry-until-success": true,
"include-in-reporting": false
}
}
{% with default_index_settings={
"index.refresh_interval": "30s",
"index.number_of_shards": number_of_shards | default(6),
"index.translog.flush_threshold_size": "4g"
} %}
{{ benchmark.collect(parts="../../common_operations/common_workload_setup.json") }}
{% endwith %}
]
},
{
"name": "significant-text",
"description": "Indexes the whole document corpus using OpenSearch default settings. We only adjust the number of replicas as we benchmark a single node cluster and Benchmark will only start the benchmark if the cluster turns green. Document ids are unique so all index operations are append only.",
"schedule": [
{
"operation": "delete-index"
},
{
"operation": {
"operation-type": "create-index",
"settings": {{index_settings | default({}) | tojson}}
}
},
{
"name": "check-cluster-health",
"operation": {
"operation-type": "cluster-health",
"index": "geonames",
"request-params": {
"wait_for_status": "{{cluster_health | default('green')}}",
"wait_for_no_relocating_shards": "true"
},
"retry-until-success": true
}
},
{
"operation": "index-append",
"warmup-time-period": 120,
"clients": {{bulk_indexing_clients | default(8)}}
},
{
"operation": {
"operation-type": "force-merge",
"request-timeout": 7200
}
},
{
"name": "wait-until-merges-finish",
"operation": {
"operation-type": "index-stats",
"index": "_all",
"condition": {
"path": "_all.total.merges.current",
"expected-value": 0
},
"retry-until-success": true,
"include-in-reporting": false
}
},
{% with default_index_settings={} %}
{{ benchmark.collect(parts="../../common_operations/common_workload_setup.json") }},
{% endwith %}
{
"operation": "significant_text_selective",
"warmup-iterations": {{ significant_text_selective_warmup_iterations or warmup_iterations | default(200) | tojson }},
Expand Down