[pull] main from elastic:main #507
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![pull[bot]](https://avatars.githubusercontent.com/in/12910?s=60&v=4){kind=small}
Enable incremental bulk processing by default.
A few of today's REST handler implementations compute a new set of supported parameters on each request. This is needlessly inefficient since the set never changes. This commit fixes those implementations and adds assertions to verify that we are returning the exact same instance each time.
* validate index name in parser
…mall_withPlatformAgnosticVariant #113983
REST APIs which declare their supported parameters must consume exactly those parameters: consuming an unsupported parameter means that requests including that parameter will be rejected, whereas failing to consume a supported parameter means that this parameter has no effect and should be removed. This commit adds an assertion to verify that we are consuming the correct parameters. Closes #113854
It's possible to call `RecoveryTarget#restoreFileFromSnapshot` after the `RecoveryTarget` has been replaced with a new instance due to a retry, but before all its refs have been released. If the recovery was snapshot-based then in this situation we will have already transferred the permits to the new instance, so the assertion that this instance has permits will trip. This commit fixes the problem with a non-null placeholder value indicating that the recovery target was at least originally created while holding some snapshot recovery permits. Closes #96018
…rch.test.rest.ClientYamlTestSuiteIT #114013
…ses size (#113613) Final (I wish) part of #99815 Also, fixes #113916 ## Steps 1. Migrate TDigest classes to use a custom Array implementation. Temporarily use a simple array wrapper (#112810) 2. Implement CircuitBreaking in the `WrapperTDigestArrays` class. Add Releasable/AutoCloseable and ensure everything is closed (#113105) 3. Pass the CircuitBreaker as a parameter to TDigestState from wherever it's being used (#113387) - ESQL: Pass a real CB - Other aggs: Use the deprecated methods on `TDigestState`, that will use a No-op CB instead 4. Account remaining TDigest classes size ("SHALLOW_SIZE") (This PR) Every step should be safely mergeable to main: - The first and second steps should have no impact. - The third and fourth ones will start increasing the CB count partially. ## Remarks As TDigests are releasable now, I had to refactor all tests, adding try-with-resources or direct close() calls. That added a lot of changes, but most of them are trivial. Outside of it, in ESQL, TDigestStates are closed now. Old aggregations don't close them, as it's not trivial. However, as they are using the NoopCircuitBreaker, there's no problem with it. There's nothing to be closed. ## _Remarks 2_ I tried to follow the same pattern in how everything is accounted. On each TDigest class: - Static constant "SHALLOW_SIZE" with the object weight - Field `AtomicBoolean closed` to ensure indempotent `close()` - Static `create()` method that accounts the SHALLOW_SIZE, and returns a new isntance. And the important part: On exception, it discounts SHALLOW_SIZE again - A `ramBytesUsed()` (Accountable interface), barely used for anything really, but some assertions I believe - A constructor, that closes everything it created on exception (If it creates an array, and the next array surpasses the CB limit, the first one must be closed) - And a close() that will, well, close everything and discount SHALLOW_SIZE A lot of steps to make sure everything works well in this multi-level structure, but I believe the result was quite clean
…st {yaml=rrf/700_rrf_retriever_search_api_compatibility/rrf retriever with top-level collapse} #114019
…ithTrainedModelAndInference #114023
Copy from the 8.x branch to main
* merging * copy elser service files into elasticsearch service * Add deprecation log message for elser service * improve deprecation warning * change elasticsearch internal service elser case to use elser model * switch elasticsearch elser tests to use elasticsearch elser * Update docs/changelog/113216.yaml * alias elser service to elasticsearch * delete elser service package now that elasticsearch service supports it and has aliased it * Add deprecation warning to infer API for elser * Fix accidentally introduced NPE and retain BWC support for null model ID (with deprecation message) * change "area" to "REST API" because "Machine Learning" isn't an option for deprecation * change elser literals to static variable * change Put and Elasticsearch Internal service to pass the service name if it is elser or elasticsearch this will allow the elasticsearch service to maintain BWC for null model IDs if the service was elser. * fix up tests to match new elasticsearch service semantics regarding elser. * Move passing of service name * add persistence for elser models in elasticsearch * copy elser service files into elasticsearch service * Add deprecation log message for elser service * Add deprecation warning to infer API for elser * fix merge conflicts * fix merge
This mentions EXPLAIN ANALYZE and EXPLAIN PLAN in the docs for ESQL's `profile` option. Those are things that folks from PostgreSQL and Oracle are used to and might search for. And `profile` is the closest thing we have to them. EXPLAIN PLAN doesn't run the query - it just tells you what the plan is. ESQL's `profile` always runs the query. So that's different. But it's close! EXPLAIN ANALYZE *does* run the query. It's pretty much the same.
* start to add deberta-v2 tokenizer classes * continue to add basic tokenizer stuff * Finish adding DeBERTa-2 tokenizer Still need to review & test * Complete test setup and linting * Update docs/changelog/111852.yaml * Add serialization of deberta tokenization * fix request buillder to match model * debugging * add balanced truncation * remove full vocabulary and use tiny vocab for tests * Remove TODO * precommit * Add named writables and known tokenizers * Add deberta to list of known tokenizers in test * Add tests for balanced tokenizer and fix errors in tokenizer logic * fix order of parameters passed to deberta * Add support for byte_fallback which is enabled for DeBERTa byte_fallback decomposes unknown tokens into multiple tokens each of one byte if those bytes are in the vocabulary. * precommit * update tests to account for byte decomposition * remove sysout * fix tests for byteFallback, for real this time * Move defaultSpanForChunking into super class to avoid repitition * simplify decomposeBytePieces --------- Co-authored-by: Elastic Machine <[email protected]>
Enables end-to-end streaming for OpenAI's chat completion API. Additional changes: - InferenceServiceResults now returns a wildcard, allowing for tests to work with classes that extend ChunkedToXContent. - StreamingChatCompletionResults now defines the Results structure so that future providers can reuse the same structure. - DelegatingProcessor now cancels the upstream if `next` throws an exception and forwards the exception downstream. - Moved the streaming handler from OpenAiChatCompletionResponseHandler to OpenAiResponseHandler so that Azure Open AI can reuse it. - OpenAiStreamingProcessor now iterates over the returned choices array, handling both OpenAI and Azure response formats. - SenderService declares a helper `Set<TaskType>` for implementations to reuse when enabling streaming. - Added an InferenceEventsAssertion test helper to fluidly assert responses from the mock webserver.
These were added in #113804 and missing the `false`/`false` test case.
Let's use a vendor neutral link.
Closes ES-9623
This PR tracks the total number of query and fetch failures, in addition to the existing metrics for each shard, and exposes them through the stats API.
Fixes a validation step that might prevent creating empty aggregations iif the output format does not allows negative numbers.
This change exposes query, fetch and indexing metrics for each index mode.
…3712) This commit is a follow up to #113151 to better clarify how to deprecate HTTP routes. That change introduced RouteBuilder.deprecateAndKeep to enable the ability to deprecate an HTTP API, but not require REST compatibilty headers in the next major version. This commit deprecates the java methnod RouteBuilder.deprecated and introduces RouteBuilder.deprecatedForRemoval. The valid options are now RouteBuilder.deprecateAndKeep vs. RouteBuilder.deprecatedForRemoval where the later will require compatiblity headers to access the route in any newer REST API versions than the lastFullySupportedVersion declared. The javadoc should help to provide some clarification. Additionally, the deprecation level should not be configurable. The deprecation level of WARN, when applied to routes, is informational only (and may require compatibility headers in the next version). The deprecation level of CRITICAL means means no access what so ever in the next major version. Generally, CRTICIAL is used for any cases where the compatibility is required (which means it is the last version for any access), or no compatibility is planned. Some examples: ``` Route.builder(GET, "/foo/bar").build() -> no deprecations Route.builder(GET, "/foo/bar").deprecateAndKeep("my message").build() -> deprecated, but removal is not influenced by REST API compatibilty Route.builder(GET, "/foo/bar").deprecatedForRemoval("my message", V_8).build() -> will be available in V_8, but emit a warn level deprecation, V_9 will require compatiblity headers and emit a CRITICAL deprecation, and V_10 this will no longer be available Route.builder(GET, "/foo/bar").replaces(GET, "/foo/baz", V_8).build() -> /foo/bar will be available in all versions. /foo/baz will be available in V_8 but emit a warn level deprecation, V_9 will require compatibility headers and emit a CRITICAL deprecation, and V_10 this will no longer be available. This is effectively a short cut to register a new route ("/foo/bar") and deprecatedForRemoval the path being replaced. ``` The functionality remains unchanged and this refactoring only provides better contracts and cleans up some of the implementation.
Today there are a handful of integer settings for `repository-s3` repositories whose docs link to the page about numeric field types. Yet these settings are not fields, and do not support floating-point values either. The convention throughout the rest of the docs is to just call these things `integer` without linking to anything. This commit aligns the `repository-s3` docs with this convention.
…operations (#114075) * Took time and cluster details get updated for coordinator only query operations The ComputeService.runCompute pathway for coordinator only operations (such as `FROM foo | LIMIT 0` or a ROW command) get updated with overall took time. This also includes support for cross-cluster coordinator only operations, which come about with queries like `FROM foo,remote:foo | LIMIT 0`. The _clusters metadata is now properly updated for those cases as well. Fixes #114014
This fixes tests so that they can work with multiple shards.
Adds a REVERSE string function
…ulkHighWatermarkBackOff #114073
It is possible in the incremental high watermark test that the data is submitted causing a corruption of the bulk request. This commit fixes the issue to ensure we only send new data after it has been requested. Additionally, it adds an assertion to prevent this error from happening again.
… UI (#114042) * Hide aggregated metrics generated for the APM UI * Update 30_aggregated_metrics_tests.yml * Review feedback - Introduced templates for 1 minute aggregations - Moved dynamic templates `ecs_ip` and `all_strings_to_keywords` into a dedicated file and now I pull the file in instead of repeating them - Introduced `metrics-[x]m.otel@custom` - Added tests with terms aggregation that assert by default 1 bucket (only 1m) with metricsset.interval, and with allowing hidden indices it's 3 buckets (1m, 10m, 60m) * Update 30_aggregated_metrics_tests.yml Simplify - no need to separate tests for the 3bucket queries. * Rename metrics-otel-fixed@mappings to ecs-tsdb@mappings --------- Co-authored-by: Elastic Machine <[email protected]>
…ore 8.16 (#114061) * Fix HuggingFaceMixedIT test sometimes failing when run on version before 8.16 * Fixing typo in expected error message
Re-add the `semantic_text.inner_hits` cluster feature to fix serverless test failures
…ulkLowWatermarkBackOff #114182
… test {yaml=aggregations/stats_metric_fail_formatting/fail formatting} #114187
…ticsearch.xpack.esql.action.EsqlActionBreakerIT #114194
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )