Inference API returning Unable to execute HTTP request: event executor terminated

[jeffvestal]

Elasticsearch Version

serverless

Installed Plugins

No response

Java Version

bundled

OS Version

serverless

Problem Description

Getting error

{
  "error": {
    "root_cause": [
      {
        "type": "exception",
        "reason": "Amazon Bedrock converse call failed"
      }
    ],
    "type": "exception",
    "reason": "Failed to send request from inference entity id [bedrock_cohere-cmd-r-plus_completion]",
    "caused_by": {
      "type": "exception",
      "reason": "org.elasticsearch.ElasticsearchException: Amazon Bedrock converse call failed",
      "caused_by": {
        "type": "exception",
        "reason": "Amazon Bedrock converse call failed",
        "caused_by": {
          "type": "sdk_client_exception",
          "reason": "Unable to execute HTTP request: event executor terminated",
          "caused_by": {
            "type": "rejected_execution_exception",
            "reason": "event executor terminated"
          }
        }
      }
    }
  },
  "status": 500
}

Steps to Reproduce

## Using Bedrock Service - Cohere Command R+ model
PUT _inference/completion/bedrock_cohere-cmd-r-plus_completion
{
    "service": "amazonbedrock",
    "service_settings": {
        "access_key": "${bedrock_access_key}",
        "secret_key": "${bedrock_secret_key}",
        "region": "us-east-1",
        "provider": "cohere",
        "model": "cohere.command-r-plus-v1:0"
    }
}

## Test inference api - streaming
POST _inference/completion/bedrock_cohere-cmd-r-plus_completion/
{
  "input": "How many male malard ducks fit in an american football field?"
}

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/search-inference-team (Team:Search - Inference)

[elasticsearchmachine]

Pinging @elastic/search-eng (Team:SearchOrg)

[jeffvestal]

Also, if I wait 5 minutes I get the error again

{
  "error": {
    "root_cause": [
      {
        "type": "exception",
        "reason": "Amazon Bedrock converse call failed"
      }
    ],
    "type": "exception",
    "reason": "Failed to send request from inference entity id [bedrock_cohere-cmd-r-plus_completion]",
    "caused_by": {
      "type": "exception",
      "reason": "org.elasticsearch.ElasticsearchException: Amazon Bedrock converse call failed",
      "caused_by": {
        "type": "exception",
        "reason": "Amazon Bedrock converse call failed",
        "caused_by": {
          "type": "sdk_client_exception",
          "reason": "Unable to execute HTTP request: event executor terminated",
          "caused_by": {
            "type": "rejected_execution_exception",
            "reason": "event executor terminated"
          }
        }
      }
    }
  },
  "status": 500
}

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)