Jaeger query to ES failed - Internal Server Error - [type=search_phase_execution_exception]

[rdoshi29]

I am running jaeger in a kubernetes pod, and storing jaeger data in elasticsearch backend.
The connection to elasticsearch is successful and jaeger-service-* and jaeger-span-* indices are getting created on elasticsearch.

I am getting this error same as the below github issue every time I open Jaeger UI to search traces. The solution mentioned in the below link did not work.
jaeger query es failed
Jaeger pod logs
{"level":"error","ts":1675245520.6951814,"caller":"app/http_handler.go:487","msg":"HTTP handler, Internal Server Error","error":"search services failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:487\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getServices\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:158\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\tgithub.com/opentracing-contrib/[email protected]/nethttp/server.go:154\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/[email protected]/mux.go:210\ngithub.com/jaegertracing/jaeger/cmd/query/app.additionalHeadersHandler.func1\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/gorilla/handlers.CompressHandlerLevel.func1\n\tgithub.com/gorilla/[email protected]/compress.go:141\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\tgithub.com/gorilla/[email protected]/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2878\nnet/http.(*conn).serve\n\tnet/http/server.go:1929"}

[smarri4]

I get the same error too for a fresh deployment of Jaeger using helm charts. All my efforts of trying to enable debug log level in helm charts were futile. Would be helful if anyone can help enable debug at the very least. Below is my values.yaml:

jaeger:
provisionDataStore:
cassandra: false
storage:
type: elasticsearch
elasticsearch:
host: elasticsearch.monitoring.svc.cluster.local
port: 9200
usePassword: false
extraEnv:
- name: ES_VERSION
value: "7"
- name: ES_CREATE_INDEX_TEMPLATES
value: "false"
cassandra:
schemaJobEnabled: false
query:
nodeSelector:
process: CPU
serviceMonitor:
enabled: true
extraEnv:
- name: LOG-LEVEL
value: debug
cmdlineParams:
es:
log-level: debug

collector:
nodeSelector:
process: CPU
serviceMonitor:
enabled: true
extraEnv:
- name: LOG-LEVEL
value: debug
cmdlineParams:
es:
log-level: debug

agent:
nameOverride: "jaeger-tracing-agent"
nodeSelector:
process: CPU
serviceMonitor:
enabled: true
extraEnv:
- name: LOG-LEVEL
value: debug
cmdlineParams:
es:
log-level: debug

spark:
enabled: true
nodeSelector:
process: CPU

esIndexCleaner:
enabled: true
nodeSelector:
process: CPU

hotrod:
enabled: true
replicaCount: 1
image:
repository: jaegertracing/example-hotrod
pullPolicy: IfNotPresent
pullSecrets: []
service:
annotations: {}
name: hotrod
type: ClusterIP
loadBalancerSourceRanges: []
port: 80
extraEnv:
- name: LOG-LEVEL
value: debug