[CI] EsqlSpecIT test {lookup-join.LookupIPFromIndexKeep ASYNC} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #13857 / part3
• elasticsearch-intake #13811 / part3

Reproduction Line:

./gradlew ":x-pack:plugin:esql:qa:server:multi-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.multi_node.EsqlSpecIT" -Dtests.method="test {lookup-join.LookupIPFromIndexKeep ASYNC}" -Dtests.seed=BE0A9C1F9174B2FD -Dtests.locale=fr-MQ -Dtests.timezone=Asia/Riyadh -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Data mismatch:
row 0 column 0:a list containing
row 0 column 0:0: expected "2023-10-23T13:55:01.543Z" but was "2023-10-23T12:27:28.948Z"
row 0 column 1:a list containing
row 0 column 1:0: expected "172.21.3.15" but was "172.21.2.113"
row 0 column 2:a list containing
row 0 column 2:0: expected "1756467" but was "2764889"
row 0 column 3:a list containing
row 0 column 3:0: expected "Connected to 10.1.0.1" but was "Connected to 10.1.0.2"
row 0 column 4:a list containing
row 0 column 4:0: expected "Production" but was "QA"
row 1 column 0:a list containing
row 1 column 0:0: expected "2023-10-23T13:53:55.832Z" but was "2023-10-23T12:15:03.360Z"
row 1 column 1:a list containing
row 1 column 1:0: expected "172.21.3.15" but was "172.21.2.162"
row 1 column 2:a list containing
row 1 column 2:0: expected "5033755" but was "3450233"
row 1 column 3:a list containing
row 1 column 3:0: expected "Connection error" but was "Connected to 10.1.0.3"
row 1 column 4:a list containing
row 1 column 4:0: expected "Production" but was "QA"
row 2 column 0:a list containing
row 2 column 0:0: expected "2023-10-23T13:52:55.015Z" but was "2023-10-23T13:55:01.543Z"

Issue Reasons:

• [main] 2 failures in test test {lookup-join.LookupIPFromIndexKeep ASYNC} (1.3% fail rate in 149 executions)
• [main] 2 failures in step part3 (6.9% fail rate in 29 executions)
• [main] 2 failures in pipeline elasticsearch-intake (6.9% fail rate in 29 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test test {lookup-join.LookupIPFromIndexKeep ASYNC} (1.3% fail rate in 149 executions)
• [main] 2 failures in step part3 (6.9% fail rate in 29 executions)
• [main] 2 failures in pipeline elasticsearch-intake (6.9% fail rate in 29 executions)

Build Scans:

• elasticsearch-intake #13857 / part3
• elasticsearch-intake #13811 / part3

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)