[Bug]: Cannot run delete-transform-group-by-terms in transform test procedure for the eventdata workload

[OVI3D0]

Describe the bug

When running the transform test procedure for eventdata, I get the following error when getting to the delete-transform-group-by-terms operation:

Running delete-transform-group-by-terms                                        [  0% done]
[ERROR] Cannot execute-test. Error in load generator [0]
        Cannot run task [delete-transform-group-by-terms]: 'request_start'

To reproduce

opensearch-benchmark execute-test --target-hosts={cluster} --workload=eventdata --kill-running-processes --test-procedure=transform

Personally, I tried this with a 1 node 2.10 cluster, and a 3 node 2.5 cluster. Both resulted in the same error

Relevant log output

2024-12-18 19:56:27,458 -not-actor-/PID:20048 osbenchmark.benchmark ERROR Cannot run subcommand [execute-test].
Traceback (most recent call last):
  File "/workplace/mikeovi/opensearch-benchmark/osbenchmark/benchmark.py", line 1024, in dispatch_sub_command
    execute_test(cfg, args.kill_running_processes)
  File "/workplace/mikeovi/opensearch-benchmark/osbenchmark/benchmark.py", line 746, in execute_test
    with_actor_system(test_execution_orchestrator.run, cfg)
  File "/workplace/mikeovi/opensearch-benchmark/osbenchmark/benchmark.py", line 773, in with_actor_system
    runnable(cfg)
  File "/workplace/mikeovi/opensearch-benchmark/osbenchmark/test_execution_orchestrator.py", line 381, in run
    raise e
  File "/workplace/mikeovi/opensearch-benchmark/osbenchmark/test_execution_orchestrator.py", line 378, in run
    pipeline(cfg)
  File "/workplace/mikeovi/opensearch-benchmark/osbenchmark/test_execution_orchestrator.py", line 69, in __call__
    self.target(cfg)
  File "/workplace/mikeovi/opensearch-benchmark/osbenchmark/test_execution_orchestrator.py", line 314, in benchmark_only
    return execute_test(cfg, external=True)
  File "/workplace/mikeovi/opensearch-benchmark/osbenchmark/test_execution_orchestrator.py", line 273, in execute_test
    raise exceptions.BenchmarkError(result.message, result.cause)
osbenchmark.exceptions.BenchmarkError: Error in load generator [0]

[OVI3D0]

I wanted to also add that I'm getting the same error in the http_logs workload when attempting the append-no-conflicts-index-reindex-only test procedure. Specifically when running the reindex operation!

[IanHoang]

AttributeError: 'BenchmarkAsyncOpenSearch' object has no attribute 'transform'

For event-data, seems like we need to update opensearch.transform.put_transform to opensearch.transform.put to match what the current API looks like. https://github.com/opensearch-project/opensearch-py/blob/7815c6abe8079e84b55e8790e25596b9f70e5b1d/opensearchpy/plugins/transforms.py#L24

[OVI3D0]

Tried matching up with the current API, but there seems to still be an issue with 'transforms'. Here is another snippet from my log file:

  File "/workplace/mikeovi/opensearch-benchmark/osbenchmark/worker_coordinator/runner.py", line 2200, in __call__
    await opensearch.transforms.put(id=id, body=body, params=params)

AttributeError: 'BenchmarkAsyncOpenSearch' object has no attribute 'transforms'

[IanHoang]

I'm getting it to work when using the path:

        await opensearch.plugins.transforms.put(id=transform_id, body=body, params=params)

However, getting a param or body 400 error with the request. Complaining about the "index" field, which might be related to "dest" being in the params.

[IanHoang]

Our documentation is currently lacking pages for transform API in OpenSearch Py https://opensearch-project.github.io/opensearch-py/api-ref.html

[IanHoang]

Created issue with opensearch-py repository to see opensearch-project/opensearch-py#872. The unittests did not capture this because it uses assert_called_once_with that checks if path to the API request was intereacted with and doesn't actually test the API call.

# from runner_test.py
opensearch.transform.put_transform.assert_called_once_with(transform_id=params["transform-id"], body=params["body"], defer_validation=params["defer-validation"])

[IanHoang]

We can find a short term fix for this. After applying the fix, we consider contributing to this opensearch-project/opensearch-py#872 (comment)