[ENHANCEMENT] Enable QA To Create the Helm Chart with OpenSearch via GHA

[slolatte]

Describe the use case:

I am currently working on expanding the cross component test suite to cover OpenSearch. For context, please see the following ticket. However, I am running into some Helm Chart installation errors. I have followed the setup instructions in our documentation, and using the OpenSearch instance provided by infra.

Describe the enhancement/feature:

For more context, please see the following Slack thread.

Desired outcome and acceptance tests:

This will enable QA on a nightly basis and during release time, to test OpenSearch.

Tasks

Preview Give feedback

1. feat: adding OpenSearch test to SM nightly e2e tests c8-cross-component-e2e-tests#574
   +0
   
2. ci: ability to skip elasticsearch in integration tests #2764
   area/ci tool/github-actions version/8.6 +3
   
3. Clean up DB after nightly runs

[hamza-m-masood]

I will create an OpenSearch instance for QA team to use. The database must be wiped via a github actions workflow before every nightly run.

[hamza-m-masood]

OpenSearch instance is now created.
I will work on modifying the github actions workflow now.

[hamza-m-masood]

Please note: This github action workflow can only be run alone. It will not be possible to run many instances of this github action at the same time because the DB would need to be wiped before every run.

[hamza-m-masood]

I can trigger a run from the e2e QA repo with the correct values.yaml so the platform uses OpenSearch as a DB. The problem that I have just discovered is that our integration tests are not built to support DBs other than elasticsearch. I see that two pre-flight tests are failing because of the missing elasticsearch pods:

1. TEST-Liveness
2. TEST-Readiness

The core test suite will probably have more failing tests, but I can only see that when I fix the pre-flight tests.

Another PR must be created to fix these integration tests and support other databases.

[slolatte]

@hamza-m-masood Thanks for the update and for the support with this!

[hamza-m-masood]

PR created: #2764
It is a WIP

[hisImminence]

Removing the medic label as this is no medic task anymore but a prioritized issue.