E2E Testing for Vector

[mjnagel]

E2E testing for Vector should only be run on the all package test only since it focuses on interactions between components - see #211 for some thoughts on this.

Initial thoughts on testing areas:

• Input handling: Ensure that Vector is picking up the expected files (pod + node logs), possibly via the API?
• Output handling: Ensure that Vector is able to ship to the destinations, possibly via the API or a test with Vector connected to MinIO and checking bucket contents?
• Other testing: This is not super well defined but we should evaluate if there are other health endpoints or metrics that would be helpful to look at

Vector's API is not enabled by default so we may need to enable it in CI at minimum and use it to access some of these data points.

[noahpb]

Evaluating the following:

• GraphQL API should be healthy
• Vector should be collecting node logs
• Vector should be collecting pod logs
• Loki Host sink should be receiving node logs
• Loki Pod sink should be receiving pod logs
• Vector should be able to ship logs to Loki

Noting that I will have to deploy a privileged pod to generate node logs as there are no node logs being generated by the time vector is deployed in CI. I will also have to expose a new zarf var on the vector package to toggle enabling the API.

[mjnagel]

Noting that I will have to deploy a privileged pod to generate node logs as there are no node logs being generated by the time vector is deployed in CI

This is surprising to me - there should be a very short log file for k3d startup?

[noahpb]

Noting that I will have to deploy a privileged pod to generate node logs as there are no node logs being generated by the time vector is deployed in CI

This is surprising to me - there should be a very short log file for k3d startup?

There is - about a dozen lines. But from what I found in testing, vector only collects logs that were generated after it starts up. I could be missing something though. Maybe there's a parameter I can add that allows me to search back in time before it was deployed.

[noahpb]

Discussed offline - found there is no need to manually generate node logs at this time.

[mjnagel]

Should also note - log generation for testing could be a good path if we want to test additional log scrape targets in the future, I've definitely seen similar tests in other projects where a pod will just generate some noise to ensure there is always something happening.

[noahpb]

Noting here context - we switched the vector e2e tests to querying loki directly during the loki e2e tests. We noticed that it took a while for vector metrics to populate which was causing tests to fail. Running the tests from loki ensures that same cases are validated and gives vector adequate time to collect logs.