feat(e2e): add comprehensive end-to-end tests and improve CI workflows

[lvlcn-t]

Motivation

This PR adds E2E tests to Sparrow to make sure its core features work as expected in (almost) real-world conditions. These tests simulate how the application is used and improving reliability and stability.

I didn’t include E2E tests for the target manager component to keep the PR manageable. Those will come in a follow-up PR. This one is already quite large because I included both the E2E tests and the framework setup in one place to avoid splitting discussions across multiple smaller PRs.

---

Changes

E2E Testing Framework

A new testing framework was added to make it easier to write and run E2E tests. It includes:

• Server Management Tools: Makes sure the sparrow is ready before tests start.
• Remote Config Testing: Handles tests for dynamic configurations.
• Modular Design: Makes it simple to add new tests / assertions.

New E2E Tests

Here are the tests added in this PR:

• Health Check Tests: Verify sparrows health check.
• Latency Tests: Verify sparrows latency check runs correctly.
• DNS Tests: Make sure DNS resolutions work.
• File Loader Tests: Test configuration loading from local files.
• Remote Config Loader Tests: Validate fetching and applying remote configurations.
• Check Reconfiguration Tests: Test how the app handles updates to configuration during runtime.

In the future, this framework might be extended to cover existing unit tests, but that may require major restructuring (e.g., moving tests into <package>_test or reorganizing packages).

---

Bug Fixes

This PR also fixes several issues:

• Fixed race conditions during shutdown and configuration updates.
• Added the OpenAPI version to make the API spec compliant.
• Fixed metrics API content type to return application/json instead of text/plain.

---

Refactoring and Code Simplifications

Some general improvements were made to simplify the code:

• Combined test workflow files to make CI/CD setups cleaner.
• Improved naming across the checks.
• Unexposed private functions and types for better encapsulation.

---

CI/CD Pipeline Updates

The CI/CD pipeline was improved with these changes:

• Renamed workflows to kebab-case for consistency.
• Separated go unit and go E2E tests into separate jobs for faster execution.
• Added tparse for cleaner test output formatting.
• Started uploading gosec results to GitHub’s security page and scheduled regular SAST scans.
• Added customizable logging options to test workflows.

---

Dependency Updates

The YAML library was updated from yaml.v3 to go-yaml because it’s actively maintained, and works better with modern Go versions.

---

Tests Done

All tests passed:

• Unit tests succeeded
• E2E tests succeeded

---

Next Steps

1. Add E2E tests for the target manager component in a future PR.
2. Explore extending the E2E framework to existing unit tests, which might need some bigger changes to the code structure.

[github-advanced-security]

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.