[FEATURE] Externalize and Expose Test Utilities

[dannymeijer]

Is your feature request related to a problem? Please describe.

Currently, the test utilities and fixtures in the koheesio library's conftest.py files are not easily accessible for engineers who want to write tests in their own pipelines. This limits the reusability and efficiency of the existing test setups, making it harder for engineers to leverage the robust testing utilities already developed.

Describe the solution you'd like

I propose creating a set of testing utilities that externalize and expose the useful setups and fixtures from the conftest.py files. These utilities should be importable and documented, allowing engineers to easily integrate them into their own test suites. This would involve refactoring the current conftest.py setups into a separate module or package dedicated to testing utilities.

Describe alternatives you've considered

1. Copying and Pasting: Engineers could manually copy the necessary fixtures and setups from the conftest.py files into their own projects. However, this approach is error-prone and leads to code duplication.
2. Documentation: Providing detailed documentation on how to replicate the setups in conftest.py. While helpful, this still requires engineers to manually implement the setups, which is less efficient than having reusable utilities.

Additional context

The koheesio library has a comprehensive and well-structured test suite that includes various fixtures and setups for Spark, Delta Lake, and other components. By externalizing these utilities, we can enhance the testing capabilities of other projects and promote best practices in test writing. This initiative aligns with the principles of reusability and modularity in software development.