[FEA]: Improve Testing Coverage and Flexibility for Isolated Pipeline Stages

[cwharris]

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

High

Please provide a clear description of problem this feature solves

I want Morpheus to be able to detect issues in individual pipeline stages more accurately, even in complex workflows, because currently, our unit tests pass while example pipelines sometimes fail. This discrepancy likely stems from missing test coverage on isolated stages, Addressing this issue would reduce release risks and minimize troubleshooting time.

Describe your ideal solution

1. Flexible and Reusable Scaffolding for Isolated Stage Testing:

• Invest in scaffolding that facilitates testing stages in isolation. This would allow developers to test each stage independently without incurring significant setup costs, making it more likely isolated stage testing becomes a regular testing practice.

2. Automated Intermediate Data Capture and Replay System:

• Implement a system that injects sensors between nodes in the pipeline, capturing intermediate data to disk. This would enable:
  • Automatic generation of tests that use this intermediate data to identify which specific stages fail if a pipeline breaks.
  • Higher-level workflows, such as those written by Data Science (DS) teams, to effectively serve as implicit unit tests.
• This approach could be extended to external API calls, allowing for a replay mechanism where previously recorded request/response pairs are used to validate tests.

3. Benefits for Customers:

• If this system is designed with user-friendliness in mind, it could be useful for external customers as well. It would enable users to quickly identify and troubleshoot which stages may be responsible for issues in their pipelines.

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request