As a dev team, we want to introduce proper performance testing for the analysis pipeline.

[Endlessflow]

Context:

The FertiScan analysis pipeline, which includes image mergin, OCR and GPT model processing, lacks adequate performance testing. This makes it difficult to assess the efficiency and accuracy of the pipeline's outputs. Currently, there is no dedicated framework for performance testing, and the code is intertwined with other parts of the application, making it challenging to isolate and test modifications.

We need to accurately assess the state of our pipeline and understand the impact of new modifications on performance. Migrating the pipeline code to a dedicated repository and implementing performance testing will help identify bottlenecks and optimize processing time and accuracy.

Desired Outcomes:

Code Migration

• The analysis pipeline code is successfully migrated to its own repository.
• Integration with the main application is seamless and well-documented.

Performance Testing Framework

• The testing framework accurately assesses the time required to run the pipeline, both for each component and end-to-end.
• The testing framework accurately assesses the accuracy of the pipeline outputs, both for each component and end-to-end.

Documentation

• Detailed documentation of the performance testing process, benchmarks, and results.
• Guidelines for running performance tests and interpreting the results are provided.

Tasks

• Implement Basic Performance Testing Framework #18

[k-allagbe]

Is this complete? Do we want to add the aggregation issue(s) here?