Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split AutoTuner for Profiling and Qualification and Override BATCH_SIZE_BYTES #1471

Open
wants to merge 3 commits into
base: dev
Choose a base branch
from

Conversation

parthosa
Copy link
Collaborator

@parthosa parthosa commented Dec 18, 2024

Fixes #1470.
Fixes #1399.

This PR refactors the AutoTuner into different classes for the Qualification and Profiling tools. This separation allows us to provide recommendations based on the specific tool being used.

For example, with this change, the AutoTuner for the Qualification tool will now recommend BATCH_SIZE_BYTES as 1GB, while the AutoTuner for Profiling tool will recommend 2GB.

Changes:

  1. Refactored object AutoTuner into trait AutoTunerConfigsProvider:

    • Introduced concrete objects QualificationAutoTunerConfigsProvider and ProfilingAutoTunerConfigsProvider.
    • Overridden BATCH_SIZE_BYTES in QualificationAutoTunerConfigsProvider.
  2. Extended class AutoTuner to QualificationAutoTuner:

    • This can be to add logic specific to the Qualification AutoTuner in the future.
  3. Refactored AutoTunerSuite into BaseAutoTunerSuite:

    • Created test suites ProfilingAutoTunerSuite and QualificationAutoTunerSuite.
    • Added unit test for batch size in Qualification AutoTuner.
  4. Renamed the existing class QualificationAutoTuner to QualificationAutoTunerRunner:

    • This class was not an implementation of the Qualification AutoTuner but a wrapper for running the AutoTuner with the Qualification tool.

@parthosa parthosa added feature request New feature or request core_tools Scope the core module (scala) labels Dec 18, 2024
@parthosa parthosa self-assigned this Dec 18, 2024
@parthosa parthosa marked this pull request as ready for review December 18, 2024 17:29
amahussein
amahussein previously approved these changes Dec 20, 2024
Copy link
Collaborator

@amahussein amahussein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @parthosa
LGTME!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core_tools Scope the core module (scala) feature request New feature or request
Projects
None yet
2 participants