Support for Multiple scoring Metrics in OptunaSearchCV (similar to GridSearchCV)

[jckkvs]

Motivation

Currently, OptunaSearchCV only supports a single scoring metric, either a string or a callable. This limitation makes it challenging to evaluate models using multiple evaluation metrics simultaneously, which is a common requirement in complex machine learning tasks.

In contrast, GridSearchCV supports multiple scoring metrics by allowing a dictionary, list, or tuple as input for the scoring parameter. This flexibility is particularly useful for tasks where multiple metrics must be considered (e.g., accuracy and f1_macro for imbalanced classification problems).

Providing support for multiple scoring metrics in OptunaSearchCV would make it a more versatile tool for hyperparameter optimization and align its functionality more closely with GridSearchCV.

Description

I propose extending OptunaSearchCV to support multiple scoring metrics, following the approach used in GridSearchCV.

Key Features:

1. Enhanced scoring parameter:
   Allow scoring to accept:

   • A dictionary with metric names as keys and callables as values.
   • A list or tuple of unique metric names.
   • A callable returning a dictionary where keys are metric names and values are metric scores.

2. Refit behavior:
   Add a refit parameter to specify which metric should be used to select the best hyperparameters. This should mirror the behavior of GridSearchCV (e.g., refit="accuracy").

3. Comprehensive results:
   Include all specified scoring metrics in the cv_results_ attribute to provide a detailed evaluation of model performance for each metric.

Example Usage:

from sklearn.svm import SVC
from sklearn.datasets import load_iris
from sklearn.metrics import make_scorer, f1_score

# Load dataset
X, y = load_iris(return_X_y=True)

# Define model
clf = SVC()

# Define hyperparameter search space
param_distributions = {
    "C": [0.1, 1, 10],
    "kernel": ["linear", "rbf"]
}

# Define multiple scoring metrics
scoring = {
    "accuracy": "accuracy",
    "f1_macro": make_scorer(f1_score, average="macro"),
}

# OptunaSearchCV with multiple scoring metrics
optuna_search = OptunaSearchCV(
    estimator=clf,
    param_distributions=param_distributions,
    scoring=scoring,
    refit="accuracy",  # Use accuracy to select the best parameters
    cv=3,
)

# Fit the model
optuna_search.fit(X, y)

# Print results
print("Best parameters:", optuna_search.best_params_)
print("Best score:", optuna_search.best_score_)

Alternatives (optional)

No response

Additional context (optional)

I propose this feature because allowing multiple scoring metrics in OptunaSearchCV would make it easier to implement evaluation methods such as Tibshirani-Tibshirani and BBC-CV.

[nzw0301]

Here is my previous attempt: optuna/optuna#4340

[jckkvs]

@nzw0301
Sorry for late reply.

Thank you for sharing your code regarding multiple scoring.
I have reviewed it and found no major technical issues.
As you mentioned, I understand that this feature has relatively low urgency or importance.

Since the issue is labeled as contribution-welcome, could you clarify what kind of contributions you are expecting?

[nzw0301]

Thank you for your comment! My previous PR was closed because the core team (excluding myself) could not notice the PR so I've forgotten the PR to update it.

So I think sending the same changes to https://github.com/optuna/optuna-integration is expected for the team, I believe. I'm a bit busy nowadays, so anyone can takeover the pull request if interested in it.

[jckkvs]

Thank you for the clarification! While this is not an essential feature, I find it quite interesting and valuable. I'm also a bit busy at the moment, so I may not be able to act on it immediately, but I would like to take over the pull request.