Quantus evaluation for Lime

[Akshatha-Mohan]

from captum.attr import Saliency, IntegratedGradients, Lime

a_batch = Lime(model, interpretable_model=SkLearnLasso(alpha=0.08)).attribute(inputs=x_batch, target=y_batch, feature_mask=feature_mask.to(device).long(), n_samples=16, perturbations_per_eval=16, return_input_shape=True, show_progress=True).sum(axis=1).cpu().numpy()

xai_methods ={"Lime": a_batch}
results = evaluate_xai_methods(model, metrics_avg,
                        xai_methods,
                        x_batch,
                        y_batch,
                        s_batch)

def evaluate_xai_methods(
                         model, metrics: dict, 
                         xai_methods: dict, 
                         x_batch: np.ndarray, 
                         y_batch: np.ndarray,
                         s_batch: np.ndarray,
                         nr_samples: int = 1,
                         ):
    """Evaluate explanations."""
    results = {}
    for method, explainer in xai_methods.items():
        results[method] = {}
        for metric, metric_func in metrics.items():
            print(f"Evaluating the {metric} of {method} method...")
            
            scores = metric_func(
                model=model,
                x_batch=x_batch[:nr_samples],
                y_batch=y_batch[:nr_samples],
                s_batch=s_batch[:nr_samples],
                a_batch = explainer[:nr_samples],
                explain_func=quantus.explain,
                explain_func_kwargs={"method": method}
            )
            results[method][metric] = scores #results[method][metric] = np.random.random(size=nr_test_samples)
    return results
    

Is there a better method to send parameters to Lime? I'd like to use the pre-computed explanation values (a_batch). How to set feature_mask and other Lime attributes.

[annahedstroem]

Hi @Akshatha122

Regarding setting the Lime attributes, such as feature_mask, you can extend the dictionary in the explain_func_kwargs dictionary as follows:

explain_func_kwargs = {"xai_lib": "captum", "method": "Lime", "xai_lib_kwargs": {"feature_mask": feature_mask.to(device).long()}}
results = evaluate_xai_methods(model, ...., s_batch, nr_samples=1, explain_func_kwargs=explain_func_kwargs)

Read more in our official documentation: https://quantus.readthedocs.io/en/latest/docs_api/quantus.functions.explanation_func.html#quantus.functions.explanation_func.explain

LIME hyperparams: https://captum.ai/api/lime.html

I hope this helps. Let me know if you have any further questions!

[Akshatha-Mohan]

Thank you for your response. It helped me fix the problem

…

On Wed, 17 May, 2023, 3:44 am Anna Hedström, ***@***.***> wrote: Hi @Akshatha122 <https://github.com/Akshatha122> Regarding setting the Lime attributes, such as feature_mask, you can extend the dictionary in the explain_func_kwargs dictionary as follows: explain_func_kwargs = {"xai_lib": "captum", "method": "Lime", "xai_lib_kwargs": {"feature_mask": feature_mask.to(device).long()}} results = evaluate_xai_methods(model, ...., s_batch, nr_samples=1, explain_func_kwargs=explain_func_kwargs) Read more in our official documentation: https://quantus.readthedocs.io/en/latest/docs_api/quantus.functions.explanation_func.html#quantus.functions.explanation_func.explain LIME hyperparams: https://captum.ai/api/lime.html I hope this helps. Let me know if you have any further questions! — Reply to this email directly, view it on GitHub <#265 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMPSWQM7S6F3BFJ3A47VHGDXGSFXBANCNFSM6AAAAAAYD5JUNU> . You are receiving this because you were mentioned.Message ID: <understandable-machine-intelligence-lab/Quantus/repo-discussions/265/comments/5925053 @github.com>