NLP data and model support

[sabrinagoellner]

Hi, I read in your documentation that you are planning/working on supporting NLP data and models for evaluation. When will this feature be available? I am very excited about this toolkit and soon I will start my master's thesis research also on this topic. I'm looking forward to hearing back from you.

[annahedstroem]

Hi @sabrinagoellner,

Thank you for your interest in Quantus and apologies for my delay in getting back to you! We are currently working on adding support for NLP data and models, but we do not have a date for when this feature will be available. We will be sure to announce it on our documentation once it is released.

In the meantime, you can take a look at our NLP demonstration tutorial, which provides an example of how to use Quantus with NLP models. Link: https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/tutorials/Tutorial_NLP_Demonstration.ipynb. Additionally, you can use Quantus with metrics that do not rely on perturbing the input, such as randomisation metrics and complexity metrics. This may be useful in your master's thesis research as well!

[abarbosa94]

Hi @annahedstroem, Are there any updates about that?

I'm researching NLP explainability (coincidentally, a Master project as well :D) to avoid implementing some stuff from scratch. I decided to look into some available projects. This library is the best one available related to XAI objective metrics quantification :)

I tried to see if there is any open issue, but this was the only one that I found so far, #103, without any updates

I'm willing to help, so I don't need to implement things in my library :)
Please let me know if I can start digging into that.

[annahedstroem]

Hi @abarbosa94! Awesome to hear you're working in XAI/ NLP intersection and great that you want to get involved 💯

I'd like to also loop in @aaarrti here, who has been working on this in the past months.

Let's all connect here https://discord.gg/scj2BKZn8Q?

[annahedstroem]

I started an NLP channel in the discord --> https://discord.com/channels/1185623299808579694/1185623862793211934 where we should continue our conversation. Also, I shared a Google Docs there with some thoughts.