Explainability techniques for image classification such as saliency maps provide insight into which regions of the image influenced the prediction. Saliency techniques can assist data scientists in model debugging by identifying features in images that are not relevant to the image class. However, the static presentation of saliency maps withhold data scientists from further exploring model behavior on out-of-distribution data. Incorporating saliency maps in an interactive tool will enable data scientists to debug their models while understanding how their model behaves on out-of-distribution data. We present a protoype that incorporates saliency maps in an interactive, exploratory tool to showcase novel interactions that data scientists can experience with saliency maps.
Project Report: PDF
Project Demo Video: Video
- View this Google Colab to see how we generated the saliency maps.
- View this Google Colab to see how we approximated image similarity.
- View this Google Colab to see how we approximated the top-K colors in an image.
- View this Google Colab to see how we generated stylized images.
- View this Google Colab to see how we predicted the class of the stylized images.
-
Make sure you have
npmon your computer. If you don't, you can install it here. -
Clone this repo
-
In the terminal, navigate to this repo.
-
Run the following commands:
3.1.
npm install3.2.
npm run startornpm run devornpm run -
Copy and paste the local host url and open up in a browser.
We leveraged the GitHub projects features to outline features, bugs, and next steps throughout developing our project. You can view our project with a history of our tasks and issues here.
In terms of work split between the two developers:
Swetha designed and fully implemented the stylize an image tab and explain an image tab. She also managed all of the UI/UX for our site. This includes disbaling certain features under certain conditions, setting the theme of the tabs, and designing the look of the website.
Katelyn designed and helped implement the Saliency Similiarity tab and the Model comparison tab. Swetha assisted with the UI/UX for these tabs. Katelyn managed all of the machine learning related source code including generating stylized images using neural style transfer, generating the XRAI saliency maps, and the predictions for each image. Katelyn also brought her experience of working with vision models into the project. Her expertise informed the development and direction of the page and she put work into ensuring that all explanations were consistent with industry knowledge in the field.
- Swetha Kannan
- Katelyn Morrison