Sensie is toolset for probing the sensitivity of a deep neural network model to various properties of the data.
An active area of research (and not a little hand-wringing) in deep learning at present is better understanding how to interpret the key features learned by DNNs, particularly in order to better understand and predict failure modes. Various algorithms and toolsets exist for interpreting DNNs; in the computer vision arena, saliency maps are an important and well-motivated technique for understanding the key features of the images used by the network in order to make its decisions. However, these maps are not equally informative in all applications, for instance in some scientific applications where inputs are not RGB images, or physical parameters are otherwise not readily discerned from a saliency map.
Sensie probes the sensitivity of a network to inputs with a particular property, 
. This property can be a feature of the data; an otherwise known property of a test or training set that is not provided explicitly in training; or a function that can artificially vary this property for a supplied test set. The effect of a particular property is measured according to the variance it introduces to the correct output of the network, such as the score for the correct class 
. Quantitatively, we seek the output 
for a supplied property, .
Although the algorithm is simple, it has the potential to highlight and quantify gaps in the training set or areas of increasing unrealiability in a trained model. Sensie was written with scientific applications in mind, but can be used on any trained model.
Sensie can operate on models trained for discrete classes (classifier); similar tools for regression will be added in the next release.
Sensie can be adapted to works on any predictive model.
Main outstanding issues:
- Proper support for regression models
Sensie assumes a model object with a predict method (as per Keras), but with a user supplied predictor function any framework should be usable. Python prequisites are listed in requirements.txt. The pymc3 probabilistic programming framework is required for the (optional) determination of credible intervals for feature sensitivities. This can be used to assess the significance of a sensitivity analysis.
Sensie requires python 3.6 or above.
After cloning the repository, I recommend creating a virtual environment for Sensie (currently, Sensie requires PyMC3 3.7). Clone the repository and install with:
pip install -e .
Which will install Sensie and the required python libraries. Optionally, install pytest with pip install pytest, then run the tests with pytest test from the repository root.
See the example below, and the notebooks in examples/ for some detailed examples of how Sensie can be used. In summary:
- Create a
sensie.Probeinstance to wrap a pre-trained model:
probe = sensie.Probe(model=model)
- Pass the Probe a test set, ground truths, and either a vector/tensor containing the property/properties to test, or a function that mutates a training example, taking a scalar parameter indicating the size of the effect.
probe.predict_and_measure_perturbed(X_test, y_test, peturb_func, ...)
- Sensie will return a
sensie.SensitivityMeasureobject, a collection of the results of each test represented bysensie.SingleTestobjects. - Examine a plot of the outcome of each test, or print the
summaryto quantify the size of the effect. - Is the effect significant? The sensitivity, i.e. the gradient of mean correct score with property
, is determined using ordinary linear regression (sensie.Probe.get_gradient) or using Bayesian inference with PyMC3; this in turn supplies a 50%/95% credible interval for the sensitivity. - Where the relationship is non-linear, polynomial fits can also be visualised in order to identify regions of the parameter space where the network is most sensitive to the supplied property.
Sensie has several options for plotting the results. See the docs and examples for more information.
Sensie assumes that model.predict returns a tensor t with categorical scores, such that t[i, j] returns the score for class j for test example i. If this is not the case, supply a predictor function at instantiation time that does: probe = sensie.Probe(model, predictor) where predictor is a function with the signature predictor(model, x_test) and returns a tensor of dimensions (N, n_classes).
Detailed documentation can be accessed at readthedocs.org.
When Sensie reports the significance of the effect, it is reporting whether there is a detectable linear relationship between the property measured and model's score for the correct class for each example. In other words, if we assume the accuracy is a function of some property of the inputs, is the function affecting the scores in a significant way?
Formally, Sensie tests the assumption that
where
and uses Bayesian linear regression to determine the gradient 
along with
credible intervals. If the effect of property is small or neglible, then
the result should be consistent with 
.
For properties without a natural ordering, like classes, Sensie can first re-order the classes/properties by the mean scores then measure the significance of the effect, i.e. whether there is a significant gradient under this ordering. Whether this is meaningful is highly context dependent! [1]
How sensitive is a model trained on MNIST digits to the orientation of the digits?
def rotate(x, angle):
# code that rotates the image by _angle_ degrees goes here, see examples/MNIST
# x_rotated = ...
return x_rotated
model = load_model() # a trained model
sensie_mnist = sensie.Probe(model)
sensie_mnist.predict_and_measure_perturbed(X_test, y_test,
rotate, p_min=0, p_max=180, steps=10,
label='rotation', plot=True)
From this we can see that the trained model is sensitive to the orientation of an input; beyond about 20 degrees, accuracy suffers significantly.
For this and some more complex examples, see the Jupyter notebooks in the examples directory.
Module docs can be found at sensie.readthedocs.org.
If you use Sensie and run into any problems, please open an issue here.
Any questions, comments and suggestions can be sent via GitHub or to colin(@)coljac.net.
Contributions are welcome. Fork this repository to your own machine, make some changes, and push your work back up to the fork and open a pull request so that I can review the changes.
[1]: Consider plt.scatter(np.arange(0, 100), np.sort(np.random.random(100))).