Active Learning
Active learning as a framework has a straight objective to select the data points, or images, that would be most beneficial for the deep learning model to learn from. As a cyclic framework it consists of Training pool, a machine learning, a strategy to estimate uncertainty of unlabeled data points, unlabeled pool of data and an Oracle that annotates the data. The cyclic framework is in short a data engine that is designed to keep improving the machine learning model.
In MONAI label, the developer creates their own scoring strategy based on which the next sample can be chosen. All sample-apps are equipped with a basic skeleton to allow for the development of a scoring strategy. For example here in the DeepEdit App.
Currently, MONAI Label offers two Active Learning strategies that researchers can use to accelerate the training process. One is based on the uncertainty that comes from the dataset (Aleatoric uncertainty) using Test Time Augmentation (TTA) and the other one is based on the uncertainty that comes from the model (Epistemic Uncertainty) using Dropout.
Aleatoric uncertainty estimation using Test Time Augmentation (TTA)
Test time augmentations are a useful feature for computing uncertainty, as well as observing the network's dependency on the applied random transforms. MONAI Label offers TTA as one of the scoring methods. TTA uses transform(s) that are applied to each image. Those transforms should have their corresponding inverse so the image can be returned to the same spatial reference. Then metrics such as the volume variation coefficient (vvc) can be computed and used as a scoring method.
Reference: Wang et al., Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks, https://doi.org/10.1016/j.neucom.2019.01.103
