RandAugment Magnitude

[vchiley]

For RandAugment you have a _MAX_LEVEL that clips the Magnitude of RandAugment to 10.
Many papers I read that use RandAugment set their Magnitude to 15 (eg Table 8 here), and the original RandAugment paper consistently talks about testing magnitudes "ranging from [0, 30]".
It almost seems like the RandAugment implemented here is different from the RandAugment everyone else is talking about.
How does the limit of RandAugment here compare to the limit of the RandAugment everyone else is referring to? Is it like a 3 to 1 conversion factor? As an example, what would be the RandAugment argument if I was trying to replicate the version described in Table 8 here?

Note: I've read the disclaimer here.

[vchiley]

I actually think this might be a bug in your code here.

If we're adapting the EfficientNet rand augment code, they too have a _MAX_LEVEL = 10. setting, BUT it is not used to clip the rand augment magnitude level (which is done in this code here); In the EfficientNet work they allow the RandAugment level to be much higher than 10 (here they quote randaugment levels of 28)
The timm library includes this clipping because it is "testing" sampling the magnitude from a distribution. The sampled magnitude must be above 0, but below some ridiculously high value.
The BUG: _MAX_LEVEL = 10. should be "the max integer that the controller RNN could predict for the augmentation scheme", NOT the absolute max level of the augmentation magnitude (which, according from the randaugment paper, can be as high as 31).
Possible solution: Since we probably want to keep mstd in randaugment, we can probable do something like:

class RAugmentOp(AugmentOp):
    _MAX_LEVEL = 31  # or whatever the true max level should be

    def __call__(self, img):
        if self.prob < 1.0 and random.random() > self.prob:
            return img
        magnitude = self.magnitude
        if self.magnitude_std:
            if self.magnitude_std == float('inf'):
                magnitude = random.uniform(0, magnitude)
            elif self.magnitude_std > 0:
                magnitude = random.gauss(magnitude, self.magnitude_std)
        magnitude = min(self._MAX_LEVEL, max(0, magnitude))  # clip to valid range
        level_args = self.level_fn(magnitude, self.hparams) if self.level_fn is not None else tuple()
        return self.aug_fn(img, *level_args, **self.kwargs)

[rwightman]

@vchiley I have purposely diverged from the standard Tensorflow / 'Google' RandAugment implementations. I should really update some comments and document why, but have not prioritized that.

The TF RA magnitude is not intuitive, if you actually look at the augmentations deployed for say M0, M5, M10, M15 it is VERY counter to what you might think, some augmentations go up in strength, but quite a few also go down (or are completely disabled) as you increase the magnitude due to bugs/oversights in the original impl wrt to some augs like the enhancements, posterize/solarize, etc. Each M is essentially it's own thing and so applying sampling to that scale doesn't work well. At M0 there are actually a few augs at max strength, and I believe some completely off at M5, etc.

With inc1 (increasing mode) enabled in my impl, moving across the magnitude scale makes more sense, all augs should decrease/increase in strength with corresponding decrease/increase in M (or sampling of). I use this extensively in all my training and so have a growing number of researchers (Facebook Research in DeiT/CaiT/ResMLP and SlowFast repos, Microsoft in Swin Transformers (and related), BeiT (for fine-tune stage) are a few examples). I have not done a controlled set of experiments to determine which approach is better, but I have met or surpassed results with Google's RA on numerous occasions using my training pipeline (but w/ other differences that make fair comparison challenging).

Worth also pointing out that I purposely remove cutout from RA, it harms the image stats (mean/std) by erasing large regions w/ constant values before the images are normalized. I use random erasing w/ mean 0, std-dev 1.0 after normalization (standardization) as a replacement for that.

I don't see the max level = 10 to be an issue, it's actually challenging to limit some of the augs to a sensible, symmetric range if you don't normalize appropriately. However, I do have a TODO to experiment and add a 'boost' or some means of extending magnitudes of some augs where it would/could make sense. I'm not sure if that would be allowing M > 10 for a subset of the augs and clipping for those where it doesn't make sense, or adding an extra 'turbo boost' magnitude for a specified/fixed subset of the augs....

[rwightman]

Another point,

Instead of using the higher M values (or the proposed boost mode), I have found that adding extra aug layers is quite effective. I've been training with N3 and N4 on some models (NFNet and ViT especially). For the 'Train your own ViT' paper I was running some high augreg train from scratch comparison baselines / explorations to train on small datasets from scratch and was able to extend training (without overfitting, and boost end result) by going up to N5/N6 but a lower M (4-7) and lots and lots of iterations on a small dataset (200 dataset passes per-'epoch' for 600 'epochs)

[vchiley]

Thank you for the response.
To summarize, the Tensorflow / 'Google' RandAugment has sooo many issues / bugs that instead of trying to fix it, you've just modified it beyond recognition.
That makes sense.

So when if I ask: "How do I use the RandAugment used in (for instance) the ResNetRS paper"
Is the answer: "There is no one to one correspondence, because of how much modifications have happened since"?
Or maybe is it something like RandAugment with Magnitude = 15 is similar to (but NOT the same as) rand-m<M>-n2-mstd0.-inc0 where the original Magnitude of 15 is approximately 15 / 31 * 10.

[rwightman]

@vchiley I wouldnt' say it's beyond recognition, lots of additions and tweaks but most of the original behaviour was left as an option... as per your second question I believe that you can get pretty close to the original efficientnet impl w/ rand-m<M> . inc0 is still default, mstd0. is default, and n2 is default.

To use M values > 10 you can change the clipping line as you say and choose a max that's higher.

I'm open to adding a config to modify clipping upper bound so you could do something like rand-m30-mmax30

[vchiley]

👍 Thank you.

[rwightman]

@vchiley I've added mmax .... 3cdaf5e

so rand-m<M>-mmax30 would allows you to choose M values up to 30 and should behave fairly similar to the EfficientNet TF original, probably the biggest difference would be the handling of the translate, but you could fix that by switching the rel translates to abs and making sure your translate const matches the efficientnet defaults.

For others reading this too... these should also work with some of my other additions so rand-m15-mmax30-mstd5.0-inc1 would give you increasing aug scales with lots of m value varation. mstd values > 100 will switch it to unform sampling between 0 and m rand-m15-mmax30-mstd101-inc1.

Let me know if any issues come up.