can't reproduce Nest transformer results on Cifar10

[abdohelmy]

I have used the exact same augmentations as the original repo and changed the depthdrop rate to 0.1 as the paper/
used adamW optimizer and cosine annealing scheduler
optimizer = optim.AdamW(model.parameters(), lr=0.00025,weight_decay=0.05)
lr_sch=lr_scheduler.CosineAnnealingLR(optimizer,steps)

Still can't get higher than 86%

[alexander-soare]

Hi @abdohelmy maybe this discussion will give you the insight you're looking for #751

[abdohelmy]

Thank you so much for the quick answer, will try check it