Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CIFAR-10 evaluation #12

Open
wants to merge 144 commits into
base: master
Choose a base branch
from
Open

CIFAR-10 evaluation #12

wants to merge 144 commits into from

Conversation

rohinmshah
Copy link
Member

Adds run_cifar.py, which runs representation learning with a ResNet-18 on CIFAR-10, finetunes a linear layer on top of it, and evaluates the accuracy of the resulting classifier. Hyperparameters are set to mimic SimCLR: https://github.com/google-research/simclr/

The current implementation still depends on an incorrect loss function (to be fixed in #10 ) and augments the examples at a batch level instead of a dataset level (to be fixed in a different PR).

Copy link
Member

@decodyng decodyng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of small questions/suggestions, but otherwise LGTM!

algos/decoders.py Outdated Show resolved Hide resolved
algos/encoders.py Outdated Show resolved Hide resolved
algos/utils.py Outdated Show resolved Hide resolved
algos/utils.py Outdated Show resolved Hide resolved
run_cifar.py Outdated Show resolved Hide resolved
run_cifar.py Outdated Show resolved Hide resolved
algos/utils.py Outdated Show resolved Hide resolved
run_cifar.py Outdated Show resolved Hide resolved
@RPC2
Copy link
Collaborator

RPC2 commented Aug 27, 2020

The code looks good to me!

Just that when I try to plot the lr curve, it seems the linear scaling part's minimum lr is eta_min and the cosine part's minimum lr is 0 (I've seen people who end the cosine part at eta_min). I'm not sure if this is your intended behavior, but if it is then you can ignore this comment :)

08-27-2020-15-35-58 lr

The line is produced by:

    import torchvision.models as models

    resnet18 = models.resnet18()
    optimizer = torch.optim.Adam(resnet18.parameters(), lr=0.03)
    scheduler = LinearWarmupCosine(optimizer, 5, 30)

    lr = []
    for _ in range(30):
        optimizer.step()
        lr.append(optimizer.param_groups[0]['lr'])
        scheduler.step()

    plot([lr], ['lr'], 'lr', 'test_scheduler')

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants