kagle

https://www.kaggle.com/c/bengaliai-cv19/leaderboard

write up

https://www.kaggle.com/c/bengaliai-cv19/discussion/135960

Kaggle-Bengali

EXP	grapheme_root	vowel_diacritic	consonant_diacritic	CV	LB
EXP_200	0.976273	0.990575	0.985638	0.982190	0.9745
EXP_200_20	0.969665	0.992616	0.990397	0.980586	0.9701
EXP_200_CUTMIX	0.977921	0.990718	0.985638	0.983230	0.950

Setup

17 JAN 2019 9:40 AM

Images: Preprocessing seems to make model stuck around local and lb 0.969 If we dont preprocess images atleads to higher local score abd lb sc

EXP_200.ipynb

MODEL:           se_resnext50_32x4d
BS:              1024
SZ:              128 (1 CH)
VALID:           5 FOLD CV (FOLD=2)
TFMS:            transform(get_transforms(do_flip=False,max_warp=0.2, max_zoom=1.1, max_rotate=5, 
                 xtra_tfms=[cutout(n_holes=(10,25), length=(10, 30), p=.5)]), size=(SZ, SZ), 
                 resize_method=ResizeMethod.SQUISH, padding_mode='reflection')
MixUP:           True

PRETRAINED:      IMAGENET
NORMALIZE:       ([0.0692], [0.2051])

LOSS:            WEIGHTED [0.7, 0.1, 0.2]
TRAINING:        OPT: Over9000
                 fit_one_cycle(100, lr, wd=1e-2,  pct_start=0.0,  div_factor=100)
                 
NOTEBOOK:        EXP_200 
MODEL WEIGHTS:   [EXP_200_RESNEX_1CH_MISH_SIMPLE_ORIG_2_2.pth]
MODEL TRN_LOSS:  0.597414
MODEL VAL_LOSS:  0.071885
ACCURACY ALL  :  0.982190
LB SCORE:        0.9745 (SUB_NAME: EXP_80_SERESNET101_1CH(version 23/23))

grapheme_root	vowel_diacritic	consonant_diacritic	CV	LB
0.976273	0.990575	0.985638	0.982190	0.9745

Model Structure

Mish only for tails (body was with nn.ReLU())
n - on the last linear layers out_features depending on 3 classes [168, 11, 7]

  (head1): Head(
    (fc): Sequential(
      (0): AdaptiveConcatPool2d(
        (ap): AdaptiveAvgPool2d(output_size=1)
        (mp): AdaptiveMaxPool2d(output_size=1)
      )
      (1): Mish()
      (2): Flatten()
      (3): BatchNorm1d(4096, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (4): Dropout(p=0.2, inplace=False)
      (5): Linear(in_features=4096, out_features=512, bias=True)
      (6): Mish()
      (7): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (8): Dropout(p=0.2, inplace=False)
      (9): Linear(in_features=512, out_features=n, bias=True)

Conclusion: Defenetly Imporvement of the model

Thing to try:  - Loss without weights
               - GeM polling layer 
               - Full Mish Model                        

18 JAN 2019 12:17 AM

EXP_200_20.ipynb

MODEL:           se_resnext50_32x4d
BS:              1024
SZ:              128 (1 CH)
VALID:           5 FOLD CV (FOLD=0)
TFMS:            transform(get_transforms(do_flip=False,max_warp=0.2, max_zoom=1.1, max_rotate=5, 
                 xtra_tfms=[cutout(n_holes=(10,25), length=(10, 30), p=.5)]), size=(SZ, SZ), 
                 resize_method=ResizeMethod.SQUISH, padding_mode='reflection')
MixUP:           True

PRETRAINED:      IMAGENET
NORMALIZE:       ([0.0692], [0.2051])

LOSS:            NORMAL
TRAINING:        OPT: Over9000
                 fit_one_cycle(100, lr, wd=1e-2,  pct_start=0.0,  div_factor=100)
                 
NOTEBOOK:        EXP_200 
MODEL WEIGHTS:   [EXP_200_RESNEX_1CH_MISH_SIMPLE_ORIG_LOSS_GEM_0_0.pth]
MODEL TRN_LOSS:  1.333434
MODEL VAL_LOSS:  0.171532
ACCURACY ALL  :  0.98058
LB SCORE:        EXP_200(version 26/27)

grapheme_root	vowel_diacritic	consonant_diacritic	CV	LB
0.969665	0.992616	0.990397	0.980586	0.9701

Model Structure

Mish for everything, + 'GeM pooling layer'
n - on the last linear layers out_features depending on 3 classes [168, 11, 7]

  (head1): Head(
    (fc): Sequential(
      (0): GeM(p=3.0000, eps=1e-06)
      (1): Mish()
      (2): Flatten()
      (3): BatchNorm1d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (4): Dropout(p=0.2, inplace=False)
      (5): Linear(in_features=2048, out_features=512, bias=True)
      (6): Mish()
      (7): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (8): Dropout(p=0.2, inplace=False)
      (9): Linear(in_features=512, out_features=168, bias=True)
    )

Thing to try:  - cutmix

20 JAN 2019 10:30 AM

Images: I have implemented fastai to work with cutmix

EXP_200_CUTMIX.ipynb

same as 17 JAN 2019 9:40 AM, EXP_200.ipynb just with cutmix

grapheme_root	vowel_diacritic	consonant_diacritic	CV	LB
0.977921	0.990718	0.985638	0.983230	0.950

Model Structure

same as 17 JAN 2019 9:40 AM, EXP_200.ipynb

Thing to try:  - cutmix, with simple pool layers and no non linearity 

EXP_200_CUTMIX_CUTOUT_STRECHING.ipynb

SZ = 152 POOL = Averge no Wrap removed squish from dataloader