loading pretrained open ai model

[mehdimashayekhi]

can somebody please explain what are these parameters here

pytorch-openai-transformer-lm/model_pytorch.py

Line 303 in 561d409

def load_openai_pretrained_model(model, n_ctx=-1, n_special=-1, n_transfer=12, n_embd=768, path='./model/',

, e.g., offsets, init parameters, can you add some comments to this function?, thanks

[rodgzilla]

There you go (@thomwolf correct me if I'm wrong on any of these):

• n_ctx is the maximum number of token in an input sequence.
• n_special is the number of special tokens used to format the input properly. For example in the ROCStories problem, we use 3 additional tokens, _start_, _delimiter_ and _classify_.
• n_transfer is the number of pre-trained layers that we will be loaded, the next ones will be initialized randomly.
• n_embd is the dimension of the embedding and of the vector associated to each position in the network. It has the value 768 because the network uses multi-head attention with 12 heads and 768 = 12 * 64.

If these comments are helpful to you I will add them in the code.

[thomwolf]

Exactly!

[mehdimashayekhi]

@rodgzilla thanks!