Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Activation Function] GeLU + Commons #16

Open
15 tasks done
jinmang2 opened this issue Jun 21, 2022 · 2 comments
Open
15 tasks done

[Activation Function] GeLU + Commons #16

jinmang2 opened this issue Jun 21, 2022 · 2 comments

Comments

@jinmang2
Copy link
Owner

jinmang2 commented Jun 21, 2022

목적: transformers activation function 구현

GeLU

  • Bloom GeLU
  • NewGeLU
  • GeLU
  • FastGeLU
  • QuickGeLU
  • ClippedGeLU

Quantized

  • IntGELU
  • QuantAct

Commons

  • linear
  • mish
  • relu
  • sigmoid
  • silu
  • swish
  • tanh
@jinmang2
Copy link
Owner Author

jinmang2 commented Jun 21, 2022

jinmang2 added a commit that referenced this issue Jun 21, 2022
@jinmang2 jinmang2 changed the title [Activation Function] GeLU [Activation Function] GeLU + Commons Jun 21, 2022
@jinmang2
Copy link
Owner Author

iBERT의 경우, activation function에 들어가는 인자들이 많아 구현에 고민을 하게된다. 아직은 Transformer를 Text, Vision, Audio별로 나눌지 각 모델별 객체를 어떻게 세팅할지 결정하지 못했기 때문에 modeling_ibert.py에 있는 activation_bit에 사용되는 [8, 16, 22] 만 남긴 상태. 향후 TODO로 남겨둠.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant