Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

生成モデル　Fine Tuning に関して #3

Open
Tracked by #1
Unagi2 opened this issue Aug 14, 2022 · 2 comments
Open
Tracked by #1

生成モデル　Fine Tuning に関して #3

Unagi2 opened this issue Aug 14, 2022 · 2 comments

Comments

@Unagi2
Copy link
Owner

Unagi2 commented Aug 14, 2022 •
edited
Loading

当初予定していたgpt2-mediumは、モデルの層のサイズが非常に大きく、計算機サーバ（susanoo:rtx3080 12GB）ではメモリのオーバーフローでFine Tuning が不可能であることが判明
@Unagi2 Unagi2 mentioned this issue Aug 14, 2022
Open
9 tasks
@Unagi2
Copy link
Owner Author

Unagi2 commented Aug 14, 2022

Distlgpt2という軽量化・高速化されたモデルに変更

TrainとTest両方のデータセットをフルデータで、メモリオーバーフローを引き起こさずにFine Tuning が行えることを確認

10000epoch学習し、モデルを保存済み

@Unagi2
Copy link
Owner Author

Unagi2 commented Aug 15, 2022 •
edited
Loading

Fine Tuningに用いるデータの違いによる評価の変化

  • TrainとTestデータをマージ後FineTuneして生成したデータセット
    CV = 0.952277695
    LB = 0.4653222317277211

  • TrainデータのみでFine Tuneして生成したデータセット
    CV =
    LB =

@Unagi2 Unagi2 mentioned this issue Aug 15, 2022
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Unagi2