Is it possible to tweak the adamw_torch optimizer to change Beta1 and Beta2 values?

[jackswl]

As per title,

  optimizer: adamw_torch

How can I tweak the beta1 and beta2 values in my .yml file?

@abhishekkrthakur wondering if this is possible. thanks!

[abhishekkrthakur]

which task?

[jackswl]

sorry, forgot to specify in details. It is for LLM fine-tuning

task: llm-sft
base_model: xxx
project_name: xxx
log: none
backend: local

data:
  path: xxx
  train_split: train
  valid_split: null
  chat_template: null
  column_mapping:
    text_column: text

params:
  block_size: 1024
  model_max_length: 1024
  epochs: 3
  batch_size: 4
  lr: 1e-4
  peft: true
  quantization: int4
  target_modules: "q_proj,v_proj,o_proj,k_proj"
  padding: right
  optimizer: adamw_torch
  scheduler: cosine
  gradient_accumulation: 4
  mixed_precision: bf16     
  warmup_ratio: 0.1
  weight_decay: 0.1
  lora_r: 16
  lora_alpha: 16
  lora_dropout: 0
  merge_adapter: false
  use_flash_attention_2: true  
  logging_steps: 1
  unsloth: false

hub:
  username: xx
  token: hf_XXX
  push_to_hub: false

@abhishekkrthakur could you kindly let me know, is there a way to tweak the beta1 and beta2 values for adamw_torch to 0.9 and 0.95 respectively for beta1 and beta2? I believe the default value is 0.9 and 0.999 respectively

[jackswl]

@abhishekkrthakur any clue on this? thanks!

[abhishekkrthakur]

i can add the params. for now they are hidden. is it required?

[jackswl]

that may be good. I think the use case might be small, because I am using autotrain specifically for research. Thus, I would prefer to follow the standard convention which is 0.95 for beta2, instead of 0.999 for beta2. This follows the research community best practices. would this be possible?

facebookresearch/mae#184
https://docs.mosaicml.com/projects/composer/en/latest/api_reference/generated/composer.optim.DecoupledAdamW.html

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 20 days since being marked as stale.