How to upload model to the hub?

[ashercn97]

This is more of a question than an issue but: how do i push a model to the hub?

[ashercn97]

the Huggingface hub btw**

[ashercn97]

@NanoCode012 Or how do i upload to github from a runpod enviornment

[NanoCode012]

• See hub_model_id config if you're training
• See https://huggingface.co/docs/huggingface_hub/main/en/guides/upload#upload-a-folder otherwise

[ashercn97]

Okay ty!

[ashercn97]

@NanoCode012 How do i get git lfs on runpod? I cannot figure it out and it says I need it

[NanoCode012]

apt install git-lfs

[ashercn97]

Oh. Thanks. Lemme try

[ashercn97]

BTW @NanoCode012 Ur such a real one for helping me all these times !!

[ashercn97]

@NanoCode012 It didnt work. I tried to run the accelerate launch and then my config using the hf_repo_id and it still says i need to isntall git lfs

[ashercn97]

Traceback (most recent call last):
File "/workspace/miniconda3/envs/python39/lib/python3.9/site-packages/huggingface_hub/repository.py", line 574, in check_git_versions
lfs_version = run_subprocess("git-lfs --version", self.local_dir).stdout.strip()
File "/workspace/miniconda3/envs/python39/lib/python3.9/site-packages/huggingface_hub/utils/_subprocess.py", line 83, in run_subprocess
return subprocess.run(
File "/workspace/miniconda3/envs/python39/lib/python3.9/subprocess.py", line 505, in run
with Popen(*popenargs, **kwargs) as process:
File "/workspace/miniconda3/envs/python39/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/workspace/miniconda3/envs/python39/lib/python3.9/subprocess.py", line 1837, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'git-lfs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/axolotl/scripts/finetune.py", line 356, in
fire.Fire(train)
File "/workspace/miniconda3/envs/python39/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/workspace/miniconda3/envs/python39/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/workspace/miniconda3/envs/python39/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/workspace/axolotl/scripts/finetune.py", line 289, in train
trainer = setup_trainer(cfg, train_dataset, eval_dataset, model, tokenizer)
File "/workspace/axolotl/src/axolotl/utils/trainer.py", line 344, in setup_trainer
trainer = trainer_cls(
File "/workspace/miniconda3/envs/python39/lib/python3.9/site-packages/transformers/trainer.py", line 551, in init
self.init_git_repo(at_init=True)
File "/workspace/miniconda3/envs/python39/lib/python3.9/site-packages/transformers/trainer.py", line 3406, in init_git_repo
self.repo = Repository(self.args.output_dir, clone_from=repo_name, token=self.args.hub_token)
File "/workspace/miniconda3/envs/python39/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/workspace/miniconda3/envs/python39/lib/python3.9/site-packages/huggingface_hub/repository.py", line 504, in init
self.check_git_versions()
File "/workspace/miniconda3/envs/python39/lib/python3.9/site-packages/huggingface_hub/repository.py", line 576, in check_git_versions
raise EnvironmentError(
OSError: Looks like you do not have git-lfs installed, please install. You can install from https://git-lfs.github.com/. Then run git lfs install (you only have to do this once).
Traceback (most recent call last):
File "/workspace/miniconda3/envs/python39/bin/accelerate", line 8, in
sys.exit(main())
File "/workspace/miniconda3/envs/python39/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py", line 45, in main
args.func(args)
File "/workspace/miniconda3/envs/python39/lib/python3.9/site-packages/accelerate/commands/launch.py", line 979, in launch_command
simple_launcher(args)
File "/workspace/miniconda3/envs/python39/lib/python3.9/site-packages/accelerate/commands/launch.py", line 628, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/workspace/miniconda3/envs/python39/bin/python', 'scripts/finetune.py', 'config/orca.yml']' returned non-zero exit status 1.
(python39) root@c1eafd0302c4:/workspace/axolotl# git lfs install
git: 'lfs' is not a git command. See 'git --help'.

The most similar command is
log

[NanoCode012]

git lfs install run this after apt install.

[ashercn97]

(python39) root@c1eafd0302c4:/workspace/axolotl# git lfs install
git: 'lfs' is not a git command. See 'git --help'.

The most similar command is
log
(python39) root@

[ashercn97]

Is it somethign wrong with runpod?

[ashercn97]

it says when I do apt install:

'E: Unable to locate package git-lfs'

[NanoCode012]

Follow this: https://github.com/git-lfs/git-lfs/blob/main/INSTALLING.md#1-adding-the-packagecloud-repository

[ashercn97]

I asked chatgpt to clarify something but I get it now an its working!!!! THANK YOU SO MUCH!

[ashercn97]

Last question: How often does it checkpoint?

[NanoCode012]

up to your config of save_steps

[ashercn97]

Uh oh
I dont think i did that. What is the default?

[ashercn97]

And it wil uplaod the the HF HUB every time it saves? If not, when does it upload the the hf hub

[NanoCode012]

should only upload final.

it saves last/top three checkpoint.

if you do not pass config, it does epoch.

[ashercn97]

Ohh okay.. If i stop it early what wouod happen?

[ashercn97]

And if i set the steps to .001 it will save the model every .001 epochs?

[NanoCode012]

it uses steps to count, not ratio. Open Issue here: #38

It should save when you stop, but I don't trust that feature all the time to work.

[ashercn97]

Okay. Thank you so much. I will close this issue now!