Fine-Tuning LLM with MacBook Pro M4 MAX

[softshipper]

Hi everyone,

I have a question regarding the capability of the MacBook Pro M4 MAX with 128 GB RAM for fine-tuning large language model. Specifically, is this system sufficient to fine-tune LLaMA 3.2 with 3 billion parameters and using MLX?

Best regards

[awni]

It really depends on what you are trying to do. You can LoRA fine tune that model on a M1 with just 32Gb on a small to medium dataset. If you're dataset is larger, or has very long sequences, or you want to do full fine-tuning (train all the params, not low rank) you will probably want more RAM so 128GB will be better.

[awni]

• If you fine tune a 16-bit model (and fuse the adapters into it) it should work in most cases
• For 4-bit it has to be dequantized to work

[softshipper]

it should work in most cases

But what does in most cases mean? Does this imply that it might not work sometimes?

[awni]

Yes it might not work sometimes. MOEs are an example of when it doesn’t work.

[awni]

In general you need to convert the model back to the Transformers format. When the conversion is identity it works when it’s not it doesn’t. This is on a model by model basis.

[softshipper]

@awni Thank you for the clarification.