Fix memory leak in fp8 causing OOM (and potentially 3x vRAM usage)

[muellerzr]

What does this PR do?

Doing .copy() like we were before leads to a huge memory leak during fp8, and also putting the model on CUDA after converting the layers can reduce some memory.

Example benchmark:
Model: "meta-llama/Llama-2-7b-hf"
Note: on FP8 weights are in bf16 beforehand
Before fix:
Memory in FP32: 25.23 GB
Memory on BF16: 12.61 GB
Memory on FP8: 37.23 GB <- Warning sign that something is amiss!

After fix:
Memory on FP8:

1. Doing .cuda() before (current implementation in .prepare()): 12.87 GB
2. Doing .cuda() after: 12.61 GB

Fixes # (issue)

(Finally) fixes #1430

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@BenjaminBossan @LysandreJik

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[BenjaminBossan]

Very good find, thanks for investigating and fixing this.

The movement of the code block in accelerator.py is presumably due to the order in which convert_model needs to be called?

(note for later: the 2nd error message in that block is messed up)

[muellerzr]

@BenjaminBossan correct, it's that note about how we can save extra vram by converting to fp8 before moving the model to CUDA

[overvalidated]

So that's how it should've been done... Thank you, it's amazing to see this is finally resolved.
Even though I have already traded my 4090 for 3090s xD.

[LysandreJik]

Thanks for investigating this @muellerzr!