Fix memory leak in fp8 causing OOM (and potentially 3x vRAM usage)

Fix memory leak in fp8 causing OOM (and potentially 3x vRAM usage) #3399