expected dtype float for end but got dtype c10::BFloat16

[tcxia]

Running the gitcode encountered the above error, the full error is as follows:

transformers == 4.32.1
torch==2.2.0

[Vashistht]

getting the same error with the newest torch and transformers==4.36.2

[daegonYu]

The same error occurs in the version below.

---

absl-py 2.1.0
accelerate 1.1.0
addict 2.4.0
aiobotocore 2.15.2
aiofiles 23.2.1
aiohappyeyeballs 2.4.3
aiohttp 3.11.7
aioitertools 0.12.0
aiosignal 1.3.1
altair 5.5.0
annotated-types 0.7.0
antlr4-python3-runtime 4.13.2
anyio 4.6.2.post1
art 6.4
asttokens 2.4.1
async-timeout 5.0.1
attrs 24.2.0
axolotl 0.5.2 /home/dgon/LLM/gits/axolotl/src
bitsandbytes 0.44.1
botocore 1.35.36
cachetools 5.5.0
certifi 2024.8.30
chardet 5.2.0
charset-normalizer 3.4.0
click 8.1.7
cmake 3.31.1
colorama 0.4.6
coloredlogs 15.0.1
comm 0.2.2
contourpy 1.3.1
cycler 0.12.1
DataProperty 1.0.1
datasets 3.1.0
debugpy 1.8.9
decorator 5.1.1
deepspeed 0.15.4
deepspeed-kernels 0.0.1.dev1698255861
dill 0.3.8
docker-pycreds 0.4.0
einops 0.8.0
evaluate 0.4.1
exceptiongroup 1.2.2
executing 2.1.0
fastapi 0.115.5
fastcore 1.7.20
ffmpy 0.4.0
filelock 3.13.1
fire 0.7.0
flash-attn 2.7.0.post2
fonttools 4.55.0
frozenlist 1.5.0
fsspec 2024.9.0
gcsfs 2024.9.0.post1
gitdb 4.0.11
GitPython 3.1.43
google-api-core 2.23.0
google-auth 2.36.0
google-auth-oauthlib 1.2.1
google-cloud-core 2.4.1
google-cloud-storage 2.18.2
google-crc32c 1.6.0
google-resumable-media 2.7.2
googleapis-common-protos 1.66.0
gradio 3.50.2
gradio_client 0.6.1
grpcio 1.68.0
h11 0.14.0
hf_transfer 0.1.8
hjson 3.1.0
httpcore 1.0.7
httpx 0.27.2
huggingface-hub 0.26.2
humanfriendly 10.0
idna 3.10
immutabledict 4.2.0
importlib_metadata 8.5.0
importlib_resources 6.4.5
ipykernel 6.29.5
ipython 8.29.0
jedi 0.19.2
Jinja2 3.1.3
jmespath 1.0.1
joblib 1.4.2
jsonlines 4.0.0
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
jupyter_client 8.6.3
jupyter_core 5.7.2
kiwisolver 1.4.7
langdetect 1.0.9
liger_kernel 0.4.2
llvmlite 0.43.0
lm_eval 0.4.4
lxml 5.3.0
Markdown 3.7
markdown-it-py 3.0.0
MarkupSafe 2.1.5
matplotlib 3.9.2
matplotlib-inline 0.1.7
mbstrdecoder 1.1.3
mdurl 0.1.2
more-itertools 10.5.0
mpmath 1.3.0
msgpack 1.1.0
multidict 6.1.0
multiprocess 0.70.16
narwhals 1.14.2
nest_asyncio 1.6.0
networkx 3.2.1
ninja 1.11.1.2
nltk 3.9.1
numba 0.60.0
numexpr 2.10.2
numpy 1.26.3
nvidia-cublas-cu11 11.11.3.6
nvidia-cuda-cupti-cu11 11.8.87
nvidia-cuda-nvrtc-cu11 11.8.89
nvidia-cuda-runtime-cu11 11.8.89
nvidia-cudnn-cu11 9.1.0.70
nvidia-cufft-cu11 10.9.0.58
nvidia-curand-cu11 10.3.0.86
nvidia-cusolver-cu11 11.4.1.48
nvidia-cusparse-cu11 11.7.5.86
nvidia-ml-py 12.560.30
nvidia-nccl-cu11 2.21.5
nvidia-nvtx-cu11 11.8.86
oauthlib 3.2.2
optimum 1.16.2
orjson 3.10.12
packaging 23.2
pandas 2.2.3
parso 0.8.4
pathvalidate 3.2.1
peft 0.13.2
pexpect 4.9.0
pickleshare 0.7.5
pillow 10.2.0
pip 24.3.1
platformdirs 4.3.6
portalocker 3.0.0
prompt_toolkit 3.0.48
propcache 0.2.0
proto-plus 1.25.0
protobuf 5.29.0
psutil 6.1.0
ptyprocess 0.7.0
pure_eval 0.2.3
py-cpuinfo 9.0.0
pyarrow 18.1.0
pyasn1 0.6.1
pyasn1_modules 0.4.1
pybind11 2.13.6
pydantic 2.6.3
pydantic_core 2.16.3
pydub 0.25.1
Pygments 2.18.0
pynvml 11.5.3
pyparsing 3.2.0
pytablewriter 1.2.0
python-dateutil 2.9.0.post0
python-dotenv 1.0.1
python-multipart 0.0.17
pytz 2024.2
PyYAML 6.0.2
pyzmq 26.2.0
referencing 0.35.1
regex 2024.11.6
requests 2.32.3
requests-oauthlib 2.0.0
responses 0.18.0
rich 13.9.4
rouge_score 0.1.2
rpds-py 0.21.0
rsa 4.9
s3fs 2024.9.0
sacrebleu 2.4.3
safetensors 0.4.5
schedulefree 1.3
scikit-learn 1.4.2
scipy 1.14.1
semantic-version 2.10.0
sentencepiece 0.2.0
sentry-sdk 2.19.0
setproctitle 1.3.4
setuptools 75.6.0
six 1.16.0
smmap 5.0.1
sniffio 1.3.1
sqlitedict 2.1.0
stack-data 0.6.2
starlette 0.41.3
sympy 1.13.1
tabledata 1.3.3
tabulate 0.9.0
tcolorpy 0.1.6
tensorboard 2.18.0
tensorboard-data-server 0.7.2
termcolor 2.5.0
threadpoolctl 3.5.0
tokenizers 0.20.4
torch 2.5.1+cu118
torchao 0.5.0
torchaudio 2.5.1+cu118
torchvision 0.20.1+cu118
tornado 6.4.2
tqdm 4.67.1
tqdm-multiprocess 0.0.11
traitlets 5.14.3
transformers 4.46.3
triton 3.1.0
trl 0.12.0
typepy 1.3.2
typing_extensions 4.12.2
tzdata 2024.2
urllib3 2.2.3
uvicorn 0.32.1
wandb 0.18.7
wcwidth 0.2.13
websockets 11.0.3
Werkzeug 3.1.3
wheel 0.45.1
word2number 1.1
wrapt 1.17.0
xformers 0.0.28.post3
xxhash 3.5.0
yarl 1.18.0
zipp 3.21.0
zstandard 0.22.0

[daegonYu]

I'm getting this error at the end of epoch 1. What could be the problem?

[kfchenhn]

I'm getting this error at the end of epoch 1. What could be the problem?

Me too. Have you solved this problem？

[daegonYu]

huggingface/transformers#34702

Take a look here!