Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update OpenXLA-pin to Nov24 #6012

Merged
merged 10 commits into from
Dec 5, 2023
Merged

Update OpenXLA-pin to Nov24 #6012

merged 10 commits into from
Dec 5, 2023

Conversation

ManfeiBai
Copy link
Collaborator

@ManfeiBai ManfeiBai commented Dec 4, 2023 •
edited
Loading

update OpenXLA-pin update to Nov24, with GPU test speed up after #5970

@ManfeiBai ManfeiBai mentioned this pull request Dec 4, 2023
Closed
@ManfeiBai ManfeiBai changed the title [test] test openxla-pin update to nov24 [OpenXLA PIN]update openxla-pin to nov24 Dec 4, 2023
@ManfeiBai ManfeiBai marked this pull request as ready for review December 4, 2023 07:33
@ManfeiBai ManfeiBai requested a review from vanbasten23 December 4, 2023 18:22
@ManfeiBai ManfeiBai changed the title [OpenXLA PIN]update openxla-pin to nov24 [OpenXLA PIN] Update OpenXLA-pin to Nov24 Dec 4, 2023
@ManfeiBai ManfeiBai changed the title [OpenXLA PIN] Update OpenXLA-pin to Nov24 Update OpenXLA-pin to Nov24 Dec 4, 2023
@ManfeiBai ManfeiBai mentioned this pull request Dec 4, 2023
Closed
@JackCaoG
Copy link
Collaborator

JackCaoG commented Dec 4, 2023

@ManfeiBai did you manually check the resnet perfomrance before and after this update on TPU?

@ManfeiBai
Copy link
Collaborator Author

ManfeiBai commented Dec 5, 2023 •
edited
Loading

this is resent train before update openxla-pin on TPU:

PJRT_DEVICE=TPU python test/test_train_mp_imagenet.py --fake_data --num_epochs=1
==> Preparing data..
==> Preparing data..
==> Preparing data..
==> Preparing data..
Epoch 1 train begin 23:51:16
| Training Device=xla:0/3 Epoch=1 Step=0 Loss=6.89059 Rate=2.02 GlobalRate=2.02 Time=23:52:16
| Training Device=xla:0/1 Epoch=1 Step=0 Loss=6.89059 Rate=2.03 GlobalRate=2.03 Time=23:52:16
| Training Device=xla:0/0 Epoch=1 Step=0 Loss=6.89059 Rate=2.12 GlobalRate=2.12 Time=23:52:16
| Training Device=xla:0/2 Epoch=1 Step=0 Loss=6.89059 Rate=2.15 GlobalRate=2.15 Time=23:52:16
| Training Device=xla:0/0 Epoch=1 Step=20 Loss=6.50001 Rate=24.84 GlobalRate=21.60 Time=23:53:20
| Training Device=xla:0/2 Epoch=1 Step=20 Loss=6.50001 Rate=24.85 GlobalRate=21.75 Time=23:53:20
| Training Device=xla:0/3 Epoch=1 Step=20 Loss=6.50001 Rate=24.80 GlobalRate=21.10 Time=23:53:20
| Training Device=xla:0/1 Epoch=1 Step=20 Loss=6.50001 Rate=24.81 GlobalRate=21.15 Time=23:53:20
| Training Device=xla:0/1 Epoch=1 Step=40 Loss=5.07708 Rate=1066.64 GlobalRate=40.83 Time=23:53:22
| Training Device=xla:0/0 Epoch=1 Step=40 Loss=5.07708 Rate=1066.00 GlobalRate=41.69 Time=23:53:22
| Training Device=xla:0/3 Epoch=1 Step=40 Loss=5.07708 Rate=1066.57 GlobalRate=40.73 Time=23:53:22
| Training Device=xla:0/2 Epoch=1 Step=40 Loss=5.07708 Rate=1065.94 GlobalRate=41.98 Time=23:53:22
| Training Device=xla:0/0 Epoch=1 Step=60 Loss=2.73656 Rate=1488.76 GlobalRate=61.33 Time=23:53:23
| Training Device=xla:0/1 Epoch=1 Step=60 Loss=2.73656 Rate=1488.92 GlobalRate=60.07 Time=23:53:23
| Training Device=xla:0/2 Epoch=1 Step=60 Loss=2.73656 Rate=1489.04 GlobalRate=61.74 Time=23:53:23
| Training Device=xla:0/3 Epoch=1 Step=60 Loss=2.73656 Rate=1488.46 GlobalRate=59.93 Time=23:53:23
| Training Device=xla:0/0 Epoch=1 Step=80 Loss=0.57112 Rate=1651.54 GlobalRate=80.51 Time=23:53:25
| Training Device=xla:0/1 Epoch=1 Step=80 Loss=0.57112 Rate=1651.59 GlobalRate=78.89 Time=23:53:25
| Training Device=xla:0/2 Epoch=1 Step=80 Loss=0.57112 Rate=1651.69 GlobalRate=81.05 Time=23:53:25
| Training Device=xla:0/3 Epoch=1 Step=80 Loss=0.57112 Rate=1651.30 GlobalRate=78.70 Time=23:53:25
| Training Device=xla:0/0 Epoch=1 Step=100 Loss=0.11328 Rate=1716.82 GlobalRate=99.27 Time=23:53:26
| Training Device=xla:0/1 Epoch=1 Step=100 Loss=0.11328 Rate=1716.73 GlobalRate=97.29 Time=23:53:26
| Training Device=xla:0/2 Epoch=1 Step=100 Loss=0.11328 Rate=1716.76 GlobalRate=99.92 Time=23:53:26
| Training Device=xla:0/3 Epoch=1 Step=100 Loss=0.11328 Rate=1716.96 GlobalRate=97.06 Time=23:53:26
| Training Device=xla:0/0 Epoch=1 Step=120 Loss=0.05495 Rate=1742.00 GlobalRate=117.61 Time=23:53:28
| Training Device=xla:0/3 Epoch=1 Step=120 Loss=0.05495 Rate=1742.29 GlobalRate=115.02 Time=23:53:28
| Training Device=xla:0/2 Epoch=1 Step=120 Loss=0.05495 Rate=1741.83 GlobalRate=118.38 Time=23:53:28
| Training Device=xla:0/1 Epoch=1 Step=120 Loss=0.05495 Rate=1741.51 GlobalRate=115.29 Time=23:53:28
| Training Device=xla:0/3 Epoch=1 Step=140 Loss=0.03776 Rate=1751.14 GlobalRate=132.59 Time=23:53:29
| Training Device=xla:0/1 Epoch=1 Step=140 Loss=0.03776 Rate=1751.19 GlobalRate=132.90 Time=23:53:29
| Training Device=xla:0/0 Epoch=1 Step=140 Loss=0.03776 Rate=1750.79 GlobalRate=135.56 Time=23:53:29
| Training Device=xla:0/2 Epoch=1 Step=140 Loss=0.03776 Rate=1751.08 GlobalRate=136.42 Time=23:53:29
| Training Device=xla:0/3 Epoch=1 Step=160 Loss=0.02887 Rate=1753.10 GlobalRate=149.80 Time=23:53:31
| Training Device=xla:0/0 Epoch=1 Step=160 Loss=0.02887 Rate=1753.06 GlobalRate=153.11 Time=23:53:31
| Training Device=xla:0/1 Epoch=1 Step=160 Loss=0.02887 Rate=1753.21 GlobalRate=150.14 Time=23:53:31
| Training Device=xla:0/2 Epoch=1 Step=160 Loss=0.02887 Rate=1753.16 GlobalRate=154.08 Time=23:53:31
| Training Device=xla:0/2 Epoch=1 Step=180 Loss=0.02308 Rate=1763.19 GlobalRate=171.36 Time=23:53:32
| Training Device=xla:0/1 Epoch=1 Step=180 Loss=0.02308 Rate=1763.12 GlobalRate=167.03 Time=23:53:32
| Training Device=xla:0/0 Epoch=1 Step=180 Loss=0.02308 Rate=1763.05 GlobalRate=170.29 Time=23:53:32
| Training Device=xla:0/3 Epoch=1 Step=180 Loss=0.02308 Rate=1762.95 GlobalRate=166.65 Time=23:53:32
| Training Device=xla:0/3 Epoch=1 Step=200 Loss=0.01895 Rate=1768.43 GlobalRate=183.16 Time=23:53:33
| Training Device=xla:0/0 Epoch=1 Step=200 Loss=0.01895 Rate=1768.20 GlobalRate=187.12 Time=23:53:33
| Training Device=xla:0/1 Epoch=1 Step=200 Loss=0.01895 Rate=1768.17 GlobalRate=183.58 Time=23:53:33
| Training Device=xla:0/2 Epoch=1 Step=200 Loss=0.01895 Rate=1768.10 GlobalRate=188.28 Time=23:53:33
| Training Device=xla:0/1 Epoch=1 Step=220 Loss=0.01588 Rate=1746.74 GlobalRate=199.74 Time=23:53:35
| Training Device=xla:0/3 Epoch=1 Step=220 Loss=0.01588 Rate=1746.61 GlobalRate=199.29 Time=23:53:35
| Training Device=xla:0/2 Epoch=1 Step=220 Loss=0.01588 Rate=1746.67 GlobalRate=204.80 Time=23:53:35
| Training Device=xla:0/0 Epoch=1 Step=220 Loss=0.01588 Rate=1746.20 GlobalRate=203.55 Time=23:53:35
| Training Device=xla:0/0 Epoch=1 Step=240 Loss=0.01353 Rate=1758.85 GlobalRate=219.69 Time=23:53:36
| Training Device=xla:0/2 Epoch=1 Step=240 Loss=0.01353 Rate=1758.60 GlobalRate=221.02 Time=23:53:36
| Training Device=xla:0/3 Epoch=1 Step=240 Loss=0.01353 Rate=1758.41 GlobalRate=215.13 Time=23:53:36
| Training Device=xla:0/1 Epoch=1 Step=240 Loss=0.01353 Rate=1758.50 GlobalRate=215.61 Time=23:53:36
| Training Device=xla:0/0 Epoch=1 Step=260 Loss=0.01169 Rate=1763.75 GlobalRate=235.49 Time=23:53:38
| Training Device=xla:0/1 Epoch=1 Step=260 Loss=0.01169 Rate=1763.66 GlobalRate=231.16 Time=23:53:38
| Training Device=xla:0/3 Epoch=1 Step=260 Loss=0.01169 Rate=1763.60 GlobalRate=230.65 Time=23:53:38
| Training Device=xla:0/2 Epoch=1 Step=260 Loss=0.01169 Rate=1763.27 GlobalRate=236.90 Time=23:53:38
| Training Device=xla:0/3 Epoch=1 Step=280 Loss=0.01022 Rate=1761.39 GlobalRate=245.86 Time=23:53:39
| Training Device=xla:0/0 Epoch=1 Step=280 Loss=0.01022 Rate=1761.34 GlobalRate=250.96 Time=23:53:39
| Training Device=xla:0/1 Epoch=1 Step=280 Loss=0.01022 Rate=1761.17 GlobalRate=246.39 Time=23:53:39
| Training Device=xla:0/2 Epoch=1 Step=280 Loss=0.01022 Rate=1760.83 GlobalRate=252.45 Time=23:53:39
| Training Device=xla:0/0 Epoch=1 Step=300 Loss=0.00902 Rate=1766.55 GlobalRate=266.13 Time=23:53:41
| Training Device=xla:0/3 Epoch=1 Step=300 Loss=0.00902 Rate=1766.52 GlobalRate=260.78 Time=23:53:41
| Training Device=xla:0/2 Epoch=1 Step=300 Loss=0.00902 Rate=1766.96 GlobalRate=267.70 Time=23:53:41
| Training Device=xla:0/1 Epoch=1 Step=300 Loss=0.00902 Rate=1766.33 GlobalRate=261.34 Time=23:53:41
| Training Device=xla:0/0 Epoch=1 Step=320 Loss=0.00804 Rate=1766.69 GlobalRate=281.01 Time=23:53:42
| Training Device=xla:0/3 Epoch=1 Step=320 Loss=0.00804 Rate=1766.74 GlobalRate=275.41 Time=23:53:42
| Training Device=xla:0/1 Epoch=1 Step=320 Loss=0.00804 Rate=1766.86 GlobalRate=275.99 Time=23:53:42
| Training Device=xla:0/2 Epoch=1 Step=320 Loss=0.00804 Rate=1766.90 GlobalRate=282.64 Time=23:53:42
| Training Device=xla:0/2 Epoch=1 Step=340 Loss=0.00722 Rate=1758.62 GlobalRate=297.27 Time=23:53:44
| Training Device=xla:0/1 Epoch=1 Step=340 Loss=0.00722 Rate=1758.58 GlobalRate=290.34 Time=23:53:44
| Training Device=xla:0/0 Epoch=1 Step=340 Loss=0.00722 Rate=1758.47 GlobalRate=295.56 Time=23:53:44
| Training Device=xla:0/3 Epoch=1 Step=340 Loss=0.00722 Rate=1758.36 GlobalRate=289.73 Time=23:53:44
| Training Device=xla:0/3 Epoch=1 Step=360 Loss=0.00654 Rate=1762.25 GlobalRate=303.80 Time=23:53:45
| Training Device=xla:0/2 Epoch=1 Step=360 Loss=0.00654 Rate=1762.41 GlobalRate=311.63 Time=23:53:45
| Training Device=xla:0/1 Epoch=1 Step=360 Loss=0.00654 Rate=1762.40 GlobalRate=304.43 Time=23:53:45
| Training Device=xla:0/0 Epoch=1 Step=360 Loss=0.00654 Rate=1761.69 GlobalRate=309.85 Time=23:53:45
| Training Device=xla:0/0 Epoch=1 Step=380 Loss=0.00595 Rate=1737.85 GlobalRate=323.79 Time=23:53:47
| Training Device=xla:0/1 Epoch=1 Step=380 Loss=0.00595 Rate=1737.66 GlobalRate=318.18 Time=23:53:47
| Training Device=xla:0/3 Epoch=1 Step=380 Loss=0.00595 Rate=1737.47 GlobalRate=317.52 Time=23:53:47
| Training Device=xla:0/2 Epoch=1 Step=380 Loss=0.00595 Rate=1737.58 GlobalRate=325.62 Time=23:53:47
| Training Device=xla:0/2 Epoch=1 Step=400 Loss=0.00545 Rate=1757.26 GlobalRate=339.44 Time=23:53:48
| Training Device=xla:0/1 Epoch=1 Step=400 Loss=0.00545 Rate=1757.05 GlobalRate=331.75 Time=23:53:48
| Training Device=xla:0/3 Epoch=1 Step=400 Loss=0.00545 Rate=1757.29 GlobalRate=331.07 Time=23:53:48
| Training Device=xla:0/0 Epoch=1 Step=400 Loss=0.00545 Rate=1757.24 GlobalRate=337.55 Time=23:53:48
| Training Device=xla:0/3 Epoch=1 Step=420 Loss=0.00502 Rate=1756.16 GlobalRate=344.35 Time=23:53:49
| Training Device=xla:0/0 Epoch=1 Step=420 Loss=0.00502 Rate=1756.23 GlobalRate=351.01 Time=23:53:49
| Training Device=xla:0/1 Epoch=1 Step=420 Loss=0.00502 Rate=1756.04 GlobalRate=345.05 Time=23:53:49
| Training Device=xla:0/2 Epoch=1 Step=420 Loss=0.00502 Rate=1755.84 GlobalRate=352.96 Time=23:53:49
| Training Device=xla:0/0 Epoch=1 Step=440 Loss=0.00465 Rate=1738.37 GlobalRate=364.17 Time=23:53:51
| Training Device=xla:0/3 Epoch=1 Step=440 Loss=0.00465 Rate=1738.35 GlobalRate=357.32 Time=23:53:51
| Training Device=xla:0/1 Epoch=1 Step=440 Loss=0.00465 Rate=1738.29 GlobalRate=358.04 Time=23:53:51
| Training Device=xla:0/2 Epoch=1 Step=440 Loss=0.00465 Rate=1738.25 GlobalRate=366.18 Time=23:53:51
| Training Device=xla:0/1 Epoch=1 Step=460 Loss=0.00432 Rate=1757.14 GlobalRate=370.87 Time=23:53:52
| Training Device=xla:0/2 Epoch=1 Step=460 Loss=0.00432 Rate=1757.36 GlobalRate=379.22 Time=23:53:52
| Training Device=xla:0/0 Epoch=1 Step=460 Loss=0.00432 Rate=1757.12 GlobalRate=377.17 Time=23:53:52
| Training Device=xla:0/3 Epoch=1 Step=460 Loss=0.00432 Rate=1757.14 GlobalRate=370.13 Time=23:53:52
| Training Device=xla:0/3 Epoch=1 Step=480 Loss=0.00403 Rate=1763.55 GlobalRate=382.72 Time=23:53:54
| Training Device=xla:0/2 Epoch=1 Step=480 Loss=0.00403 Rate=1763.53 GlobalRate=392.03 Time=23:53:54
| Training Device=xla:0/1 Epoch=1 Step=480 Loss=0.00403 Rate=1763.46 GlobalRate=383.47 Time=23:53:54
| Training Device=xla:0/0 Epoch=1 Step=480 Loss=0.00403 Rate=1763.43 GlobalRate=389.92 Time=23:53:54
| Training Device=xla:0/1 Epoch=1 Step=500 Loss=0.00378 Rate=1766.55 GlobalRate=395.85 Time=23:53:55
| Training Device=xla:0/3 Epoch=1 Step=500 Loss=0.00378 Rate=1766.58 GlobalRate=395.07 Time=23:53:55
| Training Device=xla:0/0 Epoch=1 Step=500 Loss=0.00378 Rate=1766.71 GlobalRate=402.45 Time=23:53:55
| Training Device=xla:0/2 Epoch=1 Step=500 Loss=0.00378 Rate=1766.58 GlobalRate=404.60 Time=23:53:55
| Training Device=xla:0/2 Epoch=1 Step=520 Loss=0.00356 Rate=1769.50 GlobalRate=416.95 Time=23:53:57
| Training Device=xla:0/1 Epoch=1 Step=520 Loss=0.00356 Rate=1769.63 GlobalRate=408.01 Time=23:53:57
| Training Device=xla:0/3 Epoch=1 Step=520 Loss=0.00356 Rate=1769.56 GlobalRate=407.22 Time=23:53:57
| Training Device=xla:0/0 Epoch=1 Step=520 Loss=0.00356 Rate=1769.42 GlobalRate=414.75 Time=23:53:57
| Training Device=xla:0/0 Epoch=1 Step=540 Loss=0.00336 Rate=1768.18 GlobalRate=426.83 Time=23:53:58
| Training Device=xla:0/1 Epoch=1 Step=540 Loss=0.00336 Rate=1767.94 GlobalRate=419.95 Time=23:53:58
| Training Device=xla:0/2 Epoch=1 Step=540 Loss=0.00336 Rate=1767.94 GlobalRate=429.07 Time=23:53:58
| Training Device=xla:0/3 Epoch=1 Step=540 Loss=0.00336 Rate=1767.08 GlobalRate=419.14 Time=23:53:58
| Training Device=xla:0/0 Epoch=1 Step=560 Loss=0.00318 Rate=1766.28 GlobalRate=438.68 Time=23:54:00
| Training Device=xla:0/2 Epoch=1 Step=560 Loss=0.00318 Rate=1766.24 GlobalRate=440.97 Time=23:54:00
| Training Device=xla:0/1 Epoch=1 Step=560 Loss=0.00318 Rate=1766.22 GlobalRate=431.68 Time=23:54:00
| Training Device=xla:0/3 Epoch=1 Step=560 Loss=0.00318 Rate=1766.63 GlobalRate=430.86 Time=23:54:00
| Training Device=xla:0/0 Epoch=1 Step=580 Loss=0.00302 Rate=1769.43 GlobalRate=450.35 Time=23:54:01
| Training Device=xla:0/1 Epoch=1 Step=580 Loss=0.00302 Rate=1769.57 GlobalRate=443.22 Time=23:54:01
| Training Device=xla:0/3 Epoch=1 Step=580 Loss=0.00302 Rate=1769.63 GlobalRate=442.38 Time=23:54:01
| Training Device=xla:0/2 Epoch=1 Step=580 Loss=0.00302 Rate=1769.51 GlobalRate=452.67 Time=23:54:01
| Training Device=xla:0/0 Epoch=1 Step=600 Loss=0.00287 Rate=1770.77 GlobalRate=461.81 Time=23:54:03
| Training Device=xla:0/1 Epoch=1 Step=600 Loss=0.00287 Rate=1770.84 GlobalRate=454.56 Time=23:54:03
| Training Device=xla:0/3 Epoch=1 Step=600 Loss=0.00287 Rate=1770.79 GlobalRate=453.71 Time=23:54:03
| Training Device=xla:0/2 Epoch=1 Step=600 Loss=0.00287 Rate=1770.78 GlobalRate=464.17 Time=23:54:03
| Training Device=xla:0/1 Epoch=1 Step=620 Loss=0.00274 Rate=1770.04 GlobalRate=465.71 Time=23:54:04
| Training Device=xla:0/2 Epoch=1 Step=620 Loss=0.00274 Rate=1769.97 GlobalRate=475.47 Time=23:54:04
| Training Device=xla:0/0 Epoch=1 Step=620 Loss=0.00274 Rate=1769.78 GlobalRate=473.07 Time=23:54:04
| Training Device=xla:0/3 Epoch=1 Step=620 Loss=0.00274 Rate=1770.09 GlobalRate=464.84 Time=23:54:04
| Training Device=xla:0/0 Epoch=1 Step=640 Loss=0.00263 Rate=1766.56 GlobalRate=484.12 Time=23:54:05
| Training Device=xla:0/1 Epoch=1 Step=640 Loss=0.00263 Rate=1766.26 GlobalRate=476.65 Time=23:54:05
| Training Device=xla:0/2 Epoch=1 Step=640 Loss=0.00263 Rate=1766.43 GlobalRate=486.56 Time=23:54:05
| Training Device=xla:0/3 Epoch=1 Step=640 Loss=0.00263 Rate=1766.20 GlobalRate=475.77 Time=23:54:05
| Training Device=xla:0/0 Epoch=1 Step=660 Loss=0.00252 Rate=1760.11 GlobalRate=494.97 Time=23:54:07
| Training Device=xla:0/3 Epoch=1 Step=660 Loss=0.00252 Rate=1760.22 GlobalRate=486.51 Time=23:54:07
| Training Device=xla:0/1 Epoch=1 Step=660 Loss=0.00252 Rate=1760.02 GlobalRate=487.40 Time=23:54:07
| Training Device=xla:0/2 Epoch=1 Step=660 Loss=0.00252 Rate=1760.04 GlobalRate=497.44 Time=23:54:07
| Training Device=xla:0/2 Epoch=1 Step=680 Loss=0.00242 Rate=1766.11 GlobalRate=508.17 Time=23:54:08
| Training Device=xla:0/1 Epoch=1 Step=680 Loss=0.00242 Rate=1766.04 GlobalRate=498.00 Time=23:54:08
| Training Device=xla:0/0 Epoch=1 Step=680 Loss=0.00242 Rate=1766.01 GlobalRate=505.67 Time=23:54:08
| Training Device=xla:0/3 Epoch=1 Step=680 Loss=0.00242 Rate=1766.12 GlobalRate=497.09 Time=23:54:08
| Training Device=xla:0/0 Epoch=1 Step=700 Loss=0.00234 Rate=1771.28 GlobalRate=516.20 Time=23:54:10
| Training Device=xla:0/3 Epoch=1 Step=700 Loss=0.00234 Rate=1771.40 GlobalRate=507.52 Time=23:54:10
| Training Device=xla:0/1 Epoch=1 Step=700 Loss=0.00234 Rate=1771.13 GlobalRate=508.43 Time=23:54:10
| Training Device=xla:0/2 Epoch=1 Step=700 Loss=0.00234 Rate=1771.22 GlobalRate=518.73 Time=23:54:10
| Training Device=xla:0/0 Epoch=1 Step=720 Loss=0.00226 Rate=1771.44 GlobalRate=526.55 Time=23:54:11
| Training Device=xla:0/2 Epoch=1 Step=720 Loss=0.00226 Rate=1771.42 GlobalRate=529.11 Time=23:54:11
| Training Device=xla:0/1 Epoch=1 Step=720 Loss=0.00226 Rate=1771.49 GlobalRate=518.69 Time=23:54:11
| Training Device=xla:0/3 Epoch=1 Step=720 Loss=0.00226 Rate=1771.32 GlobalRate=517.77 Time=23:54:11
| Training Device=xla:0/0 Epoch=1 Step=740 Loss=0.00218 Rate=1771.23 GlobalRate=536.73 Time=23:54:13
| Training Device=xla:0/1 Epoch=1 Step=740 Loss=0.00218 Rate=1771.30 GlobalRate=528.78 Time=23:54:13
| Training Device=xla:0/2 Epoch=1 Step=740 Loss=0.00218 Rate=1771.27 GlobalRate=539.32 Time=23:54:13
| Training Device=xla:0/3 Epoch=1 Step=740 Loss=0.00218 Rate=1770.39 GlobalRate=527.84 Time=23:54:13
| Training Device=xla:0/0 Epoch=1 Step=760 Loss=0.00211 Rate=1749.11 GlobalRate=546.65 Time=23:54:14
| Training Device=xla:0/1 Epoch=1 Step=760 Loss=0.00211 Rate=1748.99 GlobalRate=538.62 Time=23:54:14
| Training Device=xla:0/2 Epoch=1 Step=760 Loss=0.00211 Rate=1749.07 GlobalRate=549.26 Time=23:54:14
| Training Device=xla:0/3 Epoch=1 Step=760 Loss=0.00211 Rate=1749.10 GlobalRate=537.68 Time=23:54:14
| Training Device=xla:0/2 Epoch=1 Step=780 Loss=0.00205 Rate=1759.58 GlobalRate=559.13 Time=23:54:16
| Training Device=xla:0/0 Epoch=1 Step=780 Loss=0.00205 Rate=1759.47 GlobalRate=556.49 Time=23:54:16
| Training Device=xla:0/3 Epoch=1 Step=780 Loss=0.00205 Rate=1760.09 GlobalRate=547.43 Time=23:54:16
| Training Device=xla:0/1 Epoch=1 Step=780 Loss=0.00205 Rate=1759.49 GlobalRate=548.38 Time=23:54:16
| Training Device=xla:0/2 Epoch=1 Step=800 Loss=0.00200 Rate=1767.13 GlobalRate=568.85 Time=23:54:17
| Training Device=xla:0/3 Epoch=1 Step=800 Loss=0.00200 Rate=1767.27 GlobalRate=557.04 Time=23:54:17
| Training Device=xla:0/0 Epoch=1 Step=800 Loss=0.00200 Rate=1767.11 GlobalRate=566.19 Time=23:54:17
| Training Device=xla:0/1 Epoch=1 Step=800 Loss=0.00200 Rate=1767.17 GlobalRate=558.00 Time=23:54:17
| Training Device=xla:0/2 Epoch=1 Step=820 Loss=0.00194 Rate=1771.22 GlobalRate=578.42 Time=23:54:18
| Training Device=xla:0/1 Epoch=1 Step=820 Loss=0.00194 Rate=1771.38 GlobalRate=567.48 Time=23:54:18
| Training Device=xla:0/0 Epoch=1 Step=820 Loss=0.00194 Rate=1771.22 GlobalRate=575.74 Time=23:54:18
| Training Device=xla:0/3 Epoch=1 Step=820 Loss=0.00194 Rate=1771.37 GlobalRate=566.51 Time=23:54:18
| Training Device=xla:0/2 Epoch=1 Step=840 Loss=0.00190 Rate=1773.58 GlobalRate=587.85 Time=23:54:20
| Training Device=xla:0/1 Epoch=1 Step=840 Loss=0.00190 Rate=1773.66 GlobalRate=576.81 Time=23:54:20
| Training Device=xla:0/0 Epoch=1 Step=840 Loss=0.00190 Rate=1773.60 GlobalRate=585.14 Time=23:54:20
| Training Device=xla:0/3 Epoch=1 Step=840 Loss=0.00190 Rate=1773.64 GlobalRate=575.83 Time=23:54:20
| Training Device=xla:0/3 Epoch=1 Step=860 Loss=0.00185 Rate=1771.25 GlobalRate=585.00 Time=23:54:21
| Training Device=xla:0/1 Epoch=1 Step=860 Loss=0.00185 Rate=1771.20 GlobalRate=585.99 Time=23:54:21
| Training Device=xla:0/2 Epoch=1 Step=860 Loss=0.00185 Rate=1771.18 GlobalRate=597.11 Time=23:54:21
| Training Device=xla:0/0 Epoch=1 Step=860 Loss=0.00185 Rate=1771.19 GlobalRate=594.38 Time=23:54:21
| Training Device=xla:0/2 Epoch=1 Step=880 Loss=0.00181 Rate=1773.43 GlobalRate=606.24 Time=23:54:23
| Training Device=xla:0/0 Epoch=1 Step=880 Loss=0.00181 Rate=1773.33 GlobalRate=603.49 Time=23:54:23
| Training Device=xla:0/1 Epoch=1 Step=880 Loss=0.00181 Rate=1773.28 GlobalRate=595.04 Time=23:54:23
| Training Device=xla:0/3 Epoch=1 Step=880 Loss=0.00181 Rate=1773.36 GlobalRate=594.04 Time=23:54:23
| Training Device=xla:0/1 Epoch=1 Step=900 Loss=0.00178 Rate=1771.24 GlobalRate=603.93 Time=23:54:24
| Training Device=xla:0/2 Epoch=1 Step=900 Loss=0.00178 Rate=1771.15 GlobalRate=615.22 Time=23:54:24
| Training Device=xla:0/0 Epoch=1 Step=900 Loss=0.00178 Rate=1771.21 GlobalRate=612.45 Time=23:54:24
| Training Device=xla:0/3 Epoch=1 Step=900 Loss=0.00178 Rate=1771.17 GlobalRate=602.93 Time=23:54:24
| Training Device=xla:0/3 Epoch=1 Step=920 Loss=0.00174 Rate=1758.06 GlobalRate=611.64 Time=23:54:26
| Training Device=xla:0/1 Epoch=1 Step=920 Loss=0.00174 Rate=1758.08 GlobalRate=612.65 Time=23:54:26
| Training Device=xla:0/2 Epoch=1 Step=920 Loss=0.00174 Rate=1758.04 GlobalRate=624.01 Time=23:54:26
| Training Device=xla:0/0 Epoch=1 Step=920 Loss=0.00174 Rate=1758.03 GlobalRate=621.22 Time=23:54:26
| Training Device=xla:0/3 Epoch=1 Step=940 Loss=0.00171 Rate=1758.98 GlobalRate=620.24 Time=23:54:27
| Training Device=xla:0/1 Epoch=1 Step=940 Loss=0.00171 Rate=1759.00 GlobalRate=621.25 Time=23:54:27
| Training Device=xla:0/0 Epoch=1 Step=940 Loss=0.00171 Rate=1758.91 GlobalRate=629.88 Time=23:54:27
| Training Device=xla:0/2 Epoch=1 Step=940 Loss=0.00171 Rate=1758.77 GlobalRate=632.68 Time=23:54:27
| Training Device=xla:0/3 Epoch=1 Step=960 Loss=0.00168 Rate=1768.00 GlobalRate=628.75 Time=23:54:29
| Training Device=xla:0/2 Epoch=1 Step=960 Loss=0.00168 Rate=1768.10 GlobalRate=641.27 Time=23:54:29
| Training Device=xla:0/1 Epoch=1 Step=960 Loss=0.00168 Rate=1767.92 GlobalRate=629.77 Time=23:54:29
| Training Device=xla:0/0 Epoch=1 Step=960 Loss=0.00168 Rate=1767.96 GlobalRate=638.45 Time=23:54:29
| Training Device=xla:0/1 Epoch=1 Step=980 Loss=0.00165 Rate=1768.73 GlobalRate=638.15 Time=23:54:30
| Training Device=xla:0/3 Epoch=1 Step=980 Loss=0.00165 Rate=1768.68 GlobalRate=637.12 Time=23:54:30
| Training Device=xla:0/2 Epoch=1 Step=980 Loss=0.00165 Rate=1768.61 GlobalRate=649.72 Time=23:54:30
| Training Device=xla:0/0 Epoch=1 Step=980 Loss=0.00165 Rate=1768.81 GlobalRate=646.88 Time=23:54:30
| Training Device=xla:0/2 Epoch=1 Step=1000 Loss=0.00163 Rate=1768.87 GlobalRate=658.03 Time=23:54:32
| Training Device=xla:0/1 Epoch=1 Step=1000 Loss=0.00163 Rate=1768.74 GlobalRate=646.40 Time=23:54:32
| Training Device=xla:0/3 Epoch=1 Step=1000 Loss=0.00163 Rate=1768.74 GlobalRate=645.37 Time=23:54:32
| Training Device=xla:0/0 Epoch=1 Step=1000 Loss=0.00163 Rate=1768.76 GlobalRate=655.18 Time=23:54:32
| Training Device=xla:0/2 Epoch=1 Step=1020 Loss=0.00161 Rate=1760.17 GlobalRate=666.19 Time=23:54:33
| Training Device=xla:0/3 Epoch=1 Step=1020 Loss=0.00161 Rate=1760.14 GlobalRate=653.46 Time=23:54:33
| Training Device=xla:0/0 Epoch=1 Step=1020 Loss=0.00161 Rate=1759.98 GlobalRate=663.32 Time=23:54:33
| Training Device=xla:0/1 Epoch=1 Step=1020 Loss=0.00161 Rate=1759.99 GlobalRate=654.50 Time=23:54:33
| Training Device=xla:0/1 Epoch=1 Step=1040 Loss=0.00158 Rate=1764.27 GlobalRate=662.51 Time=23:54:34
| Training Device=xla:0/2 Epoch=1 Step=1040 Loss=0.00158 Rate=1764.34 GlobalRate=674.26 Time=23:54:34
| Training Device=xla:0/3 Epoch=1 Step=1040 Loss=0.00158 Rate=1764.35 GlobalRate=661.47 Time=23:54:34
| Training Device=xla:0/0 Epoch=1 Step=1040 Loss=0.00158 Rate=1764.30 GlobalRate=671.38 Time=23:54:34
| Training Device=xla:0/0 Epoch=1 Step=1060 Loss=0.00156 Rate=1755.95 GlobalRate=679.27 Time=23:54:36
| Training Device=xla:0/3 Epoch=1 Step=1060 Loss=0.00156 Rate=1755.82 GlobalRate=669.32 Time=23:54:36
| Training Device=xla:0/1 Epoch=1 Step=1060 Loss=0.00156 Rate=1755.87 GlobalRate=670.37 Time=23:54:36
| Training Device=xla:0/2 Epoch=1 Step=1060 Loss=0.00156 Rate=1755.71 GlobalRate=682.16 Time=23:54:36
| Training Device=xla:0/2 Epoch=1 Step=1080 Loss=0.00155 Rate=1767.48 GlobalRate=690.02 Time=23:54:37
| Training Device=xla:0/0 Epoch=1 Step=1080 Loss=0.00155 Rate=1767.44 GlobalRate=687.12 Time=23:54:37
| Training Device=xla:0/1 Epoch=1 Step=1080 Loss=0.00155 Rate=1767.48 GlobalRate=678.18 Time=23:54:37
| Training Device=xla:0/3 Epoch=1 Step=1080 Loss=0.00155 Rate=1767.35 GlobalRate=677.12 Time=23:54:37
| Training Device=xla:0/2 Epoch=1 Step=1100 Loss=0.00153 Rate=1770.58 GlobalRate=697.77 Time=23:54:39
| Training Device=xla:0/3 Epoch=1 Step=1100 Loss=0.00153 Rate=1770.56 GlobalRate=684.81 Time=23:54:39
| Training Device=xla:0/1 Epoch=1 Step=1100 Loss=0.00153 Rate=1770.33 GlobalRate=685.87 Time=23:54:39
| Training Device=xla:0/0 Epoch=1 Step=1100 Loss=0.00153 Rate=1770.48 GlobalRate=694.85 Time=23:54:39
| Training Device=xla:0/0 Epoch=1 Step=1120 Loss=0.00152 Rate=1770.99 GlobalRate=702.46 Time=23:54:40
| Training Device=xla:0/2 Epoch=1 Step=1120 Loss=0.00152 Rate=1770.92 GlobalRate=705.39 Time=23:54:40
| Training Device=xla:0/1 Epoch=1 Step=1120 Loss=0.00152 Rate=1770.98 GlobalRate=693.45 Time=23:54:40
| Training Device=xla:0/3 Epoch=1 Step=1120 Loss=0.00152 Rate=1770.77 GlobalRate=692.39 Time=23:54:40
| Training Device=xla:0/0 Epoch=1 Step=1140 Loss=0.00150 Rate=1772.97 GlobalRate=709.98 Time=23:54:42
| Training Device=xla:0/3 Epoch=1 Step=1140 Loss=0.00150 Rate=1772.96 GlobalRate=699.87 Time=23:54:42
| Training Device=xla:0/1 Epoch=1 Step=1140 Loss=0.00150 Rate=1772.94 GlobalRate=700.93 Time=23:54:42
| Training Device=xla:0/2 Epoch=1 Step=1140 Loss=0.00150 Rate=1772.92 GlobalRate=712.92 Time=23:54:42
| Training Device=xla:0/2 Epoch=1 Step=1160 Loss=0.00149 Rate=1772.36 GlobalRate=720.34 Time=23:54:43
| Training Device=xla:0/0 Epoch=1 Step=1160 Loss=0.00149 Rate=1772.31 GlobalRate=717.39 Time=23:54:43
| Training Device=xla:0/3 Epoch=1 Step=1160 Loss=0.00149 Rate=1772.42 GlobalRate=707.24 Time=23:54:43
| Training Device=xla:0/1 Epoch=1 Step=1160 Loss=0.00149 Rate=1772.00 GlobalRate=708.31 Time=23:54:43
| Training Device=xla:0/1 Epoch=1 Step=1180 Loss=0.00147 Rate=1769.48 GlobalRate=715.57 Time=23:54:45
| Training Device=xla:0/0 Epoch=1 Step=1180 Loss=0.00147 Rate=1769.09 GlobalRate=724.68 Time=23:54:45
| Training Device=xla:0/2 Epoch=1 Step=1180 Loss=0.00147 Rate=1769.12 GlobalRate=727.64 Time=23:54:45
| Training Device=xla:0/3 Epoch=1 Step=1180 Loss=0.00147 Rate=1769.13 GlobalRate=714.49 Time=23:54:45
| Training Device=xla:0/2 Epoch=1 Step=1200 Loss=0.00146 Rate=1771.68 GlobalRate=734.85 Time=23:54:46
| Training Device=xla:0/3 Epoch=1 Step=1200 Loss=0.00146 Rate=1771.74 GlobalRate=721.67 Time=23:54:46
| Training Device=xla:0/0 Epoch=1 Step=1200 Loss=0.00146 Rate=1771.72 GlobalRate=731.89 Time=23:54:46
| Training Device=xla:0/1 Epoch=1 Step=1200 Loss=0.00146 Rate=1771.71 GlobalRate=722.75 Time=23:54:46
| Training Device=xla:0/3 Epoch=1 Step=1220 Loss=0.00145 Rate=1773.39 GlobalRate=728.75 Time=23:54:47
| Training Device=xla:0/1 Epoch=1 Step=1220 Loss=0.00145 Rate=1773.47 GlobalRate=729.83 Time=23:54:47
| Training Device=xla:0/2 Epoch=1 Step=1220 Loss=0.00145 Rate=1773.33 GlobalRate=741.97 Time=23:54:47
| Training Device=xla:0/0 Epoch=1 Step=1220 Loss=0.00145 Rate=1773.28 GlobalRate=739.00 Time=23:54:47
| Training Device=xla:0/3 Epoch=1 Step=1240 Loss=0.00144 Rate=1771.17 GlobalRate=735.73 Time=23:54:49
| Training Device=xla:0/0 Epoch=1 Step=1240 Loss=0.00144 Rate=1771.21 GlobalRate=746.00 Time=23:54:49
| Training Device=xla:0/1 Epoch=1 Step=1240 Loss=0.00144 Rate=1771.22 GlobalRate=736.81 Time=23:54:49
| Training Device=xla:0/2 Epoch=1 Step=1240 Loss=0.00144 Rate=1771.07 GlobalRate=748.98 Time=23:54:49
| Training Device=xla:0/0 Epoch=1 Step=1260 Loss=0.00144 Rate=1766.04 GlobalRate=752.89 Time=23:54:50
| Training Device=xla:0/1 Epoch=1 Step=1260 Loss=0.00144 Rate=1766.02 GlobalRate=743.68 Time=23:54:50
| Training Device=xla:0/2 Epoch=1 Step=1260 Loss=0.00144 Rate=1766.01 GlobalRate=755.88 Time=23:54:50
| Training Device=xla:0/3 Epoch=1 Step=1260 Loss=0.00144 Rate=1765.77 GlobalRate=742.59 Time=23:54:50
| Training Device=xla:0/0 Epoch=1 Step=1280 Loss=0.00143 Rate=1771.64 GlobalRate=759.72 Time=23:54:52
| Training Device=xla:0/1 Epoch=1 Step=1280 Loss=0.00143 Rate=1771.69 GlobalRate=750.49 Time=23:54:52
| Training Device=xla:0/3 Epoch=1 Step=1280 Loss=0.00143 Rate=1771.75 GlobalRate=749.40 Time=23:54:52
| Training Device=xla:0/2 Epoch=1 Step=1280 Loss=0.00143 Rate=1771.61 GlobalRate=762.72 Time=23:54:52
| Training Device=xla:0/3 Epoch=1 Step=1300 Loss=0.00142 Rate=1773.97 GlobalRate=756.11 Time=23:54:53
| Training Device=xla:0/1 Epoch=1 Step=1300 Loss=0.00142 Rate=1773.94 GlobalRate=757.21 Time=23:54:53
| Training Device=xla:0/2 Epoch=1 Step=1300 Loss=0.00142 Rate=1774.01 GlobalRate=769.46 Time=23:54:53
| Training Device=xla:0/0 Epoch=1 Step=1300 Loss=0.00142 Rate=1773.83 GlobalRate=766.46 Time=23:54:53
| Training Device=xla:0/2 Epoch=1 Step=1320 Loss=0.00141 Rate=1775.15 GlobalRate=776.12 Time=23:54:55
| Training Device=xla:0/0 Epoch=1 Step=1320 Loss=0.00141 Rate=1775.19 GlobalRate=773.11 Time=23:54:55
| Training Device=xla:0/3 Epoch=1 Step=1320 Loss=0.00141 Rate=1775.12 GlobalRate=762.74 Time=23:54:55
| Training Device=xla:0/1 Epoch=1 Step=1320 Loss=0.00141 Rate=1774.42 GlobalRate=763.84 Time=23:54:55
| Training Device=xla:0/0 Epoch=1 Step=1340 Loss=0.00141 Rate=1763.79 GlobalRate=779.62 Time=23:54:56
| Training Device=xla:0/3 Epoch=1 Step=1340 Loss=0.00141 Rate=1763.75 GlobalRate=769.23 Time=23:54:56
| Training Device=xla:0/1 Epoch=1 Step=1340 Loss=0.00141 Rate=1763.99 GlobalRate=770.33 Time=23:54:56
| Training Device=xla:0/2 Epoch=1 Step=1340 Loss=0.00141 Rate=1763.76 GlobalRate=782.64 Time=23:54:56
| Training Device=xla:0/2 Epoch=1 Step=1360 Loss=0.00140 Rate=1769.04 GlobalRate=789.11 Time=23:54:58
| Training Device=xla:0/0 Epoch=1 Step=1360 Loss=0.00140 Rate=1769.04 GlobalRate=786.09 Time=23:54:58
| Training Device=xla:0/3 Epoch=1 Step=1360 Loss=0.00140 Rate=1768.93 GlobalRate=775.69 Time=23:54:58
| Training Device=xla:0/1 Epoch=1 Step=1360 Loss=0.00140 Rate=1769.19 GlobalRate=776.78 Time=23:54:58
| Training Device=xla:0/0 Epoch=1 Step=1380 Loss=0.00140 Rate=1771.16 GlobalRate=792.48 Time=23:54:59
| Training Device=xla:0/1 Epoch=1 Step=1380 Loss=0.00140 Rate=1771.32 GlobalRate=783.16 Time=23:54:59
| Training Device=xla:0/3 Epoch=1 Step=1380 Loss=0.00140 Rate=1771.20 GlobalRate=782.06 Time=23:54:59
| Training Device=xla:0/2 Epoch=1 Step=1380 Loss=0.00140 Rate=1771.01 GlobalRate=795.50 Time=23:54:59
| Training Device=xla:0/0 Epoch=1 Step=1400 Loss=0.00139 Rate=1751.82 GlobalRate=798.69 Time=23:55:00
| Training Device=xla:0/2 Epoch=1 Step=1400 Loss=0.00139 Rate=1751.93 GlobalRate=801.71 Time=23:55:00
| Training Device=xla:0/3 Epoch=1 Step=1400 Loss=0.00139 Rate=1751.85 GlobalRate=788.25 Time=23:55:00
| Training Device=xla:0/1 Epoch=1 Step=1400 Loss=0.00139 Rate=1751.86 GlobalRate=789.35 Time=23:55:00
| Training Device=xla:0/0 Epoch=1 Step=1420 Loss=0.00139 Rate=1761.29 GlobalRate=804.89 Time=23:55:02
| Training Device=xla:0/2 Epoch=1 Step=1420 Loss=0.00139 Rate=1761.28 GlobalRate=807.93 Time=23:55:02
| Training Device=xla:0/3 Epoch=1 Step=1420 Loss=0.00139 Rate=1761.30 GlobalRate=794.44 Time=23:55:02
| Training Device=xla:0/1 Epoch=1 Step=1420 Loss=0.00139 Rate=1761.17 GlobalRate=795.55 Time=23:55:02
| Training Device=xla:0/2 Epoch=1 Step=1440 Loss=0.00139 Rate=1769.79 GlobalRate=814.08 Time=23:55:03
| Training Device=xla:0/0 Epoch=1 Step=1440 Loss=0.00139 Rate=1769.73 GlobalRate=811.05 Time=23:55:03
| Training Device=xla:0/3 Epoch=1 Step=1440 Loss=0.00139 Rate=1769.81 GlobalRate=800.58 Time=23:55:03
| Training Device=xla:0/1 Epoch=1 Step=1440 Loss=0.00139 Rate=1769.83 GlobalRate=801.69 Time=23:55:03
| Training Device=xla:0/0 Epoch=1 Step=1460 Loss=0.00138 Rate=1772.55 GlobalRate=817.12 Time=23:55:05
| Training Device=xla:0/1 Epoch=1 Step=1460 Loss=0.00138 Rate=1772.61 GlobalRate=807.75 Time=23:55:05
| Training Device=xla:0/2 Epoch=1 Step=1460 Loss=0.00138 Rate=1772.58 GlobalRate=820.16 Time=23:55:05
| Training Device=xla:0/3 Epoch=1 Step=1460 Loss=0.00138 Rate=1772.57 GlobalRate=806.64 Time=23:55:05
| Training Device=xla:0/3 Epoch=1 Step=1480 Loss=0.00138 Rate=1772.39 GlobalRate=812.62 Time=23:55:06
| Training Device=xla:0/2 Epoch=1 Step=1480 Loss=0.00138 Rate=1772.37 GlobalRate=826.15 Time=23:55:06
| Training Device=xla:0/0 Epoch=1 Step=1480 Loss=0.00138 Rate=1772.32 GlobalRate=823.11 Time=23:55:06
| Training Device=xla:0/1 Epoch=1 Step=1480 Loss=0.00138 Rate=1772.37 GlobalRate=813.73 Time=23:55:06
| Training Device=xla:0/1 Epoch=1 Step=1500 Loss=0.00138 Rate=1773.33 GlobalRate=819.64 Time=23:55:08
| Training Device=xla:0/2 Epoch=1 Step=1500 Loss=0.00138 Rate=1773.23 GlobalRate=832.08 Time=23:55:08
| Training Device=xla:0/0 Epoch=1 Step=1500 Loss=0.00138 Rate=1773.29 GlobalRate=829.03 Time=23:55:08
| Training Device=xla:0/3 Epoch=1 Step=1500 Loss=0.00138 Rate=1773.09 GlobalRate=818.53 Time=23:55:08
| Training Device=xla:0/3 Epoch=1 Step=1520 Loss=0.00137 Rate=1773.87 GlobalRate=824.37 Time=23:55:09
| Training Device=xla:0/2 Epoch=1 Step=1520 Loss=0.00137 Rate=1773.78 GlobalRate=837.93 Time=23:55:09
| Training Device=xla:0/0 Epoch=1 Step=1520 Loss=0.00137 Rate=1773.76 GlobalRate=834.88 Time=23:55:09
| Training Device=xla:0/1 Epoch=1 Step=1520 Loss=0.00137 Rate=1773.59 GlobalRate=825.48 Time=23:55:09
| Training Device=xla:0/2 Epoch=1 Step=1540 Loss=0.00137 Rate=1774.89 GlobalRate=843.71 Time=23:55:11
| Training Device=xla:0/0 Epoch=1 Step=1540 Loss=0.00137 Rate=1774.88 GlobalRate=840.66 Time=23:55:11
| Training Device=xla:0/3 Epoch=1 Step=1540 Loss=0.00137 Rate=1774.90 GlobalRate=830.14 Time=23:55:11
| Training Device=xla:0/1 Epoch=1 Step=1540 Loss=0.00137 Rate=1774.82 GlobalRate=831.25 Time=23:55:11
| Training Device=xla:0/0 Epoch=1 Step=1560 Loss=0.00137 Rate=1775.75 GlobalRate=846.37 Time=23:55:12
| Training Device=xla:0/3 Epoch=1 Step=1560 Loss=0.00137 Rate=1775.79 GlobalRate=835.85 Time=23:55:12
| Training Device=xla:0/2 Epoch=1 Step=1560 Loss=0.00137 Rate=1775.74 GlobalRate=849.42 Time=23:55:12
| Training Device=xla:0/1 Epoch=1 Step=1560 Loss=0.00137 Rate=1775.78 GlobalRate=836.96 Time=23:55:12
| Training Device=xla:0/2 Epoch=1 Step=1580 Loss=0.00136 Rate=1775.81 GlobalRate=855.07 Time=23:55:13
| Training Device=xla:0/1 Epoch=1 Step=1580 Loss=0.00136 Rate=1775.91 GlobalRate=842.59 Time=23:55:13
| Training Device=xla:0/0 Epoch=1 Step=1580 Loss=0.00136 Rate=1775.81 GlobalRate=852.01 Time=23:55:13
| Training Device=xla:0/3 Epoch=1 Step=1580 Loss=0.00136 Rate=1775.79 GlobalRate=841.48 Time=23:55:13
| Training Device=xla:0/0 Epoch=1 Step=1600 Loss=0.00136 Rate=1775.53 GlobalRate=857.59 Time=23:55:15
| Training Device=xla:0/1 Epoch=1 Step=1600 Loss=0.00136 Rate=1775.57 GlobalRate=848.16 Time=23:55:15
| Training Device=xla:0/3 Epoch=1 Step=1600 Loss=0.00136 Rate=1775.52 GlobalRate=847.05 Time=23:55:15
| Training Device=xla:0/2 Epoch=1 Step=1600 Loss=0.00136 Rate=1775.48 GlobalRate=860.64 Time=23:55:15
| Training Device=xla:0/2 Epoch=1 Step=1620 Loss=0.00136 Rate=1772.06 GlobalRate=866.13 Time=23:55:16
| Training Device=xla:0/3 Epoch=1 Step=1620 Loss=0.00136 Rate=1772.04 GlobalRate=852.53 Time=23:55:16
| Training Device=xla:0/0 Epoch=1 Step=1620 Loss=0.00136 Rate=1772.02 GlobalRate=863.07 Time=23:55:16
| Training Device=xla:0/1 Epoch=1 Step=1620 Loss=0.00136 Rate=1771.92 GlobalRate=853.64 Time=23:55:16
| Training Device=xla:0/0 Epoch=1 Step=1640 Loss=0.00136 Rate=1773.68 GlobalRate=868.51 Time=23:55:18
| Training Device=xla:0/1 Epoch=1 Step=1640 Loss=0.00136 Rate=1773.65 GlobalRate=859.08 Time=23:55:18
| Training Device=xla:0/2 Epoch=1 Step=1640 Loss=0.00136 Rate=1773.63 GlobalRate=871.57 Time=23:55:18
| Training Device=xla:0/3 Epoch=1 Step=1640 Loss=0.00136 Rate=1773.66 GlobalRate=857.96 Time=23:55:18
| Training Device=xla:0/2 Epoch=1 Step=1660 Loss=0.00136 Rate=1773.49 GlobalRate=876.94 Time=23:55:19
| Training Device=xla:0/3 Epoch=1 Step=1660 Loss=0.00136 Rate=1773.47 GlobalRate=863.33 Time=23:55:19
| Training Device=xla:0/1 Epoch=1 Step=1660 Loss=0.00136 Rate=1773.51 GlobalRate=864.45 Time=23:55:19
| Training Device=xla:0/0 Epoch=1 Step=1660 Loss=0.00136 Rate=1773.29 GlobalRate=873.88 Time=23:55:19
| Training Device=xla:0/1 Epoch=1 Step=1680 Loss=0.00136 Rate=1772.00 GlobalRate=869.74 Time=23:55:21
| Training Device=xla:0/3 Epoch=1 Step=1680 Loss=0.00136 Rate=1771.90 GlobalRate=868.63 Time=23:55:21
| Training Device=xla:0/0 Epoch=1 Step=1680 Loss=0.00136 Rate=1771.90 GlobalRate=879.18 Time=23:55:21
| Training Device=xla:0/2 Epoch=1 Step=1680 Loss=0.00136 Rate=1771.88 GlobalRate=882.24 Time=23:55:21
| Training Device=xla:0/2 Epoch=1 Step=1700 Loss=0.00136 Rate=1773.69 GlobalRate=887.48 Time=23:55:22
| Training Device=xla:0/0 Epoch=1 Step=1700 Loss=0.00136 Rate=1773.79 GlobalRate=884.43 Time=23:55:22
| Training Device=xla:0/3 Epoch=1 Step=1700 Loss=0.00136 Rate=1773.68 GlobalRate=873.87 Time=23:55:22
| Training Device=xla:0/1 Epoch=1 Step=1700 Loss=0.00136 Rate=1773.54 GlobalRate=874.99 Time=23:55:22
| Training Device=xla:0/3 Epoch=1 Step=1720 Loss=0.00136 Rate=1774.45 GlobalRate=879.06 Time=23:55:24
| Training Device=xla:0/1 Epoch=1 Step=1720 Loss=0.00136 Rate=1774.55 GlobalRate=880.18 Time=23:55:24
| Training Device=xla:0/0 Epoch=1 Step=1720 Loss=0.00136 Rate=1774.49 GlobalRate=889.61 Time=23:55:24
| Training Device=xla:0/2 Epoch=1 Step=1720 Loss=0.00136 Rate=1774.44 GlobalRate=892.67 Time=23:55:24
| Training Device=xla:0/2 Epoch=1 Step=1740 Loss=0.00135 Rate=1774.86 GlobalRate=897.80 Time=23:55:25
| Training Device=xla:0/3 Epoch=1 Step=1740 Loss=0.00135 Rate=1774.81 GlobalRate=884.19 Time=23:55:25
| Training Device=xla:0/1 Epoch=1 Step=1740 Loss=0.00135 Rate=1774.86 GlobalRate=885.30 Time=23:55:25
| Training Device=xla:0/0 Epoch=1 Step=1740 Loss=0.00135 Rate=1774.63 GlobalRate=894.74 Time=23:55:25
| Training Device=xla:0/3 Epoch=1 Step=1760 Loss=0.00135 Rate=1774.09 GlobalRate=889.25 Time=23:55:26
| Training Device=xla:0/1 Epoch=1 Step=1760 Loss=0.00135 Rate=1774.12 GlobalRate=890.37 Time=23:55:26
| Training Device=xla:0/2 Epoch=1 Step=1760 Loss=0.00135 Rate=1774.05 GlobalRate=902.86 Time=23:55:26
| Training Device=xla:0/0 Epoch=1 Step=1760 Loss=0.00135 Rate=1774.22 GlobalRate=899.80 Time=23:55:26
| Training Device=xla:0/2 Epoch=1 Step=1780 Loss=0.00135 Rate=1777.10 GlobalRate=907.88 Time=23:55:28
| Training Device=xla:0/0 Epoch=1 Step=1780 Loss=0.00135 Rate=1777.11 GlobalRate=904.83 Time=23:55:28
| Training Device=xla:0/3 Epoch=1 Step=1780 Loss=0.00135 Rate=1777.10 GlobalRate=894.27 Time=23:55:28
| Training Device=xla:0/1 Epoch=1 Step=1780 Loss=0.00135 Rate=1777.00 GlobalRate=895.39 Time=23:55:28
| Training Device=xla:0/2 Epoch=1 Step=1800 Loss=0.00135 Rate=1776.21 GlobalRate=912.84 Time=23:55:29
| Training Device=xla:0/3 Epoch=1 Step=1800 Loss=0.00135 Rate=1776.17 GlobalRate=899.23 Time=23:55:29
| Training Device=xla:0/1 Epoch=1 Step=1800 Loss=0.00135 Rate=1776.22 GlobalRate=900.35 Time=23:55:29
| Training Device=xla:0/0 Epoch=1 Step=1800 Loss=0.00135 Rate=1776.24 GlobalRate=909.78 Time=23:55:29
| Training Device=xla:0/2 Epoch=1 Step=1820 Loss=0.00135 Rate=1775.42 GlobalRate=917.73 Time=23:55:31
| Training Device=xla:0/3 Epoch=1 Step=1820 Loss=0.00135 Rate=1775.47 GlobalRate=904.13 Time=23:55:31
| Training Device=xla:0/0 Epoch=1 Step=1820 Loss=0.00135 Rate=1775.43 GlobalRate=914.68 Time=23:55:31
| Training Device=xla:0/1 Epoch=1 Step=1820 Loss=0.00135 Rate=1775.43 GlobalRate=905.24 Time=23:55:31
| Training Device=xla:0/3 Epoch=1 Step=1840 Loss=0.00135 Rate=1775.45 GlobalRate=908.98 Time=23:55:32
| Training Device=xla:0/1 Epoch=1 Step=1840 Loss=0.00135 Rate=1775.44 GlobalRate=910.09 Time=23:55:32
| Training Device=xla:0/0 Epoch=1 Step=1840 Loss=0.00135 Rate=1775.46 GlobalRate=919.52 Time=23:55:32
| Training Device=xla:0/2 Epoch=1 Step=1840 Loss=0.00135 Rate=1775.33 GlobalRate=922.57 Time=23:55:32
| Training Device=xla:0/3 Epoch=1 Step=1860 Loss=0.00135 Rate=1775.79 GlobalRate=913.77 Time=23:55:34
| Training Device=xla:0/2 Epoch=1 Step=1860 Loss=0.00135 Rate=1775.79 GlobalRate=927.36 Time=23:55:34
| Training Device=xla:0/1 Epoch=1 Step=1860 Loss=0.00135 Rate=1775.79 GlobalRate=914.88 Time=23:55:34
| Training Device=xla:0/0 Epoch=1 Step=1860 Loss=0.00135 Rate=1775.75 GlobalRate=924.31 Time=23:55:34
| Training Device=xla:0/3 Epoch=1 Step=1880 Loss=0.00135 Rate=1775.74 GlobalRate=918.51 Time=23:55:35
| Training Device=xla:0/1 Epoch=1 Step=1880 Loss=0.00135 Rate=1775.77 GlobalRate=919.62 Time=23:55:35
| Training Device=xla:0/0 Epoch=1 Step=1880 Loss=0.00135 Rate=1775.74 GlobalRate=929.05 Time=23:55:35
| Training Device=xla:0/2 Epoch=1 Step=1880 Loss=0.00135 Rate=1775.70 GlobalRate=932.10 Time=23:55:35
| Training Device=xla:0/3 Epoch=1 Step=1900 Loss=0.00135 Rate=1775.87 GlobalRate=923.20 Time=23:55:37
| Training Device=xla:0/0 Epoch=1 Step=1900 Loss=0.00135 Rate=1775.92 GlobalRate=933.73 Time=23:55:37
| Training Device=xla:0/2 Epoch=1 Step=1900 Loss=0.00135 Rate=1775.89 GlobalRate=936.78 Time=23:55:37
| Training Device=xla:0/1 Epoch=1 Step=1900 Loss=0.00135 Rate=1775.85 GlobalRate=924.31 Time=23:55:37
| Training Device=xla:0/1 Epoch=1 Step=1920 Loss=0.00135 Rate=1749.98 GlobalRate=928.83 Time=23:55:38
| Training Device=xla:0/2 Epoch=1 Step=1920 Loss=0.00135 Rate=1750.01 GlobalRate=941.28 Time=23:55:38
| Training Device=xla:0/3 Epoch=1 Step=1920 Loss=0.00135 Rate=1749.87 GlobalRate=927.71 Time=23:55:38
| Training Device=xla:0/0 Epoch=1 Step=1920 Loss=0.00135 Rate=1749.46 GlobalRate=938.23 Time=23:55:38
| Training Device=xla:0/2 Epoch=1 Step=1940 Loss=0.00135 Rate=1760.66 GlobalRate=945.84 Time=23:55:39
| Training Device=xla:0/3 Epoch=1 Step=1940 Loss=0.00135 Rate=1760.68 GlobalRate=932.28 Time=23:55:39
| Training Device=xla:0/0 Epoch=1 Step=1940 Loss=0.00135 Rate=1760.92 GlobalRate=942.79 Time=23:55:39
| Training Device=xla:0/1 Epoch=1 Step=1940 Loss=0.00135 Rate=1760.45 GlobalRate=933.39 Time=23:55:39
| Training Device=xla:0/1 Epoch=1 Step=1960 Loss=0.00135 Rate=1769.43 GlobalRate=937.93 Time=23:55:41
| Training Device=xla:0/0 Epoch=1 Step=1960 Loss=0.00135 Rate=1769.39 GlobalRate=947.32 Time=23:55:41
| Training Device=xla:0/2 Epoch=1 Step=1960 Loss=0.00135 Rate=1769.30 GlobalRate=950.37 Time=23:55:41
| Training Device=xla:0/3 Epoch=1 Step=1960 Loss=0.00135 Rate=1768.85 GlobalRate=936.81 Time=23:55:41
| Training Device=xla:0/3 Epoch=1 Step=1980 Loss=0.00135 Rate=1771.13 GlobalRate=941.29 Time=23:55:42
| Training Device=xla:0/2 Epoch=1 Step=1980 Loss=0.00135 Rate=1770.89 GlobalRate=954.84 Time=23:55:42
| Training Device=xla:0/1 Epoch=1 Step=1980 Loss=0.00135 Rate=1770.90 GlobalRate=942.40 Time=23:55:42
| Training Device=xla:0/0 Epoch=1 Step=1980 Loss=0.00135 Rate=1770.96 GlobalRate=951.80 Time=23:55:42
| Training Device=xla:0/0 Epoch=1 Step=2000 Loss=0.00135 Rate=1773.49 GlobalRate=956.23 Time=23:55:44
| Training Device=xla:0/2 Epoch=1 Step=2000 Loss=0.00135 Rate=1773.43 GlobalRate=959.27 Time=23:55:44
| Training Device=xla:0/3 Epoch=1 Step=2000 Loss=0.00135 Rate=1773.52 GlobalRate=945.73 Time=23:55:44
| Training Device=xla:0/1 Epoch=1 Step=2000 Loss=0.00135 Rate=1773.38 GlobalRate=946.84 Time=23:55:44
| Training Device=xla:0/2 Epoch=1 Step=2020 Loss=0.00135 Rate=1773.20 GlobalRate=963.64 Time=23:55:45
| Training Device=xla:0/0 Epoch=1 Step=2020 Loss=0.00135 Rate=1773.14 GlobalRate=960.61 Time=23:55:45
| Training Device=xla:0/1 Epoch=1 Step=2020 Loss=0.00135 Rate=1773.28 GlobalRate=951.23 Time=23:55:45
| Training Device=xla:0/3 Epoch=1 Step=2020 Loss=0.00135 Rate=1773.24 GlobalRate=950.12 Time=23:55:45
| Training Device=xla:0/0 Epoch=1 Step=2040 Loss=0.00135 Rate=1770.56 GlobalRate=964.93 Time=23:55:47
| Training Device=xla:0/2 Epoch=1 Step=2040 Loss=0.00135 Rate=1770.50 GlobalRate=967.96 Time=23:55:47
| Training Device=xla:0/1 Epoch=1 Step=2040 Loss=0.00135 Rate=1770.40 GlobalRate=955.56 Time=23:55:47
| Training Device=xla:0/3 Epoch=1 Step=2040 Loss=0.00135 Rate=1770.46 GlobalRate=954.45 Time=23:55:47
| Training Device=xla:0/0 Epoch=1 Step=2060 Loss=0.00135 Rate=1770.75 GlobalRate=969.21 Time=23:55:48
| Training Device=xla:0/2 Epoch=1 Step=2060 Loss=0.00135 Rate=1770.77 GlobalRate=972.24 Time=23:55:48
| Training Device=xla:0/1 Epoch=1 Step=2060 Loss=0.00135 Rate=1770.81 GlobalRate=959.85 Time=23:55:48
| Training Device=xla:0/3 Epoch=1 Step=2060 Loss=0.00135 Rate=1770.73 GlobalRate=958.74 Time=23:55:48
| Training Device=xla:0/0 Epoch=1 Step=2080 Loss=0.00135 Rate=1771.31 GlobalRate=973.45 Time=23:55:50
| Training Device=xla:0/2 Epoch=1 Step=2080 Loss=0.00135 Rate=1771.26 GlobalRate=976.47 Time=23:55:50
| Training Device=xla:0/1 Epoch=1 Step=2080 Loss=0.00135 Rate=1771.24 GlobalRate=964.09 Time=23:55:50
| Training Device=xla:0/3 Epoch=1 Step=2080 Loss=0.00135 Rate=1771.26 GlobalRate=962.98 Time=23:55:50
| Training Device=xla:0/0 Epoch=1 Step=2100 Loss=0.00135 Rate=1769.88 GlobalRate=977.63 Time=23:55:51
| Training Device=xla:0/3 Epoch=1 Step=2100 Loss=0.00135 Rate=1769.99 GlobalRate=967.18 Time=23:55:51
| Training Device=xla:0/1 Epoch=1 Step=2100 Loss=0.00135 Rate=1769.94 GlobalRate=968.29 Time=23:55:51
| Training Device=xla:0/2 Epoch=1 Step=2100 Loss=0.00135 Rate=1769.91 GlobalRate=980.66 Time=23:55:51
| Training Device=xla:0/2 Epoch=1 Step=2120 Loss=0.00135 Rate=1772.07 GlobalRate=984.81 Time=23:55:52
| Training Device=xla:0/3 Epoch=1 Step=2120 Loss=0.00135 Rate=1772.10 GlobalRate=971.34 Time=23:55:52
| Training Device=xla:0/1 Epoch=1 Step=2120 Loss=0.00135 Rate=1772.10 GlobalRate=972.45 Time=23:55:52
| Training Device=xla:0/0 Epoch=1 Step=2120 Loss=0.00135 Rate=1772.04 GlobalRate=981.79 Time=23:55:52
| Training Device=xla:0/2 Epoch=1 Step=2140 Loss=0.00135 Rate=1772.75 GlobalRate=988.91 Time=23:55:54
| Training Device=xla:0/1 Epoch=1 Step=2140 Loss=0.00135 Rate=1772.79 GlobalRate=976.57 Time=23:55:54
| Training Device=xla:0/0 Epoch=1 Step=2140 Loss=0.00135 Rate=1772.80 GlobalRate=985.90 Time=23:55:54
| Training Device=xla:0/3 Epoch=1 Step=2140 Loss=0.00135 Rate=1772.76 GlobalRate=975.46 Time=23:55:54
| Training Device=xla:0/3 Epoch=1 Step=2160 Loss=0.00135 Rate=1772.39 GlobalRate=979.54 Time=23:55:55
| Training Device=xla:0/1 Epoch=1 Step=2160 Loss=0.00135 Rate=1772.29 GlobalRate=980.64 Time=23:55:55
| Training Device=xla:0/2 Epoch=1 Step=2160 Loss=0.00135 Rate=1772.13 GlobalRate=992.98 Time=23:55:55
| Training Device=xla:0/0 Epoch=1 Step=2160 Loss=0.00135 Rate=1772.34 GlobalRate=989.96 Time=23:55:55
| Training Device=xla:0/0 Epoch=1 Step=2180 Loss=0.00135 Rate=1772.17 GlobalRate=993.99 Time=23:55:57
| Training Device=xla:0/1 Epoch=1 Step=2180 Loss=0.00135 Rate=1772.22 GlobalRate=984.68 Time=23:55:57
| Training Device=xla:0/2 Epoch=1 Step=2180 Loss=0.00135 Rate=1772.25 GlobalRate=997.00 Time=23:55:57
| Training Device=xla:0/3 Epoch=1 Step=2180 Loss=0.00135 Rate=1772.17 GlobalRate=983.57 Time=23:55:57
| Training Device=xla:0/2 Epoch=1 Step=2200 Loss=0.00135 Rate=1773.98 GlobalRate=1000.98 Time=23:55:58
| Training Device=xla:0/1 Epoch=1 Step=2200 Loss=0.00135 Rate=1773.91 GlobalRate=988.68 Time=23:55:58
| Training Device=xla:0/3 Epoch=1 Step=2200 Loss=0.00135 Rate=1773.87 GlobalRate=987.57 Time=23:55:58
| Training Device=xla:0/0 Epoch=1 Step=2200 Loss=0.00135 Rate=1773.80 GlobalRate=997.98 Time=23:55:58
| Training Device=xla:0/0 Epoch=1 Step=2220 Loss=0.00135 Rate=1775.72 GlobalRate=1001.93 Time=23:56:00
| Training Device=xla:0/1 Epoch=1 Step=2220 Loss=0.00135 Rate=1775.55 GlobalRate=992.64 Time=23:56:00
| Training Device=xla:0/3 Epoch=1 Step=2220 Loss=0.00135 Rate=1775.67 GlobalRate=991.54 Time=23:56:00
| Training Device=xla:0/2 Epoch=1 Step=2220 Loss=0.00135 Rate=1775.57 GlobalRate=1004.93 Time=23:56:00
| Training Device=xla:0/1 Epoch=1 Step=2240 Loss=0.00135 Rate=1773.21 GlobalRate=996.55 Time=23:56:01
| Training Device=xla:0/0 Epoch=1 Step=2240 Loss=0.00135 Rate=1772.96 GlobalRate=1005.83 Time=23:56:01
| Training Device=xla:0/2 Epoch=1 Step=2240 Loss=0.00135 Rate=1773.17 GlobalRate=1008.83 Time=23:56:01
| Training Device=xla:0/3 Epoch=1 Step=2240 Loss=0.00135 Rate=1773.07 GlobalRate=995.45 Time=23:56:01
| Training Device=xla:0/1 Epoch=1 Step=2260 Loss=0.00135 Rate=1773.37 GlobalRate=1000.43 Time=23:56:03
| Training Device=xla:0/2 Epoch=1 Step=2260 Loss=0.00135 Rate=1773.43 GlobalRate=1012.69 Time=23:56:03
| Training Device=xla:0/0 Epoch=1 Step=2260 Loss=0.00135 Rate=1773.42 GlobalRate=1009.70 Time=23:56:03
| Training Device=xla:0/3 Epoch=1 Step=2260 Loss=0.00135 Rate=1773.31 GlobalRate=999.33 Time=23:56:03
| Training Device=xla:0/2 Epoch=1 Step=2280 Loss=0.00135 Rate=1771.91 GlobalRate=1016.51 Time=23:56:04
| Training Device=xla:0/1 Epoch=1 Step=2280 Loss=0.00135 Rate=1771.92 GlobalRate=1004.26 Time=23:56:04
| Training Device=xla:0/0 Epoch=1 Step=2280 Loss=0.00135 Rate=1771.99 GlobalRate=1013.52 Time=23:56:04
| Training Device=xla:0/3 Epoch=1 Step=2280 Loss=0.00135 Rate=1771.73 GlobalRate=1003.16 Time=23:56:04
| Training Device=xla:0/0 Epoch=1 Step=2300 Loss=0.00135 Rate=1773.79 GlobalRate=1017.31 Time=23:56:05
| Training Device=xla:0/3 Epoch=1 Step=2300 Loss=0.00135 Rate=1773.98 GlobalRate=1006.97 Time=23:56:05
| Training Device=xla:0/1 Epoch=1 Step=2300 Loss=0.00135 Rate=1773.77 GlobalRate=1008.06 Time=23:56:05
| Training Device=xla:0/2 Epoch=1 Step=2300 Loss=0.00135 Rate=1773.76 GlobalRate=1020.30 Time=23:56:05
| Training Device=xla:0/3 Epoch=1 Step=2320 Loss=0.00135 Rate=1775.56 GlobalRate=1010.74 Time=23:56:07
| Training Device=xla:0/2 Epoch=1 Step=2320 Loss=0.00135 Rate=1775.53 GlobalRate=1024.05 Time=23:56:07
| Training Device=xla:0/0 Epoch=1 Step=2320 Loss=0.00135 Rate=1775.48 GlobalRate=1021.07 Time=23:56:07
| Training Device=xla:0/1 Epoch=1 Step=2320 Loss=0.00135 Rate=1775.35 GlobalRate=1011.83 Time=23:56:07
| Training Device=xla:0/3 Epoch=1 Step=2340 Loss=0.00135 Rate=1775.95 GlobalRate=1014.48 Time=23:56:08
| Training Device=xla:0/1 Epoch=1 Step=2340 Loss=0.00135 Rate=1775.97 GlobalRate=1015.57 Time=23:56:08
| Training Device=xla:0/0 Epoch=1 Step=2340 Loss=0.00135 Rate=1775.91 GlobalRate=1024.79 Time=23:56:08
| Training Device=xla:0/2 Epoch=1 Step=2340 Loss=0.00135 Rate=1775.92 GlobalRate=1027.77 Time=23:56:08
Epoch 1 train end 23:56:08
| Test Device=xla:0/3 Step=0 Epoch=1 Time=23:56:15
| Test Device=xla:0/2 Step=0 Epoch=1 Time=23:56:15
| Test Device=xla:0/1 Step=0 Epoch=1 Time=23:56:15
| Test Device=xla:0/0 Step=0 Epoch=1 Time=23:56:15
| Test Device=xla:0/2 Step=20 Epoch=1 Time=23:56:21
| Test Device=xla:0/3 Step=20 Epoch=1 Time=23:56:21
| Test Device=xla:0/1 Step=20 Epoch=1 Time=23:56:21
| Test Device=xla:0/2 Step=40 Epoch=1 Time=23:56:21
| Test Device=xla:0/3 Step=40 Epoch=1 Time=23:56:21
| Test Device=xla:0/1 Step=40 Epoch=1 Time=23:56:22
| Test Device=xla:0/2 Step=60 Epoch=1 Time=23:56:22
| Test Device=xla:0/3 Step=60 Epoch=1 Time=23:56:22
| Test Device=xla:0/0 Step=20 Epoch=1 Time=23:56:22
| Test Device=xla:0/1 Step=60 Epoch=1 Time=23:56:22
| Test Device=xla:0/2 Step=80 Epoch=1 Time=23:56:22
| Test Device=xla:0/3 Step=80 Epoch=1 Time=23:56:22
| Test Device=xla:0/0 Step=40 Epoch=1 Time=23:56:22
| Test Device=xla:0/1 Step=80 Epoch=1 Time=23:56:22
| Test Device=xla:0/0 Step=60 Epoch=1 Time=23:56:23
| Test Device=xla:0/0 Step=80 Epoch=1 Time=23:56:23
Epoch 1 test end 23:56:24, Accuracy=100.00
Max Accuracy: 100.00%

this is resnet train after openxla-pin update to nov24 on TPU:

PJRT_DEVICE=TPU python test/test_train_mp_imagenet.py --fake_data --num_epochs=1
==> Preparing data..
==> Preparing data..
==> Preparing data..
==> Preparing data..
Epoch 1 train begin 00:34:11
| Training Device=xla:0/0 Epoch=1 Step=0 Loss=6.89059 Rate=2.16 GlobalRate=2.16 Time=00:35:11
| Training Device=xla:0/3 Epoch=1 Step=0 Loss=6.89059 Rate=2.15 GlobalRate=2.15 Time=00:35:11
| Training Device=xla:0/1 Epoch=1 Step=0 Loss=6.89059 Rate=2.16 GlobalRate=2.16 Time=00:35:11
| Training Device=xla:0/2 Epoch=1 Step=0 Loss=6.89059 Rate=2.11 GlobalRate=2.11 Time=00:35:11
| Training Device=xla:0/0 Epoch=1 Step=20 Loss=6.50001 Rate=26.60 GlobalRate=22.61 Time=00:36:10
| Training Device=xla:0/1 Epoch=1 Step=20 Loss=6.50001 Rate=26.60 GlobalRate=22.62 Time=00:36:10
| Training Device=xla:0/3 Epoch=1 Step=20 Loss=6.50001 Rate=26.60 GlobalRate=22.52 Time=00:36:10
| Training Device=xla:0/2 Epoch=1 Step=20 Loss=6.50001 Rate=26.58 GlobalRate=22.33 Time=00:36:10
| Training Device=xla:0/1 Epoch=1 Step=40 Loss=5.07708 Rate=1069.27 GlobalRate=43.63 Time=00:36:12
| Training Device=xla:0/3 Epoch=1 Step=40 Loss=5.07708 Rate=1069.67 GlobalRate=43.45 Time=00:36:12
| Training Device=xla:0/0 Epoch=1 Step=40 Loss=5.07708 Rate=1069.15 GlobalRate=43.60 Time=00:36:12
| Training Device=xla:0/2 Epoch=1 Step=40 Loss=5.07708 Rate=1069.52 GlobalRate=43.08 Time=00:36:12
| Training Device=xla:0/0 Epoch=1 Step=60 Loss=2.73656 Rate=1487.42 GlobalRate=64.10 Time=00:36:13
| Training Device=xla:0/3 Epoch=1 Step=60 Loss=2.73656 Rate=1487.63 GlobalRate=63.88 Time=00:36:13
| Training Device=xla:0/1 Epoch=1 Step=60 Loss=2.73656 Rate=1487.34 GlobalRate=64.14 Time=00:36:13
| Training Device=xla:0/2 Epoch=1 Step=60 Loss=2.73656 Rate=1487.56 GlobalRate=63.34 Time=00:36:13
| Training Device=xla:0/3 Epoch=1 Step=80 Loss=0.57112 Rate=1658.88 GlobalRate=83.83 Time=00:36:15
| Training Device=xla:0/2 Epoch=1 Step=80 Loss=0.57112 Rate=1659.12 GlobalRate=83.13 Time=00:36:15
| Training Device=xla:0/1 Epoch=1 Step=80 Loss=0.57112 Rate=1658.69 GlobalRate=84.18 Time=00:36:15
| Training Device=xla:0/0 Epoch=1 Step=80 Loss=0.57112 Rate=1658.53 GlobalRate=84.12 Time=00:36:15
| Training Device=xla:0/2 Epoch=1 Step=100 Loss=0.11328 Rate=1722.88 GlobalRate=102.47 Time=00:36:16
| Training Device=xla:0/3 Epoch=1 Step=100 Loss=0.11328 Rate=1722.59 GlobalRate=103.32 Time=00:36:16
| Training Device=xla:0/1 Epoch=1 Step=100 Loss=0.11328 Rate=1722.67 GlobalRate=103.74 Time=00:36:16
| Training Device=xla:0/0 Epoch=1 Step=100 Loss=0.11328 Rate=1722.42 GlobalRate=103.67 Time=00:36:16
| Training Device=xla:0/2 Epoch=1 Step=120 Loss=0.05495 Rate=1752.70 GlobalRate=121.37 Time=00:36:18
| Training Device=xla:0/1 Epoch=1 Step=120 Loss=0.05495 Rate=1752.63 GlobalRate=122.86 Time=00:36:18
| Training Device=xla:0/0 Epoch=1 Step=120 Loss=0.05495 Rate=1752.82 GlobalRate=122.78 Time=00:36:18
| Training Device=xla:0/3 Epoch=1 Step=120 Loss=0.05495 Rate=1752.62 GlobalRate=122.36 Time=00:36:18
| Training Device=xla:0/2 Epoch=1 Step=140 Loss=0.03776 Rate=1763.61 GlobalRate=139.85 Time=00:36:19
| Training Device=xla:0/3 Epoch=1 Step=140 Loss=0.03776 Rate=1763.62 GlobalRate=140.98 Time=00:36:19
| Training Device=xla:0/0 Epoch=1 Step=140 Loss=0.03776 Rate=1763.69 GlobalRate=141.45 Time=00:36:19
| Training Device=xla:0/1 Epoch=1 Step=140 Loss=0.03776 Rate=1763.63 GlobalRate=141.54 Time=00:36:19
| Training Device=xla:0/1 Epoch=1 Step=160 Loss=0.02887 Rate=1765.08 GlobalRate=159.80 Time=00:36:20
| Training Device=xla:0/0 Epoch=1 Step=160 Loss=0.02887 Rate=1765.15 GlobalRate=159.70 Time=00:36:20
| Training Device=xla:0/2 Epoch=1 Step=160 Loss=0.02887 Rate=1765.13 GlobalRate=157.91 Time=00:36:20
| Training Device=xla:0/3 Epoch=1 Step=160 Loss=0.02887 Rate=1764.87 GlobalRate=159.17 Time=00:36:20
| Training Device=xla:0/2 Epoch=1 Step=180 Loss=0.02308 Rate=1769.13 GlobalRate=175.58 Time=00:36:22
| Training Device=xla:0/1 Epoch=1 Step=180 Loss=0.02308 Rate=1769.22 GlobalRate=177.66 Time=00:36:22
| Training Device=xla:0/0 Epoch=1 Step=180 Loss=0.02308 Rate=1768.97 GlobalRate=177.55 Time=00:36:22
| Training Device=xla:0/3 Epoch=1 Step=180 Loss=0.02308 Rate=1769.09 GlobalRate=176.97 Time=00:36:22
| Training Device=xla:0/2 Epoch=1 Step=200 Loss=0.01895 Rate=1765.01 GlobalRate=192.86 Time=00:36:23
| Training Device=xla:0/1 Epoch=1 Step=200 Loss=0.01895 Rate=1764.77 GlobalRate=195.12 Time=00:36:23
| Training Device=xla:0/3 Epoch=1 Step=200 Loss=0.01895 Rate=1764.99 GlobalRate=194.37 Time=00:36:23
| Training Device=xla:0/0 Epoch=1 Step=200 Loss=0.01895 Rate=1764.92 GlobalRate=195.00 Time=00:36:23
| Training Device=xla:0/2 Epoch=1 Step=220 Loss=0.01588 Rate=1766.01 GlobalRate=209.77 Time=00:36:25
| Training Device=xla:0/1 Epoch=1 Step=220 Loss=0.01588 Rate=1766.08 GlobalRate=212.20 Time=00:36:25
| Training Device=xla:0/0 Epoch=1 Step=220 Loss=0.01588 Rate=1766.15 GlobalRate=212.08 Time=00:36:25
| Training Device=xla:0/3 Epoch=1 Step=220 Loss=0.01588 Rate=1766.05 GlobalRate=211.39 Time=00:36:25
| Training Device=xla:0/1 Epoch=1 Step=240 Loss=0.01353 Rate=1769.98 GlobalRate=228.92 Time=00:36:26
| Training Device=xla:0/3 Epoch=1 Step=240 Loss=0.01353 Rate=1769.95 GlobalRate=228.06 Time=00:36:26
| Training Device=xla:0/0 Epoch=1 Step=240 Loss=0.01353 Rate=1769.83 GlobalRate=228.79 Time=00:36:26
| Training Device=xla:0/2 Epoch=1 Step=240 Loss=0.01353 Rate=1769.61 GlobalRate=226.33 Time=00:36:26
| Training Device=xla:0/1 Epoch=1 Step=260 Loss=0.01169 Rate=1773.44 GlobalRate=245.30 Time=00:36:28
| Training Device=xla:0/2 Epoch=1 Step=260 Loss=0.01169 Rate=1773.63 GlobalRate=242.55 Time=00:36:28
| Training Device=xla:0/0 Epoch=1 Step=260 Loss=0.01169 Rate=1773.55 GlobalRate=245.16 Time=00:36:28
| Training Device=xla:0/3 Epoch=1 Step=260 Loss=0.01169 Rate=1773.53 GlobalRate=244.39 Time=00:36:28
| Training Device=xla:0/3 Epoch=1 Step=280 Loss=0.01022 Rate=1771.80 GlobalRate=260.36 Time=00:36:29
| Training Device=xla:0/1 Epoch=1 Step=280 Loss=0.01022 Rate=1771.50 GlobalRate=261.32 Time=00:36:29
| Training Device=xla:0/0 Epoch=1 Step=280 Loss=0.01022 Rate=1771.60 GlobalRate=261.17 Time=00:36:29
| Training Device=xla:0/2 Epoch=1 Step=280 Loss=0.01022 Rate=1771.02 GlobalRate=258.42 Time=00:36:29
| Training Device=xla:0/1 Epoch=1 Step=300 Loss=0.00902 Rate=1770.50 GlobalRate=277.01 Time=00:36:31
| Training Device=xla:0/0 Epoch=1 Step=300 Loss=0.00902 Rate=1770.41 GlobalRate=276.85 Time=00:36:31
| Training Device=xla:0/2 Epoch=1 Step=300 Loss=0.00902 Rate=1770.39 GlobalRate=273.97 Time=00:36:31
| Training Device=xla:0/3 Epoch=1 Step=300 Loss=0.00902 Rate=1770.47 GlobalRate=276.00 Time=00:36:31
| Training Device=xla:0/0 Epoch=1 Step=320 Loss=0.00804 Rate=1766.44 GlobalRate=292.20 Time=00:36:32
| Training Device=xla:0/2 Epoch=1 Step=320 Loss=0.00804 Rate=1766.75 GlobalRate=289.19 Time=00:36:32
| Training Device=xla:0/1 Epoch=1 Step=320 Loss=0.00804 Rate=1766.15 GlobalRate=292.36 Time=00:36:32
| Training Device=xla:0/3 Epoch=1 Step=320 Loss=0.00804 Rate=1766.18 GlobalRate=291.31 Time=00:36:32
| Training Device=xla:0/1 Epoch=1 Step=340 Loss=0.00722 Rate=1767.97 GlobalRate=307.41 Time=00:36:33
| Training Device=xla:0/0 Epoch=1 Step=340 Loss=0.00722 Rate=1767.87 GlobalRate=307.25 Time=00:36:33
| Training Device=xla:0/3 Epoch=1 Step=340 Loss=0.00722 Rate=1767.88 GlobalRate=306.32 Time=00:36:33
| Training Device=xla:0/2 Epoch=1 Step=340 Loss=0.00722 Rate=1768.02 GlobalRate=304.11 Time=00:36:33
| Training Device=xla:0/3 Epoch=1 Step=360 Loss=0.00654 Rate=1764.54 GlobalRate=321.01 Time=00:36:35
| Training Device=xla:0/0 Epoch=1 Step=360 Loss=0.00654 Rate=1764.54 GlobalRate=321.97 Time=00:36:35
| Training Device=xla:0/2 Epoch=1 Step=360 Loss=0.00654 Rate=1764.64 GlobalRate=318.72 Time=00:36:35
| Training Device=xla:0/1 Epoch=1 Step=360 Loss=0.00654 Rate=1764.56 GlobalRate=322.15 Time=00:36:35
| Training Device=xla:0/0 Epoch=1 Step=380 Loss=0.00595 Rate=1759.80 GlobalRate=336.39 Time=00:36:36
| Training Device=xla:0/3 Epoch=1 Step=380 Loss=0.00595 Rate=1759.81 GlobalRate=335.40 Time=00:36:36
| Training Device=xla:0/1 Epoch=1 Step=380 Loss=0.00595 Rate=1759.84 GlobalRate=336.57 Time=00:36:36
| Training Device=xla:0/2 Epoch=1 Step=380 Loss=0.00595 Rate=1759.73 GlobalRate=333.03 Time=00:36:36
| Training Device=xla:0/0 Epoch=1 Step=400 Loss=0.00545 Rate=1766.05 GlobalRate=350.56 Time=00:36:38
| Training Device=xla:0/2 Epoch=1 Step=400 Loss=0.00545 Rate=1766.09 GlobalRate=347.08 Time=00:36:38
| Training Device=xla:0/1 Epoch=1 Step=400 Loss=0.00545 Rate=1766.02 GlobalRate=350.74 Time=00:36:38
| Training Device=xla:0/3 Epoch=1 Step=400 Loss=0.00545 Rate=1765.98 GlobalRate=349.53 Time=00:36:38
| Training Device=xla:0/1 Epoch=1 Step=420 Loss=0.00502 Rate=1768.42 GlobalRate=364.63 Time=00:36:39
| Training Device=xla:0/3 Epoch=1 Step=420 Loss=0.00502 Rate=1768.38 GlobalRate=363.38 Time=00:36:39
| Training Device=xla:0/0 Epoch=1 Step=420 Loss=0.00502 Rate=1768.36 GlobalRate=364.44 Time=00:36:39
| Training Device=xla:0/2 Epoch=1 Step=420 Loss=0.00502 Rate=1768.28 GlobalRate=360.86 Time=00:36:39
| Training Device=xla:0/3 Epoch=1 Step=440 Loss=0.00465 Rate=1749.30 GlobalRate=376.90 Time=00:36:41
| Training Device=xla:0/2 Epoch=1 Step=440 Loss=0.00465 Rate=1749.35 GlobalRate=374.31 Time=00:36:41
| Training Device=xla:0/1 Epoch=1 Step=440 Loss=0.00465 Rate=1749.08 GlobalRate=378.18 Time=00:36:41
| Training Device=xla:0/0 Epoch=1 Step=440 Loss=0.00465 Rate=1749.21 GlobalRate=377.98 Time=00:36:41
| Training Device=xla:0/1 Epoch=1 Step=460 Loss=0.00432 Rate=1758.45 GlobalRate=391.53 Time=00:36:42
| Training Device=xla:0/0 Epoch=1 Step=460 Loss=0.00432 Rate=1758.38 GlobalRate=391.33 Time=00:36:42
| Training Device=xla:0/2 Epoch=1 Step=460 Loss=0.00432 Rate=1758.02 GlobalRate=387.56 Time=00:36:42
| Training Device=xla:0/3 Epoch=1 Step=460 Loss=0.00432 Rate=1758.27 GlobalRate=390.21 Time=00:36:42
| Training Device=xla:0/0 Epoch=1 Step=480 Loss=0.00403 Rate=1737.75 GlobalRate=404.32 Time=00:36:44
| Training Device=xla:0/2 Epoch=1 Step=480 Loss=0.00403 Rate=1737.98 GlobalRate=400.47 Time=00:36:44
| Training Device=xla:0/3 Epoch=1 Step=480 Loss=0.00403 Rate=1737.81 GlobalRate=403.18 Time=00:36:44
| Training Device=xla:0/1 Epoch=1 Step=480 Loss=0.00403 Rate=1737.77 GlobalRate=404.53 Time=00:36:44
| Training Device=xla:0/1 Epoch=1 Step=500 Loss=0.00378 Rate=1756.94 GlobalRate=417.38 Time=00:36:45
| Training Device=xla:0/0 Epoch=1 Step=500 Loss=0.00378 Rate=1756.92 GlobalRate=417.17 Time=00:36:45
| Training Device=xla:0/3 Epoch=1 Step=500 Loss=0.00378 Rate=1756.88 GlobalRate=416.00 Time=00:36:45
| Training Device=xla:0/2 Epoch=1 Step=500 Loss=0.00378 Rate=1756.90 GlobalRate=413.23 Time=00:36:45
| Training Device=xla:0/0 Epoch=1 Step=520 Loss=0.00356 Rate=1759.97 GlobalRate=429.76 Time=00:36:47
| Training Device=xla:0/2 Epoch=1 Step=520 Loss=0.00356 Rate=1759.96 GlobalRate=425.74 Time=00:36:47
| Training Device=xla:0/3 Epoch=1 Step=520 Loss=0.00356 Rate=1760.02 GlobalRate=428.57 Time=00:36:47
| Training Device=xla:0/1 Epoch=1 Step=520 Loss=0.00356 Rate=1759.87 GlobalRate=429.98 Time=00:36:47
| Training Device=xla:0/0 Epoch=1 Step=540 Loss=0.00336 Rate=1766.75 GlobalRate=442.14 Time=00:36:48
| Training Device=xla:0/3 Epoch=1 Step=540 Loss=0.00336 Rate=1766.71 GlobalRate=440.93 Time=00:36:48
| Training Device=xla:0/2 Epoch=1 Step=540 Loss=0.00336 Rate=1766.71 GlobalRate=438.04 Time=00:36:48
| Training Device=xla:0/1 Epoch=1 Step=540 Loss=0.00336 Rate=1766.71 GlobalRate=442.36 Time=00:36:48
| Training Device=xla:0/1 Epoch=1 Step=560 Loss=0.00318 Rate=1770.41 GlobalRate=454.52 Time=00:36:49
| Training Device=xla:0/3 Epoch=1 Step=560 Loss=0.00318 Rate=1770.34 GlobalRate=453.06 Time=00:36:49
| Training Device=xla:0/0 Epoch=1 Step=560 Loss=0.00318 Rate=1770.36 GlobalRate=454.30 Time=00:36:49
| Training Device=xla:0/2 Epoch=1 Step=560 Loss=0.00318 Rate=1770.33 GlobalRate=450.12 Time=00:36:49
| Training Device=xla:0/2 Epoch=1 Step=580 Loss=0.00302 Rate=1755.98 GlobalRate=461.93 Time=00:36:51
| Training Device=xla:0/1 Epoch=1 Step=580 Loss=0.00302 Rate=1755.94 GlobalRate=466.40 Time=00:36:51
| Training Device=xla:0/0 Epoch=1 Step=580 Loss=0.00302 Rate=1755.91 GlobalRate=466.17 Time=00:36:51
| Training Device=xla:0/3 Epoch=1 Step=580 Loss=0.00302 Rate=1755.86 GlobalRate=464.91 Time=00:36:51
| Training Device=xla:0/1 Epoch=1 Step=600 Loss=0.00287 Rate=1768.05 GlobalRate=478.13 Time=00:36:52
| Training Device=xla:0/2 Epoch=1 Step=600 Loss=0.00287 Rate=1768.05 GlobalRate=473.59 Time=00:36:52
| Training Device=xla:0/0 Epoch=1 Step=600 Loss=0.00287 Rate=1768.05 GlobalRate=477.90 Time=00:36:52
| Training Device=xla:0/3 Epoch=1 Step=600 Loss=0.00287 Rate=1767.96 GlobalRate=476.62 Time=00:36:52
| Training Device=xla:0/0 Epoch=1 Step=620 Loss=0.00274 Rate=1766.86 GlobalRate=489.40 Time=00:36:54
| Training Device=xla:0/1 Epoch=1 Step=620 Loss=0.00274 Rate=1766.82 GlobalRate=489.63 Time=00:36:54
| Training Device=xla:0/2 Epoch=1 Step=620 Loss=0.00274 Rate=1766.77 GlobalRate=485.02 Time=00:36:54
| Training Device=xla:0/3 Epoch=1 Step=620 Loss=0.00274 Rate=1766.88 GlobalRate=488.10 Time=00:36:54
| Training Device=xla:0/0 Epoch=1 Step=640 Loss=0.00263 Rate=1770.21 GlobalRate=500.70 Time=00:36:55
| Training Device=xla:0/3 Epoch=1 Step=640 Loss=0.00263 Rate=1770.21 GlobalRate=499.39 Time=00:36:55
| Training Device=xla:0/1 Epoch=1 Step=640 Loss=0.00263 Rate=1770.23 GlobalRate=500.94 Time=00:36:55
| Training Device=xla:0/2 Epoch=1 Step=640 Loss=0.00263 Rate=1769.98 GlobalRate=496.27 Time=00:36:55
| Training Device=xla:0/3 Epoch=1 Step=660 Loss=0.00252 Rate=1771.54 GlobalRate=510.49 Time=00:36:57
| Training Device=xla:0/0 Epoch=1 Step=660 Loss=0.00252 Rate=1771.50 GlobalRate=511.82 Time=00:36:57
| Training Device=xla:0/2 Epoch=1 Step=660 Loss=0.00252 Rate=1771.59 GlobalRate=507.32 Time=00:36:57
| Training Device=xla:0/1 Epoch=1 Step=660 Loss=0.00252 Rate=1771.43 GlobalRate=512.05 Time=00:36:57
| Training Device=xla:0/0 Epoch=1 Step=680 Loss=0.00242 Rate=1771.01 GlobalRate=522.73 Time=00:36:58
| Training Device=xla:0/1 Epoch=1 Step=680 Loss=0.00242 Rate=1770.92 GlobalRate=522.97 Time=00:36:58
| Training Device=xla:0/3 Epoch=1 Step=680 Loss=0.00242 Rate=1770.99 GlobalRate=521.38 Time=00:36:58
| Training Device=xla:0/2 Epoch=1 Step=680 Loss=0.00242 Rate=1771.02 GlobalRate=518.18 Time=00:36:58
| Training Device=xla:0/3 Epoch=1 Step=700 Loss=0.00234 Rate=1749.34 GlobalRate=532.00 Time=00:37:00
| Training Device=xla:0/0 Epoch=1 Step=700 Loss=0.00234 Rate=1749.25 GlobalRate=533.36 Time=00:37:00
| Training Device=xla:0/1 Epoch=1 Step=700 Loss=0.00234 Rate=1749.35 GlobalRate=533.61 Time=00:37:00
| Training Device=xla:0/2 Epoch=1 Step=700 Loss=0.00234 Rate=1749.38 GlobalRate=528.76 Time=00:37:00
| Training Device=xla:0/1 Epoch=1 Step=720 Loss=0.00226 Rate=1761.16 GlobalRate=544.15 Time=00:37:01
| Training Device=xla:0/0 Epoch=1 Step=720 Loss=0.00226 Rate=1761.07 GlobalRate=543.90 Time=00:37:01
| Training Device=xla:0/3 Epoch=1 Step=720 Loss=0.00226 Rate=1760.99 GlobalRate=542.52 Time=00:37:01
| Training Device=xla:0/2 Epoch=1 Step=720 Loss=0.00226 Rate=1760.97 GlobalRate=539.24 Time=00:37:01
| Training Device=xla:0/1 Epoch=1 Step=740 Loss=0.00218 Rate=1767.16 GlobalRate=554.52 Time=00:37:03
| Training Device=xla:0/0 Epoch=1 Step=740 Loss=0.00218 Rate=1767.10 GlobalRate=554.27 Time=00:37:03
| Training Device=xla:0/3 Epoch=1 Step=740 Loss=0.00218 Rate=1767.16 GlobalRate=552.87 Time=00:37:03
| Training Device=xla:0/2 Epoch=1 Step=740 Loss=0.00218 Rate=1767.02 GlobalRate=549.56 Time=00:37:03
| Training Device=xla:0/0 Epoch=1 Step=760 Loss=0.00211 Rate=1771.79 GlobalRate=564.47 Time=00:37:04
| Training Device=xla:0/1 Epoch=1 Step=760 Loss=0.00211 Rate=1771.80 GlobalRate=564.72 Time=00:37:04
| Training Device=xla:0/3 Epoch=1 Step=760 Loss=0.00211 Rate=1771.82 GlobalRate=563.06 Time=00:37:04
| Training Device=xla:0/2 Epoch=1 Step=760 Loss=0.00211 Rate=1771.99 GlobalRate=559.72 Time=00:37:04
| Training Device=xla:0/1 Epoch=1 Step=780 Loss=0.00205 Rate=1773.05 GlobalRate=574.75 Time=00:37:05
| Training Device=xla:0/0 Epoch=1 Step=780 Loss=0.00205 Rate=1772.92 GlobalRate=574.50 Time=00:37:05
| Training Device=xla:0/3 Epoch=1 Step=780 Loss=0.00205 Rate=1772.93 GlobalRate=573.08 Time=00:37:05
| Training Device=xla:0/2 Epoch=1 Step=780 Loss=0.00205 Rate=1773.05 GlobalRate=569.70 Time=00:37:05
| Training Device=xla:0/1 Epoch=1 Step=800 Loss=0.00200 Rate=1769.00 GlobalRate=584.60 Time=00:37:07
| Training Device=xla:0/0 Epoch=1 Step=800 Loss=0.00200 Rate=1769.04 GlobalRate=584.34 Time=00:37:07
| Training Device=xla:0/3 Epoch=1 Step=800 Loss=0.00200 Rate=1769.03 GlobalRate=582.91 Time=00:37:07
| Training Device=xla:0/2 Epoch=1 Step=800 Loss=0.00200 Rate=1768.72 GlobalRate=579.50 Time=00:37:07
| Training Device=xla:0/1 Epoch=1 Step=820 Loss=0.00194 Rate=1770.23 GlobalRate=594.30 Time=00:37:08
| Training Device=xla:0/0 Epoch=1 Step=820 Loss=0.00194 Rate=1770.24 GlobalRate=594.04 Time=00:37:08
| Training Device=xla:0/2 Epoch=1 Step=820 Loss=0.00194 Rate=1770.09 GlobalRate=589.16 Time=00:37:08
| Training Device=xla:0/3 Epoch=1 Step=820 Loss=0.00194 Rate=1770.23 GlobalRate=592.60 Time=00:37:08
| Training Device=xla:0/0 Epoch=1 Step=840 Loss=0.00190 Rate=1749.61 GlobalRate=603.48 Time=00:37:10
| Training Device=xla:0/2 Epoch=1 Step=840 Loss=0.00190 Rate=1749.88 GlobalRate=598.56 Time=00:37:10
| Training Device=xla:0/3 Epoch=1 Step=840 Loss=0.00190 Rate=1749.57 GlobalRate=602.03 Time=00:37:10
| Training Device=xla:0/1 Epoch=1 Step=840 Loss=0.00190 Rate=1748.97 GlobalRate=603.74 Time=00:37:10
| Training Device=xla:0/3 Epoch=1 Step=860 Loss=0.00185 Rate=1760.48 GlobalRate=611.39 Time=00:37:11
| Training Device=xla:0/1 Epoch=1 Step=860 Loss=0.00185 Rate=1760.68 GlobalRate=613.12 Time=00:37:11
| Training Device=xla:0/0 Epoch=1 Step=860 Loss=0.00185 Rate=1760.40 GlobalRate=612.86 Time=00:37:11
| Training Device=xla:0/2 Epoch=1 Step=860 Loss=0.00185 Rate=1760.50 GlobalRate=607.90 Time=00:37:11
| Training Device=xla:0/2 Epoch=1 Step=880 Loss=0.00181 Rate=1766.50 GlobalRate=617.10 Time=00:37:13
| Training Device=xla:0/3 Epoch=1 Step=880 Loss=0.00181 Rate=1766.43 GlobalRate=620.61 Time=00:37:13
| Training Device=xla:0/1 Epoch=1 Step=880 Loss=0.00181 Rate=1766.49 GlobalRate=622.35 Time=00:37:13
| Training Device=xla:0/0 Epoch=1 Step=880 Loss=0.00181 Rate=1766.30 GlobalRate=622.09 Time=00:37:13
| Training Device=xla:0/1 Epoch=1 Step=900 Loss=0.00178 Rate=1768.01 GlobalRate=631.44 Time=00:37:14
| Training Device=xla:0/0 Epoch=1 Step=900 Loss=0.00178 Rate=1767.86 GlobalRate=631.17 Time=00:37:14
| Training Device=xla:0/3 Epoch=1 Step=900 Loss=0.00178 Rate=1767.81 GlobalRate=629.69 Time=00:37:14
| Training Device=xla:0/2 Epoch=1 Step=900 Loss=0.00178 Rate=1767.80 GlobalRate=626.15 Time=00:37:14
| Training Device=xla:0/1 Epoch=1 Step=920 Loss=0.00174 Rate=1771.88 GlobalRate=640.40 Time=00:37:16
| Training Device=xla:0/2 Epoch=1 Step=920 Loss=0.00174 Rate=1771.84 GlobalRate=635.08 Time=00:37:16
| Training Device=xla:0/3 Epoch=1 Step=920 Loss=0.00174 Rate=1771.83 GlobalRate=638.63 Time=00:37:16
| Training Device=xla:0/0 Epoch=1 Step=920 Loss=0.00174 Rate=1771.37 GlobalRate=640.13 Time=00:37:16
| Training Device=xla:0/1 Epoch=1 Step=940 Loss=0.00171 Rate=1767.00 GlobalRate=649.19 Time=00:37:17
| Training Device=xla:0/0 Epoch=1 Step=940 Loss=0.00171 Rate=1767.30 GlobalRate=648.92 Time=00:37:17
| Training Device=xla:0/3 Epoch=1 Step=940 Loss=0.00171 Rate=1766.99 GlobalRate=647.41 Time=00:37:17
| Training Device=xla:0/2 Epoch=1 Step=940 Loss=0.00171 Rate=1766.96 GlobalRate=643.83 Time=00:37:17
| Training Device=xla:0/0 Epoch=1 Step=960 Loss=0.00168 Rate=1768.53 GlobalRate=657.58 Time=00:37:18
| Training Device=xla:0/3 Epoch=1 Step=960 Loss=0.00168 Rate=1768.38 GlobalRate=656.07 Time=00:37:18
| Training Device=xla:0/2 Epoch=1 Step=960 Loss=0.00168 Rate=1768.25 GlobalRate=652.47 Time=00:37:18
| Training Device=xla:0/1 Epoch=1 Step=960 Loss=0.00168 Rate=1768.35 GlobalRate=657.85 Time=00:37:18
| Training Device=xla:0/1 Epoch=1 Step=980 Loss=0.00165 Rate=1769.41 GlobalRate=666.39 Time=00:37:20
| Training Device=xla:0/2 Epoch=1 Step=980 Loss=0.00165 Rate=1769.52 GlobalRate=660.98 Time=00:37:20
| Training Device=xla:0/3 Epoch=1 Step=980 Loss=0.00165 Rate=1769.44 GlobalRate=664.60 Time=00:37:20
| Training Device=xla:0/0 Epoch=1 Step=980 Loss=0.00165 Rate=1769.43 GlobalRate=666.12 Time=00:37:20
| Training Device=xla:0/1 Epoch=1 Step=1000 Loss=0.00163 Rate=1770.82 GlobalRate=674.80 Time=00:37:21
| Training Device=xla:0/2 Epoch=1 Step=1000 Loss=0.00163 Rate=1770.76 GlobalRate=669.36 Time=00:37:21
| Training Device=xla:0/3 Epoch=1 Step=1000 Loss=0.00163 Rate=1770.74 GlobalRate=673.00 Time=00:37:21
| Training Device=xla:0/0 Epoch=1 Step=1000 Loss=0.00163 Rate=1770.79 GlobalRate=674.53 Time=00:37:21
| Training Device=xla:0/1 Epoch=1 Step=1020 Loss=0.00161 Rate=1771.29 GlobalRate=683.09 Time=00:37:23
| Training Device=xla:0/3 Epoch=1 Step=1020 Loss=0.00161 Rate=1771.25 GlobalRate=681.28 Time=00:37:23
| Training Device=xla:0/0 Epoch=1 Step=1020 Loss=0.00161 Rate=1771.23 GlobalRate=682.81 Time=00:37:23
| Training Device=xla:0/2 Epoch=1 Step=1020 Loss=0.00161 Rate=1771.22 GlobalRate=677.62 Time=00:37:23
| Training Device=xla:0/0 Epoch=1 Step=1040 Loss=0.00158 Rate=1773.13 GlobalRate=690.98 Time=00:37:24
| Training Device=xla:0/3 Epoch=1 Step=1040 Loss=0.00158 Rate=1773.10 GlobalRate=689.44 Time=00:37:24
| Training Device=xla:0/2 Epoch=1 Step=1040 Loss=0.00158 Rate=1773.14 GlobalRate=685.76 Time=00:37:24
| Training Device=xla:0/1 Epoch=1 Step=1040 Loss=0.00158 Rate=1773.02 GlobalRate=691.25 Time=00:37:24
| Training Device=xla:0/3 Epoch=1 Step=1060 Loss=0.00156 Rate=1772.14 GlobalRate=697.47 Time=00:37:26
| Training Device=xla:0/1 Epoch=1 Step=1060 Loss=0.00156 Rate=1772.12 GlobalRate=699.29 Time=00:37:26
| Training Device=xla:0/0 Epoch=1 Step=1060 Loss=0.00156 Rate=1771.93 GlobalRate=699.01 Time=00:37:26
| Training Device=xla:0/2 Epoch=1 Step=1060 Loss=0.00156 Rate=1772.11 GlobalRate=693.78 Time=00:37:26
| Training Device=xla:0/1 Epoch=1 Step=1080 Loss=0.00155 Rate=1772.11 GlobalRate=707.21 Time=00:37:27
| Training Device=xla:0/3 Epoch=1 Step=1080 Loss=0.00155 Rate=1772.12 GlobalRate=705.38 Time=00:37:27
| Training Device=xla:0/0 Epoch=1 Step=1080 Loss=0.00155 Rate=1772.09 GlobalRate=706.93 Time=00:37:27
| Training Device=xla:0/2 Epoch=1 Step=1080 Loss=0.00155 Rate=1772.08 GlobalRate=701.68 Time=00:37:27
| Training Device=xla:0/1 Epoch=1 Step=1100 Loss=0.00153 Rate=1774.41 GlobalRate=715.03 Time=00:37:29
| Training Device=xla:0/3 Epoch=1 Step=1100 Loss=0.00153 Rate=1774.32 GlobalRate=713.19 Time=00:37:29
| Training Device=xla:0/2 Epoch=1 Step=1100 Loss=0.00153 Rate=1774.38 GlobalRate=709.48 Time=00:37:29
| Training Device=xla:0/0 Epoch=1 Step=1100 Loss=0.00153 Rate=1774.39 GlobalRate=714.75 Time=00:37:29
| Training Device=xla:0/1 Epoch=1 Step=1120 Loss=0.00152 Rate=1774.25 GlobalRate=722.73 Time=00:37:30
| Training Device=xla:0/0 Epoch=1 Step=1120 Loss=0.00152 Rate=1774.36 GlobalRate=722.44 Time=00:37:30
| Training Device=xla:0/3 Epoch=1 Step=1120 Loss=0.00152 Rate=1774.26 GlobalRate=720.88 Time=00:37:30
| Training Device=xla:0/2 Epoch=1 Step=1120 Loss=0.00152 Rate=1774.24 GlobalRate=717.15 Time=00:37:30
| Training Device=xla:0/0 Epoch=1 Step=1140 Loss=0.00150 Rate=1756.14 GlobalRate=729.94 Time=00:37:31
| Training Device=xla:0/2 Epoch=1 Step=1140 Loss=0.00150 Rate=1756.12 GlobalRate=724.63 Time=00:37:31
| Training Device=xla:0/1 Epoch=1 Step=1140 Loss=0.00150 Rate=1756.06 GlobalRate=730.22 Time=00:37:31
| Training Device=xla:0/3 Epoch=1 Step=1140 Loss=0.00150 Rate=1755.77 GlobalRate=728.37 Time=00:37:31
| Training Device=xla:0/1 Epoch=1 Step=1160 Loss=0.00149 Rate=1741.90 GlobalRate=737.57 Time=00:37:33
| Training Device=xla:0/3 Epoch=1 Step=1160 Loss=0.00149 Rate=1742.23 GlobalRate=735.72 Time=00:37:33
| Training Device=xla:0/2 Epoch=1 Step=1160 Loss=0.00149 Rate=1742.06 GlobalRate=731.97 Time=00:37:33
| Training Device=xla:0/0 Epoch=1 Step=1160 Loss=0.00149 Rate=1741.85 GlobalRate=737.29 Time=00:37:33
| Training Device=xla:0/2 Epoch=1 Step=1180 Loss=0.00147 Rate=1760.12 GlobalRate=739.32 Time=00:37:34
| Training Device=xla:0/0 Epoch=1 Step=1180 Loss=0.00147 Rate=1760.07 GlobalRate=744.65 Time=00:37:34
| Training Device=xla:0/3 Epoch=1 Step=1180 Loss=0.00147 Rate=1760.16 GlobalRate=743.08 Time=00:37:34
| Training Device=xla:0/1 Epoch=1 Step=1180 Loss=0.00147 Rate=1760.11 GlobalRate=744.94 Time=00:37:34
| Training Device=xla:0/1 Epoch=1 Step=1200 Loss=0.00146 Rate=1765.80 GlobalRate=752.19 Time=00:37:36
| Training Device=xla:0/0 Epoch=1 Step=1200 Loss=0.00146 Rate=1765.75 GlobalRate=751.90 Time=00:37:36
| Training Device=xla:0/2 Epoch=1 Step=1200 Loss=0.00146 Rate=1765.75 GlobalRate=746.55 Time=00:37:36
| Training Device=xla:0/3 Epoch=1 Step=1200 Loss=0.00146 Rate=1765.62 GlobalRate=750.32 Time=00:37:36
| Training Device=xla:0/1 Epoch=1 Step=1220 Loss=0.00145 Rate=1770.63 GlobalRate=759.35 Time=00:37:37
| Training Device=xla:0/0 Epoch=1 Step=1220 Loss=0.00145 Rate=1770.58 GlobalRate=759.07 Time=00:37:37
| Training Device=xla:0/3 Epoch=1 Step=1220 Loss=0.00145 Rate=1770.69 GlobalRate=757.48 Time=00:37:37
| Training Device=xla:0/2 Epoch=1 Step=1220 Loss=0.00145 Rate=1770.35 GlobalRate=753.70 Time=00:37:37
| Training Device=xla:0/0 Epoch=1 Step=1240 Loss=0.00144 Rate=1771.40 GlobalRate=766.13 Time=00:37:39
| Training Device=xla:0/3 Epoch=1 Step=1240 Loss=0.00144 Rate=1771.32 GlobalRate=764.54 Time=00:37:39
| Training Device=xla:0/2 Epoch=1 Step=1240 Loss=0.00144 Rate=1771.31 GlobalRate=760.75 Time=00:37:39
| Training Device=xla:0/1 Epoch=1 Step=1240 Loss=0.00144 Rate=1771.13 GlobalRate=766.41 Time=00:37:39
| Training Device=xla:0/0 Epoch=1 Step=1260 Loss=0.00144 Rate=1774.29 GlobalRate=773.10 Time=00:37:40
| Training Device=xla:0/3 Epoch=1 Step=1260 Loss=0.00144 Rate=1774.27 GlobalRate=771.51 Time=00:37:40
| Training Device=xla:0/2 Epoch=1 Step=1260 Loss=0.00144 Rate=1774.33 GlobalRate=767.71 Time=00:37:40
| Training Device=xla:0/1 Epoch=1 Step=1260 Loss=0.00144 Rate=1774.27 GlobalRate=773.38 Time=00:37:40
| Training Device=xla:0/1 Epoch=1 Step=1280 Loss=0.00143 Rate=1775.53 GlobalRate=780.26 Time=00:37:42
| Training Device=xla:0/2 Epoch=1 Step=1280 Loss=0.00143 Rate=1775.45 GlobalRate=774.58 Time=00:37:42
| Training Device=xla:0/0 Epoch=1 Step=1280 Loss=0.00143 Rate=1775.44 GlobalRate=779.98 Time=00:37:42
| Training Device=xla:0/3 Epoch=1 Step=1280 Loss=0.00143 Rate=1775.42 GlobalRate=778.38 Time=00:37:42
| Training Device=xla:0/0 Epoch=1 Step=1300 Loss=0.00142 Rate=1735.18 GlobalRate=786.55 Time=00:37:43
| Training Device=xla:0/1 Epoch=1 Step=1300 Loss=0.00142 Rate=1735.14 GlobalRate=786.83 Time=00:37:43
| Training Device=xla:0/2 Epoch=1 Step=1300 Loss=0.00142 Rate=1735.04 GlobalRate=781.14 Time=00:37:43
| Training Device=xla:0/3 Epoch=1 Step=1300 Loss=0.00142 Rate=1734.61 GlobalRate=784.95 Time=00:37:43
| Training Device=xla:0/3 Epoch=1 Step=1320 Loss=0.00141 Rate=1756.18 GlobalRate=791.62 Time=00:37:45
| Training Device=xla:0/2 Epoch=1 Step=1320 Loss=0.00141 Rate=1756.01 GlobalRate=787.80 Time=00:37:45
| Training Device=xla:0/0 Epoch=1 Step=1320 Loss=0.00141 Rate=1755.70 GlobalRate=793.22 Time=00:37:45
| Training Device=xla:0/1 Epoch=1 Step=1320 Loss=0.00141 Rate=1755.70 GlobalRate=793.50 Time=00:37:45
| Training Device=xla:0/2 Epoch=1 Step=1340 Loss=0.00141 Rate=1765.54 GlobalRate=794.38 Time=00:37:46
| Training Device=xla:0/0 Epoch=1 Step=1340 Loss=0.00141 Rate=1765.45 GlobalRate=799.81 Time=00:37:46
| Training Device=xla:0/1 Epoch=1 Step=1340 Loss=0.00141 Rate=1765.50 GlobalRate=800.09 Time=00:37:46
| Training Device=xla:0/3 Epoch=1 Step=1340 Loss=0.00141 Rate=1765.50 GlobalRate=798.20 Time=00:37:46
| Training Device=xla:0/3 Epoch=1 Step=1360 Loss=0.00140 Rate=1764.75 GlobalRate=804.68 Time=00:37:47
| Training Device=xla:0/2 Epoch=1 Step=1360 Loss=0.00140 Rate=1764.61 GlobalRate=800.85 Time=00:37:47
| Training Device=xla:0/1 Epoch=1 Step=1360 Loss=0.00140 Rate=1764.50 GlobalRate=806.57 Time=00:37:47
| Training Device=xla:0/0 Epoch=1 Step=1360 Loss=0.00140 Rate=1764.69 GlobalRate=806.28 Time=00:37:47
| Training Device=xla:0/2 Epoch=1 Step=1380 Loss=0.00140 Rate=1768.14 GlobalRate=807.25 Time=00:37:49
| Training Device=xla:0/3 Epoch=1 Step=1380 Loss=0.00140 Rate=1768.18 GlobalRate=811.09 Time=00:37:49
| Training Device=xla:0/0 Epoch=1 Step=1380 Loss=0.00140 Rate=1768.15 GlobalRate=812.69 Time=00:37:49
| Training Device=xla:0/1 Epoch=1 Step=1380 Loss=0.00140 Rate=1768.16 GlobalRate=812.98 Time=00:37:49
| Training Device=xla:0/2 Epoch=1 Step=1400 Loss=0.00139 Rate=1771.42 GlobalRate=813.58 Time=00:37:50
| Training Device=xla:0/0 Epoch=1 Step=1400 Loss=0.00139 Rate=1771.43 GlobalRate=819.03 Time=00:37:50
| Training Device=xla:0/1 Epoch=1 Step=1400 Loss=0.00139 Rate=1771.41 GlobalRate=819.32 Time=00:37:50
| Training Device=xla:0/3 Epoch=1 Step=1400 Loss=0.00139 Rate=1771.38 GlobalRate=817.42 Time=00:37:50
| Training Device=xla:0/0 Epoch=1 Step=1420 Loss=0.00139 Rate=1768.29 GlobalRate=825.26 Time=00:37:52
| Training Device=xla:0/2 Epoch=1 Step=1420 Loss=0.00139 Rate=1768.28 GlobalRate=819.81 Time=00:37:52
| Training Device=xla:0/1 Epoch=1 Step=1420 Loss=0.00139 Rate=1768.27 GlobalRate=825.54 Time=00:37:52
| Training Device=xla:0/3 Epoch=1 Step=1420 Loss=0.00139 Rate=1768.21 GlobalRate=823.65 Time=00:37:52
| Training Device=xla:0/3 Epoch=1 Step=1440 Loss=0.00139 Rate=1769.01 GlobalRate=829.80 Time=00:37:53
| Training Device=xla:0/2 Epoch=1 Step=1440 Loss=0.00139 Rate=1768.88 GlobalRate=825.96 Time=00:37:53
| Training Device=xla:0/0 Epoch=1 Step=1440 Loss=0.00139 Rate=1768.96 GlobalRate=831.41 Time=00:37:53
| Training Device=xla:0/1 Epoch=1 Step=1440 Loss=0.00139 Rate=1768.97 GlobalRate=831.70 Time=00:37:53
| Training Device=xla:0/0 Epoch=1 Step=1460 Loss=0.00138 Rate=1767.41 GlobalRate=837.48 Time=00:37:55
| Training Device=xla:0/3 Epoch=1 Step=1460 Loss=0.00138 Rate=1767.48 GlobalRate=835.87 Time=00:37:55
| Training Device=xla:0/1 Epoch=1 Step=1460 Loss=0.00138 Rate=1767.46 GlobalRate=837.77 Time=00:37:55
| Training Device=xla:0/2 Epoch=1 Step=1460 Loss=0.00138 Rate=1767.37 GlobalRate=832.02 Time=00:37:55
| Training Device=xla:0/0 Epoch=1 Step=1480 Loss=0.00138 Rate=1765.61 GlobalRate=843.47 Time=00:37:56
| Training Device=xla:0/1 Epoch=1 Step=1480 Loss=0.00138 Rate=1765.62 GlobalRate=843.76 Time=00:37:56
| Training Device=xla:0/3 Epoch=1 Step=1480 Loss=0.00138 Rate=1765.57 GlobalRate=841.85 Time=00:37:56
| Training Device=xla:0/2 Epoch=1 Step=1480 Loss=0.00138 Rate=1765.60 GlobalRate=838.00 Time=00:37:56
| Training Device=xla:0/2 Epoch=1 Step=1500 Loss=0.00138 Rate=1767.93 GlobalRate=843.92 Time=00:37:58
| Training Device=xla:0/3 Epoch=1 Step=1500 Loss=0.00138 Rate=1767.93 GlobalRate=847.77 Time=00:37:58
| Training Device=xla:0/1 Epoch=1 Step=1500 Loss=0.00138 Rate=1767.92 GlobalRate=849.68 Time=00:37:58
| Training Device=xla:0/0 Epoch=1 Step=1500 Loss=0.00138 Rate=1767.82 GlobalRate=849.39 Time=00:37:58
| Training Device=xla:0/0 Epoch=1 Step=1520 Loss=0.00137 Rate=1770.34 GlobalRate=855.24 Time=00:37:59
| Training Device=xla:0/1 Epoch=1 Step=1520 Loss=0.00137 Rate=1770.33 GlobalRate=855.53 Time=00:37:59
| Training Device=xla:0/3 Epoch=1 Step=1520 Loss=0.00137 Rate=1770.30 GlobalRate=853.63 Time=00:37:59
| Training Device=xla:0/2 Epoch=1 Step=1520 Loss=0.00137 Rate=1770.25 GlobalRate=849.77 Time=00:37:59
| Training Device=xla:0/0 Epoch=1 Step=1540 Loss=0.00137 Rate=1772.86 GlobalRate=861.03 Time=00:38:00
| Training Device=xla:0/2 Epoch=1 Step=1540 Loss=0.00137 Rate=1772.86 GlobalRate=855.56 Time=00:38:00
| Training Device=xla:0/1 Epoch=1 Step=1540 Loss=0.00137 Rate=1772.82 GlobalRate=861.32 Time=00:38:00
| Training Device=xla:0/3 Epoch=1 Step=1540 Loss=0.00137 Rate=1772.66 GlobalRate=859.41 Time=00:38:00
| Training Device=xla:0/0 Epoch=1 Step=1560 Loss=0.00137 Rate=1773.79 GlobalRate=866.75 Time=00:38:02
| Training Device=xla:0/1 Epoch=1 Step=1560 Loss=0.00137 Rate=1773.79 GlobalRate=867.04 Time=00:38:02
| Training Device=xla:0/3 Epoch=1 Step=1560 Loss=0.00137 Rate=1773.76 GlobalRate=865.13 Time=00:38:02
| Training Device=xla:0/2 Epoch=1 Step=1560 Loss=0.00137 Rate=1773.28 GlobalRate=861.27 Time=00:38:02
| Training Device=xla:0/0 Epoch=1 Step=1580 Loss=0.00136 Rate=1758.70 GlobalRate=872.31 Time=00:38:03
| Training Device=xla:0/2 Epoch=1 Step=1580 Loss=0.00136 Rate=1759.03 GlobalRate=866.84 Time=00:38:03
| Training Device=xla:0/3 Epoch=1 Step=1580 Loss=0.00136 Rate=1758.79 GlobalRate=870.70 Time=00:38:03
| Training Device=xla:0/1 Epoch=1 Step=1580 Loss=0.00136 Rate=1758.37 GlobalRate=872.60 Time=00:38:03
| Training Device=xla:0/0 Epoch=1 Step=1600 Loss=0.00136 Rate=1766.18 GlobalRate=877.88 Time=00:38:05
| Training Device=xla:0/1 Epoch=1 Step=1600 Loss=0.00136 Rate=1766.40 GlobalRate=878.17 Time=00:38:05
| Training Device=xla:0/3 Epoch=1 Step=1600 Loss=0.00136 Rate=1766.23 GlobalRate=876.26 Time=00:38:05
| Training Device=xla:0/2 Epoch=1 Step=1600 Loss=0.00136 Rate=1766.17 GlobalRate=872.40 Time=00:38:05
| Training Device=xla:0/2 Epoch=1 Step=1620 Loss=0.00136 Rate=1768.19 GlobalRate=877.89 Time=00:38:06
| Training Device=xla:0/1 Epoch=1 Step=1620 Loss=0.00136 Rate=1768.18 GlobalRate=883.66 Time=00:38:06
| Training Device=xla:0/0 Epoch=1 Step=1620 Loss=0.00136 Rate=1768.00 GlobalRate=883.37 Time=00:38:06
| Training Device=xla:0/3 Epoch=1 Step=1620 Loss=0.00136 Rate=1768.08 GlobalRate=881.75 Time=00:38:06
| Training Device=xla:0/1 Epoch=1 Step=1640 Loss=0.00136 Rate=1772.22 GlobalRate=889.10 Time=00:38:08
| Training Device=xla:0/2 Epoch=1 Step=1640 Loss=0.00136 Rate=1772.15 GlobalRate=883.33 Time=00:38:08
| Training Device=xla:0/0 Epoch=1 Step=1640 Loss=0.00136 Rate=1772.24 GlobalRate=888.81 Time=00:38:08
| Training Device=xla:0/3 Epoch=1 Step=1640 Loss=0.00136 Rate=1772.16 GlobalRate=887.19 Time=00:38:08
| Training Device=xla:0/2 Epoch=1 Step=1660 Loss=0.00136 Rate=1773.26 GlobalRate=888.71 Time=00:38:09
| Training Device=xla:0/0 Epoch=1 Step=1660 Loss=0.00136 Rate=1773.22 GlobalRate=894.18 Time=00:38:09
| Training Device=xla:0/1 Epoch=1 Step=1660 Loss=0.00136 Rate=1773.17 GlobalRate=894.47 Time=00:38:09
| Training Device=xla:0/3 Epoch=1 Step=1660 Loss=0.00136 Rate=1773.14 GlobalRate=892.57 Time=00:38:09
| Training Device=xla:0/0 Epoch=1 Step=1680 Loss=0.00136 Rate=1773.26 GlobalRate=899.49 Time=00:38:11
| Training Device=xla:0/2 Epoch=1 Step=1680 Loss=0.00136 Rate=1773.27 GlobalRate=894.01 Time=00:38:11
| Training Device=xla:0/3 Epoch=1 Step=1680 Loss=0.00136 Rate=1773.28 GlobalRate=897.87 Time=00:38:11
| Training Device=xla:0/1 Epoch=1 Step=1680 Loss=0.00136 Rate=1773.21 GlobalRate=899.78 Time=00:38:11
| Training Device=xla:0/0 Epoch=1 Step=1700 Loss=0.00136 Rate=1773.67 GlobalRate=904.73 Time=00:38:12
| Training Device=xla:0/3 Epoch=1 Step=1700 Loss=0.00136 Rate=1773.75 GlobalRate=903.12 Time=00:38:12
| Training Device=xla:0/2 Epoch=1 Step=1700 Loss=0.00136 Rate=1773.68 GlobalRate=899.26 Time=00:38:12
| Training Device=xla:0/1 Epoch=1 Step=1700 Loss=0.00136 Rate=1773.67 GlobalRate=905.02 Time=00:38:12
| Training Device=xla:0/0 Epoch=1 Step=1720 Loss=0.00136 Rate=1774.57 GlobalRate=909.92 Time=00:38:14
| Training Device=xla:0/1 Epoch=1 Step=1720 Loss=0.00136 Rate=1774.58 GlobalRate=910.21 Time=00:38:14
| Training Device=xla:0/3 Epoch=1 Step=1720 Loss=0.00136 Rate=1774.69 GlobalRate=908.30 Time=00:38:14
| Training Device=xla:0/2 Epoch=1 Step=1720 Loss=0.00136 Rate=1774.68 GlobalRate=904.44 Time=00:38:14
| Training Device=xla:0/0 Epoch=1 Step=1740 Loss=0.00135 Rate=1775.33 GlobalRate=915.04 Time=00:38:15
| Training Device=xla:0/1 Epoch=1 Step=1740 Loss=0.00135 Rate=1775.34 GlobalRate=915.33 Time=00:38:15
| Training Device=xla:0/3 Epoch=1 Step=1740 Loss=0.00135 Rate=1775.19 GlobalRate=913.43 Time=00:38:15
| Training Device=xla:0/2 Epoch=1 Step=1740 Loss=0.00135 Rate=1775.08 GlobalRate=909.57 Time=00:38:15
| Training Device=xla:0/1 Epoch=1 Step=1760 Loss=0.00135 Rate=1777.22 GlobalRate=920.41 Time=00:38:16
| Training Device=xla:0/3 Epoch=1 Step=1760 Loss=0.00135 Rate=1777.18 GlobalRate=918.50 Time=00:38:16
| Training Device=xla:0/0 Epoch=1 Step=1760 Loss=0.00135 Rate=1777.03 GlobalRate=920.12 Time=00:38:16
| Training Device=xla:0/2 Epoch=1 Step=1760 Loss=0.00135 Rate=1777.00 GlobalRate=914.64 Time=00:38:16
| Training Device=xla:0/0 Epoch=1 Step=1780 Loss=0.00135 Rate=1775.97 GlobalRate=925.12 Time=00:38:18
| Training Device=xla:0/2 Epoch=1 Step=1780 Loss=0.00135 Rate=1776.04 GlobalRate=919.65 Time=00:38:18
| Training Device=xla:0/1 Epoch=1 Step=1780 Loss=0.00135 Rate=1775.86 GlobalRate=925.41 Time=00:38:18
| Training Device=xla:0/3 Epoch=1 Step=1780 Loss=0.00135 Rate=1775.79 GlobalRate=923.51 Time=00:38:18
| Training Device=xla:0/0 Epoch=1 Step=1800 Loss=0.00135 Rate=1774.64 GlobalRate=930.06 Time=00:38:19
| Training Device=xla:0/3 Epoch=1 Step=1800 Loss=0.00135 Rate=1774.61 GlobalRate=928.45 Time=00:38:19
| Training Device=xla:0/1 Epoch=1 Step=1800 Loss=0.00135 Rate=1774.63 GlobalRate=930.35 Time=00:38:19
| Training Device=xla:0/2 Epoch=1 Step=1800 Loss=0.00135 Rate=1774.60 GlobalRate=924.60 Time=00:38:19
| Training Device=xla:0/2 Epoch=1 Step=1820 Loss=0.00135 Rate=1774.47 GlobalRate=929.48 Time=00:38:21
| Training Device=xla:0/0 Epoch=1 Step=1820 Loss=0.00135 Rate=1774.39 GlobalRate=934.95 Time=00:38:21
| Training Device=xla:0/3 Epoch=1 Step=1820 Loss=0.00135 Rate=1774.41 GlobalRate=933.33 Time=00:38:21
| Training Device=xla:0/1 Epoch=1 Step=1820 Loss=0.00135 Rate=1774.38 GlobalRate=935.24 Time=00:38:21
| Training Device=xla:0/2 Epoch=1 Step=1840 Loss=0.00135 Rate=1766.90 GlobalRate=934.28 Time=00:38:22
| Training Device=xla:0/1 Epoch=1 Step=1840 Loss=0.00135 Rate=1766.79 GlobalRate=940.03 Time=00:38:22
| Training Device=xla:0/3 Epoch=1 Step=1840 Loss=0.00135 Rate=1766.83 GlobalRate=938.13 Time=00:38:22
| Training Device=xla:0/0 Epoch=1 Step=1840 Loss=0.00135 Rate=1766.82 GlobalRate=939.74 Time=00:38:22
| Training Device=xla:0/2 Epoch=1 Step=1860 Loss=0.00135 Rate=1773.31 GlobalRate=939.07 Time=00:38:24
| Training Device=xla:0/0 Epoch=1 Step=1860 Loss=0.00135 Rate=1773.32 GlobalRate=944.52 Time=00:38:24
| Training Device=xla:0/3 Epoch=1 Step=1860 Loss=0.00135 Rate=1773.36 GlobalRate=942.91 Time=00:38:24
| Training Device=xla:0/1 Epoch=1 Step=1860 Loss=0.00135 Rate=1773.31 GlobalRate=944.81 Time=00:38:24
| Training Device=xla:0/2 Epoch=1 Step=1880 Loss=0.00135 Rate=1775.32 GlobalRate=943.80 Time=00:38:25
| Training Device=xla:0/1 Epoch=1 Step=1880 Loss=0.00135 Rate=1775.34 GlobalRate=949.54 Time=00:38:25
| Training Device=xla:0/3 Epoch=1 Step=1880 Loss=0.00135 Rate=1775.29 GlobalRate=947.64 Time=00:38:25
| Training Device=xla:0/0 Epoch=1 Step=1880 Loss=0.00135 Rate=1775.30 GlobalRate=949.25 Time=00:38:25
| Training Device=xla:0/1 Epoch=1 Step=1900 Loss=0.00135 Rate=1771.51 GlobalRate=954.19 Time=00:38:27
| Training Device=xla:0/2 Epoch=1 Step=1900 Loss=0.00135 Rate=1771.42 GlobalRate=948.45 Time=00:38:27
| Training Device=xla:0/0 Epoch=1 Step=1900 Loss=0.00135 Rate=1771.50 GlobalRate=953.90 Time=00:38:27
| Training Device=xla:0/3 Epoch=1 Step=1900 Loss=0.00135 Rate=1771.38 GlobalRate=952.29 Time=00:38:27
| Training Device=xla:0/2 Epoch=1 Step=1920 Loss=0.00135 Rate=1774.81 GlobalRate=953.08 Time=00:38:28
| Training Device=xla:0/1 Epoch=1 Step=1920 Loss=0.00135 Rate=1774.77 GlobalRate=958.81 Time=00:38:28
| Training Device=xla:0/0 Epoch=1 Step=1920 Loss=0.00135 Rate=1774.71 GlobalRate=958.52 Time=00:38:28
| Training Device=xla:0/3 Epoch=1 Step=1920 Loss=0.00135 Rate=1774.84 GlobalRate=956.92 Time=00:38:28
| Training Device=xla:0/0 Epoch=1 Step=1940 Loss=0.00135 Rate=1777.08 GlobalRate=963.10 Time=00:38:29
| Training Device=xla:0/2 Epoch=1 Step=1940 Loss=0.00135 Rate=1777.01 GlobalRate=957.66 Time=00:38:29
| Training Device=xla:0/1 Epoch=1 Step=1940 Loss=0.00135 Rate=1777.03 GlobalRate=963.39 Time=00:38:29
| Training Device=xla:0/3 Epoch=1 Step=1940 Loss=0.00135 Rate=1776.95 GlobalRate=961.49 Time=00:38:29
| Training Device=xla:0/1 Epoch=1 Step=1960 Loss=0.00135 Rate=1777.38 GlobalRate=967.91 Time=00:38:31
| Training Device=xla:0/3 Epoch=1 Step=1960 Loss=0.00135 Rate=1777.38 GlobalRate=966.02 Time=00:38:31
| Training Device=xla:0/2 Epoch=1 Step=1960 Loss=0.00135 Rate=1777.31 GlobalRate=962.19 Time=00:38:31
| Training Device=xla:0/0 Epoch=1 Step=1960 Loss=0.00135 Rate=1777.39 GlobalRate=967.62 Time=00:38:31
| Training Device=xla:0/1 Epoch=1 Step=1980 Loss=0.00135 Rate=1777.60 GlobalRate=972.38 Time=00:38:32
| Training Device=xla:0/0 Epoch=1 Step=1980 Loss=0.00135 Rate=1777.58 GlobalRate=972.09 Time=00:38:32
| Training Device=xla:0/3 Epoch=1 Step=1980 Loss=0.00135 Rate=1777.66 GlobalRate=970.49 Time=00:38:32
| Training Device=xla:0/2 Epoch=1 Step=1980 Loss=0.00135 Rate=1777.59 GlobalRate=966.66 Time=00:38:32
| Training Device=xla:0/1 Epoch=1 Step=2000 Loss=0.00135 Rate=1779.03 GlobalRate=976.81 Time=00:38:34
| Training Device=xla:0/3 Epoch=1 Step=2000 Loss=0.00135 Rate=1779.06 GlobalRate=974.92 Time=00:38:34
| Training Device=xla:0/2 Epoch=1 Step=2000 Loss=0.00135 Rate=1779.01 GlobalRate=971.10 Time=00:38:34
| Training Device=xla:0/0 Epoch=1 Step=2000 Loss=0.00135 Rate=1779.05 GlobalRate=976.52 Time=00:38:34
| Training Device=xla:0/0 Epoch=1 Step=2020 Loss=0.00135 Rate=1776.66 GlobalRate=980.89 Time=00:38:35
| Training Device=xla:0/3 Epoch=1 Step=2020 Loss=0.00135 Rate=1776.70 GlobalRate=979.29 Time=00:38:35
| Training Device=xla:0/1 Epoch=1 Step=2020 Loss=0.00135 Rate=1776.64 GlobalRate=981.18 Time=00:38:35
| Training Device=xla:0/2 Epoch=1 Step=2020 Loss=0.00135 Rate=1776.71 GlobalRate=975.47 Time=00:38:35
| Training Device=xla:0/3 Epoch=1 Step=2040 Loss=0.00135 Rate=1772.82 GlobalRate=983.60 Time=00:38:37
| Training Device=xla:0/0 Epoch=1 Step=2040 Loss=0.00135 Rate=1772.80 GlobalRate=985.19 Time=00:38:37
| Training Device=xla:0/1 Epoch=1 Step=2040 Loss=0.00135 Rate=1772.79 GlobalRate=985.48 Time=00:38:37
| Training Device=xla:0/2 Epoch=1 Step=2040 Loss=0.00135 Rate=1772.83 GlobalRate=979.78 Time=00:38:37
| Training Device=xla:0/2 Epoch=1 Step=2060 Loss=0.00135 Rate=1775.54 GlobalRate=984.07 Time=00:38:38
| Training Device=xla:0/3 Epoch=1 Step=2060 Loss=0.00135 Rate=1775.49 GlobalRate=987.88 Time=00:38:38
| Training Device=xla:0/0 Epoch=1 Step=2060 Loss=0.00135 Rate=1775.45 GlobalRate=989.47 Time=00:38:38
| Training Device=xla:0/1 Epoch=1 Step=2060 Loss=0.00135 Rate=1775.49 GlobalRate=989.76 Time=00:38:38
| Training Device=xla:0/1 Epoch=1 Step=2080 Loss=0.00135 Rate=1774.45 GlobalRate=993.98 Time=00:38:39
| Training Device=xla:0/0 Epoch=1 Step=2080 Loss=0.00135 Rate=1774.47 GlobalRate=993.70 Time=00:38:39
| Training Device=xla:0/2 Epoch=1 Step=2080 Loss=0.00135 Rate=1774.41 GlobalRate=988.29 Time=00:38:39
| Training Device=xla:0/3 Epoch=1 Step=2080 Loss=0.00135 Rate=1774.40 GlobalRate=992.10 Time=00:38:39
| Training Device=xla:0/1 Epoch=1 Step=2100 Loss=0.00135 Rate=1775.42 GlobalRate=998.17 Time=00:38:41
| Training Device=xla:0/0 Epoch=1 Step=2100 Loss=0.00135 Rate=1775.36 GlobalRate=997.88 Time=00:38:41
| Training Device=xla:0/2 Epoch=1 Step=2100 Loss=0.00135 Rate=1775.33 GlobalRate=992.48 Time=00:38:41
| Training Device=xla:0/3 Epoch=1 Step=2100 Loss=0.00135 Rate=1775.34 GlobalRate=996.29 Time=00:38:41
| Training Device=xla:0/1 Epoch=1 Step=2120 Loss=0.00135 Rate=1774.11 GlobalRate=1002.30 Time=00:38:42
| Training Device=xla:0/0 Epoch=1 Step=2120 Loss=0.00135 Rate=1774.13 GlobalRate=1002.01 Time=00:38:42
| Training Device=xla:0/2 Epoch=1 Step=2120 Loss=0.00135 Rate=1774.13 GlobalRate=996.62 Time=00:38:42
| Training Device=xla:0/3 Epoch=1 Step=2120 Loss=0.00135 Rate=1774.14 GlobalRate=1000.42 Time=00:38:42
| Training Device=xla:0/3 Epoch=1 Step=2140 Loss=0.00135 Rate=1777.51 GlobalRate=1004.53 Time=00:38:44
| Training Device=xla:0/2 Epoch=1 Step=2140 Loss=0.00135 Rate=1777.49 GlobalRate=1000.74 Time=00:38:44
| Training Device=xla:0/1 Epoch=1 Step=2140 Loss=0.00135 Rate=1777.47 GlobalRate=1006.40 Time=00:38:44
| Training Device=xla:0/0 Epoch=1 Step=2140 Loss=0.00135 Rate=1777.40 GlobalRate=1006.12 Time=00:38:44
| Training Device=xla:0/3 Epoch=1 Step=2160 Loss=0.00135 Rate=1778.08 GlobalRate=1008.59 Time=00:38:45
| Training Device=xla:0/0 Epoch=1 Step=2160 Loss=0.00135 Rate=1778.07 GlobalRate=1010.18 Time=00:38:45
| Training Device=xla:0/2 Epoch=1 Step=2160 Loss=0.00135 Rate=1778.03 GlobalRate=1004.80 Time=00:38:45
| Training Device=xla:0/1 Epoch=1 Step=2160 Loss=0.00135 Rate=1778.06 GlobalRate=1010.46 Time=00:38:45
| Training Device=xla:0/2 Epoch=1 Step=2180 Loss=0.00135 Rate=1778.26 GlobalRate=1008.83 Time=00:38:47
| Training Device=xla:0/0 Epoch=1 Step=2180 Loss=0.00135 Rate=1778.17 GlobalRate=1014.20 Time=00:38:47
| Training Device=xla:0/3 Epoch=1 Step=2180 Loss=0.00135 Rate=1778.24 GlobalRate=1012.61 Time=00:38:47
| Training Device=xla:0/1 Epoch=1 Step=2180 Loss=0.00135 Rate=1778.06 GlobalRate=1014.48 Time=00:38:47
| Training Device=xla:0/0 Epoch=1 Step=2200 Loss=0.00135 Rate=1777.56 GlobalRate=1018.17 Time=00:38:48
| Training Device=xla:0/1 Epoch=1 Step=2200 Loss=0.00135 Rate=1777.55 GlobalRate=1018.45 Time=00:38:48
| Training Device=xla:0/2 Epoch=1 Step=2200 Loss=0.00135 Rate=1777.49 GlobalRate=1012.80 Time=00:38:48
| Training Device=xla:0/3 Epoch=1 Step=2200 Loss=0.00135 Rate=1777.48 GlobalRate=1016.59 Time=00:38:48
| Training Device=xla:0/0 Epoch=1 Step=2220 Loss=0.00135 Rate=1775.45 GlobalRate=1022.09 Time=00:38:50
| Training Device=xla:0/2 Epoch=1 Step=2220 Loss=0.00135 Rate=1775.40 GlobalRate=1016.73 Time=00:38:50
| Training Device=xla:0/1 Epoch=1 Step=2220 Loss=0.00135 Rate=1775.45 GlobalRate=1022.37 Time=00:38:50
| Training Device=xla:0/3 Epoch=1 Step=2220 Loss=0.00135 Rate=1775.42 GlobalRate=1020.51 Time=00:38:50
| Training Device=xla:0/3 Epoch=1 Step=2240 Loss=0.00135 Rate=1766.75 GlobalRate=1024.35 Time=00:38:51
| Training Device=xla:0/2 Epoch=1 Step=2240 Loss=0.00135 Rate=1766.77 GlobalRate=1020.58 Time=00:38:51
| Training Device=xla:0/0 Epoch=1 Step=2240 Loss=0.00135 Rate=1766.75 GlobalRate=1025.93 Time=00:38:51
| Training Device=xla:0/1 Epoch=1 Step=2240 Loss=0.00135 Rate=1766.78 GlobalRate=1026.21 Time=00:38:51
| Training Device=xla:0/2 Epoch=1 Step=2260 Loss=0.00135 Rate=1774.17 GlobalRate=1024.45 Time=00:38:52
| Training Device=xla:0/3 Epoch=1 Step=2260 Loss=0.00135 Rate=1774.12 GlobalRate=1028.21 Time=00:38:52
| Training Device=xla:0/1 Epoch=1 Step=2260 Loss=0.00135 Rate=1774.14 GlobalRate=1030.07 Time=00:38:52
| Training Device=xla:0/0 Epoch=1 Step=2260 Loss=0.00135 Rate=1774.12 GlobalRate=1029.79 Time=00:38:52
| Training Device=xla:0/2 Epoch=1 Step=2280 Loss=0.00135 Rate=1775.35 GlobalRate=1028.26 Time=00:38:54
| Training Device=xla:0/1 Epoch=1 Step=2280 Loss=0.00135 Rate=1775.31 GlobalRate=1033.88 Time=00:38:54
| Training Device=xla:0/3 Epoch=1 Step=2280 Loss=0.00135 Rate=1775.19 GlobalRate=1032.02 Time=00:38:54
| Training Device=xla:0/0 Epoch=1 Step=2280 Loss=0.00135 Rate=1775.15 GlobalRate=1033.59 Time=00:38:54
| Training Device=xla:0/2 Epoch=1 Step=2300 Loss=0.00135 Rate=1777.72 GlobalRate=1032.05 Time=00:38:55
| Training Device=xla:0/1 Epoch=1 Step=2300 Loss=0.00135 Rate=1777.73 GlobalRate=1037.66 Time=00:38:55
| Training Device=xla:0/0 Epoch=1 Step=2300 Loss=0.00135 Rate=1777.66 GlobalRate=1037.37 Time=00:38:55
| Training Device=xla:0/3 Epoch=1 Step=2300 Loss=0.00135 Rate=1777.78 GlobalRate=1035.80 Time=00:38:55
| Training Device=xla:0/3 Epoch=1 Step=2320 Loss=0.00135 Rate=1775.43 GlobalRate=1039.53 Time=00:38:57
| Training Device=xla:0/2 Epoch=1 Step=2320 Loss=0.00135 Rate=1775.29 GlobalRate=1035.78 Time=00:38:57
| Training Device=xla:0/0 Epoch=1 Step=2320 Loss=0.00135 Rate=1775.45 GlobalRate=1041.10 Time=00:38:57
| Training Device=xla:0/1 Epoch=1 Step=2320 Loss=0.00135 Rate=1775.25 GlobalRate=1041.38 Time=00:38:57
| Training Device=xla:0/3 Epoch=1 Step=2340 Loss=0.00135 Rate=1777.48 GlobalRate=1043.23 Time=00:38:58
| Training Device=xla:0/1 Epoch=1 Step=2340 Loss=0.00135 Rate=1777.50 GlobalRate=1045.08 Time=00:38:58
| Training Device=xla:0/2 Epoch=1 Step=2340 Loss=0.00135 Rate=1777.34 GlobalRate=1039.49 Time=00:38:58
| Training Device=xla:0/0 Epoch=1 Step=2340 Loss=0.00135 Rate=1777.42 GlobalRate=1044.80 Time=00:38:58
Epoch 1 train end 00:38:58
| Test Device=xla:0/3 Step=0 Epoch=1 Time=00:39:05
| Test Device=xla:0/2 Step=0 Epoch=1 Time=00:39:05
| Test Device=xla:0/1 Step=0 Epoch=1 Time=00:39:06
| Test Device=xla:0/0 Step=0 Epoch=1 Time=00:39:06
| Test Device=xla:0/1 Step=20 Epoch=1 Time=00:39:13
| Test Device=xla:0/2 Step=20 Epoch=1 Time=00:39:13
| Test Device=xla:0/3 Step=20 Epoch=1 Time=00:39:13
| Test Device=xla:0/1 Step=40 Epoch=1 Time=00:39:13
| Test Device=xla:0/2 Step=40 Epoch=1 Time=00:39:13
| Test Device=xla:0/3 Step=40 Epoch=1 Time=00:39:13
| Test Device=xla:0/1 Step=60 Epoch=1 Time=00:39:13
| Test Device=xla:0/2 Step=60 Epoch=1 Time=00:39:13
| Test Device=xla:0/3 Step=60 Epoch=1 Time=00:39:13
| Test Device=xla:0/1 Step=80 Epoch=1 Time=00:39:14
| Test Device=xla:0/2 Step=80 Epoch=1 Time=00:39:14
| Test Device=xla:0/3 Step=80 Epoch=1 Time=00:39:14
| Test Device=xla:0/0 Step=20 Epoch=1 Time=00:39:14
| Test Device=xla:0/0 Step=40 Epoch=1 Time=00:39:14
| Test Device=xla:0/0 Step=60 Epoch=1 Time=00:39:15
| Test Device=xla:0/0 Step=80 Epoch=1 Time=00:39:15
Epoch 1 test end 00:39:15, Accuracy=100.00
Max Accuracy: 100.00%

so before update openxla-pin, we used 5:08, and after openxla-pin update, we used 5:04, so there is no appear time regression,

@ManfeiBai ManfeiBai mentioned this pull request Dec 5, 2023
Closed
@ManfeiBai ManfeiBai merged commit 729bae5 into master Dec 5, 2023
19 checks passed
@ManfeiBai ManfeiBai mentioned this pull request Dec 5, 2023
Merged
This was referenced Dec 5, 2023
Closed
Merged
chunnienc pushed a commit to chunnienc/xla that referenced this pull request Dec 14, 2023
@ManfeiBai @chunnienc
Update OpenXLA-pin to Nov24 (pytorch#6012)
e888ada
golechwierowicz pushed a commit that referenced this pull request Jan 12, 2024
@ManfeiBai @golechwierowicz
Update OpenXLA-pin to Nov24 (#6012)
5a66cae
bhavya01 pushed a commit that referenced this pull request Apr 22, 2024
@ManfeiBai @bhavya01
Update OpenXLA-pin to Nov24 (#6012)
60eb358
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JackCaoG JackCaoG JackCaoG approved these changes

@vanbasten23 vanbasten23 Awaiting requested review from vanbasten23

Assignees
No one assigned
Labels
backport_2.2
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ManfeiBai @JackCaoG @vanbasten23