Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update libtpu version to Nov24 #6026

Merged
merged 1 commit into from
Dec 5, 2023
Merged

update libtpu version to Nov24 #6026

merged 1 commit into from
Dec 5, 2023

Conversation

ManfeiBai
Copy link
Collaborator

@ManfeiBai ManfeiBai commented Dec 5, 2023

follow up PR for #6012 to fix unstable TPUCI

@ManfeiBai ManfeiBai changed the title [test] TPU CI update libtpu version to Nov24 Dec 5, 2023
@ManfeiBai ManfeiBai requested a review from lsy323 December 5, 2023 06:37
@ManfeiBai ManfeiBai marked this pull request as ready for review December 5, 2023 06:37
@ManfeiBai
Copy link
Collaborator Author

ManfeiBai commented Dec 5, 2023

tested resnet locally with libtpu at Nov24:

PJRT_DEVICE=TPU python test/test_train_mp_imagenet.py --fake_data --num_epochs=1
==> Preparing data..
==> Preparing data..
==> Preparing data..
==> Preparing data..
Epoch 1 train begin 08:03:46
| Training Device=xla:0/0 Epoch=1 Step=0 Loss=6.89059 Rate=2.26 GlobalRate=2.26 Time=08:04:42
| Training Device=xla:0/2 Epoch=1 Step=0 Loss=6.89059 Rate=1.79 GlobalRate=1.79 Time=08:04:42
| Training Device=xla:0/3 Epoch=1 Step=0 Loss=6.89059 Rate=1.76 GlobalRate=1.76 Time=08:04:42
| Training Device=xla:0/1 Epoch=1 Step=0 Loss=6.89059 Rate=1.76 GlobalRate=1.76 Time=08:04:42
| Training Device=xla:0/0 Epoch=1 Step=20 Loss=6.50001 Rate=26.67 GlobalRate=23.14 Time=08:05:42
| Training Device=xla:0/2 Epoch=1 Step=20 Loss=6.50001 Rate=26.48 GlobalRate=20.50 Time=08:05:42
| Training Device=xla:0/1 Epoch=1 Step=20 Loss=6.50001 Rate=26.47 GlobalRate=20.31 Time=08:05:42
| Training Device=xla:0/3 Epoch=1 Step=20 Loss=6.50001 Rate=26.47 GlobalRate=20.31 Time=08:05:42
| Training Device=xla:0/1 Epoch=1 Step=40 Loss=5.07708 Rate=1072.12 GlobalRate=39.23 Time=08:05:43
| Training Device=xla:0/2 Epoch=1 Step=40 Loss=5.07708 Rate=1071.83 GlobalRate=39.60 Time=08:05:43
| Training Device=xla:0/3 Epoch=1 Step=40 Loss=5.07708 Rate=1071.92 GlobalRate=39.22 Time=08:05:43
| Training Device=xla:0/0 Epoch=1 Step=40 Loss=5.07708 Rate=1071.81 GlobalRate=44.63 Time=08:05:43
| Training Device=xla:0/1 Epoch=1 Step=60 Loss=2.73656 Rate=1493.63 GlobalRate=57.74 Time=08:05:45
| Training Device=xla:0/2 Epoch=1 Step=60 Loss=2.73656 Rate=1493.63 GlobalRate=58.28 Time=08:05:45
| Training Device=xla:0/3 Epoch=1 Step=60 Loss=2.73656 Rate=1493.59 GlobalRate=57.73 Time=08:05:45
| Training Device=xla:0/0 Epoch=1 Step=60 Loss=2.73656 Rate=1493.65 GlobalRate=65.59 Time=08:05:45
| Training Device=xla:0/2 Epoch=1 Step=80 Loss=0.57112 Rate=1661.20 GlobalRate=76.56 Time=08:05:46
| Training Device=xla:0/3 Epoch=1 Step=80 Loss=0.57112 Rate=1661.24 GlobalRate=75.85 Time=08:05:46
| Training Device=xla:0/0 Epoch=1 Step=80 Loss=0.57112 Rate=1661.17 GlobalRate=86.05 Time=08:05:46
| Training Device=xla:0/1 Epoch=1 Step=80 Loss=0.57112 Rate=1661.25 GlobalRate=75.86 Time=08:05:46
| Training Device=xla:0/0 Epoch=1 Step=100 Loss=0.11328 Rate=1728.56 GlobalRate=106.03 Time=08:05:48
| Training Device=xla:0/2 Epoch=1 Step=100 Loss=0.11328 Rate=1728.52 GlobalRate=94.45 Time=08:05:48
| Training Device=xla:0/1 Epoch=1 Step=100 Loss=0.11328 Rate=1728.29 GlobalRate=93.61 Time=08:05:48
| Training Device=xla:0/3 Epoch=1 Step=100 Loss=0.11328 Rate=1728.14 GlobalRate=93.59 Time=08:05:48
| Training Device=xla:0/3 Epoch=1 Step=120 Loss=0.05495 Rate=1753.34 GlobalRate=110.97 Time=08:05:49
| Training Device=xla:0/2 Epoch=1 Step=120 Loss=0.05495 Rate=1753.07 GlobalRate=111.97 Time=08:05:49
| Training Device=xla:0/1 Epoch=1 Step=120 Loss=0.05495 Rate=1753.03 GlobalRate=110.98 Time=08:05:49
| Training Device=xla:0/0 Epoch=1 Step=120 Loss=0.05495 Rate=1753.07 GlobalRate=125.54 Time=08:05:49
| Training Device=xla:0/1 Epoch=1 Step=140 Loss=0.03776 Rate=1766.71 GlobalRate=128.00 Time=08:05:50
| Training Device=xla:0/0 Epoch=1 Step=140 Loss=0.03776 Rate=1766.60 GlobalRate=144.60 Time=08:05:50
| Training Device=xla:0/2 Epoch=1 Step=140 Loss=0.03776 Rate=1766.57 GlobalRate=129.14 Time=08:05:50
| Training Device=xla:0/3 Epoch=1 Step=140 Loss=0.03776 Rate=1766.51 GlobalRate=127.99 Time=08:05:50
| Training Device=xla:0/2 Epoch=1 Step=160 Loss=0.02887 Rate=1764.45 GlobalRate=145.94 Time=08:05:52
| Training Device=xla:0/0 Epoch=1 Step=160 Loss=0.02887 Rate=1764.59 GlobalRate=163.21 Time=08:05:52
| Training Device=xla:0/1 Epoch=1 Step=160 Loss=0.02887 Rate=1764.29 GlobalRate=144.67 Time=08:05:52
| Training Device=xla:0/3 Epoch=1 Step=160 Loss=0.02887 Rate=1764.57 GlobalRate=144.65 Time=08:05:52
| Training Device=xla:0/3 Epoch=1 Step=180 Loss=0.02308 Rate=1759.32 GlobalRate=160.97 Time=08:05:53
| Training Device=xla:0/2 Epoch=1 Step=180 Loss=0.02308 Rate=1759.26 GlobalRate=162.39 Time=08:05:53
| Training Device=xla:0/0 Epoch=1 Step=180 Loss=0.02308 Rate=1758.84 GlobalRate=181.39 Time=08:05:53
| Training Device=xla:0/1 Epoch=1 Step=180 Loss=0.02308 Rate=1758.77 GlobalRate=160.99 Time=08:05:53
| Training Device=xla:0/2 Epoch=1 Step=200 Loss=0.01895 Rate=1765.48 GlobalRate=178.52 Time=08:05:55
| Training Device=xla:0/3 Epoch=1 Step=200 Loss=0.01895 Rate=1765.50 GlobalRate=176.98 Time=08:05:55
| Training Device=xla:0/1 Epoch=1 Step=200 Loss=0.01895 Rate=1765.90 GlobalRate=177.00 Time=08:05:55
| Training Device=xla:0/0 Epoch=1 Step=200 Loss=0.01895 Rate=1765.67 GlobalRate=199.17 Time=08:05:55
| Training Device=xla:0/3 Epoch=1 Step=220 Loss=0.01588 Rate=1770.35 GlobalRate=192.68 Time=08:05:56
| Training Device=xla:0/2 Epoch=1 Step=220 Loss=0.01588 Rate=1770.35 GlobalRate=194.34 Time=08:05:56
| Training Device=xla:0/0 Epoch=1 Step=220 Loss=0.01588 Rate=1770.39 GlobalRate=216.57 Time=08:05:56
| Training Device=xla:0/1 Epoch=1 Step=220 Loss=0.01588 Rate=1770.27 GlobalRate=192.70 Time=08:05:56
| Training Device=xla:0/1 Epoch=1 Step=240 Loss=0.01353 Rate=1771.48 GlobalRate=208.09 Time=08:05:58
| Training Device=xla:0/0 Epoch=1 Step=240 Loss=0.01353 Rate=1771.18 GlobalRate=233.59 Time=08:05:58
| Training Device=xla:0/3 Epoch=1 Step=240 Loss=0.01353 Rate=1771.19 GlobalRate=208.07 Time=08:05:58
| Training Device=xla:0/2 Epoch=1 Step=240 Loss=0.01353 Rate=1771.19 GlobalRate=209.84 Time=08:05:58
| Training Device=xla:0/0 Epoch=1 Step=260 Loss=0.01169 Rate=1771.74 GlobalRate=250.23 Time=08:05:59
| Training Device=xla:0/2 Epoch=1 Step=260 Loss=0.01169 Rate=1771.68 GlobalRate=225.05 Time=08:05:59
| Training Device=xla:0/1 Epoch=1 Step=260 Loss=0.01169 Rate=1771.74 GlobalRate=223.18 Time=08:05:59
| Training Device=xla:0/3 Epoch=1 Step=260 Loss=0.01169 Rate=1771.66 GlobalRate=223.16 Time=08:05:59
| Training Device=xla:0/2 Epoch=1 Step=280 Loss=0.01022 Rate=1772.53 GlobalRate=239.96 Time=08:06:01
| Training Device=xla:0/3 Epoch=1 Step=280 Loss=0.01022 Rate=1772.40 GlobalRate=237.96 Time=08:06:01
| Training Device=xla:0/0 Epoch=1 Step=280 Loss=0.01022 Rate=1772.52 GlobalRate=266.53 Time=08:06:01
| Training Device=xla:0/1 Epoch=1 Step=280 Loss=0.01022 Rate=1772.32 GlobalRate=237.99 Time=08:06:01
| Training Device=xla:0/0 Epoch=1 Step=300 Loss=0.00902 Rate=1771.67 GlobalRate=282.47 Time=08:06:02
| Training Device=xla:0/3 Epoch=1 Step=300 Loss=0.00902 Rate=1771.77 GlobalRate=252.49 Time=08:06:02
| Training Device=xla:0/2 Epoch=1 Step=300 Loss=0.00902 Rate=1771.55 GlobalRate=254.58 Time=08:06:02
| Training Device=xla:0/1 Epoch=1 Step=300 Loss=0.00902 Rate=1771.31 GlobalRate=252.51 Time=08:06:02
| Training Device=xla:0/1 Epoch=1 Step=320 Loss=0.00804 Rate=1769.13 GlobalRate=266.76 Time=08:06:04
| Training Device=xla:0/2 Epoch=1 Step=320 Loss=0.00804 Rate=1768.85 GlobalRate=268.92 Time=08:06:04
| Training Device=xla:0/0 Epoch=1 Step=320 Loss=0.00804 Rate=1768.74 GlobalRate=298.07 Time=08:06:04
| Training Device=xla:0/3 Epoch=1 Step=320 Loss=0.00804 Rate=1768.67 GlobalRate=266.73 Time=08:06:04
| Training Device=xla:0/3 Epoch=1 Step=340 Loss=0.00722 Rate=1765.82 GlobalRate=280.70 Time=08:06:05
| Training Device=xla:0/1 Epoch=1 Step=340 Loss=0.00722 Rate=1765.87 GlobalRate=280.73 Time=08:06:05
| Training Device=xla:0/0 Epoch=1 Step=340 Loss=0.00722 Rate=1765.69 GlobalRate=313.35 Time=08:06:05
| Training Device=xla:0/2 Epoch=1 Step=340 Loss=0.00722 Rate=1765.18 GlobalRate=282.99 Time=08:06:05
| Training Device=xla:0/3 Epoch=1 Step=360 Loss=0.00654 Rate=1759.90 GlobalRate=294.41 Time=08:06:06
| Training Device=xla:0/1 Epoch=1 Step=360 Loss=0.00654 Rate=1759.90 GlobalRate=294.44 Time=08:06:06
| Training Device=xla:0/2 Epoch=1 Step=360 Loss=0.00654 Rate=1760.06 GlobalRate=296.78 Time=08:06:06
| Training Device=xla:0/0 Epoch=1 Step=360 Loss=0.00654 Rate=1759.86 GlobalRate=328.29 Time=08:06:06
| Training Device=xla:0/3 Epoch=1 Step=380 Loss=0.00595 Rate=1766.96 GlobalRate=307.88 Time=08:06:08
| Training Device=xla:0/2 Epoch=1 Step=380 Loss=0.00595 Rate=1767.18 GlobalRate=310.34 Time=08:06:08
| Training Device=xla:0/1 Epoch=1 Step=380 Loss=0.00595 Rate=1766.94 GlobalRate=307.92 Time=08:06:08
| Training Device=xla:0/0 Epoch=1 Step=380 Loss=0.00595 Rate=1766.89 GlobalRate=342.96 Time=08:06:08
| Training Device=xla:0/2 Epoch=1 Step=400 Loss=0.00545 Rate=1766.44 GlobalRate=323.65 Time=08:06:09
| Training Device=xla:0/3 Epoch=1 Step=400 Loss=0.00545 Rate=1766.37 GlobalRate=321.11 Time=08:06:09
| Training Device=xla:0/1 Epoch=1 Step=400 Loss=0.00545 Rate=1766.31 GlobalRate=321.14 Time=08:06:09
| Training Device=xla:0/0 Epoch=1 Step=400 Loss=0.00545 Rate=1766.30 GlobalRate=357.32 Time=08:06:09
| Training Device=xla:0/1 Epoch=1 Step=420 Loss=0.00502 Rate=1762.65 GlobalRate=334.12 Time=08:06:11
| Training Device=xla:0/2 Epoch=1 Step=420 Loss=0.00502 Rate=1762.64 GlobalRate=336.70 Time=08:06:11
| Training Device=xla:0/3 Epoch=1 Step=420 Loss=0.00502 Rate=1762.37 GlobalRate=334.08 Time=08:06:11
| Training Device=xla:0/0 Epoch=1 Step=420 Loss=0.00502 Rate=1762.61 GlobalRate=371.38 Time=08:06:11
| Training Device=xla:0/1 Epoch=1 Step=440 Loss=0.00465 Rate=1767.14 GlobalRate=346.88 Time=08:06:12
| Training Device=xla:0/3 Epoch=1 Step=440 Loss=0.00465 Rate=1767.14 GlobalRate=346.84 Time=08:06:12
| Training Device=xla:0/2 Epoch=1 Step=440 Loss=0.00465 Rate=1767.00 GlobalRate=349.54 Time=08:06:12
| Training Device=xla:0/0 Epoch=1 Step=440 Loss=0.00465 Rate=1767.07 GlobalRate=385.18 Time=08:06:12
| Training Device=xla:0/1 Epoch=1 Step=460 Loss=0.00432 Rate=1764.77 GlobalRate=359.40 Time=08:06:14
| Training Device=xla:0/3 Epoch=1 Step=460 Loss=0.00432 Rate=1764.82 GlobalRate=359.37 Time=08:06:14
| Training Device=xla:0/2 Epoch=1 Step=460 Loss=0.00432 Rate=1764.71 GlobalRate=362.14 Time=08:06:14
| Training Device=xla:0/0 Epoch=1 Step=460 Loss=0.00432 Rate=1764.65 GlobalRate=398.70 Time=08:06:14
| Training Device=xla:0/0 Epoch=1 Step=480 Loss=0.00403 Rate=1770.26 GlobalRate=411.98 Time=08:06:15
| Training Device=xla:0/2 Epoch=1 Step=480 Loss=0.00403 Rate=1770.13 GlobalRate=374.53 Time=08:06:15
| Training Device=xla:0/3 Epoch=1 Step=480 Loss=0.00403 Rate=1770.07 GlobalRate=371.69 Time=08:06:15
| Training Device=xla:0/1 Epoch=1 Step=480 Loss=0.00403 Rate=1769.92 GlobalRate=371.73 Time=08:06:15
| Training Device=xla:0/1 Epoch=1 Step=500 Loss=0.00378 Rate=1772.44 GlobalRate=383.84 Time=08:06:17
| Training Device=xla:0/2 Epoch=1 Step=500 Loss=0.00378 Rate=1772.36 GlobalRate=386.71 Time=08:06:17
| Training Device=xla:0/0 Epoch=1 Step=500 Loss=0.00378 Rate=1772.44 GlobalRate=425.00 Time=08:06:17
| Training Device=xla:0/3 Epoch=1 Step=500 Loss=0.00378 Rate=1772.39 GlobalRate=383.80 Time=08:06:17
| Training Device=xla:0/2 Epoch=1 Step=520 Loss=0.00356 Rate=1771.76 GlobalRate=398.67 Time=08:06:18
| Training Device=xla:0/3 Epoch=1 Step=520 Loss=0.00356 Rate=1771.76 GlobalRate=395.70 Time=08:06:18
| Training Device=xla:0/0 Epoch=1 Step=520 Loss=0.00356 Rate=1771.75 GlobalRate=437.78 Time=08:06:18
| Training Device=xla:0/1 Epoch=1 Step=520 Loss=0.00356 Rate=1771.65 GlobalRate=395.74 Time=08:06:18
| Training Device=xla:0/0 Epoch=1 Step=540 Loss=0.00336 Rate=1772.93 GlobalRate=450.32 Time=08:06:19
| Training Device=xla:0/3 Epoch=1 Step=540 Loss=0.00336 Rate=1772.92 GlobalRate=407.40 Time=08:06:19
| Training Device=xla:0/1 Epoch=1 Step=540 Loss=0.00336 Rate=1772.93 GlobalRate=407.44 Time=08:06:19
| Training Device=xla:0/2 Epoch=1 Step=540 Loss=0.00336 Rate=1772.78 GlobalRate=410.43 Time=08:06:19
| Training Device=xla:0/3 Epoch=1 Step=560 Loss=0.00318 Rate=1773.94 GlobalRate=418.91 Time=08:06:21
| Training Device=xla:0/0 Epoch=1 Step=560 Loss=0.00318 Rate=1773.85 GlobalRate=462.62 Time=08:06:21
| Training Device=xla:0/2 Epoch=1 Step=560 Loss=0.00318 Rate=1773.94 GlobalRate=422.00 Time=08:06:21
| Training Device=xla:0/1 Epoch=1 Step=560 Loss=0.00318 Rate=1773.95 GlobalRate=418.95 Time=08:06:21
| Training Device=xla:0/1 Epoch=1 Step=580 Loss=0.00302 Rate=1767.37 GlobalRate=430.24 Time=08:06:22
| Training Device=xla:0/2 Epoch=1 Step=580 Loss=0.00302 Rate=1767.40 GlobalRate=433.34 Time=08:06:22
| Training Device=xla:0/0 Epoch=1 Step=580 Loss=0.00302 Rate=1767.40 GlobalRate=474.68 Time=08:06:22
| Training Device=xla:0/3 Epoch=1 Step=580 Loss=0.00302 Rate=1767.33 GlobalRate=430.20 Time=08:06:22
| Training Device=xla:0/2 Epoch=1 Step=600 Loss=0.00287 Rate=1769.20 GlobalRate=444.52 Time=08:06:24
| Training Device=xla:0/3 Epoch=1 Step=600 Loss=0.00287 Rate=1768.98 GlobalRate=441.31 Time=08:06:24
| Training Device=xla:0/0 Epoch=1 Step=600 Loss=0.00287 Rate=1769.14 GlobalRate=486.52 Time=08:06:24
| Training Device=xla:0/1 Epoch=1 Step=600 Loss=0.00287 Rate=1769.16 GlobalRate=441.36 Time=08:06:24
| Training Device=xla:0/0 Epoch=1 Step=620 Loss=0.00274 Rate=1768.57 GlobalRate=498.15 Time=08:06:25
| Training Device=xla:0/2 Epoch=1 Step=620 Loss=0.00274 Rate=1768.58 GlobalRate=455.50 Time=08:06:25
| Training Device=xla:0/3 Epoch=1 Step=620 Loss=0.00274 Rate=1768.59 GlobalRate=452.24 Time=08:06:25
| Training Device=xla:0/1 Epoch=1 Step=620 Loss=0.00274 Rate=1768.57 GlobalRate=452.29 Time=08:06:25
| Training Device=xla:0/2 Epoch=1 Step=640 Loss=0.00263 Rate=1772.24 GlobalRate=466.31 Time=08:06:27
| Training Device=xla:0/3 Epoch=1 Step=640 Loss=0.00263 Rate=1772.30 GlobalRate=463.01 Time=08:06:27
| Training Device=xla:0/0 Epoch=1 Step=640 Loss=0.00263 Rate=1772.23 GlobalRate=509.59 Time=08:06:27
| Training Device=xla:0/1 Epoch=1 Step=640 Loss=0.00263 Rate=1772.22 GlobalRate=463.05 Time=08:06:27
| Training Device=xla:0/2 Epoch=1 Step=660 Loss=0.00252 Rate=1773.09 GlobalRate=476.95 Time=08:06:28
| Training Device=xla:0/1 Epoch=1 Step=660 Loss=0.00252 Rate=1773.12 GlobalRate=473.64 Time=08:06:28
| Training Device=xla:0/0 Epoch=1 Step=660 Loss=0.00252 Rate=1773.10 GlobalRate=520.82 Time=08:06:28
| Training Device=xla:0/3 Epoch=1 Step=660 Loss=0.00252 Rate=1773.14 GlobalRate=473.60 Time=08:06:28
| Training Device=xla:0/1 Epoch=1 Step=680 Loss=0.00242 Rate=1773.83 GlobalRate=484.06 Time=08:06:30
| Training Device=xla:0/3 Epoch=1 Step=680 Loss=0.00242 Rate=1773.85 GlobalRate=484.02 Time=08:06:30
| Training Device=xla:0/2 Epoch=1 Step=680 Loss=0.00242 Rate=1773.78 GlobalRate=487.42 Time=08:06:30
| Training Device=xla:0/0 Epoch=1 Step=680 Loss=0.00242 Rate=1773.65 GlobalRate=531.86 Time=08:06:30
| Training Device=xla:0/1 Epoch=1 Step=700 Loss=0.00234 Rate=1743.66 GlobalRate=494.20 Time=08:06:31
| Training Device=xla:0/2 Epoch=1 Step=700 Loss=0.00234 Rate=1743.64 GlobalRate=497.60 Time=08:06:31
| Training Device=xla:0/0 Epoch=1 Step=700 Loss=0.00234 Rate=1743.73 GlobalRate=542.56 Time=08:06:31
| Training Device=xla:0/3 Epoch=1 Step=700 Loss=0.00234 Rate=1743.57 GlobalRate=494.16 Time=08:06:31
| Training Device=xla:0/3 Epoch=1 Step=720 Loss=0.00226 Rate=1755.83 GlobalRate=504.23 Time=08:06:32
| Training Device=xla:0/0 Epoch=1 Step=720 Loss=0.00226 Rate=1755.68 GlobalRate=553.18 Time=08:06:32
| Training Device=xla:0/1 Epoch=1 Step=720 Loss=0.00226 Rate=1755.64 GlobalRate=504.27 Time=08:06:32
| Training Device=xla:0/2 Epoch=1 Step=720 Loss=0.00226 Rate=1755.77 GlobalRate=507.71 Time=08:06:32
| Training Device=xla:0/3 Epoch=1 Step=740 Loss=0.00218 Rate=1765.22 GlobalRate=514.15 Time=08:06:34
| Training Device=xla:0/2 Epoch=1 Step=740 Loss=0.00218 Rate=1765.08 GlobalRate=517.68 Time=08:06:34
| Training Device=xla:0/0 Epoch=1 Step=740 Loss=0.00218 Rate=1765.31 GlobalRate=563.65 Time=08:06:34
| Training Device=xla:0/1 Epoch=1 Step=740 Loss=0.00218 Rate=1765.23 GlobalRate=514.20 Time=08:06:34
| Training Device=xla:0/3 Epoch=1 Step=760 Loss=0.00211 Rate=1767.33 GlobalRate=523.92 Time=08:06:35
| Training Device=xla:0/2 Epoch=1 Step=760 Loss=0.00211 Rate=1767.35 GlobalRate=527.48 Time=08:06:35
| Training Device=xla:0/1 Epoch=1 Step=760 Loss=0.00211 Rate=1767.26 GlobalRate=523.97 Time=08:06:35
| Training Device=xla:0/0 Epoch=1 Step=760 Loss=0.00211 Rate=1767.19 GlobalRate=573.92 Time=08:06:35
| Training Device=xla:0/3 Epoch=1 Step=780 Loss=0.00205 Rate=1771.22 GlobalRate=533.55 Time=08:06:37
| Training Device=xla:0/2 Epoch=1 Step=780 Loss=0.00205 Rate=1771.26 GlobalRate=537.15 Time=08:06:37
| Training Device=xla:0/1 Epoch=1 Step=780 Loss=0.00205 Rate=1771.29 GlobalRate=533.60 Time=08:06:37
| Training Device=xla:0/0 Epoch=1 Step=780 Loss=0.00205 Rate=1771.26 GlobalRate=584.04 Time=08:06:37
| Training Device=xla:0/1 Epoch=1 Step=800 Loss=0.00200 Rate=1773.22 GlobalRate=543.08 Time=08:06:38
| Training Device=xla:0/2 Epoch=1 Step=800 Loss=0.00200 Rate=1773.25 GlobalRate=546.67 Time=08:06:38
| Training Device=xla:0/3 Epoch=1 Step=800 Loss=0.00200 Rate=1773.17 GlobalRate=543.03 Time=08:06:38
| Training Device=xla:0/0 Epoch=1 Step=800 Loss=0.00200 Rate=1773.20 GlobalRate=593.99 Time=08:06:38
| Training Device=xla:0/0 Epoch=1 Step=820 Loss=0.00194 Rate=1774.13 GlobalRate=603.78 Time=08:06:40
| Training Device=xla:0/1 Epoch=1 Step=820 Loss=0.00194 Rate=1774.07 GlobalRate=552.42 Time=08:06:40
| Training Device=xla:0/3 Epoch=1 Step=820 Loss=0.00194 Rate=1773.92 GlobalRate=552.37 Time=08:06:40
| Training Device=xla:0/2 Epoch=1 Step=820 Loss=0.00194 Rate=1774.06 GlobalRate=556.04 Time=08:06:40
| Training Device=xla:0/0 Epoch=1 Step=840 Loss=0.00190 Rate=1775.85 GlobalRate=613.41 Time=08:06:41
| Training Device=xla:0/3 Epoch=1 Step=840 Loss=0.00190 Rate=1775.76 GlobalRate=561.57 Time=08:06:41
| Training Device=xla:0/2 Epoch=1 Step=840 Loss=0.00190 Rate=1775.81 GlobalRate=565.28 Time=08:06:41
| Training Device=xla:0/1 Epoch=1 Step=840 Loss=0.00190 Rate=1775.69 GlobalRate=561.62 Time=08:06:41
| Training Device=xla:0/0 Epoch=1 Step=860 Loss=0.00185 Rate=1776.45 GlobalRate=622.88 Time=08:06:43
| Training Device=xla:0/2 Epoch=1 Step=860 Loss=0.00185 Rate=1776.42 GlobalRate=574.37 Time=08:06:43
| Training Device=xla:0/3 Epoch=1 Step=860 Loss=0.00185 Rate=1776.36 GlobalRate=570.64 Time=08:06:43
| Training Device=xla:0/1 Epoch=1 Step=860 Loss=0.00185 Rate=1776.43 GlobalRate=570.69 Time=08:06:43
| Training Device=xla:0/2 Epoch=1 Step=880 Loss=0.00181 Rate=1776.62 GlobalRate=583.33 Time=08:06:44
| Training Device=xla:0/1 Epoch=1 Step=880 Loss=0.00181 Rate=1776.74 GlobalRate=579.62 Time=08:06:44
| Training Device=xla:0/3 Epoch=1 Step=880 Loss=0.00181 Rate=1776.64 GlobalRate=579.57 Time=08:06:44
| Training Device=xla:0/0 Epoch=1 Step=880 Loss=0.00181 Rate=1776.61 GlobalRate=632.20 Time=08:06:44
| Training Device=xla:0/1 Epoch=1 Step=900 Loss=0.00178 Rate=1774.71 GlobalRate=588.41 Time=08:06:45
| Training Device=xla:0/2 Epoch=1 Step=900 Loss=0.00178 Rate=1774.61 GlobalRate=592.15 Time=08:06:45
| Training Device=xla:0/3 Epoch=1 Step=900 Loss=0.00178 Rate=1774.77 GlobalRate=588.36 Time=08:06:45
| Training Device=xla:0/0 Epoch=1 Step=900 Loss=0.00178 Rate=1774.71 GlobalRate=641.36 Time=08:06:45
| Training Device=xla:0/1 Epoch=1 Step=920 Loss=0.00174 Rate=1774.80 GlobalRate=597.08 Time=08:06:47
| Training Device=xla:0/3 Epoch=1 Step=920 Loss=0.00174 Rate=1774.82 GlobalRate=597.03 Time=08:06:47
| Training Device=xla:0/2 Epoch=1 Step=920 Loss=0.00174 Rate=1774.81 GlobalRate=600.85 Time=08:06:47
| Training Device=xla:0/0 Epoch=1 Step=920 Loss=0.00174 Rate=1774.66 GlobalRate=650.38 Time=08:06:47
| Training Device=xla:0/2 Epoch=1 Step=940 Loss=0.00171 Rate=1766.64 GlobalRate=609.38 Time=08:06:48
| Training Device=xla:0/1 Epoch=1 Step=940 Loss=0.00171 Rate=1766.64 GlobalRate=605.59 Time=08:06:48
| Training Device=xla:0/0 Epoch=1 Step=940 Loss=0.00171 Rate=1766.63 GlobalRate=659.22 Time=08:06:48
| Training Device=xla:0/3 Epoch=1 Step=940 Loss=0.00171 Rate=1766.48 GlobalRate=605.54 Time=08:06:48
| Training Device=xla:0/1 Epoch=1 Step=960 Loss=0.00168 Rate=1773.40 GlobalRate=614.01 Time=08:06:50
| Training Device=xla:0/0 Epoch=1 Step=960 Loss=0.00168 Rate=1773.48 GlobalRate=667.97 Time=08:06:50
| Training Device=xla:0/2 Epoch=1 Step=960 Loss=0.00168 Rate=1773.41 GlobalRate=617.83 Time=08:06:50
| Training Device=xla:0/3 Epoch=1 Step=960 Loss=0.00168 Rate=1773.39 GlobalRate=613.96 Time=08:06:50
| Training Device=xla:0/0 Epoch=1 Step=980 Loss=0.00165 Rate=1768.91 GlobalRate=676.54 Time=08:06:51
| Training Device=xla:0/1 Epoch=1 Step=980 Loss=0.00165 Rate=1768.89 GlobalRate=622.29 Time=08:06:51
| Training Device=xla:0/2 Epoch=1 Step=980 Loss=0.00165 Rate=1768.86 GlobalRate=626.13 Time=08:06:51
| Training Device=xla:0/3 Epoch=1 Step=980 Loss=0.00165 Rate=1768.98 GlobalRate=622.24 Time=08:06:51
| Training Device=xla:0/2 Epoch=1 Step=1000 Loss=0.00163 Rate=1774.20 GlobalRate=634.34 Time=08:06:53
| Training Device=xla:0/1 Epoch=1 Step=1000 Loss=0.00163 Rate=1774.20 GlobalRate=630.48 Time=08:06:53
| Training Device=xla:0/3 Epoch=1 Step=1000 Loss=0.00163 Rate=1774.26 GlobalRate=630.42 Time=08:06:53
| Training Device=xla:0/0 Epoch=1 Step=1000 Loss=0.00163 Rate=1774.21 GlobalRate=685.02 Time=08:06:53
| Training Device=xla:0/0 Epoch=1 Step=1020 Loss=0.00161 Rate=1774.18 GlobalRate=693.36 Time=08:06:54
| Training Device=xla:0/1 Epoch=1 Step=1020 Loss=0.00161 Rate=1774.18 GlobalRate=638.54 Time=08:06:54
| Training Device=xla:0/2 Epoch=1 Step=1020 Loss=0.00161 Rate=1774.16 GlobalRate=642.43 Time=08:06:54
| Training Device=xla:0/3 Epoch=1 Step=1020 Loss=0.00161 Rate=1774.13 GlobalRate=638.49 Time=08:06:54
| Training Device=xla:0/0 Epoch=1 Step=1040 Loss=0.00158 Rate=1773.91 GlobalRate=701.57 Time=08:06:56
| Training Device=xla:0/3 Epoch=1 Step=1040 Loss=0.00158 Rate=1773.98 GlobalRate=646.44 Time=08:06:56
| Training Device=xla:0/2 Epoch=1 Step=1040 Loss=0.00158 Rate=1773.91 GlobalRate=650.40 Time=08:06:56
| Training Device=xla:0/1 Epoch=1 Step=1040 Loss=0.00158 Rate=1773.47 GlobalRate=646.49 Time=08:06:56
| Training Device=xla:0/3 Epoch=1 Step=1060 Loss=0.00156 Rate=1770.61 GlobalRate=654.26 Time=08:06:57
| Training Device=xla:0/0 Epoch=1 Step=1060 Loss=0.00156 Rate=1770.44 GlobalRate=709.64 Time=08:06:57
| Training Device=xla:0/2 Epoch=1 Step=1060 Loss=0.00156 Rate=1770.51 GlobalRate=658.24 Time=08:06:57
| Training Device=xla:0/1 Epoch=1 Step=1060 Loss=0.00156 Rate=1770.74 GlobalRate=654.31 Time=08:06:57
| Training Device=xla:0/2 Epoch=1 Step=1080 Loss=0.00155 Rate=1762.76 GlobalRate=665.95 Time=08:06:58
| Training Device=xla:0/1 Epoch=1 Step=1080 Loss=0.00155 Rate=1762.84 GlobalRate=662.00 Time=08:06:58
| Training Device=xla:0/0 Epoch=1 Step=1080 Loss=0.00155 Rate=1762.77 GlobalRate=717.55 Time=08:06:58
| Training Device=xla:0/3 Epoch=1 Step=1080 Loss=0.00155 Rate=1762.71 GlobalRate=661.95 Time=08:06:58
| Training Device=xla:0/2 Epoch=1 Step=1100 Loss=0.00153 Rate=1767.12 GlobalRate=673.58 Time=08:07:00
| Training Device=xla:0/0 Epoch=1 Step=1100 Loss=0.00153 Rate=1767.16 GlobalRate=725.39 Time=08:07:00
| Training Device=xla:0/3 Epoch=1 Step=1100 Loss=0.00153 Rate=1767.12 GlobalRate=669.56 Time=08:07:00
| Training Device=xla:0/1 Epoch=1 Step=1100 Loss=0.00153 Rate=1767.16 GlobalRate=669.62 Time=08:07:00
| Training Device=xla:0/3 Epoch=1 Step=1120 Loss=0.00152 Rate=1771.85 GlobalRate=677.09 Time=08:07:01
| Training Device=xla:0/2 Epoch=1 Step=1120 Loss=0.00152 Rate=1771.90 GlobalRate=681.12 Time=08:07:01
| Training Device=xla:0/1 Epoch=1 Step=1120 Loss=0.00152 Rate=1771.82 GlobalRate=677.14 Time=08:07:01
| Training Device=xla:0/0 Epoch=1 Step=1120 Loss=0.00152 Rate=1771.81 GlobalRate=733.12 Time=08:07:01
| Training Device=xla:0/2 Epoch=1 Step=1140 Loss=0.00150 Rate=1770.29 GlobalRate=688.54 Time=08:07:03
| Training Device=xla:0/3 Epoch=1 Step=1140 Loss=0.00150 Rate=1770.22 GlobalRate=684.49 Time=08:07:03
| Training Device=xla:0/1 Epoch=1 Step=1140 Loss=0.00150 Rate=1770.19 GlobalRate=684.55 Time=08:07:03
| Training Device=xla:0/0 Epoch=1 Step=1140 Loss=0.00150 Rate=1770.07 GlobalRate=740.72 Time=08:07:03
| Training Device=xla:0/0 Epoch=1 Step=1160 Loss=0.00149 Rate=1770.61 GlobalRate=748.22 Time=08:07:04
| Training Device=xla:0/3 Epoch=1 Step=1160 Loss=0.00149 Rate=1770.47 GlobalRate=691.80 Time=08:07:04
| Training Device=xla:0/1 Epoch=1 Step=1160 Loss=0.00149 Rate=1770.46 GlobalRate=691.86 Time=08:07:04
| Training Device=xla:0/2 Epoch=1 Step=1160 Loss=0.00149 Rate=1770.38 GlobalRate=695.87 Time=08:07:04
| Training Device=xla:0/1 Epoch=1 Step=1180 Loss=0.00147 Rate=1773.27 GlobalRate=699.08 Time=08:07:06
| Training Device=xla:0/3 Epoch=1 Step=1180 Loss=0.00147 Rate=1773.18 GlobalRate=699.03 Time=08:07:06
| Training Device=xla:0/2 Epoch=1 Step=1180 Loss=0.00147 Rate=1773.07 GlobalRate=703.11 Time=08:07:06
| Training Device=xla:0/0 Epoch=1 Step=1180 Loss=0.00147 Rate=1773.05 GlobalRate=755.62 Time=08:07:06
| Training Device=xla:0/0 Epoch=1 Step=1200 Loss=0.00146 Rate=1774.64 GlobalRate=762.92 Time=08:07:07
| Training Device=xla:0/1 Epoch=1 Step=1200 Loss=0.00146 Rate=1774.50 GlobalRate=706.21 Time=08:07:07
| Training Device=xla:0/3 Epoch=1 Step=1200 Loss=0.00146 Rate=1774.44 GlobalRate=706.16 Time=08:07:07
| Training Device=xla:0/2 Epoch=1 Step=1200 Loss=0.00146 Rate=1774.43 GlobalRate=710.25 Time=08:07:07
| Training Device=xla:0/3 Epoch=1 Step=1220 Loss=0.00145 Rate=1774.93 GlobalRate=713.19 Time=08:07:09
| Training Device=xla:0/2 Epoch=1 Step=1220 Loss=0.00145 Rate=1774.99 GlobalRate=717.30 Time=08:07:09
| Training Device=xla:0/0 Epoch=1 Step=1220 Loss=0.00145 Rate=1774.89 GlobalRate=770.12 Time=08:07:09
| Training Device=xla:0/1 Epoch=1 Step=1220 Loss=0.00145 Rate=1774.35 GlobalRate=713.24 Time=08:07:09
| Training Device=xla:0/2 Epoch=1 Step=1240 Loss=0.00144 Rate=1771.98 GlobalRate=724.24 Time=08:07:10
| Training Device=xla:0/0 Epoch=1 Step=1240 Loss=0.00144 Rate=1771.92 GlobalRate=777.19 Time=08:07:10
| Training Device=xla:0/3 Epoch=1 Step=1240 Loss=0.00144 Rate=1771.96 GlobalRate=720.12 Time=08:07:10
| Training Device=xla:0/1 Epoch=1 Step=1240 Loss=0.00144 Rate=1772.19 GlobalRate=720.17 Time=08:07:10
| Training Device=xla:0/3 Epoch=1 Step=1260 Loss=0.00144 Rate=1765.38 GlobalRate=726.93 Time=08:07:12
| Training Device=xla:0/2 Epoch=1 Step=1260 Loss=0.00144 Rate=1765.36 GlobalRate=731.07 Time=08:07:12
| Training Device=xla:0/1 Epoch=1 Step=1260 Loss=0.00144 Rate=1765.47 GlobalRate=726.99 Time=08:07:12
| Training Device=xla:0/0 Epoch=1 Step=1260 Loss=0.00144 Rate=1765.36 GlobalRate=784.14 Time=08:07:12
| Training Device=xla:0/1 Epoch=1 Step=1280 Loss=0.00143 Rate=1766.32 GlobalRate=733.73 Time=08:07:13
| Training Device=xla:0/0 Epoch=1 Step=1280 Loss=0.00143 Rate=1766.25 GlobalRate=791.01 Time=08:07:13
| Training Device=xla:0/3 Epoch=1 Step=1280 Loss=0.00143 Rate=1766.17 GlobalRate=733.68 Time=08:07:13
| Training Device=xla:0/2 Epoch=1 Step=1280 Loss=0.00143 Rate=1766.23 GlobalRate=737.82 Time=08:07:13
| Training Device=xla:0/0 Epoch=1 Step=1300 Loss=0.00142 Rate=1771.92 GlobalRate=797.81 Time=08:07:14
| Training Device=xla:0/3 Epoch=1 Step=1300 Loss=0.00142 Rate=1771.86 GlobalRate=740.35 Time=08:07:14
| Training Device=xla:0/1 Epoch=1 Step=1300 Loss=0.00142 Rate=1771.79 GlobalRate=740.41 Time=08:07:14
| Training Device=xla:0/2 Epoch=1 Step=1300 Loss=0.00142 Rate=1771.76 GlobalRate=744.51 Time=08:07:14
| Training Device=xla:0/3 Epoch=1 Step=1320 Loss=0.00141 Rate=1774.47 GlobalRate=746.95 Time=08:07:16
| Training Device=xla:0/1 Epoch=1 Step=1320 Loss=0.00141 Rate=1774.53 GlobalRate=747.01 Time=08:07:16
| Training Device=xla:0/2 Epoch=1 Step=1320 Loss=0.00141 Rate=1774.55 GlobalRate=751.12 Time=08:07:16
| Training Device=xla:0/0 Epoch=1 Step=1320 Loss=0.00141 Rate=1774.31 GlobalRate=804.52 Time=08:07:16
| Training Device=xla:0/3 Epoch=1 Step=1340 Loss=0.00141 Rate=1775.63 GlobalRate=753.46 Time=08:07:17
| Training Device=xla:0/0 Epoch=1 Step=1340 Loss=0.00141 Rate=1775.61 GlobalRate=811.14 Time=08:07:17
| Training Device=xla:0/2 Epoch=1 Step=1340 Loss=0.00141 Rate=1775.70 GlobalRate=757.64 Time=08:07:17
| Training Device=xla:0/1 Epoch=1 Step=1340 Loss=0.00141 Rate=1775.59 GlobalRate=753.52 Time=08:07:17
| Training Device=xla:0/2 Epoch=1 Step=1360 Loss=0.00140 Rate=1775.19 GlobalRate=764.07 Time=08:07:19
| Training Device=xla:0/0 Epoch=1 Step=1360 Loss=0.00140 Rate=1775.23 GlobalRate=817.66 Time=08:07:19
| Training Device=xla:0/3 Epoch=1 Step=1360 Loss=0.00140 Rate=1775.15 GlobalRate=759.89 Time=08:07:19
| Training Device=xla:0/1 Epoch=1 Step=1360 Loss=0.00140 Rate=1775.22 GlobalRate=759.94 Time=08:07:19
| Training Device=xla:0/3 Epoch=1 Step=1380 Loss=0.00140 Rate=1776.15 GlobalRate=766.24 Time=08:07:20
| Training Device=xla:0/2 Epoch=1 Step=1380 Loss=0.00140 Rate=1776.09 GlobalRate=770.43 Time=08:07:20
| Training Device=xla:0/0 Epoch=1 Step=1380 Loss=0.00140 Rate=1776.15 GlobalRate=824.10 Time=08:07:20
| Training Device=xla:0/1 Epoch=1 Step=1380 Loss=0.00140 Rate=1776.07 GlobalRate=766.29 Time=08:07:20
| Training Device=xla:0/1 Epoch=1 Step=1400 Loss=0.00139 Rate=1775.04 GlobalRate=772.56 Time=08:07:22
| Training Device=xla:0/3 Epoch=1 Step=1400 Loss=0.00139 Rate=1774.99 GlobalRate=772.50 Time=08:07:22
| Training Device=xla:0/2 Epoch=1 Step=1400 Loss=0.00139 Rate=1774.92 GlobalRate=776.71 Time=08:07:22
| Training Device=xla:0/0 Epoch=1 Step=1400 Loss=0.00139 Rate=1774.92 GlobalRate=830.45 Time=08:07:22
| Training Device=xla:0/3 Epoch=1 Step=1420 Loss=0.00139 Rate=1775.19 GlobalRate=778.70 Time=08:07:23
| Training Device=xla:0/0 Epoch=1 Step=1420 Loss=0.00139 Rate=1775.20 GlobalRate=836.72 Time=08:07:23
| Training Device=xla:0/2 Epoch=1 Step=1420 Loss=0.00139 Rate=1775.18 GlobalRate=782.90 Time=08:07:23
| Training Device=xla:0/1 Epoch=1 Step=1420 Loss=0.00139 Rate=1774.90 GlobalRate=778.75 Time=08:07:23
| Training Device=xla:0/0 Epoch=1 Step=1440 Loss=0.00139 Rate=1773.23 GlobalRate=842.89 Time=08:07:24
| Training Device=xla:0/2 Epoch=1 Step=1440 Loss=0.00139 Rate=1773.15 GlobalRate=789.02 Time=08:07:24
| Training Device=xla:0/3 Epoch=1 Step=1440 Loss=0.00139 Rate=1773.11 GlobalRate=784.80 Time=08:07:24
| Training Device=xla:0/1 Epoch=1 Step=1440 Loss=0.00139 Rate=1773.30 GlobalRate=784.86 Time=08:07:24
| Training Device=xla:0/2 Epoch=1 Step=1460 Loss=0.00138 Rate=1772.14 GlobalRate=795.05 Time=08:07:26
| Training Device=xla:0/0 Epoch=1 Step=1460 Loss=0.00138 Rate=1772.14 GlobalRate=848.99 Time=08:07:26
| Training Device=xla:0/3 Epoch=1 Step=1460 Loss=0.00138 Rate=1772.03 GlobalRate=790.83 Time=08:07:26
| Training Device=xla:0/1 Epoch=1 Step=1460 Loss=0.00138 Rate=1772.21 GlobalRate=790.89 Time=08:07:26
| Training Device=xla:0/3 Epoch=1 Step=1480 Loss=0.00138 Rate=1770.67 GlobalRate=796.78 Time=08:07:27
| Training Device=xla:0/2 Epoch=1 Step=1480 Loss=0.00138 Rate=1770.61 GlobalRate=801.01 Time=08:07:27
| Training Device=xla:0/0 Epoch=1 Step=1480 Loss=0.00138 Rate=1770.54 GlobalRate=854.99 Time=08:07:27
| Training Device=xla:0/1 Epoch=1 Step=1480 Loss=0.00138 Rate=1770.57 GlobalRate=796.84 Time=08:07:27
| Training Device=xla:0/1 Epoch=1 Step=1500 Loss=0.00138 Rate=1774.18 GlobalRate=802.74 Time=08:07:29
| Training Device=xla:0/0 Epoch=1 Step=1500 Loss=0.00138 Rate=1774.14 GlobalRate=860.94 Time=08:07:29
| Training Device=xla:0/2 Epoch=1 Step=1500 Loss=0.00138 Rate=1774.08 GlobalRate=806.91 Time=08:07:29
| Training Device=xla:0/3 Epoch=1 Step=1500 Loss=0.00138 Rate=1773.90 GlobalRate=802.68 Time=08:07:29
| Training Device=xla:0/0 Epoch=1 Step=1520 Loss=0.00137 Rate=1776.39 GlobalRate=866.82 Time=08:07:30
| Training Device=xla:0/1 Epoch=1 Step=1520 Loss=0.00137 Rate=1776.37 GlobalRate=808.57 Time=08:07:30
| Training Device=xla:0/3 Epoch=1 Step=1520 Loss=0.00137 Rate=1776.52 GlobalRate=808.51 Time=08:07:30
| Training Device=xla:0/2 Epoch=1 Step=1520 Loss=0.00137 Rate=1776.40 GlobalRate=812.75 Time=08:07:30
| Training Device=xla:0/3 Epoch=1 Step=1540 Loss=0.00137 Rate=1776.99 GlobalRate=814.27 Time=08:07:32
| Training Device=xla:0/0 Epoch=1 Step=1540 Loss=0.00137 Rate=1776.82 GlobalRate=872.62 Time=08:07:32
| Training Device=xla:0/2 Epoch=1 Step=1540 Loss=0.00137 Rate=1776.91 GlobalRate=818.51 Time=08:07:32
| Training Device=xla:0/1 Epoch=1 Step=1540 Loss=0.00137 Rate=1776.86 GlobalRate=814.33 Time=08:07:32
| Training Device=xla:0/0 Epoch=1 Step=1560 Loss=0.00137 Rate=1776.67 GlobalRate=878.35 Time=08:07:33
| Training Device=xla:0/3 Epoch=1 Step=1560 Loss=0.00137 Rate=1776.70 GlobalRate=819.96 Time=08:07:33
| Training Device=xla:0/1 Epoch=1 Step=1560 Loss=0.00137 Rate=1776.73 GlobalRate=820.02 Time=08:07:33
| Training Device=xla:0/2 Epoch=1 Step=1560 Loss=0.00137 Rate=1776.46 GlobalRate=824.21 Time=08:07:33
| Training Device=xla:0/3 Epoch=1 Step=1580 Loss=0.00136 Rate=1775.76 GlobalRate=825.58 Time=08:07:35
| Training Device=xla:0/2 Epoch=1 Step=1580 Loss=0.00136 Rate=1775.88 GlobalRate=829.83 Time=08:07:35
| Training Device=xla:0/1 Epoch=1 Step=1580 Loss=0.00136 Rate=1775.75 GlobalRate=825.64 Time=08:07:35
| Training Device=xla:0/0 Epoch=1 Step=1580 Loss=0.00136 Rate=1775.55 GlobalRate=884.00 Time=08:07:35
| Training Device=xla:0/1 Epoch=1 Step=1600 Loss=0.00136 Rate=1775.23 GlobalRate=831.19 Time=08:07:36
| Training Device=xla:0/3 Epoch=1 Step=1600 Loss=0.00136 Rate=1775.17 GlobalRate=831.13 Time=08:07:36
| Training Device=xla:0/2 Epoch=1 Step=1600 Loss=0.00136 Rate=1775.24 GlobalRate=835.39 Time=08:07:36
| Training Device=xla:0/0 Epoch=1 Step=1600 Loss=0.00136 Rate=1774.88 GlobalRate=889.57 Time=08:07:36
| Training Device=xla:0/2 Epoch=1 Step=1620 Loss=0.00136 Rate=1775.17 GlobalRate=840.88 Time=08:07:37
| Training Device=xla:0/1 Epoch=1 Step=1620 Loss=0.00136 Rate=1775.15 GlobalRate=836.68 Time=08:07:37
| Training Device=xla:0/3 Epoch=1 Step=1620 Loss=0.00136 Rate=1775.01 GlobalRate=836.62 Time=08:07:37
| Training Device=xla:0/0 Epoch=1 Step=1620 Loss=0.00136 Rate=1775.49 GlobalRate=895.09 Time=08:07:37
| Training Device=xla:0/3 Epoch=1 Step=1640 Loss=0.00136 Rate=1774.08 GlobalRate=842.04 Time=08:07:39
| Training Device=xla:0/2 Epoch=1 Step=1640 Loss=0.00136 Rate=1773.98 GlobalRate=846.30 Time=08:07:39
| Training Device=xla:0/0 Epoch=1 Step=1640 Loss=0.00136 Rate=1774.06 GlobalRate=900.52 Time=08:07:39
| Training Device=xla:0/1 Epoch=1 Step=1640 Loss=0.00136 Rate=1773.99 GlobalRate=842.10 Time=08:07:39
| Training Device=xla:0/2 Epoch=1 Step=1660 Loss=0.00136 Rate=1768.04 GlobalRate=851.64 Time=08:07:40
| Training Device=xla:0/3 Epoch=1 Step=1660 Loss=0.00136 Rate=1767.94 GlobalRate=847.38 Time=08:07:40
| Training Device=xla:0/1 Epoch=1 Step=1660 Loss=0.00136 Rate=1767.94 GlobalRate=847.43 Time=08:07:40
| Training Device=xla:0/0 Epoch=1 Step=1660 Loss=0.00136 Rate=1767.53 GlobalRate=905.86 Time=08:07:40
| Training Device=xla:0/1 Epoch=1 Step=1680 Loss=0.00136 Rate=1753.80 GlobalRate=852.65 Time=08:07:42
| Training Device=xla:0/2 Epoch=1 Step=1680 Loss=0.00136 Rate=1753.69 GlobalRate=856.86 Time=08:07:42
| Training Device=xla:0/3 Epoch=1 Step=1680 Loss=0.00136 Rate=1753.36 GlobalRate=852.59 Time=08:07:42
| Training Device=xla:0/0 Epoch=1 Step=1680 Loss=0.00136 Rate=1754.06 GlobalRate=911.07 Time=08:07:42
| Training Device=xla:0/3 Epoch=1 Step=1700 Loss=0.00136 Rate=1765.17 GlobalRate=857.83 Time=08:07:43
| Training Device=xla:0/0 Epoch=1 Step=1700 Loss=0.00136 Rate=1765.06 GlobalRate=916.31 Time=08:07:43
| Training Device=xla:0/1 Epoch=1 Step=1700 Loss=0.00136 Rate=1764.86 GlobalRate=857.88 Time=08:07:43
| Training Device=xla:0/2 Epoch=1 Step=1700 Loss=0.00136 Rate=1764.74 GlobalRate=862.09 Time=08:07:43
| Training Device=xla:0/3 Epoch=1 Step=1720 Loss=0.00136 Rate=1773.19 GlobalRate=863.02 Time=08:07:45
| Training Device=xla:0/1 Epoch=1 Step=1720 Loss=0.00136 Rate=1773.05 GlobalRate=863.08 Time=08:07:45
| Training Device=xla:0/2 Epoch=1 Step=1720 Loss=0.00136 Rate=1773.13 GlobalRate=867.28 Time=08:07:45
| Training Device=xla:0/0 Epoch=1 Step=1720 Loss=0.00136 Rate=1772.94 GlobalRate=921.50 Time=08:07:45
| Training Device=xla:0/0 Epoch=1 Step=1740 Loss=0.00135 Rate=1777.27 GlobalRate=926.63 Time=08:07:46
| Training Device=xla:0/1 Epoch=1 Step=1740 Loss=0.00135 Rate=1777.02 GlobalRate=868.21 Time=08:07:46
| Training Device=xla:0/2 Epoch=1 Step=1740 Loss=0.00135 Rate=1777.09 GlobalRate=872.42 Time=08:07:46
| Training Device=xla:0/3 Epoch=1 Step=1740 Loss=0.00135 Rate=1777.11 GlobalRate=868.16 Time=08:07:46
| Training Device=xla:0/2 Epoch=1 Step=1760 Loss=0.00135 Rate=1776.39 GlobalRate=877.49 Time=08:07:48
| Training Device=xla:0/0 Epoch=1 Step=1760 Loss=0.00135 Rate=1776.38 GlobalRate=931.69 Time=08:07:48
| Training Device=xla:0/1 Epoch=1 Step=1760 Loss=0.00135 Rate=1776.35 GlobalRate=873.28 Time=08:07:48
| Training Device=xla:0/3 Epoch=1 Step=1760 Loss=0.00135 Rate=1776.27 GlobalRate=873.22 Time=08:07:48
| Training Device=xla:0/0 Epoch=1 Step=1780 Loss=0.00135 Rate=1778.62 GlobalRate=936.70 Time=08:07:49
| Training Device=xla:0/2 Epoch=1 Step=1780 Loss=0.00135 Rate=1778.60 GlobalRate=882.52 Time=08:07:49
| Training Device=xla:0/1 Epoch=1 Step=1780 Loss=0.00135 Rate=1778.61 GlobalRate=878.31 Time=08:07:49
| Training Device=xla:0/3 Epoch=1 Step=1780 Loss=0.00135 Rate=1778.56 GlobalRate=878.25 Time=08:07:49
| Training Device=xla:0/2 Epoch=1 Step=1800 Loss=0.00135 Rate=1777.63 GlobalRate=887.48 Time=08:07:50
| Training Device=xla:0/3 Epoch=1 Step=1800 Loss=0.00135 Rate=1777.69 GlobalRate=883.21 Time=08:07:50
| Training Device=xla:0/1 Epoch=1 Step=1800 Loss=0.00135 Rate=1777.52 GlobalRate=883.27 Time=08:07:50
| Training Device=xla:0/0 Epoch=1 Step=1800 Loss=0.00135 Rate=1777.64 GlobalRate=941.65 Time=08:07:50
| Training Device=xla:0/0 Epoch=1 Step=1820 Loss=0.00135 Rate=1773.14 GlobalRate=946.51 Time=08:07:52
| Training Device=xla:0/1 Epoch=1 Step=1820 Loss=0.00135 Rate=1773.23 GlobalRate=888.15 Time=08:07:52
| Training Device=xla:0/2 Epoch=1 Step=1820 Loss=0.00135 Rate=1773.14 GlobalRate=892.37 Time=08:07:52
| Training Device=xla:0/3 Epoch=1 Step=1820 Loss=0.00135 Rate=1773.16 GlobalRate=888.10 Time=08:07:52
| Training Device=xla:0/1 Epoch=1 Step=1840 Loss=0.00135 Rate=1770.95 GlobalRate=892.99 Time=08:07:53
| Training Device=xla:0/3 Epoch=1 Step=1840 Loss=0.00135 Rate=1770.95 GlobalRate=892.93 Time=08:07:53
| Training Device=xla:0/2 Epoch=1 Step=1840 Loss=0.00135 Rate=1770.94 GlobalRate=897.20 Time=08:07:53
| Training Device=xla:0/0 Epoch=1 Step=1840 Loss=0.00135 Rate=1770.84 GlobalRate=951.32 Time=08:07:53
| Training Device=xla:0/0 Epoch=1 Step=1860 Loss=0.00135 Rate=1775.91 GlobalRate=956.10 Time=08:07:55
| Training Device=xla:0/3 Epoch=1 Step=1860 Loss=0.00135 Rate=1775.90 GlobalRate=897.73 Time=08:07:55
| Training Device=xla:0/1 Epoch=1 Step=1860 Loss=0.00135 Rate=1775.88 GlobalRate=897.79 Time=08:07:55
| Training Device=xla:0/2 Epoch=1 Step=1860 Loss=0.00135 Rate=1775.89 GlobalRate=902.00 Time=08:07:55
| Training Device=xla:0/1 Epoch=1 Step=1880 Loss=0.00135 Rate=1777.15 GlobalRate=902.54 Time=08:07:56
| Training Device=xla:0/2 Epoch=1 Step=1880 Loss=0.00135 Rate=1777.12 GlobalRate=906.75 Time=08:07:56
| Training Device=xla:0/3 Epoch=1 Step=1880 Loss=0.00135 Rate=1777.08 GlobalRate=902.49 Time=08:07:56
| Training Device=xla:0/0 Epoch=1 Step=1880 Loss=0.00135 Rate=1776.97 GlobalRate=960.82 Time=08:07:56
| Training Device=xla:0/3 Epoch=1 Step=1900 Loss=0.00135 Rate=1777.57 GlobalRate=907.18 Time=08:07:58
| Training Device=xla:0/0 Epoch=1 Step=1900 Loss=0.00135 Rate=1777.62 GlobalRate=965.49 Time=08:07:58
| Training Device=xla:0/1 Epoch=1 Step=1900 Loss=0.00135 Rate=1777.51 GlobalRate=907.24 Time=08:07:58
| Training Device=xla:0/2 Epoch=1 Step=1900 Loss=0.00135 Rate=1777.32 GlobalRate=911.45 Time=08:07:58
| Training Device=xla:0/1 Epoch=1 Step=1920 Loss=0.00135 Rate=1777.72 GlobalRate=911.89 Time=08:07:59
| Training Device=xla:0/2 Epoch=1 Step=1920 Loss=0.00135 Rate=1777.81 GlobalRate=916.10 Time=08:07:59
| Training Device=xla:0/0 Epoch=1 Step=1920 Loss=0.00135 Rate=1777.72 GlobalRate=970.11 Time=08:07:59
| Training Device=xla:0/3 Epoch=1 Step=1920 Loss=0.00135 Rate=1777.71 GlobalRate=911.83 Time=08:07:59
| Training Device=xla:0/1 Epoch=1 Step=1940 Loss=0.00135 Rate=1778.89 GlobalRate=916.50 Time=08:08:01
| Training Device=xla:0/0 Epoch=1 Step=1940 Loss=0.00135 Rate=1778.95 GlobalRate=974.68 Time=08:08:01
| Training Device=xla:0/2 Epoch=1 Step=1940 Loss=0.00135 Rate=1778.88 GlobalRate=920.70 Time=08:08:01
| Training Device=xla:0/3 Epoch=1 Step=1940 Loss=0.00135 Rate=1778.89 GlobalRate=916.44 Time=08:08:01
| Training Device=xla:0/1 Epoch=1 Step=1960 Loss=0.00135 Rate=1775.65 GlobalRate=921.03 Time=08:08:02
| Training Device=xla:0/3 Epoch=1 Step=1960 Loss=0.00135 Rate=1775.53 GlobalRate=920.98 Time=08:08:02
| Training Device=xla:0/2 Epoch=1 Step=1960 Loss=0.00135 Rate=1775.71 GlobalRate=925.24 Time=08:08:02
| Training Device=xla:0/0 Epoch=1 Step=1960 Loss=0.00135 Rate=1775.66 GlobalRate=979.17 Time=08:08:02
| Training Device=xla:0/3 Epoch=1 Step=1980 Loss=0.00135 Rate=1775.53 GlobalRate=925.47 Time=08:08:03
| Training Device=xla:0/2 Epoch=1 Step=1980 Loss=0.00135 Rate=1775.47 GlobalRate=929.73 Time=08:08:03
| Training Device=xla:0/0 Epoch=1 Step=1980 Loss=0.00135 Rate=1775.42 GlobalRate=983.63 Time=08:08:03
| Training Device=xla:0/1 Epoch=1 Step=1980 Loss=0.00135 Rate=1775.39 GlobalRate=925.53 Time=08:08:03
| Training Device=xla:0/2 Epoch=1 Step=2000 Loss=0.00135 Rate=1775.83 GlobalRate=934.18 Time=08:08:05
| Training Device=xla:0/1 Epoch=1 Step=2000 Loss=0.00135 Rate=1775.86 GlobalRate=929.98 Time=08:08:05
| Training Device=xla:0/0 Epoch=1 Step=2000 Loss=0.00135 Rate=1775.86 GlobalRate=988.03 Time=08:08:05
| Training Device=xla:0/3 Epoch=1 Step=2000 Loss=0.00135 Rate=1775.75 GlobalRate=929.93 Time=08:08:05
| Training Device=xla:0/3 Epoch=1 Step=2020 Loss=0.00135 Rate=1761.72 GlobalRate=934.26 Time=08:08:06
| Training Device=xla:0/1 Epoch=1 Step=2020 Loss=0.00135 Rate=1761.49 GlobalRate=934.32 Time=08:08:06
| Training Device=xla:0/2 Epoch=1 Step=2020 Loss=0.00135 Rate=1761.51 GlobalRate=938.52 Time=08:08:06
| Training Device=xla:0/0 Epoch=1 Step=2020 Loss=0.00135 Rate=1761.23 GlobalRate=992.31 Time=08:08:06
| Training Device=xla:0/1 Epoch=1 Step=2040 Loss=0.00135 Rate=1767.29 GlobalRate=938.67 Time=08:08:08
| Training Device=xla:0/0 Epoch=1 Step=2040 Loss=0.00135 Rate=1767.41 GlobalRate=996.61 Time=08:08:08
| Training Device=xla:0/2 Epoch=1 Step=2040 Loss=0.00135 Rate=1767.20 GlobalRate=942.86 Time=08:08:08
| Training Device=xla:0/3 Epoch=1 Step=2040 Loss=0.00135 Rate=1767.16 GlobalRate=938.61 Time=08:08:08
| Training Device=xla:0/0 Epoch=1 Step=2060 Loss=0.00135 Rate=1772.51 GlobalRate=1000.87 Time=08:08:09
| Training Device=xla:0/1 Epoch=1 Step=2060 Loss=0.00135 Rate=1772.41 GlobalRate=942.98 Time=08:08:09
| Training Device=xla:0/3 Epoch=1 Step=2060 Loss=0.00135 Rate=1772.29 GlobalRate=942.92 Time=08:08:09
| Training Device=xla:0/2 Epoch=1 Step=2060 Loss=0.00135 Rate=1772.34 GlobalRate=947.17 Time=08:08:09
| Training Device=xla:0/1 Epoch=1 Step=2080 Loss=0.00135 Rate=1775.76 GlobalRate=947.26 Time=08:08:11
| Training Device=xla:0/3 Epoch=1 Step=2080 Loss=0.00135 Rate=1775.70 GlobalRate=947.20 Time=08:08:11
| Training Device=xla:0/0 Epoch=1 Step=2080 Loss=0.00135 Rate=1775.63 GlobalRate=1005.09 Time=08:08:11
| Training Device=xla:0/2 Epoch=1 Step=2080 Loss=0.00135 Rate=1775.87 GlobalRate=951.45 Time=08:08:11
| Training Device=xla:0/0 Epoch=1 Step=2100 Loss=0.00135 Rate=1769.85 GlobalRate=1009.23 Time=08:08:12
| Training Device=xla:0/2 Epoch=1 Step=2100 Loss=0.00135 Rate=1769.89 GlobalRate=955.64 Time=08:08:12
| Training Device=xla:0/3 Epoch=1 Step=2100 Loss=0.00135 Rate=1769.97 GlobalRate=951.40 Time=08:08:12
| Training Device=xla:0/1 Epoch=1 Step=2100 Loss=0.00135 Rate=1769.75 GlobalRate=951.45 Time=08:08:12
| Training Device=xla:0/2 Epoch=1 Step=2120 Loss=0.00135 Rate=1772.58 GlobalRate=959.82 Time=08:08:14
| Training Device=xla:0/0 Epoch=1 Step=2120 Loss=0.00135 Rate=1772.69 GlobalRate=1013.35 Time=08:08:14
| Training Device=xla:0/3 Epoch=1 Step=2120 Loss=0.00135 Rate=1772.59 GlobalRate=955.58 Time=08:08:14
| Training Device=xla:0/1 Epoch=1 Step=2120 Loss=0.00135 Rate=1772.59 GlobalRate=955.63 Time=08:08:14
| Training Device=xla:0/1 Epoch=1 Step=2140 Loss=0.00135 Rate=1774.22 GlobalRate=959.77 Time=08:08:15
| Training Device=xla:0/3 Epoch=1 Step=2140 Loss=0.00135 Rate=1774.18 GlobalRate=959.72 Time=08:08:15
| Training Device=xla:0/0 Epoch=1 Step=2140 Loss=0.00135 Rate=1774.18 GlobalRate=1017.43 Time=08:08:15
| Training Device=xla:0/2 Epoch=1 Step=2140 Loss=0.00135 Rate=1774.04 GlobalRate=963.95 Time=08:08:15
| Training Device=xla:0/3 Epoch=1 Step=2160 Loss=0.00135 Rate=1775.45 GlobalRate=963.82 Time=08:08:16
| Training Device=xla:0/0 Epoch=1 Step=2160 Loss=0.00135 Rate=1775.46 GlobalRate=1021.47 Time=08:08:16
| Training Device=xla:0/2 Epoch=1 Step=2160 Loss=0.00135 Rate=1775.50 GlobalRate=968.05 Time=08:08:16
| Training Device=xla:0/1 Epoch=1 Step=2160 Loss=0.00135 Rate=1775.36 GlobalRate=963.87 Time=08:08:16
| Training Device=xla:0/2 Epoch=1 Step=2180 Loss=0.00135 Rate=1774.52 GlobalRate=972.10 Time=08:08:18
| Training Device=xla:0/3 Epoch=1 Step=2180 Loss=0.00135 Rate=1774.44 GlobalRate=967.87 Time=08:08:18
| Training Device=xla:0/0 Epoch=1 Step=2180 Loss=0.00135 Rate=1774.34 GlobalRate=1025.46 Time=08:08:18
| Training Device=xla:0/1 Epoch=1 Step=2180 Loss=0.00135 Rate=1774.37 GlobalRate=967.93 Time=08:08:18
| Training Device=xla:0/0 Epoch=1 Step=2200 Loss=0.00135 Rate=1775.30 GlobalRate=1029.41 Time=08:08:19
| Training Device=xla:0/1 Epoch=1 Step=2200 Loss=0.00135 Rate=1775.25 GlobalRate=971.94 Time=08:08:19
| Training Device=xla:0/3 Epoch=1 Step=2200 Loss=0.00135 Rate=1775.14 GlobalRate=971.89 Time=08:08:19
| Training Device=xla:0/2 Epoch=1 Step=2200 Loss=0.00135 Rate=1775.16 GlobalRate=976.12 Time=08:08:19
| Training Device=xla:0/0 Epoch=1 Step=2220 Loss=0.00135 Rate=1775.62 GlobalRate=1033.32 Time=08:08:21
| Training Device=xla:0/3 Epoch=1 Step=2220 Loss=0.00135 Rate=1775.61 GlobalRate=975.87 Time=08:08:21
| Training Device=xla:0/2 Epoch=1 Step=2220 Loss=0.00135 Rate=1775.59 GlobalRate=980.09 Time=08:08:21
| Training Device=xla:0/1 Epoch=1 Step=2220 Loss=0.00135 Rate=1775.62 GlobalRate=975.92 Time=08:08:21
| Training Device=xla:0/2 Epoch=1 Step=2240 Loss=0.00135 Rate=1776.59 GlobalRate=984.03 Time=08:08:22
| Training Device=xla:0/3 Epoch=1 Step=2240 Loss=0.00135 Rate=1776.63 GlobalRate=979.81 Time=08:08:22
| Training Device=xla:0/1 Epoch=1 Step=2240 Loss=0.00135 Rate=1776.66 GlobalRate=979.87 Time=08:08:22
| Training Device=xla:0/0 Epoch=1 Step=2240 Loss=0.00135 Rate=1776.63 GlobalRate=1037.20 Time=08:08:22
| Training Device=xla:0/1 Epoch=1 Step=2260 Loss=0.00135 Rate=1773.68 GlobalRate=983.75 Time=08:08:24
| Training Device=xla:0/2 Epoch=1 Step=2260 Loss=0.00135 Rate=1773.67 GlobalRate=987.91 Time=08:08:24
| Training Device=xla:0/3 Epoch=1 Step=2260 Loss=0.00135 Rate=1773.61 GlobalRate=983.70 Time=08:08:24
| Training Device=xla:0/0 Epoch=1 Step=2260 Loss=0.00135 Rate=1773.63 GlobalRate=1041.01 Time=08:08:24
| Training Device=xla:0/0 Epoch=1 Step=2280 Loss=0.00135 Rate=1772.62 GlobalRate=1044.79 Time=08:08:25
| Training Device=xla:0/3 Epoch=1 Step=2280 Loss=0.00135 Rate=1772.62 GlobalRate=987.55 Time=08:08:25
| Training Device=xla:0/2 Epoch=1 Step=2280 Loss=0.00135 Rate=1772.60 GlobalRate=991.76 Time=08:08:25
| Training Device=xla:0/1 Epoch=1 Step=2280 Loss=0.00135 Rate=1772.58 GlobalRate=987.61 Time=08:08:25
| Training Device=xla:0/0 Epoch=1 Step=2300 Loss=0.00135 Rate=1771.68 GlobalRate=1048.53 Time=08:08:27
| Training Device=xla:0/2 Epoch=1 Step=2300 Loss=0.00135 Rate=1771.71 GlobalRate=995.57 Time=08:08:27
| Training Device=xla:0/3 Epoch=1 Step=2300 Loss=0.00135 Rate=1771.71 GlobalRate=991.36 Time=08:08:27
| Training Device=xla:0/1 Epoch=1 Step=2300 Loss=0.00135 Rate=1771.69 GlobalRate=991.42 Time=08:08:27
| Training Device=xla:0/0 Epoch=1 Step=2320 Loss=0.00135 Rate=1771.61 GlobalRate=1052.23 Time=08:08:28
| Training Device=xla:0/3 Epoch=1 Step=2320 Loss=0.00135 Rate=1771.60 GlobalRate=995.14 Time=08:08:28
| Training Device=xla:0/1 Epoch=1 Step=2320 Loss=0.00135 Rate=1771.62 GlobalRate=995.19 Time=08:08:28
| Training Device=xla:0/2 Epoch=1 Step=2320 Loss=0.00135 Rate=1771.55 GlobalRate=999.34 Time=08:08:28
| Training Device=xla:0/0 Epoch=1 Step=2340 Loss=0.00135 Rate=1764.78 GlobalRate=1055.86 Time=08:08:29
| Training Device=xla:0/3 Epoch=1 Step=2340 Loss=0.00135 Rate=1764.87 GlobalRate=998.85 Time=08:08:29
| Training Device=xla:0/1 Epoch=1 Step=2340 Loss=0.00135 Rate=1764.85 GlobalRate=998.90 Time=08:08:29
| Training Device=xla:0/2 Epoch=1 Step=2340 Loss=0.00135 Rate=1764.86 GlobalRate=1003.05 Time=08:08:29
Epoch 1 train end 08:08:30
| Test Device=xla:0/0 Step=0 Epoch=1 Time=08:08:37
| Test Device=xla:0/3 Step=0 Epoch=1 Time=08:08:37
| Test Device=xla:0/2 Step=0 Epoch=1 Time=08:08:37
| Test Device=xla:0/1 Step=0 Epoch=1 Time=08:08:37
| Test Device=xla:0/0 Step=20 Epoch=1 Time=08:08:44
| Test Device=xla:0/3 Step=20 Epoch=1 Time=08:08:44
| Test Device=xla:0/0 Step=40 Epoch=1 Time=08:08:44
| Test Device=xla:0/3 Step=40 Epoch=1 Time=08:08:45
| Test Device=xla:0/0 Step=60 Epoch=1 Time=08:08:45
| Test Device=xla:0/2 Step=20 Epoch=1 Time=08:08:45
| Test Device=xla:0/3 Step=60 Epoch=1 Time=08:08:45
| Test Device=xla:0/0 Step=80 Epoch=1 Time=08:08:45
| Test Device=xla:0/2 Step=40 Epoch=1 Time=08:08:45
| Test Device=xla:0/3 Step=80 Epoch=1 Time=08:08:45
| Test Device=xla:0/1 Step=20 Epoch=1 Time=08:08:45
| Test Device=xla:0/2 Step=60 Epoch=1 Time=08:08:45
| Test Device=xla:0/1 Step=40 Epoch=1 Time=08:08:46
| Test Device=xla:0/2 Step=80 Epoch=1 Time=08:08:46
| Test Device=xla:0/1 Step=60 Epoch=1 Time=08:08:46
| Test Device=xla:0/1 Step=80 Epoch=1 Time=08:08:46
Epoch 1 test end 08:08:47, Accuracy=100.00
Max Accuracy: 100.00%

start time: 08:03:46
end time: 08:08:47,
used time: 5:01

compared with used time before update libtpu at Nov24 is 5:04, and used time before update openxla-pin to Nov24 is 5:08 mentioned in #6012, current used time 05:01 shows there is no time regression

@ManfeiBai ManfeiBai merged commit e2229fd into master Dec 5, 2023
19 checks passed
chunnienc pushed a commit to chunnienc/xla that referenced this pull request Dec 14, 2023
golechwierowicz pushed a commit that referenced this pull request Jan 12, 2024
bhavya01 pushed a commit that referenced this pull request Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants