-
Notifications
You must be signed in to change notification settings - Fork 0
/
syslog_withoutLocalAggregation
517 lines (516 loc) · 30.5 KB
/
syslog_withoutLocalAggregation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
2021-01-04 15:03:51,096 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at ip-172-31-1-54.ec2.internal/172.31.1.54:8032
2021-01-04 15:03:51,303 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-172-31-1-54.ec2.internal/172.31.1.54:10200
2021-01-04 15:03:53,071 WARN org.apache.hadoop.mapreduce.JobResourceUploader (main): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-01-04 15:03:53,080 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1609772515988_0001
2021-01-04 15:03:54,858 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total input files to process : 1
2021-01-04 15:03:55,333 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:24
2021-01-04 15:03:55,660 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1609772515988_0001
2021-01-04 15:03:55,662 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2021-01-04 15:03:55,889 INFO org.apache.hadoop.conf.Configuration (main): resource-types.xml not found
2021-01-04 15:03:55,890 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils (main): Unable to find 'resource-types.xml'.
2021-01-04 15:03:56,426 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1609772515988_0001
2021-01-04 15:03:56,568 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-172-31-1-54.ec2.internal:20888/proxy/application_1609772515988_0001/
2021-01-04 15:03:56,569 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1609772515988_0001
2021-01-04 15:04:14,748 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609772515988_0001 running in uber mode : false
2021-01-04 15:04:14,750 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2021-01-04 15:04:41,950 INFO org.apache.hadoop.mapreduce.Job (main): map 3% reduce 0%
2021-01-04 15:04:48,001 INFO org.apache.hadoop.mapreduce.Job (main): map 5% reduce 0%
2021-01-04 15:04:54,030 INFO org.apache.hadoop.mapreduce.Job (main): map 6% reduce 0%
2021-01-04 15:04:58,058 INFO org.apache.hadoop.mapreduce.Job (main): map 7% reduce 0%
2021-01-04 15:04:59,064 INFO org.apache.hadoop.mapreduce.Job (main): map 8% reduce 0%
2021-01-04 15:05:18,168 INFO org.apache.hadoop.mapreduce.Job (main): map 9% reduce 0%
2021-01-04 15:05:19,172 INFO org.apache.hadoop.mapreduce.Job (main): map 10% reduce 0%
2021-01-04 15:05:25,208 INFO org.apache.hadoop.mapreduce.Job (main): map 11% reduce 0%
2021-01-04 15:05:31,246 INFO org.apache.hadoop.mapreduce.Job (main): map 13% reduce 0%
2021-01-04 15:05:37,271 INFO org.apache.hadoop.mapreduce.Job (main): map 14% reduce 0%
2021-01-04 15:05:41,295 INFO org.apache.hadoop.mapreduce.Job (main): map 17% reduce 0%
2021-01-04 15:06:07,421 INFO org.apache.hadoop.mapreduce.Job (main): map 18% reduce 0%
2021-01-04 15:06:08,425 INFO org.apache.hadoop.mapreduce.Job (main): map 20% reduce 0%
2021-01-04 15:06:14,459 INFO org.apache.hadoop.mapreduce.Job (main): map 21% reduce 0%
2021-01-04 15:06:19,485 INFO org.apache.hadoop.mapreduce.Job (main): map 22% reduce 0%
2021-01-04 15:06:25,526 INFO org.apache.hadoop.mapreduce.Job (main): map 24% reduce 0%
2021-01-04 15:06:26,530 INFO org.apache.hadoop.mapreduce.Job (main): map 25% reduce 0%
2021-01-04 15:06:50,672 INFO org.apache.hadoop.mapreduce.Job (main): map 26% reduce 0%
2021-01-04 15:06:52,679 INFO org.apache.hadoop.mapreduce.Job (main): map 28% reduce 0%
2021-01-04 15:06:56,696 INFO org.apache.hadoop.mapreduce.Job (main): map 29% reduce 0%
2021-01-04 15:06:58,703 INFO org.apache.hadoop.mapreduce.Job (main): map 30% reduce 0%
2021-01-04 15:07:04,729 INFO org.apache.hadoop.mapreduce.Job (main): map 31% reduce 0%
2021-01-04 15:07:07,746 INFO org.apache.hadoop.mapreduce.Job (main): map 32% reduce 0%
2021-01-04 15:07:09,755 INFO org.apache.hadoop.mapreduce.Job (main): map 33% reduce 0%
2021-01-04 15:07:32,881 INFO org.apache.hadoop.mapreduce.Job (main): map 35% reduce 0%
2021-01-04 15:07:35,892 INFO org.apache.hadoop.mapreduce.Job (main): map 36% reduce 0%
2021-01-04 15:07:38,903 INFO org.apache.hadoop.mapreduce.Job (main): map 37% reduce 0%
2021-01-04 15:07:41,913 INFO org.apache.hadoop.mapreduce.Job (main): map 38% reduce 0%
2021-01-04 15:07:44,924 INFO org.apache.hadoop.mapreduce.Job (main): map 39% reduce 0%
2021-01-04 15:07:51,961 INFO org.apache.hadoop.mapreduce.Job (main): map 40% reduce 0%
2021-01-04 15:07:52,965 INFO org.apache.hadoop.mapreduce.Job (main): map 42% reduce 0%
2021-01-04 15:08:18,091 INFO org.apache.hadoop.mapreduce.Job (main): map 43% reduce 0%
2021-01-04 15:08:19,097 INFO org.apache.hadoop.mapreduce.Job (main): map 44% reduce 0%
2021-01-04 15:08:24,114 INFO org.apache.hadoop.mapreduce.Job (main): map 45% reduce 0%
2021-01-04 15:08:25,118 INFO org.apache.hadoop.mapreduce.Job (main): map 46% reduce 0%
2021-01-04 15:08:30,146 INFO org.apache.hadoop.mapreduce.Job (main): map 47% reduce 0%
2021-01-04 15:08:36,171 INFO org.apache.hadoop.mapreduce.Job (main): map 49% reduce 0%
2021-01-04 15:08:37,174 INFO org.apache.hadoop.mapreduce.Job (main): map 50% reduce 0%
2021-01-04 15:09:02,258 INFO org.apache.hadoop.mapreduce.Job (main): map 51% reduce 0%
2021-01-04 15:09:03,261 INFO org.apache.hadoop.mapreduce.Job (main): map 53% reduce 0%
2021-01-04 15:09:08,276 INFO org.apache.hadoop.mapreduce.Job (main): map 54% reduce 0%
2021-01-04 15:09:09,279 INFO org.apache.hadoop.mapreduce.Job (main): map 55% reduce 0%
2021-01-04 15:09:15,298 INFO org.apache.hadoop.mapreduce.Job (main): map 56% reduce 0%
2021-01-04 15:09:20,313 INFO org.apache.hadoop.mapreduce.Job (main): map 58% reduce 0%
2021-01-04 15:09:46,459 INFO org.apache.hadoop.mapreduce.Job (main): map 61% reduce 0%
2021-01-04 15:09:52,511 INFO org.apache.hadoop.mapreduce.Job (main): map 63% reduce 0%
2021-01-04 15:09:58,530 INFO org.apache.hadoop.mapreduce.Job (main): map 64% reduce 0%
2021-01-04 15:10:04,550 INFO org.apache.hadoop.mapreduce.Job (main): map 65% reduce 0%
2021-01-04 15:10:05,554 INFO org.apache.hadoop.mapreduce.Job (main): map 67% reduce 0%
2021-01-04 15:10:29,630 INFO org.apache.hadoop.mapreduce.Job (main): map 68% reduce 0%
2021-01-04 15:10:31,637 INFO org.apache.hadoop.mapreduce.Job (main): map 70% reduce 0%
2021-01-04 15:10:35,649 INFO org.apache.hadoop.mapreduce.Job (main): map 71% reduce 0%
2021-01-04 15:10:37,659 INFO org.apache.hadoop.mapreduce.Job (main): map 72% reduce 0%
2021-01-04 15:10:47,712 INFO org.apache.hadoop.mapreduce.Job (main): map 74% reduce 0%
2021-01-04 15:10:48,716 INFO org.apache.hadoop.mapreduce.Job (main): map 75% reduce 0%
2021-01-04 15:11:13,808 INFO org.apache.hadoop.mapreduce.Job (main): map 76% reduce 0%
2021-01-04 15:11:14,811 INFO org.apache.hadoop.mapreduce.Job (main): map 78% reduce 0%
2021-01-04 15:11:19,828 INFO org.apache.hadoop.mapreduce.Job (main): map 79% reduce 0%
2021-01-04 15:11:20,831 INFO org.apache.hadoop.mapreduce.Job (main): map 80% reduce 0%
2021-01-04 15:11:26,850 INFO org.apache.hadoop.mapreduce.Job (main): map 81% reduce 0%
2021-01-04 15:11:30,863 INFO org.apache.hadoop.mapreduce.Job (main): map 82% reduce 0%
2021-01-04 15:11:31,868 INFO org.apache.hadoop.mapreduce.Job (main): map 83% reduce 0%
2021-01-04 15:11:50,931 INFO org.apache.hadoop.mapreduce.Job (main): map 84% reduce 0%
2021-01-04 15:11:56,950 INFO org.apache.hadoop.mapreduce.Job (main): map 85% reduce 0%
2021-01-04 15:11:57,953 INFO org.apache.hadoop.mapreduce.Job (main): map 86% reduce 0%
2021-01-04 15:12:02,977 INFO org.apache.hadoop.mapreduce.Job (main): map 87% reduce 0%
2021-01-04 15:12:03,980 INFO org.apache.hadoop.mapreduce.Job (main): map 88% reduce 0%
2021-01-04 15:12:10,002 INFO org.apache.hadoop.mapreduce.Job (main): map 89% reduce 0%
2021-01-04 15:12:15,018 INFO org.apache.hadoop.mapreduce.Job (main): map 90% reduce 0%
2021-01-04 15:12:17,026 INFO org.apache.hadoop.mapreduce.Job (main): map 92% reduce 0%
2021-01-04 15:12:37,093 INFO org.apache.hadoop.mapreduce.Job (main): map 93% reduce 0%
2021-01-04 15:12:39,099 INFO org.apache.hadoop.mapreduce.Job (main): map 94% reduce 0%
2021-01-04 15:12:43,111 INFO org.apache.hadoop.mapreduce.Job (main): map 95% reduce 0%
2021-01-04 15:12:45,118 INFO org.apache.hadoop.mapreduce.Job (main): map 96% reduce 0%
2021-01-04 15:12:49,143 INFO org.apache.hadoop.mapreduce.Job (main): map 97% reduce 0%
2021-01-04 15:12:54,158 INFO org.apache.hadoop.mapreduce.Job (main): map 99% reduce 0%
2021-01-04 15:12:57,166 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2021-01-04 15:13:14,221 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 22%
2021-01-04 15:13:20,240 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 33%
2021-01-04 15:13:32,280 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 67%
2021-01-04 15:13:38,297 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 70%
2021-01-04 15:13:44,316 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 75%
2021-01-04 15:13:50,341 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 80%
2021-01-04 15:13:56,360 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 85%
2021-01-04 15:14:02,379 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 89%
2021-01-04 15:14:08,399 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 94%
2021-01-04 15:14:14,423 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 99%
2021-01-04 15:14:17,433 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2021-01-04 15:14:19,442 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609772515988_0001 completed successfully
2021-01-04 15:14:19,560 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 61
File System Counters
FILE: Number of bytes read=297278239
FILE: Number of bytes written=607148260
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=3288
HDFS: Number of bytes written=0
HDFS: Number of read operations=24
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3N: Number of bytes read=1604714281
S3N: Number of bytes written=53012247
S3N: Number of read operations=0
S3N: Number of large read operations=0
S3N: Number of write operations=0
Job Counters
Killed map tasks=1
Launched map tasks=24
Launched reduce tasks=1
Data-local map tasks=24
Total time spent by all maps in occupied slots (ms)=48554400
Total time spent by all reduces in occupied slots (ms)=7583328
Total time spent by all map tasks (ms)=1011550
Total time spent by all reduce tasks (ms)=78993
Total vcore-milliseconds taken by all map tasks=1011550
Total vcore-milliseconds taken by all reduce tasks=78993
Total megabyte-milliseconds taken by all map tasks=1553740800
Total megabyte-milliseconds taken by all reduce tasks=242666496
Map-Reduce Framework
Map input records=163471963
Map output records=71119513
Map output bytes=2400202188
Map output materialized bytes=303921087
Input split bytes=3288
Combine input records=0
Combine output records=0
Reduce input groups=1686118
Reduce shuffle bytes=303921087
Reduce input records=71119513
Reduce output records=1686118
Spilled Records=142239026
Shuffled Maps =24
Failed Shuffles=0
Merged Map outputs=24
GC time elapsed (ms)=18034
CPU time spent (ms)=800940
Physical memory (bytes) snapshot=21087199232
Virtual memory (bytes) snapshot=78798204928
Total committed heap usage (bytes)=17921212416
Peak Map Physical memory (bytes)=853663744
Peak Map Virtual memory (bytes)=3112513536
Peak Reduce Physical memory (bytes)=2058981376
Peak Reduce Virtual memory (bytes)=4459450368
FileSpliter$ReducerClass$Counter
N_COUNTER=245382239
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1604714281
File Output Format Counters
Bytes Written=53012247
2021-01-04 15:14:19,619 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at ip-172-31-1-54.ec2.internal/172.31.1.54:8032
2021-01-04 15:14:19,620 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-172-31-1-54.ec2.internal/172.31.1.54:10200
2021-01-04 15:14:19,738 WARN org.apache.hadoop.mapreduce.JobResourceUploader (main): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-01-04 15:14:19,743 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1609772515988_0002
2021-01-04 15:14:21,330 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total input files to process : 1
2021-01-04 15:14:21,352 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader (main): Loaded native gpl library
2021-01-04 15:14:21,355 INFO com.hadoop.compression.lzo.LzoCodec (main): Successfully loaded & initialized native-lzo library [hadoop-lzo rev 049362b7cf53ff5f739d6b1532457f2c6cd495e8]
2021-01-04 15:14:21,816 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:1
2021-01-04 15:14:21,908 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1609772515988_0002
2021-01-04 15:14:21,908 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2021-01-04 15:14:21,949 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1609772515988_0002
2021-01-04 15:14:21,986 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-172-31-1-54.ec2.internal:20888/proxy/application_1609772515988_0002/
2021-01-04 15:14:21,986 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1609772515988_0002
2021-01-04 15:14:40,226 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609772515988_0002 running in uber mode : false
2021-01-04 15:14:40,227 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2021-01-04 15:14:59,399 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2021-01-04 15:15:11,460 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2021-01-04 15:15:12,470 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609772515988_0002 completed successfully
2021-01-04 15:15:12,508 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 59
File System Counters
FILE: Number of bytes read=16143278
FILE: Number of bytes written=32762211
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=95
HDFS: Number of bytes written=0
HDFS: Number of read operations=1
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3N: Number of bytes read=53012247
S3N: Number of bytes written=121603
S3N: Number of read operations=0
S3N: Number of large read operations=0
S3N: Number of write operations=0
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=782928
Total time spent by all reduces in occupied slots (ms)=911520
Total time spent by all map tasks (ms)=16311
Total time spent by all reduce tasks (ms)=9495
Total vcore-milliseconds taken by all map tasks=16311
Total vcore-milliseconds taken by all reduce tasks=9495
Total megabyte-milliseconds taken by all map tasks=25053696
Total megabyte-milliseconds taken by all reduce tasks=29168640
Map-Reduce Framework
Map input records=1686118
Map output records=3372236
Map output bytes=94422608
Map output materialized bytes=16143274
Input split bytes=95
Combine input records=0
Combine output records=0
Reduce input groups=4382
Reduce shuffle bytes=16143274
Reduce input records=3372236
Reduce output records=8764
Spilled Records=6744472
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=802
CPU time spent (ms)=18240
Physical memory (bytes) snapshot=1446010880
Virtual memory (bytes) snapshot=7538221056
Total committed heap usage (bytes)=1333788672
Peak Map Physical memory (bytes)=810803200
Peak Map Virtual memory (bytes)=3086782464
Peak Reduce Physical memory (bytes)=635207680
Peak Reduce Virtual memory (bytes)=4451438592
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=53012247
File Output Format Counters
Bytes Written=121603
2021-01-04 15:15:12,545 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at ip-172-31-1-54.ec2.internal/172.31.1.54:8032
2021-01-04 15:15:12,546 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-172-31-1-54.ec2.internal/172.31.1.54:10200
2021-01-04 15:15:12,642 WARN org.apache.hadoop.mapreduce.JobResourceUploader (main): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-01-04 15:15:12,650 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1609772515988_0003
2021-01-04 15:15:14,356 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total input files to process : 2
2021-01-04 15:15:14,785 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:2
2021-01-04 15:15:15,219 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1609772515988_0003
2021-01-04 15:15:15,219 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2021-01-04 15:15:15,442 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1609772515988_0003
2021-01-04 15:15:15,445 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-172-31-1-54.ec2.internal:20888/proxy/application_1609772515988_0003/
2021-01-04 15:15:15,445 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1609772515988_0003
2021-01-04 15:15:32,622 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609772515988_0003 running in uber mode : false
2021-01-04 15:15:32,622 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2021-01-04 15:15:47,719 INFO org.apache.hadoop.mapreduce.Job (main): map 50% reduce 0%
2021-01-04 15:15:53,749 INFO org.apache.hadoop.mapreduce.Job (main): map 83% reduce 0%
2021-01-04 15:15:57,771 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2021-01-04 15:16:14,858 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 86%
2021-01-04 15:16:20,893 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2021-01-04 15:16:21,902 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609772515988_0003 completed successfully
2021-01-04 15:16:21,953 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 59
File System Counters
FILE: Number of bytes read=50166310
FILE: Number of bytes written=101011430
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=462
HDFS: Number of bytes written=0
HDFS: Number of read operations=2
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3N: Number of bytes read=53133850
S3N: Number of bytes written=128898155
S3N: Number of read operations=0
S3N: Number of large read operations=0
S3N: Number of write operations=0
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=1727616
Total time spent by all reduces in occupied slots (ms)=2117664
Total time spent by all map tasks (ms)=35992
Total time spent by all reduce tasks (ms)=22059
Total vcore-milliseconds taken by all map tasks=35992
Total vcore-milliseconds taken by all reduce tasks=22059
Total megabyte-milliseconds taken by all map tasks=55283712
Total megabyte-milliseconds taken by all reduce tasks=67765248
Map-Reduce Framework
Map input records=1694882
Map output records=3381000
Map output bytes=109054514
Map output materialized bytes=50130692
Input split bytes=462
Combine input records=0
Combine output records=0
Reduce input groups=8764
Reduce shuffle bytes=50130692
Reduce input records=3381000
Reduce output records=3372236
Spilled Records=6762000
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=1554
CPU time spent (ms)=36540
Physical memory (bytes) snapshot=2478247936
Virtual memory (bytes) snapshot=10624118784
Total committed heap usage (bytes)=2279604224
Peak Map Physical memory (bytes)=813150208
Peak Map Virtual memory (bytes)=3092774912
Peak Reduce Physical memory (bytes)=1034575872
Peak Reduce Virtual memory (bytes)=4441886720
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=128898155
2021-01-04 15:16:21,991 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at ip-172-31-1-54.ec2.internal/172.31.1.54:8032
2021-01-04 15:16:21,991 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-172-31-1-54.ec2.internal/172.31.1.54:10200
2021-01-04 15:16:22,094 WARN org.apache.hadoop.mapreduce.JobResourceUploader (main): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-01-04 15:16:22,099 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1609772515988_0004
2021-01-04 15:16:23,676 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total input files to process : 1
2021-01-04 15:16:23,701 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:2
2021-01-04 15:16:23,724 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1609772515988_0004
2021-01-04 15:16:23,724 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2021-01-04 15:16:23,745 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1609772515988_0004
2021-01-04 15:16:23,748 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-172-31-1-54.ec2.internal:20888/proxy/application_1609772515988_0004/
2021-01-04 15:16:23,748 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1609772515988_0004
2021-01-04 15:16:41,933 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609772515988_0004 running in uber mode : false
2021-01-04 15:16:41,934 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2021-01-04 15:17:04,084 INFO org.apache.hadoop.mapreduce.Job (main): map 32% reduce 0%
2021-01-04 15:17:10,117 INFO org.apache.hadoop.mapreduce.Job (main): map 67% reduce 0%
2021-01-04 15:17:13,133 INFO org.apache.hadoop.mapreduce.Job (main): map 83% reduce 0%
2021-01-04 15:17:15,142 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2021-01-04 15:17:31,215 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 87%
2021-01-04 15:17:36,242 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2021-01-04 15:17:38,253 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609772515988_0004 completed successfully
2021-01-04 15:17:38,287 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 59
File System Counters
FILE: Number of bytes read=42955237
FILE: Number of bytes written=86681214
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=468
HDFS: Number of bytes written=0
HDFS: Number of read operations=2
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3N: Number of bytes read=128913143
S3N: Number of bytes written=78906794
S3N: Number of read operations=0
S3N: Number of large read operations=0
S3N: Number of write operations=0
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=2811120
Total time spent by all reduces in occupied slots (ms)=2026944
Total time spent by all map tasks (ms)=58565
Total time spent by all reduce tasks (ms)=21114
Total vcore-milliseconds taken by all map tasks=58565
Total vcore-milliseconds taken by all reduce tasks=21114
Total megabyte-milliseconds taken by all map tasks=89955840
Total megabyte-milliseconds taken by all reduce tasks=64862208
Map-Reduce Framework
Map input records=3372236
Map output records=3372236
Map output bytes=128898155
Map output materialized bytes=43009017
Input split bytes=468
Combine input records=0
Combine output records=0
Reduce input groups=1686118
Reduce shuffle bytes=43009017
Reduce input records=3372236
Reduce output records=0
Spilled Records=6744472
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=2048
CPU time spent (ms)=55630
Physical memory (bytes) snapshot=2685050880
Virtual memory (bytes) snapshot=10651090944
Total committed heap usage (bytes)=2303197184
Peak Map Physical memory (bytes)=829853696
Peak Map Virtual memory (bytes)=3101937664
Peak Reduce Physical memory (bytes)=1029406720
Peak Reduce Virtual memory (bytes)=4452192256
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=0
2021-01-04 15:17:38,323 INFO org.apache.hadoop.yarn.client.RMProxy (main): Connecting to ResourceManager at ip-172-31-1-54.ec2.internal/172.31.1.54:8032
2021-01-04 15:17:38,324 INFO org.apache.hadoop.yarn.client.AHSProxy (main): Connecting to Application History server at ip-172-31-1-54.ec2.internal/172.31.1.54:10200
2021-01-04 15:17:38,414 WARN org.apache.hadoop.mapreduce.JobResourceUploader (main): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-01-04 15:17:38,418 INFO org.apache.hadoop.mapreduce.JobResourceUploader (main): Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1609772515988_0005
2021-01-04 15:17:39,970 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total input files to process : 2
2021-01-04 15:17:40,804 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): number of splits:3
2021-01-04 15:17:41,270 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Submitting tokens for job: job_1609772515988_0005
2021-01-04 15:17:41,270 INFO org.apache.hadoop.mapreduce.JobSubmitter (main): Executing with tokens: []
2021-01-04 15:17:41,301 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (main): Submitted application application_1609772515988_0005
2021-01-04 15:17:41,309 INFO org.apache.hadoop.mapreduce.Job (main): The url to track the job: http://ip-172-31-1-54.ec2.internal:20888/proxy/application_1609772515988_0005/
2021-01-04 15:17:41,309 INFO org.apache.hadoop.mapreduce.Job (main): Running job: job_1609772515988_0005
2021-01-04 15:17:58,462 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609772515988_0005 running in uber mode : false
2021-01-04 15:17:58,462 INFO org.apache.hadoop.mapreduce.Job (main): map 0% reduce 0%
2021-01-04 15:18:19,597 INFO org.apache.hadoop.mapreduce.Job (main): map 33% reduce 0%
2021-01-04 15:18:22,614 INFO org.apache.hadoop.mapreduce.Job (main): map 45% reduce 0%
2021-01-04 15:18:25,630 INFO org.apache.hadoop.mapreduce.Job (main): map 56% reduce 0%
2021-01-04 15:18:33,669 INFO org.apache.hadoop.mapreduce.Job (main): map 89% reduce 0%
2021-01-04 15:18:43,727 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 0%
2021-01-04 15:18:59,793 INFO org.apache.hadoop.mapreduce.Job (main): map 100% reduce 100%
2021-01-04 15:19:02,814 INFO org.apache.hadoop.mapreduce.Job (main): Job job_1609772515988_0005 completed successfully
2021-01-04 15:19:02,851 INFO org.apache.hadoop.mapreduce.Job (main): Counters: 61
File System Counters
FILE: Number of bytes read=29746193
FILE: Number of bytes written=60447566
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=707
HDFS: Number of bytes written=0
HDFS: Number of read operations=3
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
S3N: Number of bytes read=78921783
S3N: Number of bytes written=78906794
S3N: Number of read operations=0
S3N: Number of large read operations=0
S3N: Number of write operations=0
Job Counters
Killed map tasks=1
Launched map tasks=3
Launched reduce tasks=1
Other local map tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=3595344
Total time spent by all reduces in occupied slots (ms)=1641312
Total time spent by all map tasks (ms)=74903
Total time spent by all reduce tasks (ms)=17097
Total vcore-milliseconds taken by all map tasks=74903
Total vcore-milliseconds taken by all reduce tasks=17097
Total megabyte-milliseconds taken by all map tasks=115051008
Total megabyte-milliseconds taken by all reduce tasks=52521984
Map-Reduce Framework
Map input records=1686118
Map output records=1686118
Map output bytes=59275372
Map output materialized bytes=29746104
Input split bytes=707
Combine input records=0
Combine output records=0
Reduce input groups=1686118
Reduce shuffle bytes=29746104
Reduce input records=1686118
Reduce output records=0
Spilled Records=3372236
Shuffled Maps =3
Failed Shuffles=0
Merged Map outputs=3
GC time elapsed (ms)=2141
CPU time spent (ms)=57830
Physical memory (bytes) snapshot=2955317248
Virtual memory (bytes) snapshot=13719019520
Total committed heap usage (bytes)=2679635968
Peak Map Physical memory (bytes)=831881216
Peak Map Virtual memory (bytes)=3096608768
Peak Reduce Physical memory (bytes)=680972288
Peak Reduce Virtual memory (bytes)=4445622272
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=0