CDH6.1.1中的hadoop的测试例子

发表于 | | 阅读次数: | 访问次数 次
字数统计: 1.3k | 阅读时长 ≈ 7

微信公众号：关注菜鸟解说大数据
关注可了解更多的大数据相关的内容。问题或建议，请公众号留言;
如果你觉得我写的文章对你有帮助，欢迎关注和赞赏我[^1]

[TOC]

推荐阅读

1.找到jar的目录

1
/opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/jars

2.如何使用

1
2
3
4
5
6
7
hadoop jar hadoop-mapreduce-examples-3.0.0-cdh6.1.1.jar  wordcount /tmp/data/wc.txt /tmp/data/out
--说明：
hadoop jar:执行jar的命令;
hadoop-mapreduce-examples-3.0.0-cdh6.1.1.jar:执行的jar包的名字，这里因为我就在当前目录中有这个jar包所以我没有使用绝对路径，如果你不是在这个jar包所在的路径的话，需要使用绝对路径
wordcount：程序主类的名字
/tmp/data/wc.txt：文件的输入路径
/tmp/data/out：文件的输出路径

3.运行过程

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
WARNING: Use "yarn jar" to launch YARN applications.
19/09/26 16:25:18 INFO client.RMProxy: Connecting to ResourceManager at hadoop01/192.168.163.100:8032
19/09/26 16:25:20 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/hdfs/.staging/job_1562050681525_0001
19/09/26 16:25:21 INFO input.FileInputFormat: Total input files to process : 1
19/09/26 16:25:22 INFO mapreduce.JobSubmitter: number of splits:1
19/09/26 16:25:22 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
19/09/26 16:25:22 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1562050681525_0001
19/09/26 16:25:22 INFO mapreduce.JobSubmitter: Executing with tokens: []
19/09/26 16:25:23 INFO conf.Configuration: resource-types.xml not found
19/09/26 16:25:23 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
19/09/26 16:25:25 INFO impl.YarnClientImpl: Submitted application application_1562050681525_0001
19/09/26 16:25:26 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1562050681525_0001/
19/09/26 16:25:26 INFO mapreduce.Job: Running job: job_1562050681525_0001
19/09/26 16:26:14 INFO mapreduce.Job: Job job_1562050681525_0001 running in uber mode : false
19/09/26 16:26:14 INFO mapreduce.Job:  map 0% reduce 0%
19/09/26 16:27:14 INFO mapreduce.Job:  map 100% reduce 0%
19/09/26 16:27:23 INFO mapreduce.Job:  map 100% reduce 25%
19/09/26 16:27:28 INFO mapreduce.Job:  map 100% reduce 50%
19/09/26 16:27:34 INFO mapreduce.Job:  map 100% reduce 75%
19/09/26 16:27:53 INFO mapreduce.Job:  map 100% reduce 100%
19/09/26 16:27:54 INFO mapreduce.Job: Job job_1562050681525_0001 completed successfully
19/09/26 16:27:54 INFO mapreduce.Job: Counters: 54
	File System Counters
		FILE: Number of bytes read=191
		FILE: Number of bytes written=1088483
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=221
		HDFS: Number of bytes written=72
		HDFS: Number of read operations=23
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=8
		HDFS: Number of bytes read erasure-coded=0
	Job Counters 
		Launched map tasks=1
		Launched reduce tasks=4
		Data-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=53647
		Total time spent by all reduces in occupied slots (ms)=42839
		Total time spent by all map tasks (ms)=53647
		Total time spent by all reduce tasks (ms)=42839
		Total vcore-milliseconds taken by all map tasks=53647
		Total vcore-milliseconds taken by all reduce tasks=42839
		Total megabyte-milliseconds taken by all map tasks=54934528
		Total megabyte-milliseconds taken by all reduce tasks=43867136
	Map-Reduce Framework
		Map input records=10
		Map output records=20
		Map output bytes=190
		Map output materialized bytes=175
		Input split bytes=101
		Combine input records=20
		Combine output records=10
		Reduce input groups=10
		Reduce shuffle bytes=175
		Reduce input records=10
		Reduce output records=10
		Spilled Records=20
		Shuffled Maps =4
		Failed Shuffles=0
		Merged Map outputs=4
		GC time elapsed (ms)=711
		CPU time spent (ms)=12920
		Physical memory (bytes) snapshot=1241501696
		Virtual memory (bytes) snapshot=12939522048
		Total committed heap usage (bytes)=904396800
		Peak Map Physical memory (bytes)=450433024
		Peak Map Virtual memory (bytes)=2578866176
		Peak Reduce Physical memory (bytes)=206004224
		Peak Reduce Virtual memory (bytes)=2591559680
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=120
	File Output Format Counters 
		Bytes Written=72
您在 /var/spool/mail/root 中有新邮件

4.结果

55b156eedbe6fde87c3da2046a1b0be.png

源码分析

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
package org.apache.hadoop.examples;

import java.io.IOException;
import java.util.Iterator;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class WordCount {
    public WordCount() {
    }

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        String[] otherArgs = (new GenericOptionsParser(conf, args)).getRemainingArgs();
        if (otherArgs.length < 2) {
            System.err.println("Usage: wordcount <in> [<in>...] <out>");
            System.exit(2);
        }

        Job job = Job.getInstance(conf, "word count");
        job.setJarByClass(WordCount.class);
        job.setMapperClass(WordCount.TokenizerMapper.class);
        job.setCombinerClass(WordCount.IntSumReducer.class);
        job.setReducerClass(WordCount.IntSumReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        for(int i = 0; i < otherArgs.length - 1; ++i) {
            FileInputFormat.addInputPath(job, new Path(otherArgs[i]));
        }

        FileOutputFormat.setOutputPath(job, new Path(otherArgs[otherArgs.length - 1]));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }

    public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
        private IntWritable result = new IntWritable();

        public IntSumReducer() {
        }

        public void reduce(Text key, Iterable<IntWritable> values, Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException {
            int sum = 0;

            IntWritable val;
            for(Iterator var5 = values.iterator(); var5.hasNext(); sum += val.get()) {
                val = (IntWritable)var5.next();
            }

            this.result.set(sum);
            context.write(key, this.result);
        }
    }

    public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> {
        private static final IntWritable one = new IntWritable(1);
        private Text word = new Text();

        public TokenizerMapper() {
        }

        public void map(Object key, Text value, Mapper<Object, Text, Text, IntWritable>.Context context) throws IOException, InterruptedException {
            StringTokenizer itr = new StringTokenizer(value.toString());

            while(itr.hasMoreTokens()) {
                this.word.set(itr.nextToken());
                context.write(this.word, one);
            }

        }
    }
}


关注菜鸟解说大数据

如果你觉得到作者的文章对你有帮助，欢迎赞赏，有你的支持，公众号一定会越来越好！
公众号二维码

JunMoXiao wechat
Donate comment here
-------------　　　　本文结束　　感谢您的阅读　　　　-------------