君莫笑的博客

CDH安装

2019-09-30T06:33:08.746Z

微信公众号：关注菜鸟解说大数据
关注可了解更多的大数据相关的内容。问题或建议，请公众号留言;
如果你觉得我写的文章对你有帮助，欢迎关注和赞赏我[^1]

[TOC]

推荐阅读

1. CDH简介

Cloudera Manager 最主要的功能就是安装和管理CDH服务，可以使用Cloudera Manager 来为新环境安装服务也可以用来升级现有的服务，在安装方式上，Cloudera Manager支持两种格式：packages和parcels。

Package：是一种二进制的软件包，包括了编译后的代码和元信息，比如包的描述、版本和依赖，通过这些元信息，我们可以使用包管理器来搜索和安装需要的软件包，Cloudera Manager使用操作系统的包管理器来安装软件，如Redhat的rpm包管理和Ubuntu的apt-get等。

Parcel：是一种包含二进制程序文件的包，同时还包含额外的元数据信息，是Cloudera Manager使用的一种包管理和安装CDH服务的方式。它和package的不同之处主要有以下几点：

Parcels是一组包含特定版本组件的包，也可以下载多个不同的版本到同一个parcel目录下，在安装时可以选择要激活的组件版本。

可以安装parcels到系统的任意位置，默认安装位置为：/opt/cloudera/parcels，package包的默认安装位置为：/usr/lib。

当从Cloudera Manager的Parcels page安装时，Cloudera Manager会为集群中的每个节点自动下载、分发和激活选定的parcel。

2. 总的架构

3. 集群规划

HDFS NameNode (2, HA)
HDFS JournalNode (3)
HDFS FailoverController (2, collocated with NN services)
YARN Resource Manager (2, HA)
YARN History Server
HBase Master (3, HA)
Sentry Service
ZooKeeper (3)

4. 集群部署

4.1. 安装介质准备

1.JDK1.8 下载地址：
链接：https://pan.baidu.com/s/1wiM5de1B8Tk67y1TyPQQUg 密码：skoe
2.Cloudera Manager 下载地址：

https://archive.cloudera.com/cm5/cm/5/

说明：CM的各个平台，版本的包都在这里了，各取所需。根据我的环境，下载的是 cloudera-manager-centos7-cm5.15.2_x86_64.tar.gz

3.CDH 下载地址：

http://archive.cloudera.com/cdh5/parcels/5.14/

说明：和上面CM一样，各取所需。需要下载3个文件，我下载的是 CDH-5.15.2-1.cdh5.15.5.p0.3-el7.parcel.sha1（这个文件下载下来后，一定要把后缀 .sha1 改为 .sha）， CDH-5.15.2-1.cdh5.15.2.p0.3-el7.parcel， manifest.json

这里要注意CM和CDH，大版本应该要一致，比如CM为5.15，CDH也是5.15

4.Mysql
如果你的机器能连接外网，那么这里可以暂时不用准备，下面联网安装。如果不能连接外网，那么这里就可以先安装好MySQL（为了方便，安装在server上）。
5.数据库驱动下载地址：

链接：https://pan.baidu.com/s/1yAS_LG88-uOInu5Ee4vzzQ 密码：1bf6

说明：注意默认的数据驱动是有版本号的，比如mysql-connector-java-xxx.jar，这里要把版本号去掉

6.mysql-community-libs-compat-5.7.23-1.el7.x86_64.rpm 下载地址：

链接：https://pan.baidu.com/s/1PqHf3XCoAsDeEqwiDMH_5Q 密码：qgcq

说明：这是MySQL的一个组件的安装包，不是核心组件，不安装也不影响MySQL使用。但是如果缺少这个包，后面hue安装的时候很可能报错，所以先下载下来备用。如果想找不同的版本，可以去MySQL官网找。

4.2 安装目录规划

1.JDK安装目录：/usr/lib/jdk
2.CDH Manager安装目录：/opt/cloudera-manager
3.CDH 组件安装目录：/opt/cloudera

4.3 配置主机名和hosts

1) 执行下面命令，配置静态IP等

1	vi /etc/sysconfig/network-scripts/ifcfg-ens33

下面这是我的网卡配置信息，仅供参考

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=4cbacedd-16ea-4204-9829-223958abbf62
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.79.110
GATEWAY=192.168.79.2
NETMASK=255.255.255.0
DNS1=192.168.79.2

2)修改hosts文件，输入IP和主机名

1	vi /etc/hosts

下面是我的hosts文件，仅供参考：

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.79.100    hadoop01
192.168.79.110    hadoop02
192.168.79.120    hadoop03

4.4 关闭防火墙

1 2	systemctl stop firewalld systemctl disable firewalld

重启机器。然后保证主机之间互ping 主机名，互ping ip能够ping通

4.5 配置免密码登陆（每台主机都要做）

由于时间的原因，后面的内容，我们下期再见。

参考文章

https://blog.csdn.net/u010936936/article/details/81939880#%E5%BC%80%E5%A7%8B%E5%AE%89%E8%A3%85CM%E5%92%8CCDH
https://www.cnblogs.com/xiqing/p/9645718.html

关注菜鸟解说大数据

如果你觉得到作者的文章对你有帮助，欢迎赞赏，有你的支持，公众号一定会越来越好！

CDH6.1.1中的hadoop的测试例子

2019-09-30T06:33:08.743Z

微信公众号：关注菜鸟解说大数据
关注可了解更多的大数据相关的内容。问题或建议，请公众号留言;
如果你觉得我写的文章对你有帮助，欢迎关注和赞赏我[^1]

[TOC]

推荐阅读

1.找到jar的目录

1	/opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/jars

2.如何使用

hadoop jar hadoop-mapreduce-examples-3.0.0-cdh6.1.1.jar  wordcount /tmp/data/wc.txt /tmp/data/out
--说明：
hadoop jar:执行jar的命令;
hadoop-mapreduce-examples-3.0.0-cdh6.1.1.jar:执行的jar包的名字，这里因为我就在当前目录中有这个jar包所以我没有使用绝对路径，如果你不是在这个jar包所在的路径的话，需要使用绝对路径
wordcount：程序主类的名字
/tmp/data/wc.txt：文件的输入路径
/tmp/data/out：文件的输出路径

3.运行过程

WARNING: Use "yarn jar" to launch YARN applications.
19/09/26 16:25:18 INFO client.RMProxy: Connecting to ResourceManager at hadoop01/192.168.163.100:8032
19/09/26 16:25:20 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/hdfs/.staging/job_1562050681525_0001
19/09/26 16:25:21 INFO input.FileInputFormat: Total input files to process : 1
19/09/26 16:25:22 INFO mapreduce.JobSubmitter: number of splits:1
19/09/26 16:25:22 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
19/09/26 16:25:22 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1562050681525_0001
19/09/26 16:25:22 INFO mapreduce.JobSubmitter: Executing with tokens: []
19/09/26 16:25:23 INFO conf.Configuration: resource-types.xml not found
19/09/26 16:25:23 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
19/09/26 16:25:25 INFO impl.YarnClientImpl: Submitted application application_1562050681525_0001
19/09/26 16:25:26 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1562050681525_0001/
19/09/26 16:25:26 INFO mapreduce.Job: Running job: job_1562050681525_0001
19/09/26 16:26:14 INFO mapreduce.Job: Job job_1562050681525_0001 running in uber mode : false
19/09/26 16:26:14 INFO mapreduce.Job:  map 0% reduce 0%
19/09/26 16:27:14 INFO mapreduce.Job:  map 100% reduce 0%
19/09/26 16:27:23 INFO mapreduce.Job:  map 100% reduce 25%
19/09/26 16:27:28 INFO mapreduce.Job:  map 100% reduce 50%
19/09/26 16:27:34 INFO mapreduce.Job:  map 100% reduce 75%
19/09/26 16:27:53 INFO mapreduce.Job:  map 100% reduce 100%
19/09/26 16:27:54 INFO mapreduce.Job: Job job_1562050681525_0001 completed successfully
19/09/26 16:27:54 INFO mapreduce.Job: Counters: 54
File System Counters
FILE: Number of bytes read=191
FILE: Number of bytes written=1088483
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=221
HDFS: Number of bytes written=72
HDFS: Number of read operations=23
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
HDFS: Number of bytes read erasure-coded=0
Job Counters 
Launched map tasks=1
Launched reduce tasks=4
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=53647
Total time spent by all reduces in occupied slots (ms)=42839
Total time spent by all map tasks (ms)=53647
Total time spent by all reduce tasks (ms)=42839
Total vcore-milliseconds taken by all map tasks=53647
Total vcore-milliseconds taken by all reduce tasks=42839
Total megabyte-milliseconds taken by all map tasks=54934528
Total megabyte-milliseconds taken by all reduce tasks=43867136
Map-Reduce Framework
Map input records=10
Map output records=20
Map output bytes=190
Map output materialized bytes=175
Input split bytes=101
Combine input records=20
Combine output records=10
Reduce input groups=10
Reduce shuffle bytes=175
Reduce input records=10
Reduce output records=10
Spilled Records=20
Shuffled Maps =4
Failed Shuffles=0
Merged Map outputs=4
GC time elapsed (ms)=711
CPU time spent (ms)=12920
Physical memory (bytes) snapshot=1241501696
Virtual memory (bytes) snapshot=12939522048
Total committed heap usage (bytes)=904396800
Peak Map Physical memory (bytes)=450433024
Peak Map Virtual memory (bytes)=2578866176
Peak Reduce Physical memory (bytes)=206004224
Peak Reduce Virtual memory (bytes)=2591559680
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters 
Bytes Read=120
File Output Format Counters 
Bytes Written=72
您在 /var/spool/mail/root 中有新邮件

4.结果

源码分析

package org.apache.hadoop.examples;

import java.io.IOException;
import java.util.Iterator;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class WordCount {
    public WordCount() {
    }

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        String[] otherArgs = (new GenericOptionsParser(conf, args)).getRemainingArgs();
        if (otherArgs.length < 2) {
            System.err.println("Usage: wordcount  [...] ");
            System.exit(2);
        }

        Job job = Job.getInstance(conf, "word count");
        job.setJarByClass(WordCount.class);
        job.setMapperClass(WordCount.TokenizerMapper.class);
        job.setCombinerClass(WordCount.IntSumReducer.class);
        job.setReducerClass(WordCount.IntSumReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        for(int i = 0; i < otherArgs.length - 1; ++i) {
            FileInputFormat.addInputPath(job, new Path(otherArgs[i]));
        }

        FileOutputFormat.setOutputPath(job, new Path(otherArgs[otherArgs.length - 1]));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }

    public static class IntSumReducer extends Reducer {
        private IntWritable result = new IntWritable();

        public IntSumReducer() {
        }

        public void reduce(Text key, Iterable values, Reducer.Context context) throws IOException, InterruptedException {
            int sum = 0;

            IntWritable val;
            for(Iterator var5 = values.iterator(); var5.hasNext(); sum += val.get()) {
                val = (IntWritable)var5.next();
            }

            this.result.set(sum);
            context.write(key, this.result);
        }
    }

    public static class TokenizerMapper extends Mapper {
        private static final IntWritable one = new IntWritable(1);
        private Text word = new Text();

        public TokenizerMapper() {
        }

        public void map(Object key, Text value, Mapper.Context context) throws IOException, InterruptedException {
            StringTokenizer itr = new StringTokenizer(value.toString());

            while(itr.hasMoreTokens()) {
                this.word.set(itr.nextToken());
                context.write(this.word, one);
            }

        }
    }
}

关注菜鸟解说大数据

如果你觉得到作者的文章对你有帮助，欢迎赞赏，有你的支持，公众号一定会越来越好！