一、安装前准备：操作系统：CentOS 6.5 64位操作系统环境：jdk1.7.0_45以上，本次采用jdk-7u55-linux-x64.tar.gzmaster01 10.10.2.57 namenode 节点master02 10.10.2.58 namenode 节点slave01:10.10.2.173 datanode 节点slave02:10.10.2.59 datanode 节点slave03: 10.10.2.60 datanode 节点注：Hadoop2.0以上采用的是jdk环境是1.7,Linux自带的jdk卸载掉，重新安装下载地址：http://www.oracle.com/technetwork/java/javase/downloads/index.html软件版本：hadoop-2.3.0-cdh5.1.0.tar.gz, zookeeper-3.4.5-cdh5.1.0.tar.gz下载地址：http://archive.cloudera.com/cdh5/cdh/5/开始安装：二、jdk安装1、检查是否自带jdkrpm -qa | grep jdkjava-1.6.0-openjdk-1.6.0.0-1.45.1.11.1.el6.i686 2、卸载自带jdkyum -y remove java-1.6.0-openjdk-1.6.0.0-1.45.1.11.1.el6.i6863、安装jdk-7u55-linux-x64.tar.gz在usr/目录下创建文件夹java,在java文件夹下运行tar –zxvf jdk-7u55-linux-x64.tar.gz解压到java目录下[root@master01 java]# lsjdk1.7.0_55三、配置环境变量远行vi /etc/profile# /etc/profile# System wide environment and startup programs, for login setup# Functions and aliases go in /etc/bashrcexport JAVA_HOME=/usr/java/jdk1.7.0_55export JRE_HOME=/usr/java/jdk1.7.0_55/jreexport CLASSPATH=/usr/java/jdk1.7.0_55/libexport PATH=$JAVA_HOME/bin: $PATH保存修改，运行source /etc/profile 重新加载环境变量运行java -version[root@master01 java]# java -versionjava version "1.7.0_55"Java(TM) SE Runtime Environment (build 1.7.0_55-b13)Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)Jdk配置成功四、系统配置预先准备5台机器，并配置IP关闭防火墙chkconfig iptables off(永久性关闭)配置主机名和hosts文件[root@master01 java]# vi /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain610.10.2.57 master0110.10.2.58 master0210.10.2.173 slave0110.10.2.59 slave0210.10.2.60 slave03按照不同机器IP配置不同的主机名3、SSH无密码验证配置因为Hadoop运行过程需要远程管理Hadoop的守护进程，NameNode节点需要通过SSH（Secure Shell）链接各个DataNode节点，停止或启动他们的进程，所以SSH必须是没有密码的，所以我们要把NameNode节点和DataNode节点配制成无秘密通信，同理DataNode也需要配置无密码链接NameNode节点。在每一台机器上配置：vi /etc/ssh/sshd_config打开RSAAuthentication yes # 启用 RSA 认证，PubkeyAuthentication yes # 启用公钥私钥配对认证方式Master01:运行：ssh-keygen –t rsa –P '' 不输入密码直接enter默认存放在 ／root／.ssh目录下,cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys[root@master01 .ssh]# lsauthorized_keys id_rsa id_rsa.pub known_hostsslave01执行相同的操作，然后将master01 ／root／.ssh/目录下的id_rsa.pub放到 slave01 相同目录下的authorized_keys这样slave01就持有了master01的公钥 然后直接ssh slave01测试是否可以无密码连接到slave01上，然后将slave01 上的id_rsa.pub 追加到master01的authorized_keys中，测试ssh master01 是否可以直接连上slave01.[root@master01 ~]# ssh slave01Last login: Tue Aug 19 14:28:15 2014 from master01[root@slave01 ~]# Master01-master02Master01-slave01Master01-slave02Master01-slave03Master02-slave01Master02-slave02Master02-slave03执行相同的操作。 五、安装Hadoop建立文件目录 /usr/local/cloud 创建文件夹data，存放数据、日志文件，haooop原文件，zookeeper原文件[root@slave01 cloud]# lsdata hadoop tar zookeeper5.1、配置hadoop-env.sh进入到/usr/local/cloud/hadoop/etc/hadoop目录下配置vi hadoop-env.sh hadoop运行环境加载export JAVA_HOME=/usr/java/jdk1.7.0_555.2、配置core-site.xmlhadoop.tmp.dir /usr/local/cloud/data/hadoop/tmp fs.defaultFS hdfs://zzg (2)hdfs-site.xml配置 ha.zookeeper.quorum master01:2181,slave01:2181,slave02:2181 dfs.namenode.name.dir /usr/local/cloud/data/hadoop/dfs/nn dfs.datanode.data.dir /usr/local/cloud/data/hadoop/dfs/dn dfs.replication 3 dfs.webhdfs.enabled true dfs.permissions false dfs.permissions.enabled false dfs.nameservices zzg dfs.ha.namenodes.zzg nn1,nn2 dfs.namenode.rpc-address.zzg.nn1 master01:9000 dfs.namenode.rpc-address.zzg.nn2 master02:9000 dfs.namenode.http-address.zzg.nn1 master01:50070 dfs.namenode.http-address.zzg.nn2 master02:50070 dfs.namenode.servicerpc-address.zzg.nn1 master01:53310 dfs.namenode.servicerpc-address.zzg.nn2 master02:53310 dfs.namenode.shared.edits.dir qjournal://master01:8485;slave01:8485;slave02:8485/zzg dfs.journalnode.edits.dir /usr/local/cloud/data/hadoop/ha/journal dfs.client.failover.proxy.provider.zzg org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider dfs.ha.automatic-failover.enabled true ha.zookeeper.quorum master01:2181,slave01:2181,slave02:2181 dfs.ha.fencing.methods sshfence 5.3 配置maped-site.xml dfs.ha.fencing.ssh.private-key-files /root/.ssh/id_rsa 5.4配置yarn HA 配置yarn-en.sh java环境# some Java parameters export JAVA_HOME=/usr/java/jdk1.7.0_555.5配置yarn-site.xml mapreduce.framework.name yarn yarn.resourcemanager.connect.retry-interval.ms 2000 yarn.resourcemanager.ha.enabled true yarn.resourcemanager.ha.automatic-failover.enabled true yarn.resourcemanager.ha.rm-ids rm1,rm2 yarn.resourcemanager.ha.id rm1 If we want to launch more than one RM in single node, we need this configuration yarn.resourcemanager.recovery.enabled true yarn.resourcemanager.zk-state-store.address localhost:2181 yarn.resourcemanager.store.class org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore yarn.resourcemanager.zk-address localhost:2181 yarn.resourcemanager.cluster-id yarn-cluster yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms 5000 yarn.resourcemanager.address.rm1 master01:23140 yarn.resourcemanager.scheduler.address.rm1 master01:23130 yarn.resourcemanager.webapp.address.rm1 master01:23188 yarn.resourcemanager.resource-tracker.address.rm1 master01:23125 yarn.resourcemanager.admin.address.rm1 master01:23141 yarn.resourcemanager.ha.admin.address.rm1 master01:23142 yarn.resourcemanager.address.rm2 master02:23140 yarn.resourcemanager.scheduler.address.rm2 master02:23130 yarn.resourcemanager.webapp.address.rm2 master02:23188 yarn.resourcemanager.resource-tracker.address.rm2 master02:23125 yarn.resourcemanager.admin.address.rm2 master02:23141 yarn.resourcemanager.ha.admin.address.rm2 master02:23142 Address where the localizer IPC is. yarn.nodemanager.localizer.address 0.0.0.0:23344 NM Webapp address. yarn.nodemanager.webapp.address 0.0.0.0:23999 yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.aux-services.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler yarn.nodemanager.local-dirs /usr/local/cloud/data/hadoop/yarn/local yarn.nodemanager.log-dirs /usr/local/cloud/data/logs/hadoop mapreduce.shuffle.port 23080 六、配置zookeeper集群在zookeeper目录下建立data目录 和logs目录，配置zoo.cnfdataDir=/usr/local/cloud/zookeeper/datadataLogDir=/usr/local/cloud/zookeeper/logs# the port at which the clients will connectclientPort=2181server.1=master01:2888:3888server.2=master02:2888:3888server.3=slave01:2888:3888server.4=slave02:2888:3888server.5=slave03:2888:3888在data目录下创建myid文件，并在对应的机器上填写数字，如上配置master01 server01 的myid写入1，master02 中的data的myid写入2，依次在其他机子上执行相同操作。在各个机器下zookeeper目录下的bin目录下执行zkServer.sh start命令再运行zkServer.sh status如果出现leader 或fllower 则说明集群配置正确。 到此各个配置文件配置完毕七、启动Hadoop集群严格按照以下顺序执行（第一次）(1)各个节点启动zookeeper，在zookeeper/bin/zkServer.sh start(2) 在hadoop/bin/hdfs zkfc –formatZK 进行格式化创建命名空间(3)在配置了journalnode的节点启动，master01，slave01,slave02 在hadoop/sbin/hadoop-daemon.sh journalnode（4）在主namenode节点执行格式化./bin/hadoop namenode -format zzg 主机器上启动namenode hadoop/sbin/ hadoop-daemon.sh start namenode(5)将主namenode节点格式化的目录拷贝到从主namenode节点上hadoop/bin/hdfs namenode –bootstrapStandbyhadoop/sbin/hadoop-daemon.sh start namenode(6) 在两个namenode节点都执行以下命令./sbin/hadoop-daemon.sh start zkfc(7) 在所有datanode节点都执行以下命令启动datanode./sbin/hadoop-daemon.sh start datanode(8)在主namenode节点启动yarn，运行yarn-start.sh命令jps可以看到namenode节点[root@master01 ~]# jps38972 JournalNode38758 NameNode39166 DFSZKFailoverController37473 QuorumPeerMain39778 ResourceManager42620 Jpsdatanode节点[root@slave01 ~]# jps33440 DataNode35277 Jps32681 QuorumPeerMain33568 JournalNode34231 NodeManager yarn.client.failover-proxy-provider org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider