IT博客汇 | 搭建hadoop-2.6.0-cdh5.4.7伪分布式

搭建hadoop-2.6.0-cdh5.4.7伪分布式

李阳发表于 2015-09-22 07:29:38

首先说一下配置吧：

OS：Centos6.6，1核1G内存（有些吃力）

JDK:1.7_79

已经关闭防火墙，selinux！

那么废话不多说直接开始了，首先下载hadoop-2.6.0-cdh5.4.7.tar.gz

地址：http://archive.cloudera.com/cdh5/cdh/5/自己找哈！我下载的是最新的。

下载完成后，解压

tar -zxvf  hadoop-2.6.0-cdh5.4.7.tar.gz

刷屏完毕后，进入 hadoop-2.6.0-cdh5.4.7/etc/hadoop目录开始配置。

首先请确定是否已经成功配置JDK环境变量，JDK没问题后编辑hadoop-env.sh

vi hadoop-env.sh

修改如下配置：

export JAVA_HOME=/usr/local/java/jdk1.7.0_79

JAVA_HOME请修改成自己的。:wp保存并退出。

然后编辑core-site.xml。

vi core-site.xml

添加如下配置：

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://node2:9000</value>
        </property>
</configuration>

node2说明，如果没有配置hosts，请将node2换成IP地址:wp保存并退出。

然后编辑hdfs-site.xml 。

vi hdfs-site.xml

添加如下配置：

<configuration>
    <property>
        <!--开启web hdfs-->
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/opt/cdh/hadoop/name</value>
        <description> namenode 存放name table(fsimage)本地目录（需要修改）</description>
    </property>
    <property>
        <name>dfs.namenode.edits.dir</name>
        <value>${dfs.namenode.name.dir}</value>
        <description>namenode存放 transactionfile(edits)本地目录（请自行修改）</description>
    </property>
    <property>
       <name>dfs.datanode.data.dir</name>
       <value>/opt/cdh/hadoop/data</value>
       <description>datanode存放block本地目录（请自行修改）</description>
   </property>
</configuration>

以上配置完成后，请按照配置内容创建文件夹：

mkdir -p cdh/hadoop/name
mkdir  cdh/hadoop/data

然后复制mapred-site.xml.template 并且更名为mapred-site.xml:

cp mapred-site.xml.template mapred-site.xml

并且加入一下配置：

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

编辑yarn-site.xml

vi yarn-site.xml

加入以下配置：

<configuration>
    <!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

到此位置，Hadoop-2.6.0-cdh5.4.7的配置已经结束，请检查配置文件是否合格，合格的话进入下面的步骤：

首先，格式化HDFS：

bin/hdfs namenode -format

看到如下信息：

************************************************************/
15/09/22 14:59:46 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
15/09/22 14:59:46 INFO namenode.NameNode: createNameNode [-format]
15/09/22 14:59:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/09/22 14:59:49 WARN common.Util: Path /opt/cdh/hadoop/name should be specified as a URI in configuration files. Please update hdfs configuration.
15/09/22 14:59:49 WARN common.Util: Path /opt/cdh/hadoop/name should be specified as a URI in configuration files. Please update hdfs configuration.
Formatting using clusterid: CID-41ea6672-a32e-4b16-b704-962381ed409a
15/09/22 14:59:49 INFO namenode.FSNamesystem: No KeyProvider found.
15/09/22 14:59:49 INFO namenode.FSNamesystem: fsLock is fair:true
15/09/22 14:59:49 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
15/09/22 14:59:49 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
15/09/22 14:59:49 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
15/09/22 14:59:49 INFO blockmanagement.BlockManager: The block deletion will start around 2015 九月 22 14:59:49
15/09/22 14:59:49 INFO util.GSet: Computing capacity for map BlocksMap
15/09/22 14:59:49 INFO util.GSet: VM type       = 64-bit
15/09/22 14:59:49 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
15/09/22 14:59:49 INFO util.GSet: capacity      = 2^21 = 2097152 entries
15/09/22 14:59:50 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
15/09/22 14:59:50 INFO blockmanagement.BlockManager: defaultReplication         = 1
15/09/22 14:59:50 INFO blockmanagement.BlockManager: maxReplication             = 512
15/09/22 14:59:50 INFO blockmanagement.BlockManager: minReplication             = 1
15/09/22 14:59:50 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
15/09/22 14:59:50 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false
15/09/22 14:59:50 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
15/09/22 14:59:50 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
15/09/22 14:59:50 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
15/09/22 14:59:50 INFO namenode.FSNamesystem: fsOwner             = root (auth:SIMPLE)
15/09/22 14:59:50 INFO namenode.FSNamesystem: supergroup          = supergroup
15/09/22 14:59:50 INFO namenode.FSNamesystem: isPermissionEnabled = true
15/09/22 14:59:50 INFO namenode.FSNamesystem: HA Enabled: false
15/09/22 14:59:50 INFO namenode.FSNamesystem: Append Enabled: true
15/09/22 14:59:50 INFO util.GSet: Computing capacity for map INodeMap
15/09/22 14:59:50 INFO util.GSet: VM type       = 64-bit
15/09/22 14:59:50 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
15/09/22 14:59:50 INFO util.GSet: capacity      = 2^20 = 1048576 entries
15/09/22 14:59:50 INFO namenode.NameNode: Caching file names occuring more than 10 times
15/09/22 14:59:50 INFO util.GSet: Computing capacity for map cachedBlocks
15/09/22 14:59:50 INFO util.GSet: VM type       = 64-bit
15/09/22 14:59:50 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
15/09/22 14:59:50 INFO util.GSet: capacity      = 2^18 = 262144 entries
15/09/22 14:59:50 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
15/09/22 14:59:50 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
15/09/22 14:59:50 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
15/09/22 14:59:50 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
15/09/22 14:59:50 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
15/09/22 14:59:50 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
15/09/22 14:59:50 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
15/09/22 14:59:50 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
15/09/22 14:59:50 INFO util.GSet: Computing capacity for map NameNodeRetryCache
15/09/22 14:59:50 INFO util.GSet: VM type       = 64-bit
15/09/22 14:59:50 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
15/09/22 14:59:50 INFO util.GSet: capacity      = 2^15 = 32768 entries
15/09/22 14:59:50 INFO namenode.NNConf: ACLs enabled? false
15/09/22 14:59:50 INFO namenode.NNConf: XAttrs enabled? true
15/09/22 14:59:50 INFO namenode.NNConf: Maximum size of an xattr: 16384
15/09/22 14:59:51 INFO namenode.FSImage: Allocated new BlockPoolId: BP-314159059-192.168.1.3-1442905191056
15/09/22 14:59:51 INFO common.Storage: Storage directory /opt/cdh/hadoop/name has been successfully formatted.
15/09/22 14:59:51 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
15/09/22 14:59:51 INFO util.ExitUtil: Exiting with status 0
15/09/22 14:59:51 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node2/192.168.1.3
************************************************************/

如果不报错，证明格式化成功。

然后分别启动HDFS和Yarn:

sbin/start-dfs.sh
sbin/start-yarn.sh

启动过程也会输出一下日志，如果没发现错误，请输入命令jps查看所有的节点是否都已经成功启动，成功启动的进程为：

[root@node2 hadoop-2.6.0-cdh5.4.7]# jps
2422 SecondaryNameNode
3021 Jps
2676 NodeManager
2177 NameNode
2591 ResourceManager
2259 DataNode

OK，打开地址http://192.168.1.3:8088/cluster查看yarn，http://192.168.1.3:50070/dfshealth.html#tab-overview，查看HDFS（datenode）状态，到此为止，hadoop-2.6.0-cdh5.4.7伪分布式已经搭建成功，大家可以运行自带的例子（wordcount）来测试mapreduce！过几天再把HBASE的伪分布式分享给大家！

未经允许不得转载：李阳博客 » 搭建hadoop-2.6.0-cdh5.4.7伪分布式