hadoop安裝

NO IMAGE

準備:hadoop-2.7.1.tar.gz檔案,jdk1.7、linux環境等

 

1、  建立hadoop使用者用來,執行hadoop叢集

useradd hadoop

passwd hadoop

 

2、  所有機器之間做雙機互信,master通過遠端啟動datanode程序和tasktracker程序,如果不做雙機互信,會導致每次啟動叢集服務都會需要密碼

分別執行:

vim /etc/hosts

172.19.34.185 hadoop185

172.19.34.109 hadoop109

 

su hadoop

ssh-keygen -t rsa # 設定ssh無密碼登入

ssh-copy-id -i hadoop185

 

3、  各機器做時間同步

su root

/usr/sbin/ntpdate ntp.api.bz

crontab -e

*/5 * * * * /usr/sbin/ntpdate ntp.api.bz &>/dev/null

 

4、  配置hadoop-master

tar xf hadoop-2.7.1.tar.gz  -C /usr/local/

chown -R hadoop:hadoop /usr/local/hadoop-2.7.1

mv /usr/local/hadoop-2.7.1 /usr/local/hadoop

 

su hadoop

vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh

# 配置:exportJAVA_HOME=${JAVA_HOME} ,可通過echo ${JAVA_HOME}檢視

 

vim /usr/local/hadoop/etc/hadoop/core-site.xml

<configuration>

<property>

    <name>hadoop.tmp.dir</name>

    <value>/usr/local/hadoop/tmp</value>

</property>

<property>

    <name>fs.default.name</name>

    <value>hdfs://linux-3:9000</value>

</property>

</configuration>

# hadoop.tmp.dir屬性用於定義Hadoop的臨時目錄,其預設為/tmp/hadoop-${username}。HDFS程序的許多目錄預設都在此目錄中,要保證執行Hadoop程序的使用者對其具有全部訪問許可權。

# fs.default.name屬性用於定義HDFS的名稱節點和其預設的檔案系統,其值是一個URI,即NameNode的RPC伺服器監聽的地址(可以是主機名)和埠(預設為8020)。其預設值為file:///,即本地檔案系統。

 

 

vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml

<configuration>

       <property>

<!–定義的HDFS後設資料持久儲存路徑,預設為${hadoop.tmp.dir}/dfs/name–>

              <name>dfs.data.dir</name>

                <value>/usr/local/hadoop/data</value>

        </property>

        <property>

<!–儲存副本的數量,預設是儲存3份,根據實際slave數量配置–>

              <name>dfs.replication</name>

               <value>2</value>

        </property>

</configuration>

 

 

vi/usr/local/hadoop/etc/hadoop/mapred-site.xml

<configuration>

       <property>

<!– 指定mr框架為yarn方式–>

               <name>mapreduce.framework.name</name>

               <value>yarn</value>

       </property>

</configuration>

 

vi/usr/local/hadoop/etc/hadoop/yarn-site.xml

<configuration>

<!– 指定resourcemanager地址–>

<property>

<name>yarn.resourcemanager.hostname</name>

<value>ubuntu3</value>

</property>

<!– 指定nodemanager啟動時載入server的方式為shuffle server –>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

 

 

vi /usr/local/hadoop/etc/hadoop/slaves

hadoop185

hadoop109

 

 

5、  將已配置好的hadoop,scp到其它節機器

scp -r usr/local/hadoop 
[email protected]:/usr/local/

 

6、  到bin目錄下執行format檢查是否報錯

cd /usr/local/hadoop/bin

./hadoop namenode –format

# 輸出檢測結果

15/07/17 09:37:08 INFO metrics.TopMetrics:NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25

15/07/17 09:37:08INFO namenode.FSNamesystem: Retry cache on namenode is enabled

15/07/17 09:37:08 INFO namenode.FSNamesystem:Retry cache will use 0.03 of total heap and retry cache entry expiry time is600000 millis

15/07/17 09:37:08INFO util.GSet: Computing capacity for map NameNodeRetryCache

15/07/17 09:37:08 INFO util.GSet: VM type       = 64-bit

15/07/17 09:37:08 INFO util.GSet: 0.029999999329447746%max memory 888.9 MB = 273.1 KB

15/07/17 09:37:08INFO util.GSet: capacity      = 2^15 =32768 entries

15/07/17 09:37:08 INFO namenode.FSImage:Allocated new BlockPoolId: BP-660215815-172.19.34.185-1437125828943

15/07/17 09:37:09 INFO common.Storage: Storagedirectory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.

15/07/17 09:37:09 INFOnamenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

15/07/17 09:37:09 INFO util.ExitUtil: Exitingwith status 0

15/07/17 09:37:09 INFO namenode.NameNode:SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG:Shutting down NameNode at linux-3/172.19.34.185

************************************************************/

 

# 報錯則根據具體報錯情況調整

 

# 啟動hadoop

# 主節點執行 , 執行成功後兩個節點執行jps看服務狀態

./start-all.sh

 

 [[email protected] sbin]$ jps

27709 NameNode

28628 Jps

28186 ResourceManager

28024 SecondaryNameNode

27845 DataNode

28297 NodeManager

 

[[email protected] sbin]$ jps

3825 NodeManager

3677 DataNode

4087 Jps

 

http://NameNode:8088/ 檢視yarn

http://NameNode:50070/ 檢視hdfs