Hadoop的安裝和配置,Hadoop叢集搭建

NO IMAGE

一,Hadoop的安裝:

vim /etc/hosts
172.25.38.7    server7
useradd hadoop
su - hadoop
pwd 
/home/hadoop
tar zxf hadoop-2.7.3.tar.gz
tar zxf jdk-7u79-linux-x64.tar.gz 
ln -s jdk1.7.0_79/ jdk
ln -s hadoop-2.7.3 hadoop
vim /home/hadoop/.bash_profile 
10 PATH=$PATH:$HOME/bin:/home/hadoop/jdk/bin
vim hadoop/etc/hadoop/hadoop-env.sh   
25 export JAVA_HOME=/home/hadoop/jdk

測試使用Hadoop自帶的wordcount:

cd /home/hadoop/hadoop
mkdir input
cp etc/hadoop/* input/
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount input output
cd output
[email protected] output]$ ls
part-r-00000  _SUCCESS              #對input執行mapreduce的結果
cat part-r-00000        #可以看見wordcount功能已經完成了mapreduce任務;
!=      3
""      6
"".     4
"$HADOOP_CLASSPATH"     1
"$JAVA_HOME"    2
"$YARN_HEAPSIZE"        1
"$YARN_LOGFILE" 1
"$YARN_LOG_DIR" 1
"$YARN_POLICYFILE"      1
"*"     18
"AS     24
"Error: 1
"License");     24
"alice,bob      18
"console"       1
"dfs"   3
"hadoop.root.logger".   1
"jks".  4
"jvm"   3
"mapred"        3
"rpc"   3
"run    1
"ugi"   3
"x"     1

單節點叢集:NN DN SNN在一臺主機上:

cd /home/hadoop/hadoop/etc/hadoop
vim core-site.xml19 <configuration>
20 <property>
21         <name>fs.defaultFS</name>
22                 <value>hdfs://172.25.38.7:9000</value>
23                     </property>
24 </configuration>
vim hdfs-site.xml
19 <configuration>
20 <property>
21         <name>dfs.replication</name>
22                 <value>1</value>        #此時資料塊僅僅儲存一份,預設儲存三份
23                     </property>
24 </configuration>
vim /home/hadoop/hadoop/etc/hadoop/slaves 
172.25.38.7
ssh-keygen
ssh-copy-id 172.25.38.7
ssh-copy-id server7
bin/hdfs namenode -format
/home/hadoop/hadoop/sbin/start-dfs.sh 
jps

Hadoop叢集搭建:

server7:namenode,SecondaryNameNode
server8,server9:datanode
1,配置叢集節點時間同步:設定叢集節點同步主機的時間:

主機上:vim /etc/chrony.conf
7 server time1.aliyun.com  iburst    #設定主機同步阿里雲的時間
23 allow 172.25.38/24           #允許同步的主機網段

server7,server8,server9上:

yum install -y ntp
vim /etc/ntp.conf 22 server 172.25.38.250 iburst      設定叢集節點同步主機的時間
/etc/init.d/ntpd start

2,配置nfs共享配置:
server7上:

yum install -y nfs-utils
/etc/init.d/rpcbind  start
/etc/init.d/nfs start
/home/hadoop    *(rw,anonuid=500,anongid=500)
exportfs -rv

server8,server9上:

useradd hadoop
yum install -y nfs-utils
/etc/init.d/rpcbind  start
/etc/init.d/nfs start
showmount -e 172.25.38.7
mount 172.25.38.7:/home/hadoop/ /home/hadoop/
vim /home/hadoop/.bash_profile 
PATH=$PATH:$HOME/bin:/home/hadoop/jdk/bin

Hadoop增加叢集節點,擴容

Hadoop縮榮,減少叢集節點