NO IMAGE

本次大資料環境有三個節點,所扮演的角色如下:

hadoop01所扮演的角色如下:

[[email protected] ~]# jps
18186 DataNode
18606 DFSZKFailoverController
5780 Jps
17905 QuorumPeerMain
18708 ResourceManager
18068 NameNode
18400 JournalNode
19241 HRegionServer
18833 NodeManager

hadoop02所扮演的角色如下:

[[email protected] ~]# jps
23016 ResourceManager
22487 DataNode
1391 Jps
23120 HRegionServer
22576 JournalNode
22405 NameNode
22695 DFSZKFailoverController
22847 NodeManager
22339 QuorumPeerMain

hadoop03所扮演的角色如下:

[[email protected] ~]# jps
19117 NodeManager
19011 JournalNode
18909 DataNode
19571 HRegionServer
18838 QuorumPeerMain
24682 Jps
19427 HMaster

基於這個叢集安裝Sqoop1.4.6

所使用的軟體是:sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

          依賴包是:mysql-connector-java-5.1.32-bin.jar

一,安裝

1,解壓軟體到指定的位置

[[email protected] bigdata]# ll
total 1005240
-rw-r--r--. 1 root root  92834839 May 28 10:26 apache-hive-1.2.1-bin.tar.gz
-rw-r--r--. 1 root root 231535691 May 29 15:46 apache-hive-2.3.0-bin.tar.gz
-rw-r--r--. 1 root root 231740978 May 31 14:35 apache-hive-2.3.2-bin.tar.gz
-rw-r--r--. 1 root root  53092828 Jun  1 16:07 apache-tez-0.7.1-bin.tar.gz
-rw-r--r--. 1 root root 216929574 May 24 11:42 hadoop-2.7.5.tar.gz
-rw-r--r--. 1 root root  88957416 Jul  2 14:13 hbase-0.98.15-hadoop2-bin.tar.gz
-rw-r--r--. 1 root root    969020 May 28 10:26 mysql-connector-java-5.1.32-bin.jar
-rw-r--r--. 1 root root  16870735 May 28 17:12 sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
-rw-r--r--. 1 root root  29909691 May 29 10:54 sqoop-1.4.6-cdh5.5.2.tar.gz
-rw-r--r--. 1 root root   2197063 May 31 10:28 sqoop-1.4.6.tar.gz
-rw-r--r--. 1 root root  46600150 Jun  6 15:20 tez-0.8.3.tar.gz
-rw-r--r--. 1 root root  17699306 May 24 11:44 zookeeper-3.4.6.tar.gz
[[email protected] bigdata]# pwd
/home/tools/bigdata
[[email protected] bigdata]# tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /home/bigdata/

2,配置環境變數

[[email protected] ~]# cd /home/bigdata/
[[email protected] bigdata]# mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha sqoop-1.4.6
vi /etc/profile
export SQOOP_HOME=/home/bigdata/sqoop-1.4.6
export PATH=$PATH:$SQOOP_HOME/bin
source /etc/profile

3,配置檔案的修改

[[email protected] conf]# cp sqoop-env-template.sh sqoop-env.sh
[[email protected] conf]# vi sqoop-env.sh 
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# included in all the hadoop scripts with source command
# should not be executable directly
# also should not be passed any arguments, since we need original $*
# Set Hadoop-specific environment variables here.
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/home/bigdata/hadoop-2.7.5
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/home/bigdata/hadoop-2.7.5
#set the path to where bin/hbase is available
export HBASE_HOME=/home/bigdata/hbase-0.98
#Set the path to where bin/hive is available
export HIVE_HOME=/home/bigdata/hive-1.2.1
#Set the path for where zookeper config dir is
export ZOOCFGDIR=/home/bigdata/zookeeper-3.4.6

4,上傳jar到sqoop的lib目錄下

[[email protected] ~]# cd /home/bigdata/sqoop-1.4.6/lib/
[[email protected] lib]# cp /home/tools/bigdata/mysql-connector-java-5.1.32-bin.jar .

二,基本應用(基於使用者測試應用)

1,檢視Sqoop的版本

[[email protected] ~]$ sqoop version
Warning: /home/bigdata/sqoop-1.4.6/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/bigdata/sqoop-1.4.6/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
18/07/03 08:27:43 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
Sqoop 1.4.6
git commit id c0c5a81723759fa575844a0a1eae8f510fa32c25
Compiled by root on Mon Apr 27 14:38:36 CST 2015

2,測試是否能連到MySql

[[email protected] ~]$ sqoop list-databases --connect jdbc:mysql://hadoop01:3306/ --username root --password 000000
Warning: /home/bigdata/sqoop-1.4.6/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/bigdata/sqoop-1.4.6/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
18/07/03 08:29:13 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/07/03 08:29:13 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/07/03 08:29:13 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
hive_remote
mysql
oozie
performance_schema
sys
temp
test

3,MySql的資料匯入Hive

sqoop import --connect jdbc:mysql://hadoop01:3306/test?characterEncoding=utf-8  --username root --password '000000'  -table student -hive-import -hive-table tuoming.temp_student -m 1 --fields-terminated-by ","

4,Hive的資料匯入MySql

sqoop export --connect jdbc:mysql://hadoop01:3306/test  --username root --password 000000  --table  student --export-dir /user/hive_remote/warehouse/tuoming.db/temp_student --input-fields-terminated-by ','