一、安装概述

安装sqoop，为实现将mysql、hive数据库之间数据传送功能。

二、下载及安装

官网下载地址，http://archive.apache.org/dist/sqoop/，我选择的版本是1.4.7
image.png
image.png
将已下载的安装包上传到hadoop的根节点，并解压

[hadoop@hadoop01 ~]$ tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

名字太长了，看着不爽，改个名儿

[hadoop@hadoop01 ~]$ mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop-1.4.7

修改sqoop-env.sh，将#注释的环境变量解开并赋值

[hadoop@hadoop01 ~]$ cd sqoop-1.4.7/conf/
[hadoop@hadoop01 conf]$ mv sqoop-env-template.sh sqoop-env.sh
[hadoop@hadoop01 conf]$ vim sqoop-env.sh

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/home/hadoop/hadoop-2.10.1
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/home/hadoop/hadoop-2.10.1
#set the path to where bin/hbase is available
#export HBASE_HOME=
#Set the path to where bin/hive is available
export HIVE_HOME=/home/hadoop/apache-hive-2.3.9-bin
#Set the path for where zookeper config dir is
#export ZOOCFGDIR=

将mysql的驱动包上传到sqoop的lib包下
将sqoop的环境变量添加到当前用户的环境变量下

[hadoop@hadoop01 conf]$ vim ~/.bashrc 

export JAVA_HOME=/usr/local/java
export HADOOP_HOME=/home/hadoop/hadoop-2.10.1
export SCALA_HOME=/home/hadoop/scala-2.12.2
export SPARK_HOME=/home/hadoop/spark-3.0.3-bin-hadoop2.7
export HIVE_HOME=/home/hadoop/apache-hive-2.3.9-bin
export SQOOP_HOME=/home/hadoop/sqoop-1.4.7
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SCALA_HOME/bin:$SPARK_HOME/bin:$HIVE_HOME/bin:$SQOOP_HOME/bin

[hadoop@hadoop01 conf]$ source ~/.bashrc

验证sqoop安装结果

[hadoop@hadoop01 conf]$ sqoop version
Warning: /home/hadoop/sqoop-1.4.7/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/hadoop/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/hadoop/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/hadoop/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
21/12/06 11:50:20 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017
You have new mail in /var/spool/mail/root
[hadoop@hadoop01 conf]$

三、基本命令

列出mysql数据库

sqoop list-databases \
--connect jdbc:mysql://hadoop01:3306/ \
--username root \
--password 123456

将mysql库表中的数据导入到hive数据库中

sqoop import  \
--connect jdbc:mysql://172.16.100.19:3306/test  \
--username queryuser  \
--password abcde12345  \
--table t_sitenumber_distance  \
--hive-import  \
--hive-overwrite  \
--create-hive-table  \
--delete-target-dir \
--hive-database  sitelight \
--hive-table t_sitenumber_distance

将hive数据库表中的数据导出到mysql数据库中

sqoop export \
--connect jdbc:mysql://172.16.100.19:3306/test  \
--username queryuser  \
--password abcde12345  \
--table t_sitenumber_distance  \
--export-dir /usr/local/warehouse/sitelight.db/t_sitenumber_distance \
--fields-terminated-by '\001'