下载sqoop
因为官方并不建议在生产环境中使用sqoop2，即1.99.7，所以这里我用的是sqoop1,即1.4.7
点击下载：http://mirror.bit.edu.cn/apache/sqoop/
解压和上传
我用的xshell辅助工具，先在本地解压改了名称之后上传到Linux上的
sqoop.png
上传linux
linux下.png
也可上传压缩包，执行解压缩，执行命令：
cd /home/hadoop
tar -xvf sqoop-1.4.7.bin__hadoop-2.6.0.tar
配置环境变量
1.配置环境变量
编辑/etc/profile文件，添加SQOOP_HOME变量，并且将$SQOOP_HOME/bin添加到PATH变量中
添加后的profile文件内容
export JAVA_HOME=/home/hadoop/jdk1.8.0_144
export HADOOP_HOME=/home/hadoop/hadoop-2.7.2
export HBASE_HOME=/home/hadoop/hbase-1.2.2
export HIVE_HOME=/home/hadoop/hive-2.1.0
export SQOOP_HOME=/home/hadoop/sqoop-1.4.7
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$HIVE_HOME/bin:$SQOOP_HOME/bin:$PATH
编辑完成后，执行命令： source /etc/profile
2.sqoop配置文件修改
2.1 sqoop-env.sh 文件
进入 /home/hadoop/sqoop-1.4.7/conf 目录下，也就是执行：
cd /home/hadoop/sqoop-1.4.7/conf
将sqoop-env-template.sh复制一份，并取名为sqoop-env.sh，也就是执行命令：
cp sqoop-env-template.sh sqoop-env.sh
文件末尾加入一下配置：
#Set the path for where zookeper config dir is
#export ZOOCFGDIR=
export HADOOP_COMMON_HOME=/home/hadoop/hadoop-2.7.2
export HADOOP_MAPRED_HOME=/home/hadoop/hadoop-2.7.2
export HIVE_HOME=/home/hadoop/hive-2.1.0
export HBASE_HOME=/home/hadoop/hbase-1.2.2
注：上面的路径要改为自己的路径
3.把MySQL的驱动包上传到sqoop的lib下
驱动包.png
本人测试这个版本在这里是可以的
4.使用sqoop
安装后，如果命令不涉及hive和hdfs的，可以不启动，例如sqoop help命令：
此命令帮助查看sqoop有哪些命令
[hadoop@master ~]$ sqoop help
Warning: /home/hadoop/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/hadoop/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/hadoop/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/02/24 16:39:16 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
usage: sqoop COMMAND [ARGS]
Available commands:
codegen Generate code to interact with database records
create-hive-table Import a table definition into Hive
eval Evaluate a SQL statement and display the results
export Export an HDFS directory to a database table
help List available commands
import Import a table from a database to HDFS
import-all-tables Import tables from a database to HDFS
import-mainframe Import datasets from a mainframe server to HDFS
job Work with saved jobs
list-databases List available databases on a server
list-tables List available tables in a database
merge Merge results of incremental imports
metastore Run a standalone Sqoop metastore
version Display version information
See 'sqoop help COMMAND' for information on a specific command.
5.使用sqoop查看mysql中的数据表：
进入$SQOOP_HOME/bin目录下执行如下命令：连接mysql看有多少个表
./sqoop list-databases --connect jdbc:mysql://192.168.1.34:3306/test?characterEncoding=UTF-8 --username root --password '123'
命令.png
6.把MySQL中的表导入hdfs中
前提：一定要启动hdfs和yarn,本人忘了启动yarn，一直显示连接超时，搞了半天才反应过来.......
sqoop import -m 1 --connect jdbc:mysql://192.168.1.34:3306/test --username root --password 123 --table name --target-dir /user/sqoop/datatest
显示：
18/02/24 15:52:32 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/02/24 15:52:33 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/02/24 15:52:33 INFO client.RMProxy: Connecting to ResourceManager at master.hadoop/192.168.139.128:8032
18/02/24 15:52:38 INFO db.DBInputFormat: Using read commited transaction isolation
18/02/24 15:52:38 INFO mapreduce.JobSubmitter: number of splits:1
18/02/24 15:52:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1519458728102_0001
18/02/24 15:52:39 INFO impl.YarnClientImpl: Submitted application application_1519458728102_0001
18/02/24 15:52:39 INFO mapreduce.Job: The url to track the job: http://master.hadoop:8088/proxy/application_1519458728102_0001/
18/02/24 15:52:39 INFO mapreduce.Job: Running job: job_1519458728102_0001
18/02/24 15:52:51 INFO mapreduce.Job: Job job_1519458728102_0001 running in uber mode : false
18/02/24 15:52:51 INFO mapreduce.Job: map 0% reduce 0%
18/02/24 15:52:57 INFO mapreduce.Job: map 100% reduce 0%
18/02/24 15:52:57 INFO mapreduce.Job: Job job_1519458728102_0001 completed successfully
18/02/24 15:52:58 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=136482
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=87
HDFS: Number of bytes written=16
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=3648
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=3648
Total vcore-milliseconds taken by all map tasks=3648
Total megabyte-milliseconds taken by all map tasks=3735552
Map-Reduce Framework
Map input records=1
Map output records=1
Input split bytes=87
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=88
CPU time spent (ms)=630
Physical memory (bytes) snapshot=100278272
Virtual memory (bytes) snapshot=2060853248
Total committed heap usage (bytes)=16719872
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=16
18/02/24 15:52:58 INFO mapreduce.ImportJobBase: Transferred 16 bytes in 24.5469 seconds (0.6518 bytes/sec)
18/02/24 15:52:58 INFO mapreduce.ImportJobBase: Retrieved 1 records.
网友评论