sqoop2 配置

---

title: sqoop2 配置
tags: sqoop2,环境配置,1.99.7,官网
grammar_cjkRuby: true

---

1 Server installation

sqoop2 官网下载

2 解压到 /usr/tools 文件夹下

tar -zxvf sqoop-1.99.7-bin-hadoop200.tar.gz -C /usr/tools

3 Hadoop dependencies

If the environment $HADOOP_HOME is set, Sqoop will usee the following locations: $HADOOP_HOME/share/hadoop/common, $HADOOP_HOME/share/hadoop/hdfs, $HADOOP_HOME/share/hadoop/mapreduce and $HADOOP_HOME/share/hadoop/yarn.
如果你已经在/etc/profile中配置了 $HADOOP_HOME那你将不需要额外配置以上参数

4 Hadoop configuration

core-site.xml 添加

<property>
  <name>hadoop.proxyuser.sqoop2.hosts</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.sqoop2.groups</name>
  <value>*</value>
</property>

yarn-site.xml 添加

如果，不加上面的属性，会产生什么问题呢？可能在数据迁移的过程中出现“Aggregation is not enabled”的问题，导致数据迁移失败

<property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
</property>

配置完后需要重新启动hadoop

5 Third party jars 第三方数据库驱动配置

mkdir /usr/tools/sqoop-1.99.7-bin-hadoop200/extralib
cp mysql-jdbc*.jar /usr/tools/sqoop-1.99.7-bin-hadoop200/extralib

6 Configuring PATH 修改/etc/profile

#sqoop
export SQOOP_HOME=/usr/tools/sqoop-1.99.7-bin-hadoop200
export SQOOP_SERVER_EXTRA_LIB=$SQOOP_HOME/extralib/
export PATH=$PATH:$SQOOP_HOME/bin

不要忘了source 生效

7 Configuring Server 服务器参数配置

路径/usr/tools/sqoop-1.99.7-bin-hadoop200 下面的sqoop.properties文件

143 # Hadoop configuration directory
144 org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/hadoop-2.6.4/etc/hadoop
145 

元数据数据库，换成mysql需要在这个地方进行配置。暂时不用修改使用系统自带的derby数据库

# JDBC repository provider configuration
org.apache.sqoop.repository.jdbc.handler=org.apache.sqoop.repository.derby.DerbyRepositoryHandler
org.apache.sqoop.repository.jdbc.transaction.isolation=READ_COMMITTED
org.apache.sqoop.repository.jdbc.maximum.connections=10
org.apache.sqoop.repository.jdbc.url=jdbc:derby:@BASEDIR@/repository/db;create=true
org.apache.sqoop.repository.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriver
org.apache.sqoop.repository.jdbc.user=sa
org.apache.sqoop.repository.jdbc.password=

为了将日志输出到/usr/tools/sqoop-1.99.7-bin-hadoop200/logs
进入末行模式 将@LOGDIR@全部替换为../logs

:% s/@LOGDIR@/..\/logs/g

8 验证 sqoop2-tool verify

[root@master conf]# sqoop2-tool verify
Setting conf dir: /usr/tools/sqoop-1.99.7-bin-hadoop200/bin/../conf
Sqoop home directory: /usr/tools/sqoop-1.99.7-bin-hadoop200
Sqoop tool executor:
        Version: 1.99.7
        Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
        Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
13   [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hadoop-2.6.4/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/tools/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.

9 启动服务 sqoop2-server start

[root@master logs]# sqoop2-server start
Setting conf dir: /usr/tools/sqoop-1.99.7-bin-hadoop200/bin/../conf
Sqoop home directory: /usr/tools/sqoop-1.99.7-bin-hadoop200
Starting the Sqoop2 server...
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
13   [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hadoop-2.6.4/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/tools/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Sqoop2 server started.
[root@master logs]# jps
7088 Jps
4982 ResourceManager
4664 NameNode
7069 SqoopJettyServer
4813 SecondaryNameNode

相关链接

Sqoop2 官网配置教程

Sqoop2的安装与使用