BigData~09：Sqoop

使用 Sqoop 把数据从 Hive 导出到 MySQL，中间遇到了一些坑，现总结如下：

一、报错总结

1. 在 CDH 上添加了 sqoop 服务之后：

   [root@db1 ~]# sqoop help
   Warning: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
   Please set $ACCUMULO_HOME to the root of your Accumulo installation.
   Error: Could not find or load main class org.apache.sqoop.Sqoop
   

   • 后来，在另外一个机器上做的，这个问题就没有遇到，遇到在添加吧
2. -bash: --username: command not found

   [root@db2 sqoop]# sqoop export \
   > --connect jdbc:mysql://localhost:3306/fact?useUnicode=true&characterEncoding=utf8 \
   > --username txdevfactroot \
   > --password 'admin' \
   > --table wechart_user_increase_fact  \
   > --input-fields-terminated-by '\001' \
   > --update-key id \
   > --update-mode allowinsert \
   > --export-dir /user/hive/warehouse/test.db/user_increase_wechart_fact/
   [1] 28195
   -bash: --username: command not found
   [root@db2 sqoop]# Warning: /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
   Please set $ACCUMULO_HOME to the root of your Accumulo installation.
   19/05/18 12:22:44 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.15.1
   Export requires a --table or a --call argument.
   Try --help for usage instructions.
   
   [1]+  Exit 1                  sqoop export --connect jdbc:mysql://172.25.48.2:3306/fact?useUnicode=true
   

   • 凡是遇到--参数找不到的，把上一条命令使用双引号括起来即可
3. ERROR tool.ExportTool: Error during export:

   19/05/18 10:52:28 ERROR tool.ExportTool: Error during export:
   Export job failed!
       at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:439)
       at org.apache.sqoop.manager.MySQLManager.upsertTable(MySQLManager.java:145)
       at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:85)
       at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:111)
       at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
       at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
       at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
       at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
       at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
       at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
   

   • 当然了，这个原因应该有好多，我的原因是：hive 的表中没有 id，MySQL 中有 id，这个 id 是自增的，想当然的认为没有问题，后来问题一直解决不了，就在 Hive 得到表中添加了一个使用 UUID 的 id 字段，当然了，MySQL 中需要把自增干掉；