1.测试Oracle连接
bin/sqoop list-databases --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password
2.导入hdfs
3.1 导入
bin/sqoop import --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password --table SITTS.ESB_SERVICE_PARAM --target-dir /sqoop/oracle/esb_service_param --split-by PARAM_ID
3.2验证
hdfs dfs -ls /sqoop/oracle/esb_service_param
hdfs dfs -cat /sqoop/oracle/esb_service_param/part-m-00000
4.导入hbase
4.0直接新建并导入
bin/sqoop import --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password --table SITTS.ESB_SERVICE_PARAM --split-by PARAM_ID --hbase-create-table --hbase-table orcl_esb_service_param --hbase-row-key PARAM_ID --column-family info
报错
16/09/07 14:05:30 INFO mapreduce.HBaseImportJob: Creating missing HBase table orcl_esb_service_param
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.HTableDescriptor.addFamily(Lorg/apache/hadoop/hbase/HColumnDescriptor;)V
原因
版本不匹配，环境上版本固定，所以选用手动新建hbase表，再导入数据
4.1新建hbase表
hbase shell
create 'orcl_esb_service_param', 'region'
4.2导入mysql数据到hbase
bin/sqoop import --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password --table SITTS.ESB_SERVICE_PARAM --split-by PARAM_ID --hbase-table orcl_esb_service_param --hbase-row-key PARAM_ID --column-family region
额外例子，导入hbase生成复合的rowKey
bin/sqoop import -D sqoop.hbase.add.row.key=true --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password --table SITTS.ESB_SERVICE_PARAM --split-by PARAM_ID --hbase-table lsy_test_multiple_table --hbase-row-key PARAM_ID,SERVICE_ID --column-family data
4.3验证
scan 'orcl_esb_service_param'
count 'orcl_esb_service_param'
5.导入hive
5.1新建hive表，并导入数据
bin/sqoop import --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password --table SITTS.ESB_SERVICE_PARAM --split-by PARAM_ID --hive-import --create-hive-table --target-dir /user/hive/warehouse/orcl_esb_service_param --hive-table orcl_esb_service_param
5.2 验证
show tables;
select * from orcl_esb_service_param;
select count(*) from orcl_esb_service_param;
6.增量
增量到hdfs
bin/sqoop import --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password --table SITTS.ESB_SERVICE_PARAM --target-dir /sqoop/oracle/esb_service_param --split-by PARAM_ID --incremental append --check-column PARAM_ID --last-value 33
返回
16/09/07 15:27:17 INFO tool.ImportTool: --incremental append
16/09/07 15:27:17 INFO tool.ImportTool: --check-column PARAM_ID
16/09/07 15:27:17 INFO tool.ImportTool: --last-value 34
16/09/07 15:27:17 INFO tool.ImportTool: (Consider saving this with 'sqoop job --create')
校验
hdfs dfs -cat /sqoop/oracle/esb_service_param/part-m-00004
增量到hbase
bin/sqoop import --connect jdbc:oracle:thin:@192.168.16.223:1521/orcl --username sitts --password password --table SITTS.ESB_SERVICE_PARAM --split-by PARAM_ID --hbase-table orcl_esb_service_param --hbase-row-key PARAM_ID --column-family region --incremental append --check-column PARAM_ID --last-value 33
返回
16/09/07 15:59:49 INFO tool.ImportTool: --incremental append
16/09/07 15:59:49 INFO tool.ImportTool: --check-column PARAM_ID
16/09/07 15:59:49 INFO tool.ImportTool: --last-value 34
16/09/07 15:59:49 INFO tool.ImportTool: (Consider saving this with 'sqoop job --create')
校验
scan 'orcl_esb_service_param'
网友评论