CarbonData supports read and wri

1、CarbonData SDK with OSS

Prerequisites

• Building CarbonData. Alternatively you could get it from Carbondata website.
• Hadoop should be installed.
• OSS bucket, access key and secret key creation.

1.1 run sdk example to read and write with OSS

java -classpath /home/carbondata/carbondata/assembly/target/scala-2.11/apache-carbondata-1.6.0-SNAPSHOT-bin-spark2.2.1-hadoop2.7.2.jar:/home/carbondata/carbondata/examples/spark2/target/carbondata-examples-1.6.0-SNAPSHOT.jar:/home/carbondata/hadoop-2.7.2/share/hadoop/tools/lib/:/home/carbondata/hadoop-2.7.2/share/hadoop/common/:/home/carbondata/hadoop-2.7.2/share/hadoop/mapreduce/* org.apache.carbondata.examples.sdk.SDKS3Example <oss-access-key> <oss-secret-key> <oss-endpoint> <table-path-on-oss>
oss-endpoint example: oss-cn-shanghai.aliyuncs.com
table-path-on-oss: s3a://demo20190203/carbon/sdk/data

1.2 check oss write successfully

image.png

1.3 check oss read successfully

image.png

2、CarbonData session with OSS

Prerequisites

• Building CarbonData. Alternatively you could get it from Carbondata website.
• OSS bucket, access key and secret key creation.
• Installing and Configuring CarbonData on Spark on YARN Cluster.
• Create a sample.csv file using the following commands. The CSV file is required for loading data into CarbonData

cd carbondata
cat > sample.csv << EOF
id,name,city,age
1,david,shenzhen,31
2,eason,shenzhen,27
3,jarry,wuhan,35
EOF
#上传sample.csv至hdfs
hadoop fs -put ./sample.csv /tmp

1.1 run carbondata thrift server

./spark-2.2.1-bin-hadoop2.7/bin/spark-submit --class org.apache.carbondata.spark.thriftserver.CarbonThriftServer $SPARK_HOME/carbonlib/apache-carbondata-1.6.0-SNAPSHOT-bin-spark2.2.1-hadoop2.7.2.jar <table-path-on-oss> <oss-access-key> <oss-secret-key> <oss-endpoint>
oss-endpoint example: oss-cn-shanghai.aliyuncs.com
table-path-on-oss: s3a://demo20190203/carbon/data/store

1.2 use beeline to connect thrift server

cd $SPARK_HOME
./bin/beeline -u jdbc:hive2://localhost:10000

1.3 create table in beeline

CREATE TABLE IF NOT EXISTS test_table ( id string, name string,city string,age Int) STORED AS carbondata LOCATION 's3a://demo20190203/carbon/session/data/store';

check oss data:

image.png

1.4 Load data into oss in beeline

LOAD DATA INPATH '[hdfs://localhost:9000/tmp/sample.csv'](hdfs://localhost:9000/tmp/sample.csv') INTO TABLE test_table;

check oss data:

image.png image.png

1.5 Read data fron oss in beeline

SELECT city, avg(age), sum(age) FROM test_table GROUP BY city;

check oss data:

image.png