Hive中order by、sort by、distribute

Order By语法

对全局数据的排序，只有一个reduce
colOrder: ( ASC | DESC )
colNullOrder: (NULLS FIRST | NULLS LAST) -- (Note: Available in Hive 2.1.0 and later)
orderBy: ORDER BY colName colOrder? colNullOrder? (',' colName colOrder? colNullOrder?)*
query: SELECT expression (',' expression)* FROM src orderBy

order by 样例

select * from emp;

Sort By

对每一个Reduce内部进行排序，对全局结果集来说不是排序的
设置 reduce 执行的个数
set mapreduce.job.reduces=<number>

sort by样例

hive>set mapreduce.job.reduces=3;
hive>insert overwrite local directory '/opt/datas/hive_exp_emp0308' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY '\n'
select * from emp sort by empno asc

Distribute By

也就是分区partition，类似MapReduce中分区partition，对数据进行分区后，结合sort by 进行排序使用。

insert overwrite local directory '/opt/datas/hive_exp_distribute_emp0308' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY '\n'
select * from emp distribute by deptno sort by empno asc

第一个分区数据000000_0

第二个分区000001_0

第三个分区000002_0

Cluster By

当sort by 和 distribute by的字段相同时，就可以使用Cluster By替换。

insert overwrite local directory '/opt/datas/hive_exp_cluster_emp0308' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY '\n'
select * from emp cluster by empno

总结

Hive中select新特性
Order By 全局排序，一个Reduce
Sort By 每个reduce内部进行排序，全局不是排序 Distribute By 类似MR中partition，进行分区，结合sort by使用 Cluster By 当distribute和sort字段相同时，使用方式