一、Local sort设计

二、localSort BenchMark测试
测试代码sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/LocalSortBenchmark.scala
基于50万条数据测试情况
1、'none' compress
Create local-sort table or not and use 'none' compress: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative------------------------------------------------------------------------------------------------Create no local-sort table 829 / 930 -0.0 -828575541.0 1.0XCreate local-sort table with one column 687 / 725 -0.0 -686940503.0 1.2XCreate local-sort table with two columns 702 / 727 -0.0 -702051948.0 1.2XUse local-sort table or not and 'none' compress: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative------------------------------------------------------------------------------------------------Use no local-sort table 727 / 830 -0.0 -727380314.0 1.0XUse local-sort table with one column 646 / 771 -0.0 -646038168.0 1.1XUse local-sort table with two columns 595 / 659 -0.0 -594576302.0 1.2X
2、'uncompressed' compress
Create local-sort table or not and use 'uncompressed' compress: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative------------------------------------------------------------------------------------------------Create no local-sort table 646 / 670 -0.0 -645599722.0 1.0XCreate local-sort table with one column 657 / 695 -0.0 -656976360.0 1.0XCreate local-sort table with two columns 640 / 682 -0.0 -639984996.0 1.0XUse local-sort table or not and 'uncompressed' compress: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative------------------------------------------------------------------------------------------------Use no local-sort table 652 / 733 -0.0 -652089070.0 1.0XUse local-sort table with one column 538 / 613 -0.0 -538297266.0 1.2XUse local-sort table with two columns 521 / 549 -0.0 -520702701.0 1.3X
3、'snappy' compress
Create local-sort table or not and use 'snappy' compress: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative------------------------------------------------------------------------------------------------Create no local-sort table 657 / 674 -0.0 -656778039.0 1.0XCreate local-sort table with one column 695 / 736 -0.0 -695051941.0 0.9XCreate local-sort table with two columns 654 / 698 -0.0 -653894042.0 1.0XUse local-sort table or not and 'snappy' compress: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative------------------------------------------------------------------------------------------------Use no local-sort table 504 / 562 -0.0 -503619019.0 1.0XUse local-sort table with one column 454 / 508 -0.0 -453511415.0 1.1XUse local-sort table with two columns 482 / 531 -0.0 -481824674.0 1.0X
4、'zlib' compress
Create local-sort table or not and use 'zlib' compress: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative------------------------------------------------------------------------------------------------Create no local-sort table 724 / 737 -0.0 -723502111.0 1.0XCreate local-sort table with one column 739 / 741 -0.0 -738642143.0 1.0XCreate local-sort table with two columns 719 / 731 -0.0 -718866255.0 1.0XUse local-sort table or not and 'zlib' compress: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative------------------------------------------------------------------------------------------------Use no local-sort table 465 / 575 -0.0 -465250793.0 1.0XUse local-sort table with one column 467 / 519 -0.0 -467351210.0 1.0XUse local-sort table with two columns 461 / 523 -0.0 -460964183.0 1.0X
5、'lzo' compress
Create local-sort table or not and use 'lzo' compress: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative------------------------------------------------------------------------------------------------Create no local-sort table 681 / 692 -0.0 -681079691.0 1.0XCreate local-sort table with one column 695 / 738 -0.0 -695109679.0 1.0XCreate local-sort table with two columns 661 / 781 -0.0 -660778989.0 1.0XUse local-sort table or not and 'lzo' compress: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative------------------------------------------------------------------------------------------------Use no local-sort table 498 / 560 -0.0 -498172655.0 1.0XUse local-sort table with one column 534 / 778 -0.0 -534378198.0 0.9XUse local-sort table with two columns 516 / 581 -0.0 -515501123.0 1.0X
网友评论