HDFS小文件合并时遇到的几个小问题

1. 如何“非正则”地遍历一个目录下某些特定的文件？

在spark中newHadoopAPI()或者FSShell其实都是支持正则话输入路径的，不要怕输入的路径太长，一定要正则化输入路径；

• https://stackoverflow.com/questions/31782763/how-to-use-regex-to-include-exclude-some-input-files-in-sc-textfile
• https://stackoverflow.com/questions/24224392/why-does-spark-throw-notserializableexception-org-apache-hadoop-io-nullwritable

2. 在读取->repartition->重新再写入的过程中，遇到序列化失败该如何处理？

尽量使用kyro序列化，兼容性还是比较好；
同时注意null值的处理；

• http://apache-spark-developers-list.1001551.n3.nabble.com/NullWritable-not-serializable-td8412.html
• https://stackoverflow.com/questions/22592811/task-not-serializable-java-io-notserializableexception-when-calling-function-ou/22594142#22594142
• https://issues.apache.org/jira/browse/SPARK-2421
• https://spark.apache.org/docs/latest/tuning.html