【Spark】Spark应用报错及解决

【1】 Expected only partition pruning predicates
解决方案：设置spark.sql.hive.metastorePartitionPruning=false

【2】 Error in query: Detected cartesian product for INNER join between logical plans
Project-Join condition is missing or trivial.
Use the CROSS JOIN syntax to allow cartesian products between these relations
解决方案：设置spark.sql.crossJoin.enabled=true

【3】 ERROR ApplicationMaster: User class threw exception: java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]

• image.png
• image.png

解决方案：设置spark.sql.autoBroadcastJoinThreshold为-1，尝试关闭BroadCast Join

【4】org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLException: No suitable driver found for
bigquery打包时，生成了spark-1.0.3的包，用它起thriftserver，里面逻辑涉及到访问mysql时，报No suitable driver found for错误，看错误是没拿到mysql的url。检查jar包，common模块的resource中的配置文件没有打进去，看了下是spark模块pom.xml打包时没有加上common模块的resource包。

image.png