**

hive on spark 执行报错问题解决

**

原因一

Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client due to invalid resource request: Required executor memory (8192 MB), offHeap memory (0) MB, overhead (819 MB), and PySpark memory (0 MB) is above the max threshold (8192 MB))'

由于物理资源不足，一般就是内存、磁盘，或者修改 hive/conf/spark-defaults.conf 中的配置

原因二

hive提示

Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 3b65a4c8-8e36-4191-8ab0-b22bb4f819c6)'
FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 3b65a4c8-8e36-4191-8ab0-b22bb4f819c6

查看 yarn的resourcemanager日志

Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
错误: 找不到或无法加载主类 org.apache.spark.deploy.yarn.ApplicationMaster

原因：由于hive/conf/spark-defaults.conf 设置了 spark.eventLog.dir，需要hdbs创建 spark-history 目录，并上传 /xxx/spark-3.3.0-bin-without-hadoop/jars/ 中的包

spark.master                               yarn
spark.eventLog.enabled                   true
spark.eventLog.dir                        hdfs://hadoop102:9000/spark-history
spark.executor.memory                    1g
spark.driver.memory					   1g

将包传到hdfs上
hadoop fs -put jars/* /spark-jars

原因三

查看 http://hadoop:8088/cluster/apps 调度失败日志，或 resourcemanager日志

Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :

由于hive 与 spark 版本不一致导致，gitee / github 查看你所用版本hive的pom 中 <spark.version>，这时候就麻烦了！！！需要下hive源码去修改pom依赖重新编译
查看此文章