提交spark任务时一直报错:No suitable driver
提交任务命令如下

spark-submit \
--driver-class-path /opt/sparkJob/mysql-connector-java-5.1.47.jar \
--master yarn \
--deploy-mode cluster \
--driver-memory 4G \
--executor-memory 2G \
--total-executor-cores 12 \
--class com.xxx.xxx.Demo /opt/sparkJob/xxx.jar

每次执行这个命令都会报错,经过查询资料,尝试了加入--jars /opt/sparkJob/mysql-connector-java-5.1.47.jar也不行,后来又看到相关资料,还需要加入--conf spark.executor.extraClassPath=/opt/sparkJob/mysql-connector-java-5.1.47.jar才可以
完整命令如下

spark-submit \
--jars /opt/sparkJob/mysql-connector-java-5.1.47.jar \
--driver-class-path /opt/sparkJob/mysql-connector-java-5.1.47.jar \
--conf spark.executor.extraClassPath=/opt/sparkJob/mysql-connector-java-5.1.47.jar \
--master yarn \
--deploy-mode cluster \
--driver-memory 4G \
--executor-memory 2G \
--total-executor-cores 12 \
--class com.xxx.xxx.Demo /opt/sparkJob/xxx.jar

这三个必须都加上才可以,即--jars--driver-class-path--conf

Logo

腾讯云面向开发者汇聚海量精品云计算使用和开发经验,营造开放的云计算技术生态圈。

更多推荐