Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.util.Utils$.classForName(Ljava/lang/String;)Ljava/lang/Class;
    at org.apache.spark.streaming.scheduler.JobGenerator.liftedTree1$1(JobGenerator.scala:52)
    at org.apache.spark.streaming.scheduler.JobGenerator.<init>(JobGenerator.scala:51)
    at org.apache.spark.streaming.scheduler.JobScheduler.<init>(JobScheduler.scala:55)
    at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:184)
    at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:85)
    at streaming.SparkStream01_WordCount$.main(SparkStream01_WordCount.scala:20)
    at streaming.SparkStream01_WordCount.main(SparkStream01_WordCount.scala)
 

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.util.Utils$.classForName(Ljava/lang/String;)Ljava/lang/Class;
	at org.apache.spark.streaming.scheduler.JobGenerator.liftedTree1$1(JobGenerator.scala:52)
	at org.apache.spark.streaming.scheduler.JobGenerator.<init>(JobGenerator.scala:51)
	at org.apache.spark.streaming.scheduler.JobScheduler.<init>(JobScheduler.scala:55)
	at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:184)
	at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:85)
	at streaming.SparkStream01_WordCount$.main(SparkStream01_WordCount.scala:20)
	at streaming.SparkStream01_WordCount.main(SparkStream01_WordCount.scala)

原因  maven 导入的依赖版本问题

1.因为IDEA-maven的依赖包是直接在pom文件中添加代码,在初次使用时,很容易出现版本不一致的情况;
2.我的程序是spark-scala程序,还需要spark依赖的版本与scala匹配,比如scala2.12只适用于spark2.4以上;

我的scala  SDK 是2.12

spark-streaming与scala版本对应

 

<!--2.4.8版本不可以-->
<!--        <dependency>-->
<!--            <groupId>org.apache.spark</groupId>-->
<!--            <artifactId>spark-streaming_2.12</artifactId>-->
<!--            <version>2.4.8</version>-->
<!--&lt;!&ndash;            <scope>provided</scope>&ndash;&gt;-->
<!--        </dependency>-->

<!--这个版本3.1.0可以-->
<!--        <dependency>-->
<!--            <groupId>org.apache.spark</groupId>-->
<!--            <artifactId>spark-streaming_2.12</artifactId>-->
<!--            <version>3.1.0</version>-->
<!--            &lt;!&ndash;            <scope>provided</scope>&ndash;&gt;-->
<!--        </dependency>-->

<!--        这个版本3.0.0可以-->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.12</artifactId>
            <version>3.0.0</version>
<!--            <scope>provided</scope>-->
        </dependency>

Logo

腾讯云面向开发者汇聚海量精品云计算使用和开发经验,营造开放的云计算技术生态圈。

更多推荐