采用集成了scala的eclipse编写代码

代码：

package wordcount

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext

object WordCount {
  
  def main(args: Array[String]): Unit = {
    //非常重要，是通向Spark集群的入口
    val conf=new SparkConf().setAppName("WC")
    val sc=new SparkContext(conf)
    
    sc.textFile(args(0)).flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_)
    .sortBy(_._2, false).saveAsTextFile(args(1))
    
    sc.stop()
  }
  
  
}

导出成jar包

上传到服务器

提交命令：

./spark-submit --master spark://nbdo1:7077 

--class wordcount.WordCount

 /home/hadoop/wordcount.jar 

"hdfs://nbdo1:9000/wordcount.txt" "hdfs://nbdo1:9000/out3"

运行效果：

[hadoop@nbdo1 ~]$ hdfs dfs -cat /out3/part-*
(hello,6)
(zeng,4)
(ting,2)
(miao,2)
(gen,2)
(wen,2)
(biao,2)
(zhu,2)
(ye,1)
(,1)
(zhang,1)
(ai,1)
(lai,1)
(su,1)
(qi,1)
(sheng,1)
(xiao,1)
(xiang,1)
(lu,1)
(chang,1)
(ni,1)

-------------

更多的Java，Android，大数据，J2EE，Python，数据库，Linux，Java架构师，教程，视频请访问：

http://www.cnblogs.com/zengmiaogen/p/7083694.html