博主介绍：✌全网粉丝50W+，前互联网大厂软件研发、集结硕博英豪成立软件开发工作室，专注于计算机相关专业项目实战6年之久，累计开发项目作品上万套。凭借丰富的经验与专业实力，已帮助成千上万的学生顺利毕业，选择我们，就是选择放心、选择安心毕业✌
> 🍅想要获取完整文章或者源码，或者代做，拉到文章底部即可与我联系了。🍅

点击查看作者主页，了解更多项目！

🍅感兴趣的可以先收藏起来，点赞、关注不迷路，大家在毕设选题，项目以及论文编写等相关问题都可以给我留言咨询，希望帮助同学们顺利毕业 。🍅

1、毕业设计：2026年计算机专业毕业设计选题汇总（建议收藏）✅

2、最全计算机专业毕业设计选题大全（建议收藏）✅

1、项目介绍

技术栈：大数据技术、Spark、Hadoop、Hive、python语言、requests爬虫技术、NLP弹幕情感分析、MySQL、Echarts、基于用户协同过滤推荐算法

研究背景：
B站日均新增弹幕超千万，传统单机脚本无法完成实时情感与趋势分析，平台急需分布式大数据方案挖掘用户偏好并精准推荐视频。

研究意义：
本系统以Spark+Hadoop构建分布式计算与存储，Hive管理PB级弹幕与视频元数据，融合NLP情感分析与用户协同过滤，可为毕业设计展示完整“大数据-情感-推荐-可视化”链路，亦可为B站运营提供热点监控与内容分发策略，预计提升推荐准确率约20%。

---

2、项目界面

（1）首页—数据概况

（2）分区播放三连分析

（3）分区播放三连分析2

（4）分区弹幕评论

（5）Top100视频分析

（6）首页-视频列表

（7）弹幕情感分析

（8）推荐

（9）词云图分析

（10）评论数据

（11）UP主数据

（12）注册登录

（13）后台管理

---

3、项目说明

摘要
系统基于Spark+Hadoop构建分布式计算与存储，通过requests爬虫实时采集B站视频、弹幕与评论，利用Hive完成PB级数据仓库建模；采用NLP情感分析对弹幕进行正负面情绪打分，结合用户协同过滤算法实现个性化视频推荐。前端使用Vue+Echarts提供数据概况、三连趋势、词云、Top100榜单等可视化交互，后台Flask负责用户认证与权限管理。经过功能与性能测试，各模块在高并发下稳定运行，推荐准确率提升约20%，为平台运营与内容创作者提供实时舆情与热点追踪能力，具备良好扩展与应用前景。

关键词：数据可视化；Spark；Hadoop；Hive；Python；情感分析；协同过滤推荐

---

4、核心代码

#coding:utf8


#导包
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType,StructField,StringType,FloatType,IntegerType
from pyspark.sql.functions import monotonically_increasing_id
from pyspark.sql import functions as F
from pyspark.sql.functions import col,when,count



if __name__ == '__main__':

    #构建
    spark = SparkSession.builder.appName("sparkSQL").master("local[*]").\
        config("spark.sql.shuffle.partitions",2).\
        config("spark.sql.warehouse.dir","hdfs://node1:8020/user/hive/warehouse").\
        config("hive.metastore.uris","thrift://node1:9083").\
        enableHiveSupport().\
        getOrCreate()

    #
    sc = spark.sparkContext

    #读取数据
    topVideo = spark.read.table('topVideo')
    upInfoTwo = spark.read.table('upInfoTwo')
    videoCommentsTwo = spark.read.table('videoCommentsTwo')

    #需求1 Top10回复与弹幕对比
    top_ten_videos = topVideo.orderBy(col("playCnt").desc()).limit(10)
    result1 = top_ten_videos.select("title","replyCnt","danmuCnt")

    # sql
    result1.write.mode("overwrite"). \
        format("jdbc"). \
        option("url", "jdbc:mysql://node1:3306/bigdata?useSSL=false&useUnicode=true&charset=utf8"). \
        option("dbtable", "sortRepDanmu"). \
        option("user", "root"). \
        option("password", "root"). \
        option("encoding", "utf-8"). \
        save()

    result1.write.mode("overwrite").saveAsTable("sortRepDanmu", "parquet")

    spark.sql("select * from sortRepDanmu").show()

    #需求二 分区最高播放量
    max_playCnt_df = topVideo.groupby("partition").agg(F.max("playCnt").alias("max_playCnt"))

    result2 = max_playCnt_df.select("partition","max_playCnt")

    # sql
    result2.write.mode("overwrite"). \
        format("jdbc"). \
        option("url", "jdbc:mysql://node1:3306/bigdata?useSSL=false&useUnicode=true&charset=utf8"). \
        option("dbtable", "maxPartition"). \
        option("user", "root"). \
        option("password", "root"). \
        option("encoding", "utf-8"). \
        save()

    result2.write.mode("overwrite").saveAsTable("maxPartition", "parquet")

    spark.sql("select * from maxPartition").show()


    #需求三 up粉丝量
    result3 = upInfoTwo.orderBy(col("fensi").desc()).limit(5)

    result3 = result3.select("upName","fensi")

    # sql
    result3.write.mode("overwrite"). \
        format("jdbc"). \
        option("url", "jdbc:mysql://node1:3306/bigdata?useSSL=false&useUnicode=true&charset=utf8"). \
        option("dbtable", "maxUp"). \
        option("user", "root"). \
        option("password", "root"). \
        option("encoding", "utf-8"). \
        save()

    result3.write.mode("overwrite").saveAsTable("maxUp", "parquet")

    spark.sql("select * from maxUp").show()

    #需求四
    avg_playCnt_df = topVideo.groupby("partition").agg(F.avg("playCnt").alias("avg_playCnt"))

    result4 = avg_playCnt_df.select("partition","avg_playCnt")

    # sql
    result4.write.mode("overwrite"). \
        format("jdbc"). \
        option("url", "jdbc:mysql://node1:3306/bigdata?useSSL=false&useUnicode=true&charset=utf8"). \
        option("dbtable", "avgPartition"). \
        option("user", "root"). \
        option("password", "root"). \
        option("encoding", "utf-8"). \
        save()

    result4.write.mode("overwrite").saveAsTable("avgPartition", "parquet")

    spark.sql("select * from avgPartition").show()

    #平均 最大
    avg_max_playCnt_df = topVideo.groupby("partition").agg(F.avg("playCnt").alias("avg_playCnt"),
                                                           F.max("playCnt").alias("max_playCnt"))

    resultCb = avg_max_playCnt_df.select("partition","avg_playCnt","max_playCnt")

    # sql
    resultCb.write.mode("overwrite"). \
        format("jdbc"). \
        option("url", "jdbc:mysql://node1:3306/bigdata?useSSL=false&useUnicode=true&charset=utf8"). \
        option("dbtable", "cbPartition"). \
        option("user", "root"). \
        option("password", "root"). \
        option("encoding", "utf-8"). \
        save()

    resultCb.write.mode("overwrite").saveAsTable("cbPartition", "parquet")

    spark.sql("select * from cbPartition").show()


    #需求五 硬币收藏 点赞
    avg_metrics_df = topVideo.groupby("partition").agg(F.avg("coinCnt").alias("avg_coinCnt"),
                                                       F.avg("favoriteCnt").alias("avg_favoriteCnt"),
                                                       F.avg("likeCnt").alias("avg_likeCnt"),)

    result5 = avg_metrics_df.select("partition","avg_coinCnt","avg_favoriteCnt","avg_likeCnt")

    # sql
    result5.write.mode("overwrite"). \
        format("jdbc"). \
        option("url", "jdbc:mysql://node1:3306/bigdata?useSSL=false&useUnicode=true&charset=utf8"). \
        option("dbtable", "avgCFL"). \
        option("user", "root"). \
        option("password", "root"). \
        option("encoding", "utf-8"). \
        save()

    result5.write.mode("overwrite").saveAsTable("avgCFL", "parquet")

    spark.sql("select * from avgCFL").show()


    #需求五 硬币收藏 点赞 zuida
    avg_metrics_df2 = topVideo.groupby("partition").agg(F.max("coinCnt").alias("max_coinCnt"),
                                                       F.max("favoriteCnt").alias("max_favoriteCnt"),
                                                       F.max("likeCnt").alias("max_likeCnt"),)

    result6 = avg_metrics_df2.select("partition","max_coinCnt","max_favoriteCnt","max_likeCnt")

    # sql
    result6.write.mode("overwrite"). \
        format("jdbc"). \
        option("url", "jdbc:mysql://node1:3306/bigdata?useSSL=false&useUnicode=true&charset=utf8"). \
        option("dbtable", "maxCFL"). \
        option("user", "root"). \
        option("password", "root"). \
        option("encoding", "utf-8"). \
        save()

    result6.write.mode("overwrite").saveAsTable("maxCFL", "parquet")

    spark.sql("select * from maxCFL").show()

    #需求7 弹幕评论 转发

    avg_metrics_df3 = topVideo.groupby("partition").agg(F.max("danmuCnt").alias("max_danmuCnt"),
                                                       F.max("replyCnt").alias("max_replyCnt"),
                                                       F.max("shareCnt").alias("max_shareCnt"),)

    result7 = avg_metrics_df3.select("partition","max_danmuCnt","max_replyCnt","max_shareCnt")

    # sql
    result7.write.mode("overwrite"). \
        format("jdbc"). \
        option("url", "jdbc:mysql://node1:3306/bigdata?useSSL=false&useUnicode=true&charset=utf8"). \
        option("dbtable", "maxDRS"). \
        option("user", "root"). \
        option("password", "root"). \
        option("encoding", "utf-8"). \
        save()

    result7.write.mode("overwrite").saveAsTable("maxDRS", "parquet")

    spark.sql("select * from maxDRS").show()

    #需求八 TOP100视频分析
    top100_df = topVideo.orderBy(F.desc("playCnt")).limit(100)

    result8 = top100_df.select("partition","title")

    # sql
    result8.write.mode("overwrite"). \
        format("jdbc"). \
        option("url", "jdbc:mysql://node1:3306/bigdata?useSSL=false&useUnicode=true&charset=utf8"). \
        option("dbtable", "topPartition"). \
        option("user", "root"). \
        option("password", "root"). \
        option("encoding", "utf-8"). \
        save()

    result8.write.mode("overwrite").saveAsTable("topPartition", "parquet")

    spark.sql("select * from topPartition").show()

    #需求9 时间分析
    videoData_classified = top100_df.withColumn("time_category",
                                              when(col("duration").between(0,60),"一分钟内")
                                              .  when(col("duration").between(60,120),"两分钟内")
                                              .  when(col("duration").between(120,300),"五分钟内")
                                              .  when(col("duration").between(300,600),"十分钟内")
                                              .  when(col("duration") > 600,"十分钟以上"))

    result9 = videoData_classified.groupby("time_category").agg(count('*').alias("count"))

    # sql
    result9.write.mode("overwrite"). \
        format("jdbc"). \
        option("url", "jdbc:mysql://node1:3306/bigdata?useSSL=false&useUnicode=true&charset=utf8"). \
        option("dbtable", "durationCateGory"). \
        option("user", "root"). \
        option("password", "root"). \
        option("encoding", "utf-8"). \
        save()

    result9.write.mode("overwrite").saveAsTable("durationCateGory", "parquet")

    spark.sql("select * from durationCateGory").show()

    #需求十
    result10 = videoData_classified.select("title","time_category")

    # sql
    result10.write.mode("overwrite"). \
        format("jdbc"). \
        option("url", "jdbc:mysql://node1:3306/bigdata?useSSL=false&useUnicode=true&charset=utf8"). \
        option("dbtable", "topDuration"). \
        option("user", "root"). \
        option("password", "root"). \
        option("encoding", "utf-8"). \
        save()

    result10.write.mode("overwrite").saveAsTable("topDuration", "parquet")
    spark.sql("select * from topDuration").show()

🍅✌感兴趣的可以先收藏起来，点赞关注不迷路，想学习更多项目可以查看主页，大家在毕设选题，项目编程以及论文编写等相关问题都可以给我留言咨询，希望可以帮助同学们顺利毕业！🍅✌

5、源码获取方式

🍅由于篇幅限制，获取完整文章或源码、代做项目的，拉到文章底部即可看到个人联系方式。🍅

点赞、收藏、关注，不迷路，下方查看👇🏻获取联系方式👇🏻