1.log4j.properties（spark本地配置）

在D:\spark\spark-core\src\main下新建resources目录 log4j.properties文件，然后复制以下信息

## 声明Apache许可证（标准头部，无需修改）
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# Set everything to be logged to the console
# 核心配置：根日志级别为WARN，输出到控制台
log4j.rootCategory=WARN, console
# 配置控制台输出的Appender
log4j.appender.console=org.apache.log4j.ConsoleAppender
# 输出目标为标准错误流（System.err）
log4j.appender.console.target=System.err
# 日志格式使用PatternLayout
log4j.appender.console.layout=org.apache.log4j.PatternLayout
# 日志格式：时间 日志级别 类名: 日志信息（换行）
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
# spark-shell默认日志级别设为WARN（覆盖根日志级别）
log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose
# 抑制第三方组件的冗余日志
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
# 修复你配置中连在一起的行（拆分并统一）
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
# 抑制parquet相关日志（只输出ERROR级别）
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
# SPARK-9183：解决SparkSQL+Hive查找不存在的UDF时的冗余提示
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

日志配置关键说明：
log4j.rootCategory=WARN,console：全局日志级别为WARN（只输出WARN/ERROR/FATAL级别的日志），输出到控制台；如果需要更详细的日志（比如调试），可改为INFO/DEBUG。
ConversionPattern：定义日志的输出格式，%d是时间、%p是日志级别、%c{1}是简化的类名、%m%n是日志内容+换行。
第三方日志抑制：针对Jetty、Parquet、Hive等组件的日志级别单独设置，避免冗余日志刷屏。

2.pom.xml设置

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>spark</artifactId>
        <groupId>org.example</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>spark-core</artifactId>
<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>3.0.0</version>
    </dependency>

    <dependency>
        <groupId>joda-time</groupId>
        --<!-- Joda时间工具（处理时间日期） -->
        <artifactId>joda-time</artifactId> 
        <version>2.9.7</version>
    </dependency>

    <dependency>
        <groupId>mysql</groupId>
         <!-- MySQL驱动（连接MySQL数据库） -->
        <artifactId>mysql-connector-java</artifactId> --导入mysql
        <version>5.1.44</version>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.12</artifactId> --导入spark-sql
        <version>3.0.0</version>
    </dependency>
    <!--    <dependency>-->
    <!--        <groupId>com.redislabs</groupId>-->
    <!--        <artifactId>spark-redis</artifactId>-->
    <!--        <version>2.3.1-RC1</version>-->
    <!--    </dependency>-->


    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_2.12</artifactId>
        <version>3.0.0</version>
        <!--<scope>provided</scope>-->
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.12</artifactId>
        <version>3.0.0</version>
    </dependency>

    <dependency>
        <groupId>com.redislabs</groupId>
        <artifactId>spark-redis_2.12</artifactId>
        <version>3.0.0</version>
    </dependency>
</dependencies>


<!--    <properties>-->
<!--        <maven.compiler.source>8</maven.compiler.source>-->
<!--        <maven.compiler.target>8</maven.compiler.target>-->
<!--    </properties>-->

</project>