报错信息如下:

hive> SELECT COUNT(*) FROM bigdata_user;
Query ID = hadoop_20251109114535_3b46daca-96a6-48a9-b485-b5c9d68b4edc
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Job running in-process (local Hadoop)
2025-11-09 11:45:36,957 Stage-1 map = 0%,  reduce = 0%
Ended Job = job_local1661400954_0013 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 

Stage-Stage-1:  HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

表现为:

        何涉及数据写入的操作都失败

        简单的测试表创建成功,但涉及数据操作的查询都失败。这表明问题出在 MapReduce 任务的数据写入阶段

解决:

        四个小时才找到解决方法!!!其死啦

        终端运行以下两条指令,重启所有服务

export HADOOP_CLIENT_OPTS="-Xmx2048m $HADOOP_CLIENT_OPTS"
export HADOOP_HEAPSIZE=${HADOOP_HEAPSIZE:-2048}

根本原因:   

        内存不足:复杂的 Hive 查询需要更多内存来执行 MapReduce 任务

        默认配置过低:Hadoop 默认的内存配置对于数据处理任务来说偏小

        JVM 堆限制:没有足够堆空间来处理大规模数据

Logo

腾讯云面向开发者汇聚海量精品云计算使用和开发经验,营造开放的云计算技术生态圈。

更多推荐