hive3.1.0读取paimon1.2表报错问题zstd问题,java.lang.UnsatisfiedLinkError: com.github.luben.zstd.ZstdInputStreamNoFinalizer.recommendedDInSize()J

完整的报错内容:

> select * from test_visit_log limit 2;
OK
id      tags    event_original  service log_file_path   version host    message timestamp       event_date
Exception in thread "main" java.lang.UnsatisfiedLinkError: com.github.luben.zstd.ZstdInputStreamNoFinalizer.recommendedDInSize()J
        at com.github.luben.zstd.ZstdInputStreamNoFinalizer.recommendedDInSize(Native Method)
        at com.github.luben.zstd.ZstdInputStreamNoFinalizer.<clinit>(ZstdInputStreamNoFinalizer.java:35)
        at org.apache.paimon.shade.org.apache.avro.file.ZstandardLoader.input(ZstandardLoader.java:38)
        at org.apache.paimon.shade.org.apache.avro.file.ZstandardCodec.decompress(ZstandardCodec.java:84)
        at org.apache.paimon.shade.org.apache.avro.file.DataFileStream$DataBlock.decompressUsing(DataFileStream.java:393)
        at org.apache.paimon.shade.org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:226)
        at org.apache.paimon.format.avro.AvroBulkFormat.replaceAvroRuntimeException(AvroBulkFormat.java:156)
        at org.apache.paimon.format.avro.AvroBulkFormat.access$200(AvroBulkFormat.java:43)
        at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.readNextBlock(AvroBulkFormat.java:119)
        at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.readBatch(AvroBulkFormat.java:103)
        at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.readBatch(AvroBulkFormat.java:57)
        at org.apache.paimon.reader.RecordReaderIterator.<init>(RecordReaderIterator.java:37)
        at org.apache.paimon.reader.RecordReader.toCloseableIterator(RecordReader.java:210)
        at org.apache.paimon.utils.ObjectsFile.createIterator(ObjectsFile.java:186)
        at org.apache.paimon.utils.ObjectsFile.readWithIOException(ObjectsFile.java:153)
        at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:134)
        at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:105)
        at org.apache.paimon.manifest.ManifestList.readDataManifests(ManifestList.java:90)
        at org.apache.paimon.operation.ManifestsReader.readManifests(ManifestsReader.java:128)
        at org.apache.paimon.operation.ManifestsReader.read(ManifestsReader.java:114)
        at org.apache.paimon.operation.AbstractFileStoreScan.readManifests(AbstractFileStoreScan.java:417)
        at org.apache.paimon.operation.AbstractFileStoreScan.plan(AbstractFileStoreScan.java:257)
        at org.apache.paimon.table.source.snapshot.SnapshotReaderImpl.read(SnapshotReaderImpl.java:318)
        at org.apache.paimon.table.source.snapshot.ReadPlanStartingScanner.scan(ReadPlanStartingScanner.java:45)
        at org.apache.paimon.table.source.DataTableBatchScan.plan(DataTableBatchScan.java:79)
        at org.apache.paimon.hive.utils.HiveSplitGenerator.generateSplits(HiveSplitGenerator.java:100)
        at org.apache.paimon.hive.mapred.PaimonInputFormat.getSplits(PaimonInputFormat.java:45)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.generateWrappedSplits(FetchOperator.java:425)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:395)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:314)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:540)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509)
        at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
        at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2691)
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
[root@sc001 ~]#

整合依赖的jar包
paimon-hive-connector-3.1-1.2-20250516.003010-31.jar
目录是:hive/auxlibs
主要问题是 没有 zstd的jar包
http://github.com/apache/paimon/issues/3492
需要增加
zstd-jni-1.5.6-10.jar
放到 hive/lib下重启hive就可以

Logo

腾讯云面向开发者汇聚海量精品云计算使用和开发经验,营造开放的云计算技术生态圈。

更多推荐