Java调用hadoop时出现报错:

java.lang.IllegalArgumentException: Wrong FS: hdfs://XXX:9000, expected: file:///

错误分析:

hdfs是一种uri路径,hadoop并不认识hdfs:这种格式,后面提示:file:/// 这是一种本地文件系统,由此可见这里要求我们传入一个本地文件系统。

解决方案:

1.hdfs是我们在core-site中进行配置的,而这个文件是存放在服务器中的,所以无法读取配置文件,我们可以将core-site文件copy到工程目录的src下,这样Configuration conf =new Configuration();时就会读取配置文件。
目录结构如下图:

项目目录图
获取文件系统代码如下:

Configuration conf = new Configuration();
Path path = new Path("hdfs://whb01:9000/jdk9.tar");
FileSystem fs = FileSystem.get(conf);//上下不同点
FSDataInputStream open = fs.open(path);
System.out.println(open);
2.hadoop为我们提供了分布式方式的解决方案,不需要复制配置文件,只需要通过path进行创建filesystem
Configuration conf = new Configuration();
Path path = new Path("hdfs://whb01:9000/jdk9.tar");
FileSystem fs = path.getFileSystem(conf);//上下不同点
FSDataInputStream open = fs.open(path);
System.out.println(open);
3.conf对象中指定配置信息
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://whb01:9000/");
Path path = new Path("hdfs://whb01:9000/jdk9.tar");
//FileSystem fs = path.getFileSystem(conf);
FileSystem fs = FileSystem.get(conf);
FSDataInputStream open = fs.open(path);

完整报错:

java.lang.IllegalArgumentException: Wrong FS: hdfs://whb01:9000, expected: file:///
    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:666)
    at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:86)
    at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:630)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:861)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:625)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:435)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:146)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:347)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:786)
    at top.whbweb.hadoop.HadoopComm.test(HadoopComm.java:20)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

Logo

腾讯云面向开发者汇聚海量精品云计算使用和开发经验,营造开放的云计算技术生态圈。

更多推荐