🚩无sudo权限服务器环境下深度学习C++/CUDA扩展编译报错解决实录【conda多版本gcc隔离最佳实践】

关键词:PyTorch、CUDA、conda、featup、C++扩展、gcc版本、环境变量、深度学习环境


1️⃣ 问题背景

最近在搭建 SimFeatUp 项目环境,执行 pip install git+https://github.com/likyoo/SimFeatUp.git 时一直报如下 C++/CUDA 编译相关错误:

Building wheel for featup (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [154 lines of output]
      /your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
      !!
      
              ********************************************************************************
              Please consider removing the following classifiers in favor of a SPDX license expression:
      
              License :: OSI Approved :: MIT License
      
              See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
              ********************************************************************************
      
      !!
        self._finalize_license_expression()
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-39/featup
      copying featup/__init__.py -> build/lib.linux-x86_64-cpython-39/featup
      copying featup/downsamplers.py -> build/lib.linux-x86_64-cpython-39/featup
      copying featup/layers.py -> build/lib.linux-x86_64-cpython-39/featup
      copying featup/losses.py -> build/lib.linux-x86_64-cpython-39/featup
      copying featup/plotting.py -> build/lib.linux-x86_64-cpython-39/featup
      copying featup/train_upsampler.py -> build/lib.linux-x86_64-cpython-39/featup
      copying featup/upsamplers.py -> build/lib.linux-x86_64-cpython-39/featup
      copying featup/util.py -> build/lib.linux-x86_64-cpython-39/featup
      creating build/lib.linux-x86_64-cpython-39/featup/adaptive_conv_cuda
      copying featup/adaptive_conv_cuda/__init__.py -> build/lib.linux-x86_64-cpython-39/featup/adaptive_conv_cuda
      copying featup/adaptive_conv_cuda/adaptive_conv.py -> build/lib.linux-x86_64-cpython-39/featup/adaptive_conv_cuda
      creating build/lib.linux-x86_64-cpython-39/featup/datasets
      copying featup/datasets/AID.py -> build/lib.linux-x86_64-cpython-39/featup/datasets
      copying featup/datasets/JitteredImage.py -> build/lib.linux-x86_64-cpython-39/featup/datasets
      copying featup/datasets/__init__.py -> build/lib.linux-x86_64-cpython-39/featup/datasets
      copying featup/datasets/util.py -> build/lib.linux-x86_64-cpython-39/featup/datasets
      creating build/lib.linux-x86_64-cpython-39/featup/featurizers
      copying featup/featurizers/CLIP.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
      copying featup/featurizers/DINO.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
      copying featup/featurizers/DINOv2.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
      copying featup/featurizers/DeepLabV3.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
      copying featup/featurizers/MAE.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
      copying featup/featurizers/MIDAS.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
      copying featup/featurizers/MaskCLIP.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
      copying featup/featurizers/ResNet.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
      copying featup/featurizers/XCLIP.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
      copying featup/featurizers/__init__.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
      copying featup/featurizers/util.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
      creating build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2
      copying featup/featurizers/dinov2/__init__.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2
      creating build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
      copying featup/featurizers/maskclip/__init__.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
      copying featup/featurizers/maskclip/clip.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
      copying featup/featurizers/maskclip/interpolate.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
      copying featup/featurizers/maskclip/model.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
      copying featup/featurizers/maskclip/simple_tokenizer.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
      copying featup/featurizers/maskclip/xclip.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
      copying featup/featurizers/maskclip/xmodel.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
      creating build/lib.linux-x86_64-cpython-39/featup/featurizers/modules
      copying featup/featurizers/modules/__init__.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/modules
      copying featup/featurizers/modules/layers.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/modules
      copying featup/featurizers/modules/resnet.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/modules
      copying featup/featurizers/modules/vgg.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/modules
      creating build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
      copying featup/featurizers/dinov2/layers/__init__.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
      copying featup/featurizers/dinov2/layers/attention.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
      copying featup/featurizers/dinov2/layers/block.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
      copying featup/featurizers/dinov2/layers/dino_head.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
      copying featup/featurizers/dinov2/layers/drop_path.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
      copying featup/featurizers/dinov2/layers/layer_scale.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
      copying featup/featurizers/dinov2/layers/mlp.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
      copying featup/featurizers/dinov2/layers/patch_embed.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
      copying featup/featurizers/dinov2/layers/swiglu_ffn.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
      running build_ext
      /your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/utils/cpp_extension.py:414: UserWarning: The detected CUDA version (12.4) has a minor version mismatch with the version that was used to compile PyTorch (12.1). Most likely this shouldn't be a problem.
        warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
      /your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/utils/cpp_extension.py:424: UserWarning: There are no g++ version bounds defined for CUDA version 12.4
        warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
      building 'adaptive_conv_cuda_impl' extension
      creating /tmp/pip-req-build-s0kol3am/build/temp.linux-x86_64-cpython-39/featup/adaptive_conv_cuda
      Emitting ninja build file /tmp/pip-req-build-s0kol3am/build/temp.linux-x86_64-cpython-39/build.ninja...
      Compiling objects...
      Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
      [1/2] c++ -MMD -MF /tmp/pip-req-build-s0kol3am/build/temp.linux-x86_64-cpython-39/featup/adaptive_conv_cuda/adaptive_conv_cuda.o.d -pthread -B /your/path/to/envs/SegEarth/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /your/path/to/envs/SegEarth/include -I/your/path/to/envs/SegEarth/include -fPIC -O2 -isystem /your/path/to/envs/SegEarth/include -fPIC -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/TH -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/THC -I/usr/local/cuda/include -I/your/path/to/envs/SegEarth/include/python3.9 -c -c /tmp/pip-req-build-s0kol3am/featup/adaptive_conv_cuda/adaptive_conv_cuda.cpp -o /tmp/pip-req-build-s0kol3am/build/temp.linux-x86_64-cpython-39/featup/adaptive_conv_cuda/adaptive_conv_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=adaptive_conv_cuda_impl -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
      [2/2] /usr/local/cuda/bin/nvcc  -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/TH -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/THC -I/usr/local/cuda/include -I/your/path/to/envs/SegEarth/include/python3.9 -c -c /tmp/pip-req-build-s0kol3am/featup/adaptive_conv_cuda/adaptive_conv_kernel.cu -o /tmp/pip-req-build-s0kol3am/build/temp.linux-x86_64-cpython-39/featup/adaptive_conv_cuda/adaptive_conv_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=adaptive_conv_cuda_impl -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 -std=c++17
      FAILED: /tmp/pip-req-build-s0kol3am/build/temp.linux-x86_64-cpython-39/featup/adaptive_conv_cuda/adaptive_conv_kernel.o
      /usr/local/cuda/bin/nvcc  -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/TH -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/THC -I/usr/local/cuda/include -I/your/path/to/envs/SegEarth/include/python3.9 -c -c /tmp/pip-req-build-s0kol3am/featup/adaptive_conv_cuda/adaptive_conv_kernel.cu -o /tmp/pip-req-build-s0kol3am/build/temp.linux-x86_64-cpython-39/featup/adaptive_conv_cuda/adaptive_conv_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=adaptive_conv_cuda_impl -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 -std=c++17
      /your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/ATen/core/boxing/impl/boxing.h:41:104: error: expected primary-expression before ‘>’ token
         41 | struct has_ivalue_to<T, guts::void_t<decltype(std::declval<IValue>().to<T>())>>
            |                                                                                                        ^
      /your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/ATen/core/boxing/impl/boxing.h:41:107: error: expected primary-expression before ‘)’ token
         41 | struct has_ivalue_to<T, guts::void_t<decltype(std::declval<IValue>().to<T>())>>
            |                                                                                                           ^
      ninja: build stopped: subcommand failed.
      Traceback (most recent call last):
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 2100, in _run_ninja_build
          subprocess.run(
        File "/your/path/to/envs/SegEarth/lib/python3.9/subprocess.py", line 528, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
      
      The above exception was the direct cause of the following exception:
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 35, in <module>
        File "/tmp/pip-req-build-s0kol3am/setup.py", line 4, in <module>
          setup(
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 186, in setup
          return run_commands(dist)
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
          dist.run_commands()
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
          self.run_command(cmd)
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/command/bdist_wheel.py", line 370, in run
          self.run_command("build")
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 99, in run
          _build_ext.run(self)
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
          self.build_extensions()
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 873, in build_extensions
          build_ext.build_extensions(self)
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
          self._build_extensions_serial()
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
          self.build_extension(ext)
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 264, in build_extension
          _build_ext.build_extension(self, ext)
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 565, in build_extension
          objects = self.compiler.compile(
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 686, in unix_wrap_ninja_compile
          _write_ninja_file_and_compile_objects(
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1774, in _write_ninja_file_and_compile_objects
          _run_ninja_build(
        File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 2116, in _run_ninja_build
          raise RuntimeError(message) from e
      RuntimeError: Error compiling objects for extension
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for featup
  Running setup.py clean for featup
Failed to build featup
ERROR: Failed to build installable wheels for some pyproject.toml based projects (featup)

  • 我的系统环境是 Ubuntu 24.04,Python 3.9,PyTorch 2.1.2,CUDA 12.1
  • gcc 版本为 13.3.0

2️⃣ 问题分析

  • 查了大量资料和论坛后发现,这类报错99%都是因为 gcc/g++ 版本太新,和 PyTorch/CUDA 的 C++ API 头文件兼容性有冲突。
  • PyTorch 官方一般推荐用 gcc 9.x 或 10.x 编译扩展,太新/太老都容易报错
  • 目前很多深度学习C++/CUDA扩展包(如featup、mmcv、dcn等)都需要遵循这个“gcc 9/10”惯例。

3️⃣ 尝试conda安装多版本gcc并隔离

优势:不污染全局环境!不同项目可用不同gcc。无sudo权限依然可以优雅隔离!

步骤一:conda安装gcc-9.5和g+±9.5

conda install -c conda-forge gcc_linux-64=9.5.0 gxx_linux-64=9.5.0

步骤二:查看conda gcc路径

echo $CONDA_PREFIX
ls $CONDA_PREFIX/bin/x86_64-conda-linux-gnu-gcc

步骤三:临时指定编译器变量(仅本终端会话有效)

export CC="$CONDA_PREFIX/bin/x86_64-conda-linux-gnu-gcc"
export CXX="$CONDA_PREFIX/bin/x86_64-conda-linux-gnu-g++"

步骤四:检查实际用的gcc/g++版本

$CC --version
x86_64-conda-linux-gnu-gcc (conda-forge gcc 9.5.0-19) 9.5.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$CXX --version
x86_64-conda-linux-gnu-g++ (conda-forge gcc 9.5.0-19) 9.5.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

注意!这里不要用which gcc,因为它只看PATH优先级,并不会反映pip/cmake等实际用到的编译器!

步骤五:pip编译/安装C++/CUDA扩展

pip install git+https://github.com/likyoo/SimFeatUp.git

4️⃣ which gcc 与 CC/CXX 的区别

  • which gcc 只是shell里的全局查找顺序,不会因conda环境改变。
  • pip/setuptools/cmake等工具实际会优先用你导出的CC/CXX变量指定的编译器。
  • 所以只要你export了CC/CXX,编译环境就是对的,不用在意which gcc的输出!

5️⃣ 经验总结与避坑建议

  • 推荐优先用conda安装gcc多版本做隔离,不污染系统环境,也不影响其他用户和项目。
  • 不要盲目手动降级系统gcc,conda+export CC/CXX就足够了。
  • 每次编译前都要记得export CC/CXX,保险!
  • 遇到类似boxing.h、template、primary-expression相关的C++编译错误,优先怀疑gcc不兼容,先降级试试。
  • 如果pip/conda源下不到cuda 12.x的cudatoolkit,可用pip官方nvidia-cuda-xxx系列包,或者直接用服务器系统级CUDA toolkit,并通过 CUDA_HOME 指定路径。

6️⃣ 完整流程代码块

conda activate your_env_name
conda install -c conda-forge gcc_linux-64=9.5.0 gxx_linux-64=9.5.0
export CC="$CONDA_PREFIX/bin/x86_64-conda-linux-gnu-gcc"
export CXX="$CONDA_PREFIX/bin/x86_64-conda-linux-gnu-g++"
$CC --version  # 确认是9.x
pip install git+https://github.com/likyoo/SimFeatUp.git

如果还遇到 CUDA toolkit 相关找不到,可以手动指定:

export CUDA_HOME=/usr/local/cuda-12.1
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH

7️⃣ 参考资料


8️⃣ 结语

遇到深度学习 C++/CUDA 扩展包编译报错,优先排查编译器版本与PyTorch/环境的兼容性!
conda多版本gcc+export CC/CXX 是无sudo用户的最优解,通杀99%的环境编译问题!


如果你有其他环境配置问题、pip/conda/python/cuda相关报错,欢迎评论区留言交流!
觉得有帮助的话欢迎点赞+收藏,让更多人少踩坑!


Logo

腾讯云面向开发者汇聚海量精品云计算使用和开发经验,营造开放的云计算技术生态圈。

更多推荐