无sudo权限服务器环境下深度学习C++/CUDA扩展编译报错解决实录【conda多版本gcc隔离最佳实践】
·
🚩无sudo权限服务器环境下深度学习C++/CUDA扩展编译报错解决实录【conda多版本gcc隔离最佳实践】
关键词:PyTorch、CUDA、conda、featup、C++扩展、gcc版本、环境变量、深度学习环境
1️⃣ 问题背景
最近在搭建 SimFeatUp 项目环境,执行 pip install git+https://github.com/likyoo/SimFeatUp.git 时一直报如下 C++/CUDA 编译相关错误:
Building wheel for featup (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [154 lines of output]
/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
!!
********************************************************************************
Please consider removing the following classifiers in favor of a SPDX license expression:
License :: OSI Approved :: MIT License
See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
********************************************************************************
!!
self._finalize_license_expression()
running bdist_wheel
running build
running build_py
creating build/lib.linux-x86_64-cpython-39/featup
copying featup/__init__.py -> build/lib.linux-x86_64-cpython-39/featup
copying featup/downsamplers.py -> build/lib.linux-x86_64-cpython-39/featup
copying featup/layers.py -> build/lib.linux-x86_64-cpython-39/featup
copying featup/losses.py -> build/lib.linux-x86_64-cpython-39/featup
copying featup/plotting.py -> build/lib.linux-x86_64-cpython-39/featup
copying featup/train_upsampler.py -> build/lib.linux-x86_64-cpython-39/featup
copying featup/upsamplers.py -> build/lib.linux-x86_64-cpython-39/featup
copying featup/util.py -> build/lib.linux-x86_64-cpython-39/featup
creating build/lib.linux-x86_64-cpython-39/featup/adaptive_conv_cuda
copying featup/adaptive_conv_cuda/__init__.py -> build/lib.linux-x86_64-cpython-39/featup/adaptive_conv_cuda
copying featup/adaptive_conv_cuda/adaptive_conv.py -> build/lib.linux-x86_64-cpython-39/featup/adaptive_conv_cuda
creating build/lib.linux-x86_64-cpython-39/featup/datasets
copying featup/datasets/AID.py -> build/lib.linux-x86_64-cpython-39/featup/datasets
copying featup/datasets/JitteredImage.py -> build/lib.linux-x86_64-cpython-39/featup/datasets
copying featup/datasets/__init__.py -> build/lib.linux-x86_64-cpython-39/featup/datasets
copying featup/datasets/util.py -> build/lib.linux-x86_64-cpython-39/featup/datasets
creating build/lib.linux-x86_64-cpython-39/featup/featurizers
copying featup/featurizers/CLIP.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
copying featup/featurizers/DINO.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
copying featup/featurizers/DINOv2.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
copying featup/featurizers/DeepLabV3.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
copying featup/featurizers/MAE.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
copying featup/featurizers/MIDAS.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
copying featup/featurizers/MaskCLIP.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
copying featup/featurizers/ResNet.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
copying featup/featurizers/XCLIP.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
copying featup/featurizers/__init__.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
copying featup/featurizers/util.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers
creating build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2
copying featup/featurizers/dinov2/__init__.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2
creating build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
copying featup/featurizers/maskclip/__init__.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
copying featup/featurizers/maskclip/clip.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
copying featup/featurizers/maskclip/interpolate.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
copying featup/featurizers/maskclip/model.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
copying featup/featurizers/maskclip/simple_tokenizer.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
copying featup/featurizers/maskclip/xclip.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
copying featup/featurizers/maskclip/xmodel.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/maskclip
creating build/lib.linux-x86_64-cpython-39/featup/featurizers/modules
copying featup/featurizers/modules/__init__.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/modules
copying featup/featurizers/modules/layers.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/modules
copying featup/featurizers/modules/resnet.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/modules
copying featup/featurizers/modules/vgg.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/modules
creating build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
copying featup/featurizers/dinov2/layers/__init__.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
copying featup/featurizers/dinov2/layers/attention.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
copying featup/featurizers/dinov2/layers/block.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
copying featup/featurizers/dinov2/layers/dino_head.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
copying featup/featurizers/dinov2/layers/drop_path.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
copying featup/featurizers/dinov2/layers/layer_scale.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
copying featup/featurizers/dinov2/layers/mlp.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
copying featup/featurizers/dinov2/layers/patch_embed.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
copying featup/featurizers/dinov2/layers/swiglu_ffn.py -> build/lib.linux-x86_64-cpython-39/featup/featurizers/dinov2/layers
running build_ext
/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/utils/cpp_extension.py:414: UserWarning: The detected CUDA version (12.4) has a minor version mismatch with the version that was used to compile PyTorch (12.1). Most likely this shouldn't be a problem.
warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/utils/cpp_extension.py:424: UserWarning: There are no g++ version bounds defined for CUDA version 12.4
warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
building 'adaptive_conv_cuda_impl' extension
creating /tmp/pip-req-build-s0kol3am/build/temp.linux-x86_64-cpython-39/featup/adaptive_conv_cuda
Emitting ninja build file /tmp/pip-req-build-s0kol3am/build/temp.linux-x86_64-cpython-39/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] c++ -MMD -MF /tmp/pip-req-build-s0kol3am/build/temp.linux-x86_64-cpython-39/featup/adaptive_conv_cuda/adaptive_conv_cuda.o.d -pthread -B /your/path/to/envs/SegEarth/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /your/path/to/envs/SegEarth/include -I/your/path/to/envs/SegEarth/include -fPIC -O2 -isystem /your/path/to/envs/SegEarth/include -fPIC -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/TH -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/THC -I/usr/local/cuda/include -I/your/path/to/envs/SegEarth/include/python3.9 -c -c /tmp/pip-req-build-s0kol3am/featup/adaptive_conv_cuda/adaptive_conv_cuda.cpp -o /tmp/pip-req-build-s0kol3am/build/temp.linux-x86_64-cpython-39/featup/adaptive_conv_cuda/adaptive_conv_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=adaptive_conv_cuda_impl -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
[2/2] /usr/local/cuda/bin/nvcc -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/TH -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/THC -I/usr/local/cuda/include -I/your/path/to/envs/SegEarth/include/python3.9 -c -c /tmp/pip-req-build-s0kol3am/featup/adaptive_conv_cuda/adaptive_conv_kernel.cu -o /tmp/pip-req-build-s0kol3am/build/temp.linux-x86_64-cpython-39/featup/adaptive_conv_cuda/adaptive_conv_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=adaptive_conv_cuda_impl -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 -std=c++17
FAILED: /tmp/pip-req-build-s0kol3am/build/temp.linux-x86_64-cpython-39/featup/adaptive_conv_cuda/adaptive_conv_kernel.o
/usr/local/cuda/bin/nvcc -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/TH -I/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/THC -I/usr/local/cuda/include -I/your/path/to/envs/SegEarth/include/python3.9 -c -c /tmp/pip-req-build-s0kol3am/featup/adaptive_conv_cuda/adaptive_conv_kernel.cu -o /tmp/pip-req-build-s0kol3am/build/temp.linux-x86_64-cpython-39/featup/adaptive_conv_cuda/adaptive_conv_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=adaptive_conv_cuda_impl -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 -std=c++17
/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/ATen/core/boxing/impl/boxing.h:41:104: error: expected primary-expression before ‘>’ token
41 | struct has_ivalue_to<T, guts::void_t<decltype(std::declval<IValue>().to<T>())>>
| ^
/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/include/ATen/core/boxing/impl/boxing.h:41:107: error: expected primary-expression before ‘)’ token
41 | struct has_ivalue_to<T, guts::void_t<decltype(std::declval<IValue>().to<T>())>>
| ^
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 2100, in _run_ninja_build
subprocess.run(
File "/your/path/to/envs/SegEarth/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 35, in <module>
File "/tmp/pip-req-build-s0kol3am/setup.py", line 4, in <module>
setup(
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/__init__.py", line 117, in setup
return distutils.core.setup(**attrs)
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 186, in setup
return run_commands(dist)
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
dist.run_commands()
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
self.run_command(cmd)
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/dist.py", line 1104, in run_command
super().run_command(command)
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
cmd_obj.run()
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/command/bdist_wheel.py", line 370, in run
self.run_command("build")
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
self.distribution.run_command(command)
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/dist.py", line 1104, in run_command
super().run_command(command)
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
cmd_obj.run()
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
self.distribution.run_command(command)
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/dist.py", line 1104, in run_command
super().run_command(command)
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
cmd_obj.run()
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 99, in run
_build_ext.run(self)
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
self.build_extensions()
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 873, in build_extensions
build_ext.build_extensions(self)
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
self._build_extensions_serial()
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
self.build_extension(ext)
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 264, in build_extension
_build_ext.build_extension(self, ext)
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 565, in build_extension
objects = self.compiler.compile(
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 686, in unix_wrap_ninja_compile
_write_ninja_file_and_compile_objects(
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1774, in _write_ninja_file_and_compile_objects
_run_ninja_build(
File "/your/path/to/envs/SegEarth/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 2116, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for featup
Running setup.py clean for featup
Failed to build featup
ERROR: Failed to build installable wheels for some pyproject.toml based projects (featup)
- 我的系统环境是 Ubuntu 24.04,Python 3.9,PyTorch 2.1.2,CUDA 12.1
- gcc 版本为 13.3.0
2️⃣ 问题分析
- 查了大量资料和论坛后发现,这类报错99%都是因为 gcc/g++ 版本太新,和 PyTorch/CUDA 的 C++ API 头文件兼容性有冲突。
- PyTorch 官方一般推荐用 gcc 9.x 或 10.x 编译扩展,太新/太老都容易报错。
- 目前很多深度学习C++/CUDA扩展包(如featup、mmcv、dcn等)都需要遵循这个“gcc 9/10”惯例。
3️⃣ 尝试conda安装多版本gcc并隔离
优势:不污染全局环境!不同项目可用不同gcc。无sudo权限依然可以优雅隔离!
步骤一:conda安装gcc-9.5和g+±9.5
conda install -c conda-forge gcc_linux-64=9.5.0 gxx_linux-64=9.5.0
步骤二:查看conda gcc路径
echo $CONDA_PREFIX
ls $CONDA_PREFIX/bin/x86_64-conda-linux-gnu-gcc
步骤三:临时指定编译器变量(仅本终端会话有效)
export CC="$CONDA_PREFIX/bin/x86_64-conda-linux-gnu-gcc"
export CXX="$CONDA_PREFIX/bin/x86_64-conda-linux-gnu-g++"
步骤四:检查实际用的gcc/g++版本
$CC --version
x86_64-conda-linux-gnu-gcc (conda-forge gcc 9.5.0-19) 9.5.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$CXX --version
x86_64-conda-linux-gnu-g++ (conda-forge gcc 9.5.0-19) 9.5.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
注意!这里不要用
which gcc,因为它只看PATH优先级,并不会反映pip/cmake等实际用到的编译器!
步骤五:pip编译/安装C++/CUDA扩展
pip install git+https://github.com/likyoo/SimFeatUp.git
4️⃣ which gcc 与 CC/CXX 的区别
which gcc只是shell里的全局查找顺序,不会因conda环境改变。- pip/setuptools/cmake等工具实际会优先用你导出的
CC/CXX变量指定的编译器。 - 所以只要你export了CC/CXX,编译环境就是对的,不用在意which gcc的输出!
5️⃣ 经验总结与避坑建议
- 推荐优先用conda安装gcc多版本做隔离,不污染系统环境,也不影响其他用户和项目。
- 不要盲目手动降级系统gcc,conda+export CC/CXX就足够了。
- 每次编译前都要记得export CC/CXX,保险!
- 遇到类似boxing.h、template、primary-expression相关的C++编译错误,优先怀疑gcc不兼容,先降级试试。
- 如果pip/conda源下不到cuda 12.x的cudatoolkit,可用pip官方nvidia-cuda-xxx系列包,或者直接用服务器系统级CUDA toolkit,并通过 CUDA_HOME 指定路径。
6️⃣ 完整流程代码块
conda activate your_env_name
conda install -c conda-forge gcc_linux-64=9.5.0 gxx_linux-64=9.5.0
export CC="$CONDA_PREFIX/bin/x86_64-conda-linux-gnu-gcc"
export CXX="$CONDA_PREFIX/bin/x86_64-conda-linux-gnu-g++"
$CC --version # 确认是9.x
pip install git+https://github.com/likyoo/SimFeatUp.git
如果还遇到 CUDA toolkit 相关找不到,可以手动指定:
export CUDA_HOME=/usr/local/cuda-12.1
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
7️⃣ 参考资料
8️⃣ 结语
遇到深度学习 C++/CUDA 扩展包编译报错,优先排查编译器版本与PyTorch/环境的兼容性!
conda多版本gcc+export CC/CXX 是无sudo用户的最优解,通杀99%的环境编译问题!
如果你有其他环境配置问题、pip/conda/python/cuda相关报错,欢迎评论区留言交流!
觉得有帮助的话欢迎点赞+收藏,让更多人少踩坑!
更多推荐
所有评论(0)