Need -fPIC for static builds #449
Description
Building PyTorch as a static build fails in TensorPipe:
How to repro:
Checkout latest PyTorch codebase and build with
$ USE_MKLDNN=OFF USE_CUDA=OFF BUILD_SHARED_LIBS=OFF BUILD_TESTS=OFF EXTRA_CAFFE2_CMAKE_FLAGS="-DTORCH_STATIC=1" python3 setup.py build --cmake-only
$ cmake --build build It would error with:
/usr/bin/ld: lib/libtensorpipe.a(listener_impl.cc.o): warning: relocation against `_ZTVN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12Li
stenerImplENS2_14ConnectionImplEEE' in read-only section `.text._ZN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12ListenerImplENS2_14Conn
ectionImplEED0Ev[_ZN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12ListenerImplENS2_14ConnectionImplEED5Ev]'
/usr/bin/ld: lib/libtensorpipe.a(error.cc.o): relocation R_X86_64_PC32 against symbol `_ZTVSt9basic_iosIcSt11char_traitsIcEE@@GLIBCXX_3.4' can not be used when makin
g a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
[5619/5838] Linking CXX shared library lib/libjitbackend_test.so
FAILED: lib/libjitbackend_test.so
: && /usr/bin/c++ -fPIC -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -
DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missi
ng-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno
-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiag
nostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type
-Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -Wl,--no-as-needed -rdynamic -shared -Wl,-soname,libjitbackend
_test.so -o lib/libjitbackend_test.so test_jit/CMakeFiles/jitbackend_test.dir/test_backend_lib.cpp.o -Wl,-rpath,::::::: lib/libtorch.a -Wl,--whole-archive,"/home/
anush/github/pytorch/build/lib/libtorch_cpu.a" -Wl,--no-whole-archive /usr/lib/gcc/x86_64-linux-gnu/11/libgomp.so /usr/lib/x86_64-linux-gnu/libpthread.a /usr/lib/
gcc/x86_64-linux-gnu/11/libgomp.so /usr/lib/x86_64-linux-gnu/libpthread.a -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libcaffe2_protos.a" -Wl,--no-wh
ole-archive lib/libc10.a lib/libqnnpack.a lib/libpytorch_qnnpack.a lib/libnnpack.a lib/libXNNPACK.a lib/libpthreadpool.a -lm lib/libfbgemm.a lib/libcpuinfo.
a lib/libclog.a lib/libasmjit.a lib/libtensorpipe.a lib/libtensorpipe_uv.a lib/libgloo.a -lpthread lib/libfoxi_loader.a lib/libkineto.a -lgcc_s -lgcc -lrt
-ldl -lm sleef/lib/libsleef.a -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libonnx.a" -Wl,--no-whole-archive lib/libonnx_proto.a lib/libprotobuf.
a && :
/usr/bin/ld: lib/libtensorpipe.a(listener_impl.cc.o): warning: relocation against `_ZTVN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12Li
stenerImplENS2_14ConnectionImplEEE' in read-only section `.text._ZN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12ListenerImplENS2_14Conn
ectionImplEED0Ev[_ZN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12ListenerImplENS2_14ConnectionImplEED5Ev]'
/usr/bin/ld: lib/libtensorpipe.a(error.cc.o): relocation R_X86_64_PC32 against symbol `_ZTVSt9basic_iosIcSt11char_traitsIcEE@@GLIBCXX_3.4' can not be used when makin
g a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
[5621/5838] Linking CXX shared library lib/libbackend_with_compiler.so
FAILED: lib/libbackend_with_compiler.so
: && /usr/bin/c++ -fPIC -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -
DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missi
ng-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno
-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiag
nostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type
-Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -Wl,--no-as-needed -rdynamic -shared -Wl,-soname,libbackend_wi
th_compiler.so -o lib/libbackend_with_compiler.so test_jit/CMakeFiles/backend_with_compiler.dir/test_backend_compiler_lib.cpp.o test_jit/CMakeFiles/backend_with_comp
iler.dir/test_backend_compiler_preprocess.cpp.o test_jit/CMakeFiles/backend_with_compiler.dir/__/__/__/torch/csrc/jit/mobile/profiler_edge.cpp.o -Wl,-rpath,:::::::
lib/libtorch.a -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libtorch_cpu.a" -Wl,--no-whole-archive /usr/lib/gcc/x86_64-linux-gnu/11/libgomp.so /usr/
lib/x86_64-linux-gnu/libpthread.a /usr/lib/gcc/x86_64-linux-gnu/11/libgomp.so /usr/lib/x86_64-linux-gnu/libpthread.a -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libcaffe2_protos.a" -Wl,--no-whole-archive lib/libc10.a lib/libqnnpack.a lib/libpytorch_qnnpack.a lib/libnnpack.a lib/libXNNPACK.a lib/libpthreadp
ool.a -lm lib/libfbgemm.a lib/libcpuinfo.a lib/libclog.a lib/libasmjit.a lib/libtensorpipe.a lib/libtensorpipe_uv.a lib/libgloo.a -lpthread lib/libfoxi_loa
der.a lib/libkineto.a -lgcc_s -lgcc -lrt -ldl -lm sleef/lib/libsleef.a -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libonnx.a" -Wl,--no-whole-ar
chive lib/libonnx_proto.a lib/libprotobuf.a && :
/usr/bin/ld: lib/libtensorpipe.a(listener_impl.cc.o): warning: relocation against `_ZTVN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12Li
stenerImplENS2_14ConnectionImplEEE' in read-only section `.text._ZN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12ListenerImplENS2_14Conn
ectionImplEED0Ev[_ZN10tensorpipe9transport23ListenerImplBoilerplateINS0_2uv11ContextImplENS2_12ListenerImplENS2_14ConnectionImplEED5Ev]'
/usr/bin/ld: lib/libtensorpipe.a(error.cc.o): relocation R_X86_64_PC32 against symbol `_ZTVSt9basic_iosIcSt11char_traitsIcEE@@GLIBCXX_3.4' can not be used when makin
g a shared object; recompile with -fPIC /usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
[5630/5838] Building CXX object test_tensorexpr/CMakeFiles/tutorial_tensorexpr.dir/tutorial.cpp.o
ninja: build stopped: subcommand failed. ```
I applied ```
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 77df76d..111cddb 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -9,6 +9,7 @@ cmake_minimum_required(VERSION 3.5 FATAL_ERROR)
project(tensorpipe LANGUAGES C CXX)
set(CMAKE_CXX_STANDARD 14)
+set(CMAKE_POSITION_INDEPENDENT_CODE ON)
but then there are further linking errors
FAILED: bin/test_cpp_rpc [103/47579]: && /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_P
YTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-fie
ld-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic test_cpp_rpc/CMakeFiles/test_cpp_rpc.dir/__/common/main.cpp
.o test_cpp_rpc/CMakeFiles/test_cpp_rpc.dir/e2e_test_base.cpp.o test_cpp_rpc/CMakeFiles/test_cpp_rpc.dir/test_wire_serialization.cpp.o test_cpp_rpc/CMakeFiles/test_c
pp_rpc.dir/test_e2e_tensorpipe.cpp.o test_cpp_rpc/CMakeFiles/test_cpp_rpc.dir/test_tensorpipe_serialization.cpp.o -o bin/test_cpp_rpc -Wl,-rpath,::::::: lib/libtor
ch.a lib/libgtest.a lib/libtensorpipe.a lib/libtensorpipe.a lib/libtensorpipe_uv.a -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libtorch_cpu.a" -Wl
,--no-whole-archive /usr/lib/gcc/x86_64-linux-gnu/11/libgomp.so /usr/lib/x86_64-linux-gnu/libpthread.a /usr/lib/gcc/x86_64-linux-gnu/11/libgomp.so /usr/lib/x86_64-linux-gnu/libpthread.a -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libcaffe2_protos.a" -Wl,--no-whole-archive lib/libc10.a lib/libqnnpack.a lib/l
ibpytorch_qnnpack.a lib/libnnpack.a lib/libXNNPACK.a lib/libpthreadpool.a -lm lib/libfbgemm.a lib/libcpuinfo.a lib/libclog.a lib/libasmjit.a lib/libgloo.a
-lpthread lib/libfoxi_loader.a lib/libkineto.a -lgcc_s -lgcc -lrt -ldl -lm sleef/lib/libsleef.a -Wl,--whole-archive,"/home/anush/github/pytorch/build/lib/libonnx.a" -Wl,--no-whole-archive lib/libonnx_proto.a lib/libprotobuf.a && : /usr/bin/ld: /home/anush/github/pytorch/build/lib/libtorch_cpu.a(tensorpipe_agent.cpp.o): in function `torch::distributed::rpc::TensorPipeAgent::shutdownImpl()': tensorpipe_agent.cpp:(.text+0x800): undefined reference to `tensorpipe::Context::join()'
/usr/bin/ld: /home/anush/github/pytorch/build/lib/libtorch_cpu.a(tensorpipe_agent.cpp.o): in function `torch::distributed::rpc::TensorPipeAgent::sendCompletedRespons
eMessage(std::shared_ptr<tensorpipe::Pipe>&, c10::ivalue::Future&, unsigned long, std::vector<c10::Stream, std::allocator<c10::Stream> >)::{lambda(tensorpipe::Error
const&)#2}::operator()(tensorpipe::Error const&) const':
tensorpipe_agent.cpp:(.text+0xc2c): undefined reference to `tensorpipe::Pipe::getRemoteName[abi:cxx11]()'
/usr/bin/ld: tensorpipe_agent.cpp:(.text+0xc60): undefined reference to `tensorpipe::Error::what[abi:cxx11]() const' /usr/bin/ld: tensorpipe_agent.cpp:(.text+0xd49): undefined reference to `tensorpipe::Pipe::getRemoteName[abi:cxx11]()' /usr/bin/ld: /home/anush/github/pytorch/build/lib/libtorch_cpu.a(tensorpipe_agent.cpp.o): in function `torch::distributed::rpc::TensorPipeAgent::sendCompletedResponseMessage(std::shared_ptr<tensorpipe::Pipe>&, c10::ivalue::Future&, unsigned long, std::vector<c10::Stream, std::allocator<c10::Stream> >)::{lambda(tensorpipe::Error const&)#1}::operator()(tensorpipe::Error const&) const': tensorpipe_agent.cpp:(.text+0xe5c): undefined reference to `tensorpipe::Pipe::getRemoteName[abi:cxx11]()'