Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SolveXcts with debug symbols has high memory usage during compilation #6150

Open
macedo22 opened this issue Jul 12, 2024 · 1 comment
Open

SolveXcts with debug symbols has high memory usage during compilation #6150

macedo22 opened this issue Jul 12, 2024 · 1 comment

Comments

@macedo22
Copy link
Contributor

macedo22 commented Jul 12, 2024 •
edited
Loading

Related to #5472

When attempting to build SolveXcts on ocean with gcc 11.3.0 Release, compilation succeeds but attempting to link generates this error that repeats until the process is killed:

[100%] Linking CXX executable ../../../../bin/SolveXcts
/opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/conv-static.o(.debug_info+0xd): error: relocation overflow: reference to local symbol 10 in /opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/conv-static.o
/opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/conv-static.o(.debug_info+0x12): error: relocation overflow: reference to local symbol 10 in /opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/conv-static.o
/opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/conv-static.o(.debug_info+0x16): error: relocation overflow: reference to local symbol 10 in /opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/conv-static.o
/opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/conv-static.o(.debug_info+0x65): error: relocation overflow: reference to local symbol 10 in /opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/conv-static.o
/opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/conv-static.o(.debug_info+0x72): error: relocation overflow: reference to local symbol 10 in /opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/conv-static.o
/opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/libconverse.a(commitid.C.o)(.debug_info+0xd): error: relocation overflow: reference to local symbol 11 in /opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/libconverse.a(commitid.C.o)
/opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/libconverse.a(commitid.C.o)(.debug_info+0x12): error: relocation overflow: reference to local symbol 11 in /opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/libconverse.a(commitid.C.o)
/opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/libconverse.a(commitid.C.o)(.debug_info+0x16): error: relocation overflow: reference to local symbol 11 in /opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/libconverse.a(commitid.C.o)
/opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/libconverse.a(commitid.C.o)(.debug_info+0x1f): error: relocation overflow: reference to local symbol 11 in /opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/lib/libconverse.a(commitid.C.o)
CMakeFiles/SolveXcts.dir/SolveXcts.cpp.o(.debug_info+0x1a4e): error: relocation overflow: reference to local symbol 19509 in CMakeFiles/SolveXcts.dir/SolveXcts.cpp.o
CMakeFiles/SolveXcts.dir/SolveXcts.cpp.o(.debug_info+0x1d36): error: relocation overflow: reference to local symbol 19509 in CMakeFiles/SolveXcts.dir/SolveXcts.cpp.o
CMakeFiles/SolveXcts.dir/SolveXcts.cpp.o(.debug_info+0x1d4f): error: relocation overflow: reference to local symbol 19509 in CMakeFiles/SolveXcts.dir/SolveXcts.cpp.o
.
.
.

A temporary workaround to get things to build successfully is to use -D DEBUG_SYMBOLS=OFF in the cmake command in spectre_run_cmake, which also reduces peak RAM during compilation.

Memory usage during compiling and linking is very high when debug symbols are on (default setting for Release). Here is the memory usage and time for building with and without debug symbols:

Process     | Debug symbols | Peak RAM (GB) | Time (mm:ss)
-----------------------------------------------------------
Compiling     On                  46.7           22:32
              Off                 11.2           11:37
Linking       On (killed*)        15.1*          00:16*
              Off                  2.0           00:09

*killed when linking error starts


Size of SolveXcts.cpp.o with debug symbols:     4.69 GB
Size of SolveXcts.cpp.o without debug symbols:  0.45 GB

Here is the full command that was used to compile SolveXcts with debug symbols (-g):

/opt/ohpc/pub/compiler/gcc/11.3.0/bin/g++ -DBLAZE_BLAS_INCLUDE_FILE="<gsl/gsl_cblas.h>" -DBLAZE_BLAS_MODE=1 -DBLAZE_DEFAULT_STORAGE_ORDER=blaze::columnMajor -DBLAZE_MPI_PARALLEL_MODE=0 -DBLAZE_USE_ALWAYS_INLINE=1 -DBLAZE_USE_DEFAULT_INITIALIZATON=0 -DBLAZE_USE_PADDING=0 -DBLAZE_USE_SHARED_MEMORY_PARALLELIZATION=0 -DBLAZE_USE_SLEEF=0 -DBLAZE_USE_STREAMING=1 -DBLAZE_USE_STRONG_INLINE=1 -DBOOST_ALLOW_DEPRECATED_HEADERS -DBOOST_MULTI_ARRAY_TYPES_HPP -DBOOST_PROGRAM_OPTIONS_DYN_LINK -DBOOST_PROGRAM_OPTIONS_NO_LIB -DBOOST_SP_DISABLE_THREADS -DDISABLE_OPENBLAS_MULTITHREADING -DH5_BUILT_AS_DYNAMIC_LIB -DSPECTRE_CHARM_HAS_MAIN -DSPECTRE_USE_ALWAYS_INLINE=1 -DSolveXcts_EXPORTS -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200809L -I/home/alexmacedo/spectre/src -isystem /home/alexmacedo/spectre/build_develop/src -isystem /opt/ohpc/pub/apps/spack2022/opt/spack/linux-centos7-broadwell/gcc-11.3.0/jemalloc-5.2.1-r63hempegqym5ahitm2ykc3x7x4wvkci/include -isystem /opt/ohpc/pub/apps/spack2022/opt/spack/linux-centos7-broadwell/gcc-11.3.0/blaze-3.8-y7sgzzcyehq7nosikgmqoi57pf5g2rpv/include -isystem /opt/ohpc/pub/apps/spack2022/opt/spack/linux-centos7-broadwell/gcc-11.3.0/gsl-2.7.1-sey3z3ovrt3f4gvelm6dels7elkugo7o/include -isystem /home/alexmacedo/spectre/external/brigand/include -isystem /opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/include -isystem /opt/ohpc/pub/apps/spack2022/opt/spack/linux-centos7-broadwell/gcc-11.3.0/hdf5-1.12.2-dly2yyupjmeqjt5qswpt6iwuun7gvml3/include -isystem /opt/ohpc/pub/mpi/openmpi-gnu11/4.1.4/include -isystem /opt/ohpc/pub/apps/spack2022/opt/spack/linux-centos7-broadwell/gcc-11.3.0/boost-1.79.0-ck2cccntuidtdby5falg4dooac6c7ypj/include -isystem /opt/ohpc/pub/apps/libxsmm/1.16.1/include -isystem /opt/ohpc/pub/apps/spack2022/opt/spack/linux-centos7-broadwell/gcc-11.3.0/yaml-cpp-0.7.0-a4rumor7dkzoxmila6ewami7fzvoggwl/include -O3 -DNDEBUG -fPIC -g -march=native -mno-avx512f -ftemplate-backtrace-limit=0 -fno-math-errno -freciprocal-math -W -Wall -Wcast-align -Wcast-qual -Wdisabled-optimization -Wextra -Wformat-nonliteral -Wformat-security -Wformat-y2k -Wformat=2 -Winvalid-pch -Wmissing-declarations -Wmissing-field-initializers -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-noreturn -Wno-mismatched-tags -Wno-non-template-friend -Wno-type-limits -Wnon-virtual-dtor -Wold-style-cast -Woverloaded-virtual -Wpacked -Wpedantic -Wpointer-arith -Wredundant-decls -Wshadow -Wsign-conversion -Wstack-protector -Wswitch-default -Wunreachable-code -Wwrite-strings -Wno-noexcept-type -fnon-call-exceptions -D "BLAZE_THROW(EXCEPTION)=" -pthread -std=c++20 -Winvalid-pch -include /home/alexmacedo/spectre/build_develop/CMakeFiles/SpectrePch.dir/cmake_pch.hxx -o CMakeFiles/SolveXcts.dir/SolveXcts.cpp.o -c /home/alexmacedo/spectre/src/Elliptic/Executables/Xcts/SolveXcts.cpp

And here is the full command that was used to compile SolveXcts without debug symbols (-DDEBUG_SYMBOLS=OFF used in cmake command removes -g):

/opt/ohpc/pub/compiler/gcc/11.3.0/bin/g++ -DBLAZE_BLAS_INCLUDE_FILE="<gsl/gsl_cblas.h>" -DBLAZE_BLAS_MODE=1 -DBLAZE_DEFAULT_STORAGE_ORDER=blaze::columnMajor -DBLAZE_MPI_PARALLEL_MODE=0 -DBLAZE_USE_ALWAYS_INLINE=1 -DBLAZE_USE_DEFAULT_INITIALIZATON=0 -DBLAZE_USE_PADDING=0 -DBLAZE_USE_SHARED_MEMORY_PARALLELIZATION=0 -DBLAZE_USE_SLEEF=0 -DBLAZE_USE_STREAMING=1 -DBLAZE_USE_STRONG_INLINE=1 -DBOOST_ALLOW_DEPRECATED_HEADERS -DBOOST_MULTI_ARRAY_TYPES_HPP -DBOOST_PROGRAM_OPTIONS_DYN_LINK -DBOOST_PROGRAM_OPTIONS_NO_LIB -DBOOST_SP_DISABLE_THREADS -DDISABLE_OPENBLAS_MULTITHREADING -DH5_BUILT_AS_DYNAMIC_LIB -DSPECTRE_CHARM_HAS_MAIN -DSPECTRE_USE_ALWAYS_INLINE=1 -DSolveXcts_EXPORTS -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200809L -I/home/alexmacedo/spectre/src -isystem /home/alexmacedo/spectre/build_develop_no_debug_symbols/src -isystem /opt/ohpc/pub/apps/spack2022/opt/spack/linux-centos7-broadwell/gcc-11.3.0/jemalloc-5.2.1-r63hempegqym5ahitm2ykc3x7x4wvkci/include -isystem /opt/ohpc/pub/apps/spack2022/opt/spack/linux-centos7-broadwell/gcc-11.3.0/blaze-3.8-y7sgzzcyehq7nosikgmqoi57pf5g2rpv/include -isystem /opt/ohpc/pub/apps/spack2022/opt/spack/linux-centos7-broadwell/gcc-11.3.0/gsl-2.7.1-sey3z3ovrt3f4gvelm6dels7elkugo7o/include -isystem /home/alexmacedo/spectre/external/brigand/include -isystem /opt/ohpc/pub/apps/charm_7.0.0_gnu11_022324/verbs-linux-x86_64-smp-gcc/include -isystem /opt/ohpc/pub/apps/spack2022/opt/spack/linux-centos7-broadwell/gcc-11.3.0/hdf5-1.12.2-dly2yyupjmeqjt5qswpt6iwuun7gvml3/include -isystem /opt/ohpc/pub/mpi/openmpi-gnu11/4.1.4/include -isystem /opt/ohpc/pub/apps/spack2022/opt/spack/linux-centos7-broadwell/gcc-11.3.0/boost-1.79.0-ck2cccntuidtdby5falg4dooac6c7ypj/include -isystem /opt/ohpc/pub/apps/libxsmm/1.16.1/include -isystem /opt/ohpc/pub/apps/spack2022/opt/spack/linux-centos7-broadwell/gcc-11.3.0/yaml-cpp-0.7.0-a4rumor7dkzoxmila6ewami7fzvoggwl/include -O3 -DNDEBUG -fPIC -march=native -mno-avx512f -ftemplate-backtrace-limit=0 -fno-math-errno -freciprocal-math -W -Wall -Wcast-align -Wcast-qual -Wdisabled-optimization -Wextra -Wformat-nonliteral -Wformat-security -Wformat-y2k -Wformat=2 -Winvalid-pch -Wmissing-declarations -Wmissing-field-initializers -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-noreturn -Wno-mismatched-tags -Wno-non-template-friend -Wno-type-limits -Wnon-virtual-dtor -Wold-style-cast -Woverloaded-virtual -Wpacked -Wpedantic -Wpointer-arith -Wredundant-decls -Wshadow -Wsign-conversion -Wstack-protector -Wswitch-default -Wunreachable-code -Wwrite-strings -Wno-noexcept-type -fnon-call-exceptions -D "BLAZE_THROW(EXCEPTION)=" -pthread -std=c++20 -Winvalid-pch -include /home/alexmacedo/spectre/build_develop_no_debug_symbols/CMakeFiles/SpectrePch.dir/cmake_pch.hxx -o CMakeFiles/SolveXcts.dir/SolveXcts.cpp.o -c /home/alexmacedo/spectre/src/Elliptic/Executables/Xcts/SolveXcts.cpp
@nilsvu nilsvu changed the title SolveXcts fails to link on ocean and has high memory usage during compilation SolveXcts with debug symbols has high memory usage during compilation Jul 12, 2024
@nilsvu
Copy link
Member

nilsvu commented Jul 12, 2024

Yes also on CaltechHPC I've had this problem and had to set DEBUG_SYMBOLS=OFF. I don't think the SolveXcts executable is doing anything unreasonable or goes beyond what we intend to support with the DataBox, so as you mentioned the underlying issue is #5472 and IMO it's fairly high priority to fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nilsvu @macedo22