Diverse users running across a diverse set of architectures, it seems Spack was built just for this!
I want to share my experiences with Spack, here my focus is on CP2K. My build environment is a Cray XC50 running Thunder X2 Arm CPUs.
Spack is a source-based package manager and build system, all builds take place in self-contained environments. Every installed package is defined by a spec, the spec is a string that represents the enabled features, compiler, etc.. that results in a unique package. The process of determining the packages unique hash is called concretization.
> spack compiler find
~/.spack/cray/compilers.yaml with all our compilers, Spack checks typical locations as well as your available modules.
> spack install [email protected] +openmp +mpi +plumed %[email protected] smm=blas ^fftw +openmp ^openblas threads=openmp
+openmp is default
+mpi to enable MPI support
+plumed to enable PLUMED
%[email protected] uses GCC 9.2.0, loaded via a module by Spack
smm=blas is required as libxsmm not available on arm64, noted in spec.
^fftw +openmp enables OpenMP for the FFTW dependency
^openblas threads=openmp uses OpenMP for threadding in OpenBLAS, I found the
threads= options in its spec file but I could have just run
spack info openblas.
I saw some differing results when I loaded modules into my environment before running Spack so I try avoid loading any when building packages.
Now, this build fails to concretize and Spack shows us what it tried to do and which packages are conflicting:
> spack install [email protected] +openmp +mpi +plumed %[email protected] smm=blas ^fftw +openmp ^openblas threads=openmp ==> Error: Conflicts in concretized spec "[email protected]%[email protected]~cosma~cuda~cuda_arch_35_k20x~cuda_blas~cuda_fft~elpa+libint~libvori+libxc+mpi+openmp~pexsi+plumed~sirius~spglib cuda_arch=none lmax=5 smm=blas arch=cray-cnl7-aarch64/kizgfsf" List of matching conflicts for spec: [email protected]%[email protected]~cosma~cuda~cuda_arch_35_k20x~cuda_blas~cuda_fft~elpa+libint~libvori+libxc+mpi+openmp~pexsi+plumed~sirius~spglib cuda_arch=none lmax=5 smm=blas arch=cray-cnl7-aarch64 ^[email protected]%[email protected]+mpi+openmp~pfft_patches precision=double,float arch=cray-cnl7-aarch64 ^[email protected]%[email protected]+fortran tune=cp2k-lmax-5 arch=cray-cnl7-aarch64 ... ... 1. "^openblas threads=none" conflicts with "cp2k+openmp"
You can see the concretized spec of my cp2k installation is
[email protected]%[email protected]~cosma~cuda~cuda_arch_35_k20x~cuda_blas~cuda_fft~elpa+libint~libvori+libxc+mpi+openmp~pexsi+plumed~sirius~spglib cuda_arch=none lmax=5 smm=blas arch=cray-cnl7-aarch64
This includes any defaults for every variant the package supports, I prefer to leave it up to the maintainer for variants I’m not familiar with, but if you want highly reproducible environments you will probably want to specify everything.
^openblas threads=none" conflicts with "cp2k+openmp suggests the openblas dependency won’t support threading, which is required because I’m building cp2k with openmp support.
OpenBLAS isn’t tricky to compile, so I opted to compile it manually:
> make > make PREFIX=/software/arm64/apps/openblas/0.3.15 install
Now I need to tell Spack about my installation of OpenBLAS, this kind of package is called an External and they are listed in
packages: autoconf: externals: - spec: [email protected] prefix: /usr automake: externals: - spec: [email protected] prefix: /usr bash: externals: - spec: [email protected] prefix: / ...
I added OpenBLAS and a few Cray libraries to my
mpi: externals: - spec: "[email protected]" modules: - cray-mpich/7.7.16 buildable: False openblas: externals: - spec: "[email protected]" modules: - apps/openblas/0.3.15 fftw: externals: - spec: "fftw+openmp" modules: - cray-fftw cray-libsci: externals: - spec: "cray-libsci" modules: - cray-libsci buildable: False
Each external has a spec that Spack matches against, for example if I ask for fftw explicitly without openmp support then it won’t use my fftw external as I have listed
+openmp in its spec.
buildable: False to tell Spack to never build an alternative for some packages. I want to use the supported system libraries where possible, instead of some apps using
cray-libsci and others building a custom
libsci, as an example.
This bug slowed me down, I didn’t find out the cause and it has since cleared itself up. It may be related to loading spack on different clusters with a shared home directory and it ended up loading something confusing in
==> cp2k: Executing phase: 'edit' ==> Error: JSONDecodeError: Expecting value: line 1 column 1 (char 0) /lustre/software/aarch64/spack/var/spack/repos/builtin/packages/cp2k/package.py:300, in edit: 297 ldflags += ['-hnoomp'] 298 299 if '@7:' in spec: # recent versions of CP2K use C++14 CUDA code >> 300 cxxflags.append(self.compiler.cxx14_flag) 301 nvflags.append(self.compiler.cxx14_flag) 302 303 ldflags.append(fftw.libs.search_flags)
I opened bug #23570 to track it.
Missing build dependency
I got the following error from plumed:
RuntimeError: Cannot generate configure: missing dependencies ['m4']
Spack seems to think m4 isn’t available, it turned out that the package’s
package.py was missing the
type='build' on the line where
m4 is listed as a build depdenency. This was fixed in #23953.
Lastly I had difficulty building
netlib-scalapack until I realised that
cray-libsci provides the LAPACK/ScaLAPACK capabilities cp2k requires. I manually set the dependency package with
> spack install [email protected] +openmp +mpi +plumed %[email protected] smm=blas ^cray-libsci ==> Warning: Missing a source id for [email protected] ==> Warning: Missing a source id for [email protected] ==> Warning: Missing a source id for [email protected] ==> [email protected] : has external module in ['cray-libsci'] [+] /opt/cray/pe/libsci/20.06.1/GNU/8.1/aarch64 (external cray-libsci-20.06.1-lcxhogmgo4zwfosmyp6wddo7xvga4ybx) ==> [email protected] : has external module in ['cray-fftw'] [+] /opt/cray/pe/fftw/220.127.116.11/arm_thunderx2 (external fftw-3.3.9-bwhmlzy6eik3ajeqobol527ww6s42vcn) [+] /usr (external autoconf-2.69-ememqml5q3hrhatuxevmdaeyuocnaqnk) [+] /usr (external automake-1.15.1-edymz5en4xwxwybe76d3535re7euacgl) [+] /usr (external bzip2-1.0.6-2fqflhapqghfvebfk5kvogp7g4xulquw) [+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/zlib-1.2.11-cx7za7abtiy5ezumlkfxsffzfxuj3xq6 [+] /usr (external libtool-2.4.6-utlxqeicabegxrlreyo3biinw7v4vswn) [+] /usr (external m4-1.4.18-og3jty4rquctsfxnxzwaufttw6bylag4) [+] /usr (external perl-5.26.1-lk6rvqusbsxtnumqre6ete2tvgxghgxx) ==> [email protected] : has external module in ['cray-mpich/7.7.16'] [+] /opt/cray/pe/mpt/7.7.16/gni/mpich-gnu/8.2 (external mpich-7.7.16-ifjkd3v6a3hbtqmxekneycwh3wgrhc4k) [+] /usr (external pkg-config-0.29.2-ueeszbkidqgqjkns6gh6glgaxqws2x6r) [+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/gsl-2.6-vtuwngsrcrtb5uod2hbteacifcxich5c [+] /usr (external python-3.6.10-ifrblj537zg2nvq5baegeotax4h5dzm3) [+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/boost-1.76.0-qyx6cquksudterwipejedj3cbfmutmae [+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/gmp-6.2.1-6wmrr6qexxutxbxctk5rq5d4qg5w2g3z [+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/libxc-4.3.4-tur67qh7oxjbjjtmxmgi2hjty6dt4rue [+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/py-setuptools-50.3.2-3rr4jqvqhjhjiclh2tkgl23x6uuj6aae [+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/libint-2.6.0-3ym5vlezh7eisciqwhpeu7p74vqymata [+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/py-cython-0.29.22-q4r6465kru3jpvf3ac2lliljzs4cb3an [+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/plumed-2.6.2-vkrvu45ewjb3lvj5kf4uaj3x77pf3x6e [+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/cp2k-8.1-tnejptyboq56nppfikolzk5emc6vf5sb
In the end I don’t think my OpenBLAS build was necessary, looks like libsci handled that too!
My final Spack package spec:
> spack find -vdx cp2k ==> 1 installed package -- cray-cnl7-aarch64 / [email protected] -------------------------------- [email protected]~cosma~cuda~cuda_arch_35_k20x~cuda_blas~cuda_fft~elpa+libint~libvori+libxc+mpi+openmp~pexsi+plumed~sirius~spglib cuda_arch=none lmax=5 smm=blas [email protected]~mpi~openmp+shared [email protected]+mpi+openmp~pfft_patches precision=double,float [email protected]+fortran tune=cp2k-lmax-5 [email protected]+atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=98 patches=57a8401dee8f52b0342e0c8147a5b2db834e8d8f3fbcbbc5950016bd3e9e1ef0 visibility=hidden [email protected]~debug~pic+shared [email protected]+optimize+pic+shared [email protected] [email protected]~cuda+shared cuda_arch=none [email protected]~argobots+fortran+hwloc+hydra+libxml2+pci+romio~slurm~verbs+wrapperrpath device=ch4 netmod=ofi pmi=pmi [email protected]+gsl+mpi+shared arrayfire=none optional_modules=all [email protected]~external-cblas
Now with Spack loaded, the simplest way to load this build of cp2k into my environment is with the following:
> spack load [email protected]+mpi+openmp+plumed
I could instead load
[email protected] and Spack will check against all the builds and try find match. Right now both of these commands will succeed, since I only have one build. It is best to be more specific, so I specifically loaded cp2k version 8.1 with MPI, OpenMP & Plumed support.
> which cp2k cp2k.popt cp2k.psmp cp2k_shell.psmp > which cp2k.popt /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/cp2k-8.1-tnejptyboq56nppfikolzk5emc6vf5sb/bin/cp2k.popt
Getting started with spack took some time, mostly to wrap my head around the concepts of specs, variants, externals.. You should also be prepared to contend with build bugs, but that’s nothing new!
I found the Spack community to be responsive and helpful: https://github.com/spack/spack#community
Contributing bug fixes and issues is easy, they have provided commands to print the relevant debug info, logs and discover the package maintainers which allows you to directly ping them in Github.
This work used the Isambard 2 UK National Tier-2 HPC Service (http://gw4.ac.uk/isambard/) operated by GW4 and the UK Met Office, and funded by EPSRC (EP/T022078/1)