Diverse users running across a diverse set of architectures, it seems Spack was built just for this!

I want to share my experiences with Spack, here my focus is on CP2K. My build environment is a Cray XC50 running Thunder X2 Arm CPUs.

Spack is a source-based package manager and build system, all builds take place in self-contained environments. Every installed package is defined by a spec, the spec is a string that represents the enabled features, compiler, etc.. that results in a unique package. The process of determining the packages unique hash is called concretization.

You can find all of this explained much better in the public documentation or by following the Spack Tutorial

Getting started

> spack compiler find

This populates ~/.spack/cray/compilers.yaml with all our compilers, Spack checks typical locations as well as your available modules.

First attempt

> spack install [email protected] +openmp +mpi +plumed %[email protected] smm=blas ^fftw +openmp ^openblas threads=openmp

+openmp is default

+mpi to enable MPI support

+plumed to enable PLUMED

%[email protected] uses GCC 9.2.0, loaded via a module by Spack

smm=blas is required as libxsmm not available on arm64, noted in spec .

^fftw +openmp enables OpenMP for the FFTW dependency

^openblas threads=openmp uses OpenMP for threadding in OpenBLAS, I found the threads= options in its spec file but I could have just run spack info openblas.

I saw some differing results when I loaded modules into my environment before running Spack so I try avoid loading any when building packages.

Now, this build fails to concretize and Spack shows us what it tried to do and which packages are conflicting:

> spack install [email protected] +openmp +mpi +plumed %[email protected] smm=blas ^fftw +openmp ^openblas threads=openmp
==> Error: Conflicts in concretized spec "[email protected]%[email protected]~cosma~cuda~cuda_arch_35_k20x~cuda_blas~cuda_fft~elpa+libint~libvori+libxc+mpi+openmp~pexsi+plumed~sirius~spglib cuda_arch=none lmax=5 smm=blas arch=cray-cnl7-aarch64/kizgfsf"
List of matching conflicts for spec:

    [email protected]%[email protected]~cosma~cuda~cuda_arch_35_k20x~cuda_blas~cuda_fft~elpa+libint~libvori+libxc+mpi+openmp~pexsi+plumed~sirius~spglib cuda_arch=none lmax=5 smm=blas arch=cray-cnl7-aarch64
        ^[email protected]%[email protected]+mpi+openmp~pfft_patches precision=double,float arch=cray-cnl7-aarch64
        ^[email protected]%[email protected]+fortran tune=cp2k-lmax-5 arch=cray-cnl7-aarch64
...
...
1. "^openblas threads=none" conflicts with "cp2k+openmp"

You can see the concretized spec of my cp2k installation is

[email protected]%[email protected]~cosma~cuda~cuda_arch_35_k20x~cuda_blas~cuda_fft~elpa+libint~libvori+libxc+mpi+openmp~pexsi+plumed~sirius~spglib cuda_arch=none lmax=5 smm=blas arch=cray-cnl7-aarch64

This includes any defaults for every variant the package supports, I prefer to leave it up to the maintainer for variants I’m not familiar with, but if you want highly reproducible environments you will probably want to specify everything.

The error ^openblas threads=none" conflicts with "cp2k+openmp suggests the openblas dependency won’t support threading, which is required because I’m building cp2k with openmp support.

packages.yaml

OpenBLAS isn’t tricky to compile, so I opted to compile it manually:

> make
> make PREFIX=/software/arm64/apps/openblas/0.3.15 install

Now I need to tell Spack about my installation of OpenBLAS, this kind of package is called an External and they are listed in ~/.spack/packages.yaml:

packages:
  autoconf:
    externals:
    - spec: [email protected]
      prefix: /usr
  automake:
    externals:
    - spec: [email protected]
      prefix: /usr
  bash:
    externals:
    - spec: [email protected]
      prefix: /
...

I added OpenBLAS and a few Cray libraries to my packages.yaml:

  mpi:
    externals:
    - spec: "[email protected]"
      modules:
      - cray-mpich/7.7.16
    buildable: False
  openblas:
    externals:
    - spec: "[email protected]"
      modules:
      - apps/openblas/0.3.15
  fftw:
    externals:
    - spec: "fftw+openmp"
      modules:
      - cray-fftw
  cray-libsci:
    externals:
    - spec: "cray-libsci"
      modules:
      - cray-libsci
    buildable: False

Each external has a spec that Spack matches against, for example if I ask for fftw explicitly without openmp support then it won’t use my fftw external as I have listed +openmp in its spec.

I used buildable: False to tell Spack to never build an alternative for some packages. I want to use the supported system libraries where possible, instead of some apps using cray-libsci and others building a custom libsci, as an example.

JSONDecodeError

This bug slowed me down, I didn’t find out the cause and it has since cleared itself up. It may be related to loading spack on different clusters with a shared home directory and it ended up loading something confusing in compilers.yaml? 🤷‍♂️

==> cp2k: Executing phase: 'edit'
==> Error: JSONDecodeError: Expecting value: line 1 column 1 (char 0)

/lustre/software/aarch64/spack/var/spack/repos/builtin/packages/cp2k/package.py:300, in edit:
        297            ldflags  += ['-hnoomp']
        298
        299        if '@7:' in spec:  # recent versions of CP2K use C++14 CUDA code
  >>    300            cxxflags.append(self.compiler.cxx14_flag)
        301            nvflags.append(self.compiler.cxx14_flag)
        302
        303        ldflags.append(fftw.libs.search_flags)

I opened bug #23570 to track it.

Missing build dependency

I got the following error from plumed: RuntimeError: Cannot generate configure: missing dependencies ['m4']

Spack seems to think m4 isn’t available, it turned out that the package’s package.py was missing the type='build' on the line where m4 is listed as a build depdenency. This was fixed in #23953 .

Build

Lastly I had difficulty building netlib-scalapack until I realised that cray-libsci provides the LAPACK/ScaLAPACK capabilities cp2k requires. I manually set the dependency package with ^cray-libsci.

> spack install [email protected] +openmp +mpi +plumed %[email protected] smm=blas ^cray-libsci
==> Warning: Missing a source id for [email protected]
==> Warning: Missing a source id for [email protected]
==> Warning: Missing a source id for [email protected]
==> [email protected] : has external module in ['cray-libsci']
[+] /opt/cray/pe/libsci/20.06.1/GNU/8.1/aarch64 (external cray-libsci-20.06.1-lcxhogmgo4zwfosmyp6wddo7xvga4ybx)
==> [email protected] : has external module in ['cray-fftw']
[+] /opt/cray/pe/fftw/3.3.8.5/arm_thunderx2 (external fftw-3.3.9-bwhmlzy6eik3ajeqobol527ww6s42vcn)
[+] /usr (external autoconf-2.69-ememqml5q3hrhatuxevmdaeyuocnaqnk)
[+] /usr (external automake-1.15.1-edymz5en4xwxwybe76d3535re7euacgl)
[+] /usr (external bzip2-1.0.6-2fqflhapqghfvebfk5kvogp7g4xulquw)
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/zlib-1.2.11-cx7za7abtiy5ezumlkfxsffzfxuj3xq6
[+] /usr (external libtool-2.4.6-utlxqeicabegxrlreyo3biinw7v4vswn)
[+] /usr (external m4-1.4.18-og3jty4rquctsfxnxzwaufttw6bylag4)
[+] /usr (external perl-5.26.1-lk6rvqusbsxtnumqre6ete2tvgxghgxx)
==> [email protected] : has external module in ['cray-mpich/7.7.16']
[+] /opt/cray/pe/mpt/7.7.16/gni/mpich-gnu/8.2 (external mpich-7.7.16-ifjkd3v6a3hbtqmxekneycwh3wgrhc4k)
[+] /usr (external pkg-config-0.29.2-ueeszbkidqgqjkns6gh6glgaxqws2x6r)
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/gsl-2.6-vtuwngsrcrtb5uod2hbteacifcxich5c
[+] /usr (external python-3.6.10-ifrblj537zg2nvq5baegeotax4h5dzm3)
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/boost-1.76.0-qyx6cquksudterwipejedj3cbfmutmae
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/gmp-6.2.1-6wmrr6qexxutxbxctk5rq5d4qg5w2g3z
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/libxc-4.3.4-tur67qh7oxjbjjtmxmgi2hjty6dt4rue
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/py-setuptools-50.3.2-3rr4jqvqhjhjiclh2tkgl23x6uuj6aae
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/libint-2.6.0-3ym5vlezh7eisciqwhpeu7p74vqymata
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/py-cython-0.29.22-q4r6465kru3jpvf3ac2lliljzs4cb3an
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/plumed-2.6.2-vkrvu45ewjb3lvj5kf4uaj3x77pf3x6e
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/cp2k-8.1-tnejptyboq56nppfikolzk5emc6vf5sb

Success!

In the end I don’t think my OpenBLAS build was necessary, looks like libsci handled that too!

My final Spack package spec:

> spack find -vdx cp2k
==> 1 installed package
-- cray-cnl7-aarch64 / [email protected] --------------------------------
[email protected]~cosma~cuda~cuda_arch_35_k20x~cuda_blas~cuda_fft~elpa+libint~libvori+libxc+mpi+openmp~pexsi+plumed~sirius~spglib cuda_arch=none lmax=5 smm=blas
    [email protected]~mpi~openmp+shared
    [email protected]+mpi+openmp~pfft_patches precision=double,float
    [email protected]+fortran tune=cp2k-lmax-5
        [email protected]+atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=98 patches=57a8401dee8f52b0342e0c8147a5b2db834e8d8f3fbcbbc5950016bd3e9e1ef0 visibility=hidden
            [email protected]~debug~pic+shared
            [email protected]+optimize+pic+shared
        [email protected]
    [email protected]~cuda+shared cuda_arch=none
    [email protected]~argobots+fortran+hwloc+hydra+libxml2+pci+romio~slurm~verbs+wrapperrpath device=ch4 netmod=ofi pmi=pmi
    [email protected]+gsl+mpi+shared arrayfire=none optional_modules=all
        [email protected]~external-cblas

Playtime

Now with Spack loaded, the simplest way to load this build of cp2k into my environment is with the following:

> spack load cp2k@8.1+mpi+openmp+plumed

I could instead load cp2k or [email protected] and Spack will check against all the builds and try find match. Right now both of these commands will succeed, since I only have one build. It is best to be more specific, so I specifically loaded cp2k version 8.1 with MPI, OpenMP & Plumed support.

> which cp2k
cp2k.popt        cp2k.psmp        cp2k_shell.psmp
> which cp2k.popt
/lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/cp2k-8.1-tnejptyboq56nppfikolzk5emc6vf5sb/bin/cp2k.popt

Thoughts

Getting started with spack took some time, mostly to wrap my head around the concepts of specs, variants, externals.. You should also be prepared to contend with build bugs, but that’s nothing new!

I found the Spack community to be responsive and helpful: https://github.com/spack/spack#community

Contributing bug fixes and issues is easy, they have provided commands to print the relevant debug info, logs and discover the package maintainers which allows you to directly ping them in Github.