2 Prerequisites ¶

Building HPCToolkit requires the following prerequisites.

1. The GNU gcc, g++ and gfortran compilers version 8.x or later with support for C++17. On systems with older compilers, you can use spack to build a later version of gcc.
2. GNU glibc version 2.16 or later. Note: Red Hat 6.x uses glibc 2.12 which is too old.
3. Basic build tools: make, ld, ar, objcopy, nm, etc, and shell utilities: bash, sed, awk, grep, etc. Most Linux systems have these tools, or else you couldn’t compile anything. If you are building inside a container, you may need to add them.
4. Cmake version 3.14 or later, perl version 5.x, and python version 3.8 or later. On systems that are missing these tools or have versions that are too old, you can use spack to build a later version.
5. Git and curl for downloading files.
6. (optional) Environment (TCL) or LUA (Lmod) modules, if you want to make HPCToolkit available as a module. Again, spack can install these packages if they are missing from your system.

Hpcviewer and hpctraceviewer require Java 11 or later. Spack can install Java, if needed. On Linux, the viewers also require GTK+ version 3.20 or later.

3 Spack Notation ¶

Spack uses a special notation for specifying the version, variants, compilers and dependencies when describing how to build a package. This combination of version, variants, etc is called a ’spec’ and is used both on the command line and in config files.

1. ’@’ specifies the package version. spack info <package> shows the available versions and variants for a package. In most cases, spaces are optional between elements of a spec. For example:

   boost@1.77.0    hpctoolkit @develop    papi @=6.0.0    "libiberty@2.40"
   

   Note: foo@2.1 includes all versions beginning with 2.1, including 2.1, 2.1.0, 2.1.4, 2.1.9.9.9, 2.1.stable, etc. If you want version exactly 2.1, then use the notation foo@=2.1 to differentiate 2.1 from 2.1.*.

   Note: dotted version numbers with exactly two fields that end with a 0 (@1.10, @2.120, etc) should be quoted so that yaml does not treat the version @2.40 as the floating point number 2.4.

2. ’+’, ’-’, ’~’ specify boolean (on/off) variants. Note: - (dash) and ~ (tilde) both mean ’off’. Use dash after a space and tilde after a non-space. For example:

   elfutils+bzip2~nls    elfutils +bzip2 -nls    elfutils@0.186 +bzip2~nls
   

3. ’name=value’ specifies a non-boolean variant, for example:

   dyninst+openmp build_type=RelWithDebInfo    xerces-c@3.2.2 transcoder=iconv
   

4. ’%’ specifies the build compiler and its version, for example:

   hpctoolkit@develop %gcc@8.5.0
   

5. ’cflags’, ’cxxflags’, ’fflags’, ’cppflags’, ’ldflags’ and ’ldlibs’ are special name/value variants for compiler flags. These are normally not needed, but if you do need to add a flag to the build, then one example might be:

   amg2013 cflags='-O2 -mavx512pf'
   

6. ’^’ represents a dependency spec. The spec for a dependency package is a full spec and may include its own version, variants, etc. For example:

   hpctoolkit@develop ^dyninst@12.1.0+openmp
   

7. ’arch’, ’platform’, ’os’ and ’target’ are special options for the system architecture and machine type. Platform is normally ’linux’, or else ’cray’ (or even ’darwin’). OS (or ’operating_system’) is the Linux distribution, ’rhel8’, ’sles15’, etc, and target is the machine type, ’x86_64’, ’ppc64le’, etc. Arch is a triple of platform, os and target separated by dashes, for example linux-rhel8-x86_84.

   Normally, a system has only one arch type and you don’t need to specify this. However, for systems with separate front and back-end types, the default is the back end. For example, if you wanted to build for the front end on Cray, then you could use something like this.

   python@3.7.4 arch=cray-sles15-x86_64    boost os=fe
   

   Now that spack has implemented microarchitecture targets (haswell, ivybridge, etc), you can use ’target’ to build for a generic x86_64 or a specific CPU type. For example:

   amg2013 target=x86_64    lulesh target=ivybridge
   

   You can use spack arch to display the generic, top-level families and the micro-arch targets.

   spack arch --known-targets
   

The following command gives a summary of spack spec syntax.

spack help --spec

When writing a spec (for spack spec, install, etc), spack will fully resolve all possible choices for the package and all of its dependencies and create a unique hash value for that exact configuration. This process is called ’concretization.’ To see how spack would concretize a spec, use spack spec.

spack spec hpctoolkit@develop ^elfutils@0.187 ^boost@1.77.0

https://spack.readthedocs.io/en/latest/basic_usage.html#specs-dependencies

4 Clone Spack and HPCToolkit ¶

Spack is available via git clone from GitHub. This includes the core spack machinery and recipes for building over 7,000 packages (and growing). You should also clone HPCToolkit for the packages.yaml file which is used to configure the spack build. Note: spack is on GitHub, but hpctoolkit has moved to GitLab.

git clone https://github.com/spack/spack.git
git clone https://gitlab.com/hpctoolkit/hpctoolkit.git

After cloning, add the spack/bin directory to your PATH, or else source the spack setup-env script.

(bash)   .  /path/to/spack/share/spack/setup-env.sh
(csh)    setenv SPACK_ROOT /path/to/spack/root
         source $SPACK_ROOT/share/spack/setup-env.csh

It suffices to add spack/bin to your PATH (or even symlink the spack launch script). Sourcing the setup-env script adds extra support for modules built by spack.

5 Config.yaml ¶

config.yaml is the top-level spack config file. This specifies the directory layout for installed files and the top-level spack parameters.

By default, spack installs packages inside the spack repository at spack/opt/spack. To use another location, set the root field under install_tree in config.yaml. Normally, you will want to set this.

config:
  install_tree:
    root: /path/to/top-level/install/directory

There are a few other fields that you may want to set for your local system. These are all in config.yaml.

1. build_stage – the location where spack builds packages (default is in /tmp).
2. source_cache – where spack stores downloaded source tar files.
3. connect_timeout – some download sites, especially sourceforge are often slow to connect. If you find that connections are timing out, try increasing this time to 30 or 60 seconds (default is 10 seconds).
4. url_fetch_method – by default, spack uses a python library (urllib) to fetch source files. If you have trouble downloading files, try changing this to curl.
5. build_jobs – by default, spack uses all available hardware threads for parallel make, up to a limit of 16. If you want to use a different number, then set this.

The default config.yaml file is in the spack repository at spack/etc/spack/defaults. The simplest solution is to copy this file one directory up and then edit the copy (don’t edit the default file directly).

cd spack/etc/spack
cp defaults/config.yaml .
vi config.yaml

Alternatively, you could put this file in a separate directory, outside of the spack repository and then use -C/--config-scope dir on the spack command line. (The -C option goes before the spack command name.) This is useful if you maintain multiple config files for different machines.

spack -C dir install ...

Note: if you put config.yaml in spack/etc/spack, then it will apply to every spack command for that repository (and you won’t forget). Putting it in a separate directory is more flexible because you can support multiple configurations from the same repository. But then you must use -C dir with every spack command or else you will get inconsistent results.

You can view the current configuration and see where each entry comes from with spack config.

spack [-C dir] config get config
spack [-C dir] config blame config

See the spack docs on ‘Configuration Files’ and ‘Basic Settings’.

https://spack.readthedocs.io/en/latest/configuration.html
https://spack.readthedocs.io/en/latest/config_yaml.html

6 Modules.yaml ¶

Spack supports creating module files, but does not install them by default. If you want to install module files, then you need to edit modules.yaml to specify which type of modules to use (TCL or Lmod) and the install path.

modules:
  default:
    roots:
      # normally, need only one of these
      tcl:  /path/to/top-level/tcl-module/directory
      lmod: /path/to/top-level/lmod-module/directory
    enable:
      - tcl  (or lmod)

Also, for hpctoolkit, you should also turn off autoload for dependencies. By default, autoload loads the modules for hpctoolkit’s dependencies. But hpctoolkit does not need this and loading them may interfere with an application’s dependencies. You should do this for both tcl and lmod modules.

modules:
  default:
    tcl:
      hpctoolkit:
        autoload: none
      all:
        autoload: direct

7 Packages.yaml ¶

The packages.yaml file specifies the versions and variants for the packages that spack installs and serves as a common reference point for HPCToolkit’s prerequisites. This file also specifies the paths or modules for system build tools (cmake, python, etc) to avoid rebuilding them. Put this file in the same directory as config.yaml. A sample packages.yaml file is available in the spack directory of the hpctoolkit repository.

There are two main sections to packages.yaml. The first specifies the versions and variants for hpctoolkit’s prereqs. By default, spack will choose the latest version of each package (plus any constraints from hpctoolkit’s package.py file). In most cases, this will work, but not always. If you need to specify a different version or variant, then set this in packages.yaml. For example:

packages:
  elfutils:
  version:  [0.189]
  variants: ~nls

Note: the versions and variants specified in hpctoolkit’s package.py file are hard constraints and should not be changed. Variants in packages.yaml are preferences that may be modified for your local system. (But don’t report a bug until you have first tried the versions from packages.yaml that we supply.)

• External Packages
• Micro-Architecture Targets
• Require, Reuse and Fresh

spack external find cmake

cmake:
  externals:
  - spec: cmake@3.7.2
    modules:
    - CMake/3.7.2
  buildable: False

python:
  externals:
  - spec: python@2.7.18
    prefix: /usr
  - spec: python@3.6.8
    prefix: /usr

spack arch --known-targets

spack install hpctoolkit ... target=x86_64

packages:
  all:
    target: [x86_64]
    compiler: [gcc@9.3.0]
    providers:
      mpi: [openmpi]

https://spack.readthedocs.io/en/latest/build_settings.html
https://spack.readthedocs.io/en/latest/basic_usage.html#specs-dependencies

packages:
  dyninst:
    require: "@12.3.0"

packages:
  boost:
    require:
      - one_of: ["@1.75.0", "@1.77.0"]
  elfutils:
    require:
      - any_of: ["+bzip2", "+xz"]

packages:
  all:
    require: "%gcc@9.3.0 target=x86_64"
  mpi:
    require: "mpich@4.0"

https://spack.readthedocs.io/en/latest/build_settings.html#package-requirements

8 Bootstrapping Clingo ¶

The ’concretizer’ is the part of spack that converts a partial spec into a full spec with values for the version and variants of every package in the spec plus all dependencies. The new concretizer for spack (clingo) is a third-party python library for solving answer-set logic problems (eg, satisfiability). Normally, this only needs to be set up once per machine, the first time you run spack.

The easiest way to install clingo it to use spack’s pre-built libraries. These are available for Linux (x86_64, ppc64le, aarch64) and Macos/Darwin (x86_64) for python 3.7 or later. The Macos version also requires Macos 10.13 or later and the Xcode developer package (for python and other programs).

By default, spack will automatically install (bootstrap) clingo the first time you run a command that uses it (spec or solve). However, if this fails or you want to verify the steps yourself, then follow these steps.

In config.yaml, set concretizer to clingo.

config:
  concretizer: clingo

Spack needs at least one compiler configured (see below). If this is your first time running spack on this machine, then use compiler find to detect a compiler. Finally, use spack solve to trigger bootstrapping.

spack compiler list    (to display known compilers)
spack compiler find    (to add a compiler, if needed)
spack solve zlib
==> Bootstrapping clingo from pre-built binaries
...
zlib@1.2.11%gcc@8.4.1+optimize+pic+shared arch=linux-rhel8-zen

Spack stores the clingo bootstrap files in ~/.spack/bootstrap. You can check on the status of these files or clean (reset) them with the find or clean commands.

spack find -b     (displays the status of the bootstrap files)
spack clean -b    (erases the current bootstrap files)

If the binary bootstrap fails, then try the solve step with debugging turned on.

spack -d solve zlib

If the binary bootstrap fails or if your system is not supported, then you will need to let spack build clingo from source. Reset spack-install to true and rerun spack solve zlib. This requires a compiler with support for C++14 and takes maybe 30-45 minutes to install all the packages.

https://spack.readthedocs.io/en/latest/getting_started.html#bootstrapping-clingo
https://github.com/alalazo/spack-bootstrap-mirrors#supported-platforms

9 Compilers and compilers.yaml ¶

Building HPCToolkit requires GNU gcc/g++ version 8.x or later with C++17 support. By default, spack uses the latest available version of gcc, but you can specify a different compiler, if one is available.

Spack uses a separate file, compilers.yaml to store information about available compilers. This file is normally in your home directory at ~/.spack/platform where ‘platform’ is normally ‘linux’ (or else ‘cray’).

The first time you use spack, or after adding a new compiler, you should run spack compiler find to have spack search your system for available compilers. If a compiler is provided as a module, then you should load the module before running find. Normally, you only need to run find once, unless you want to add or delete a compiler. You can also run spack compiler list and spack compiler info to see what compilers spack knows about.

For example, on one power8 system running RedHat 7.3, /usr/bin/gcc is version 4.8.5, but gcc 8.3.0 is available as module GCC/8.3.0.

module load GCC/8.3.0

spack compiler find
==> Added 2 new compilers to /home/krentel/.spack/linux/compilers.yaml
    gcc@8.3.0  gcc@4.8.5
==> Compilers are defined in the following files:
    /home/krentel/.spack/linux/compilers.yaml

spack compiler list
==> Available compilers
-- gcc rhel7-ppc64le --------------------------------------------
gcc@8.3.0  gcc@4.8.5

spack compiler info gcc@8.3
gcc@8.3.0:
    paths:
        cc = /opt/apps/software/Core/GCCcore/8.3.0/bin/gcc
        cxx = /opt/apps/software/Core/GCCcore/8.3.0/bin/g++
        f77 = /opt/apps/software/Core/GCCcore/8.3.0/bin/gfortran
        fc = /opt/apps/software/Core/GCCcore/8.3.0/bin/gfortran
    modules  = ['GCC/8.3.0']
    operating system  = rhel7

Note: for compilers from modules, spack does not fill in the modules: field in the compilers.yaml file. You need to do this manually. In the above example, after running find, I edited compilers.yaml to add GCC/8.3.0 to the modules: field as below. This is important to how spack manipulates the build environment.

- compiler:
    modules: [GCC/8.3.0]
    operating_system: rhel7
    spec: gcc@8.3.0
    ...

Spack uses % syntax to specify the build compiler and @ syntax to specify the version. For example, suppose you had gcc versions 8.5.0, 9.3.0 and 10.2.0 available and you wanted to use 9.3.0. You could write this as:

spack install package %gcc@9.3.0

You can also set the choice of compiler in the all: section of packages.yaml.

packages:
  all:
    compiler: [gcc@9.3.0]

See the spack docs on ‘Compiler Configuration’.

https://spack.readthedocs.io/en/latest/getting_started.html#compiler-configuration

10 Python ¶

Spack uses Python for two things. First, to run the Spack scripts written in Python, and second, to use as a dependency for other spack packages. These do not have to be the same python version or install.

Currently, Spack requires at a minimum Python 3.7 to run spack at all. But 3.7 is deprecated and support for it will be removed in a few months. So, the best thing to do is to upgrade to Python 3.8 or later now.

If python 3.8 or later is not available on your system, then your options to install it are: (1) load a module for a later version, (2) use yum or apt to install a later version, (3) ask your sysadmin to install a later version, or (4) as a last resort, compile a later version from source.

If a later python is available on your system but not first in your PATH or under a different name, you can set the environment variable SPACK_PYTHON to the python3 binary. For example, suppose /usr/bin/python3 is too old, but python 3.8 is available as /usr/bin/python3.8, then you could use:

export SPACK_PYTHON=/usr/bin/python3.8

If set, SPACK_PYTHON is the path to the Python interpreter used to run Spack.

11 Spack Install ¶

First, set up your config.yaml, modules.yaml, packages.yaml and compilers.yaml files as above and edit them for your system. You can see how spack will build hpctoolkit with spack spec.

spack spec hpctoolkit

Then, the “one button” method uses spack to install everything.

spack install hpctoolkit

Tip: Spack fetch is somewhat fragile and sometimes has transient problems downloading files. You can use spack fetch -D to pre-fetch all of the tar files and resolve any downloading problems before starting the full install.

spack fetch -D hpctoolkit

12 Manual Install ¶

The manual method uses spack to build hpctoolkit’s prerequisites and then uses the traditional autotools configure && make && make install to install hpctoolkit. This method is primarily for developers who want to compile hpctoolkit, edit the source code and recompile, etc. This method is also useful if you need some configure option that is not available through spack.

First, use spack to build hpctoolkit’s prerequisites as above. You can either build some version of hpctoolkit as before (which will pull in the prerequisites), or else use --only dependencies to avoid building hpctoolkit itself.

spack install --only dependencies hpctoolkit

Then, configure and build hpctoolkit as follows. Hpctoolkit uses automake and so allows for parallel make.

configure  \
   --prefix=/path/to/hpctoolkit/install/prefix  \
   --with-spack=/path/to/spack/install_tree/linux-centos9-x86_64/gcc-11.3.1  \
   ...
make -j <num>
make install

The argument to --with-spack should be the directory containing all of the individual package directories, normally two directories down from the top-level install_tree and named by the platform and compiler. This option replaces the old --with-externals. The following are other options that may be useful. For the full list of options, see configure -h.

1. --enable-all-static – build hpcprof-mpi statically linked for the compute nodes.
2. --enable-develop – compile with optimization turned off for debugging.
3. --with-package=path – specify the install prefix for some prerequisite package, mostly for developers who want to use a custom, non-spack version of some package.
4. MPICXX=compiler – specify the MPI C++ compiler for hpcprof-mpi, may be a compiler name or full path.

Note: if your spack install tree has multiple versions or variants for the same package, then --with-spack will select the one with the most recent directory time stamp (and issue a warning). If this is not what you want, then you will need to specify the correct version with a --with-package option.

13 Advanced Options ¶

• CUDA
• Level Zero
• ROCM
• OpenCL
• MPI
• PAPI vs Perfmon
• Python

spack install hpctoolkit +cuda

configure  \
   --prefix=/path/to/hpctoolkit/install/prefix  \
   --with-spack=/path/to/spack/install/dir   \
   --with-cuda=/path/to/cuda/install/prefix  \
   ...

spack install hpctoolkit +level_zero +gtpin

packages:
  oneapi-igc:
    externals:
     - spec: oneapi-igc@1.0.10409
       prefix: /usr

configure  \
    --with-level0=/path/to/oneapi-level-zero/prefix  \
    --with-gtpin=/path/to/intel-gtpin/prefix  \
    --with-igc=/usr   (or oneapi-igc prefix)  \
    ...

packages:
  hip:
    externals:
    - spec: hip@5.0.0
      prefix: /opt/rocm-5.0.0

  hsa-rocr-dev:
    externals:
    - spec: hsa-rocr-dev@5.0.0
      prefix: /opt/rocm-5.0.0

  roctracer-dev:
    externals:
    - spec: roctracer-dev@5.0.0
      prefix: /opt/rocm-5.0.0

  rocprofiler-dev:
    externals:
    - spec: rocprofiler-dev@5.0.0
      prefix: /opt/rocm-5.0.0

spack install hpctoolkit +rocm ...

configure  \
   --with-rocm=/opt/rocm \    (for all-in-one /opt/rocm)

   --with-rocm-hip=/path/to/hip/prefix  \
   --with-rocm-hsa=/path/to/hsa-rocr-dev/prefix  \
   --with-rocm-tracer=/path/to/roctracer-dev/prefix  \
   --with-rocm-profiler=/path/to/rocprofiler-dev/prefix  \
   ...

spack install hpctoolkit +cuda +opencl

configure  \
    --with-cuda=/path/to/cuda/prefix  \
    --with-opencl=/path/to/opencl-c-headers/prefix  \
    ...

spack install hpctoolkit +mpi

packages:
  mpich:
    externals:
    - spec: mpich@4.0
      modules:
      - mpich/4.0

spack install hpctoolkit ~papi

spack install hpctoolkit +python

configure  \
    --enable-python=/path/to/python-config
    ...

14 Platform Specific Notes ¶

• Cray

module swap PrgEnv-cray PrgEnv-gnu
module unload darshan

compilers:
- compiler:
    spec: gcc@11.2.0
    paths:
      cc:  /opt/cray/pe/gcc/11.2.0/bin/gcc
      cxx: /opt/cray/pe/gcc/11.2.0/bin/g++
      f77: /opt/cray/pe/gcc/11.2.0/bin/gfortran
      fc:  /opt/cray/pe/gcc/11.2.0/bin/gfortran
    modules:
      - PrgEnv-gnu/8.3.3
      - gcc/11.2.0
    operating_system: sles15
    target: x86_64
    ...

/opt/cray/pe/mpich/8.1.17/ofi/gnu/9.1/bin/mpicxx

packages:
  mpi:
    require: "cray-mpich@8.1.17"

  cray-mpich:
    externals:
      - spec: "cray-mpich@8.1.17"
        prefix: /opt/cray/pe/mpich/8.1.17/ofi/gnu/9.1
        modules:
         - cray-mpich/8.1.17

spack install hpctoolkit +mpi os=fe    (or arch=cray-sles15-x86_64)

/usr/bin/ld: warning: libfabric.so.1, needed by /opt/cray/pe/mpich/8.1.17/ofi/gnu/9.1/lib/libmpi_gnu_91.so,
not found (try using -rpath or -rpath-link)
/usr/bin/ld: /opt/cray/pe/mpich/8.1.17/ofi/gnu/9.1/lib/libmpi_gnu_91.so:
undefined reference to `fi_strerror@FABRIC_1.0'

modules:
  - PrgEnv-gnu/8.3.3
  - gcc/11.2.0
  - craype/2.7.16
  - cray-mpich/8.1.17
  - libfabric/1.15.0.0

spack install --only dependencies hpctoolkit +mpi os=fe
spack install --dirty hpctoolkit +mpi os=fe

spack install hpctoolkit +cray os=fe

configure  \
    --prefix=/path/to/install/prefix  \
    --with-spack=/path/to/linux-sles15-x86_64/gcc-11.2.0  \
    MPICXX=CC    (or /path/to/mpicxx)

spack install hpctoolkit @2022.05.15 +cray +cray-static os=fe

15 HPCToolkit GUI Interface (Hpcviewer) ¶

Since the 2020.12 release, the HPCToolkit GUI interface provides both profile and trace views in a single application, i.e. hpcviewer. Prior to that, each view was a separate program: hpcviewer to analyze the profile database, and hpctraceviewer to display the traces.

We provide binary distributions for hpcviewer on Linux (x86_64, ppc64le and aarch64), Windows (x86_64) and MacOS (x86_64, M1 and M2). HPCToolkit databases are platform-independent and it is common to run hpcrun on one machine and then view the results on another machine.

Starting with 2021.01, the viewer now requires Java 11 or later, plus GTK+ 3.20 or later on Linux.

• Spack Install
• Manual Install

spack info openjdk
spack install hpcviewer
spack install hpcviewer ^openjdk @11.0.12_7    (if needed)

/download.html

./install /path/to/install/directory

16 Building a New Compiler ¶

Some systems may have compilers that are too old for building HPCToolkit or other packages. For example, RedHat 7.x comes with gcc 4.8.x which is very old. If your system doesn’t already have modules for later compilers, then you may need to build a new compiler yourself.

First, pick a directory in which to install the modules and make subdirectories for the spack packages and module files. In this example, I’m using /opt/spack/Modules as my top-level directory and subdirectories packages and modules. Edit config.yaml and modules.yaml to add these paths.

config:
  install_tree:
    root: /opt/spack/Modules/packages

modules:
  default:
    roots:
      # just use one of these
      module_roots:
        tcl:  /opt/spack/Modules/modules
        lmod: /opt/spack/Modules/modules

Determine if your system uses TCL (environment) or Lmod modules. Normally, the module command is a shell function. TCL modules use modulecmd and Lmod modules eval LMOD_CMD. Edit modules.yaml to enable the module type for your system. Again, you only need one of these (and don’t use dotkit unless you work at LLNL).

modules:
  default:
    enable:
     - tcl
     - lmod

Then, choose a version of gcc and use the default compiler (normally /usr/bin/gcc) to build the newer version. Currently, gcc 10.x or 11.x is a good choice that builds robustly, has enough modern features but is not too new to cause problems for some packages. For example,

spack install gcc@10.4.0

Note: it is not necessary to rebuild the new compiler with itself.

• Using the New Compiler
• Bootstrapping Environment Modules

module use /opt/spack/Modules/modules/linux-rhel7-x86_64
module load gcc-8.4.0-gcc-4.8.5-qemsqrc

module use /opt/spack/Modules/modules/linux-rhel7-x86_64/Core
module load gcc/8.4.0-dan4vbm

$ spack compiler find
==> Added 1 new compiler to /home/krentel/.spack/linux/compilers.yaml
    gcc@8.4.0

- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules:
    - gcc-8.4.0-gcc-4.8.5-qemsqrc
    operating_system: rhel7
    paths:
      cc:  /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gcc
      cxx: /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/g++
      f77: /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gfortran
      fc:  /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gfortran
    spec: gcc@8.4.0
    target: x86_64

packages:
  all:
    compiler: [gcc@8.4.0]

https://spack.readthedocs.io/en/latest/getting_started.html#compiler-configuration

spack install environment-modules
cd /path/to/environment-modules-5.3.0-ism7cdy4xverxywj27jvjstqwk5oxe2v/init
(bash)  . ./bash
(csh)   source ./csh

17 Spack Mirrors ¶

A spack mirror allows you to download and save a source tar file in advance. This is useful if your system is behind a firewall, or if you need to manually agree to a license, or if you just don’t want to keep downloading the same file over and over.

A mirror has a simple directory structure and is easy to set up. Create a top-level directory with subdirectories named after the spack packages and copy the tar files into their package’s subdirectory. For example,

my-mirror/
   boost/
      boost-1.66.0.tar.bz2      (from boost_1_66_0.tar.bz2)
      boost-1.70.0.tar.bz2
   dyninst/
      dyninst-10.1.0.tar.gz     (from git checkout)
   ibm-java/
      ibm-java-8.0.5.30.None    (from ibm-java-sdk-8.0-5.30-ppc64le-archive.bin)
   intel-xed/
      intel-xed-2019.03.01.tar.gz    (from git checkout)
      mbuild-2019.03.01.tar.gz       (resource from git checkout)
   jdk/
      jdk-1.8.0_202.tar.gz      (from jdk-8u202-linux-x64.tar.gz)

Note: the names of the files in the spack mirror always follow the same, specific format, regardless of the actual name of the tar file. Version is the spack name for the version (from spack info), and extension is the same extension as the tar file (tar.gz, tar.bz2, etc) or else None for other types of files.

<package-name> - <version> . <extension>

For example, the boost 1.66.0 tar file is actually named boost_1_66_0.tar.bz2 but is stored in the mirror as boost-1.66.0.tar.bz2 and jdk-8u202-linux-x64.tar.gz is renamed to jdk-1.8.0_202.tar.gz.

For packages that use a snapshot from a git repository (tag or commit hash), clone the repository, checkout the desired version, make a tar file and gzip the file. (You should exclude the .git subdirectory.) But note that spack refuses to use a cached file for the head of a branch because it is a moving target.

Finally, after creating the mirror directory, add it to spack with spack mirror add. For example,

spack mirror add my-mirror file:///home/krentel/spack/my-mirror
spack mirror list
my-mirror    file:///home/krentel/spack/mirror

Note: by default, spack stores downloaded files inside the spack repository at spack/var/spack/cache. This directory is a full spack mirror, so instead of creating a separate directory tree, you could just copy the files into the cache directory. This is useful when spack fetch has trouble downloading a file. If you can download the file manually, or copy it from another machine, then just rename the file as above and copy it into the spack file cache.

For more information on mirrors, see:

https://spack.readthedocs.io/en/latest/mirrors.html

18 Common Problems ¶

• Unable to fetch tar file
• New releases break the build
• Failure to load modules

==> Installing m4
==> Searching for binary cache of m4
==> No binary for m4 found: installing from source
curl: (6) Could not resolve host: ftp.wayne.edu; Name or service not known
==> Fetching https://ftpmirror.gnu.org/m4/m4-1.4.18.tar.gz
==> Fetching from https://ftpmirror.gnu.org/m4/m4-1.4.18.tar.gz failed.
==> Error: FetchError: All fetchers failed for m4-1.4.18-vorbvkcjfac43b7vuswsvnm6xe7w7or5

cmake: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by cmake)
cmake: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by cmake)
cmake: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by cmake)
==> Error: ProcessError: Command exited with status 1: