[ Home | Overview | Publications | Software/Downloads ] • [ Documentation/Questions | Training Videos and Slides ] • [ People | Acks ]

Installing HPCToolkit with Spack

Table of Contents

1 Introduction

These notes describe how to build and install HPCToolkit and hpcviewer and their prerequisites with Spack. HPCToolkit proper (hpcrun, hpcstruct and hpcprof) is used to measure and analyze an application’s performance and then produce a database for hpcviewer. HPCToolkit is supported on the following platforms. IBM Blue Gene is no longer supported.

  1. Linux (64-bit) on x86_64, little-endian powerpc (power8 and 9) and ARM (aarch64). Big endian powerpc is no longer supported.
  2. Cray on x86_64 and Compute Node Linux.

We provide binary distributions for hpcviewer and hpctraceviewer on Linux (x86_64, ppc64/le and aarch64), Windows and MacOS. HPCToolkit databases are platform-independent and it is common to run hpcrun on one machine and then view the results on another machine.

We build HPCToolkit and its prerequisite libraries from source. HPCToolkit has some 15-20 base prerequisites (more for cuda or rocm) and we now use spack to build them. It is possible to use spack to install all of hpctoolkit or build just the prerequisites and then build hpctoolkit with the traditional configure ; make ; make install method from autotools. Developers will probably want to run configure and make manually, but both methods are supported.

Note: the old method of using hpctoolkit-externals to build the prerequisite libraries is now superseded by spack and is no longer supported.

These notes are written mostly from the view of using spack to build hpctoolkit and its dependencies. If you are a more experienced spack user, especially if you want to use spack to build hpctoolkit plus several other packages, then you will want to adapt these directions to your own needs.

Spack documentation is available at:


The current status of using Spack for HPCToolkit is at:


Last revised: April 14, 2022.

2 Prerequisites

Building HPCToolkit requires the following prerequisites.

  1. The GNU gcc and g++ compilers version 7.3 or later. On systems with older compilers, you can use spack to build a later version of gcc.
  2. GNU glibc version 2.16 or later. Note: Red Hat 6.x uses glibc 2.12 which is too old.
  3. Basic build tools: make, ld, ar, objcopy, nm, etc, and shell utilities: bash, sed, awk, grep, etc. Most Linux systems have these tools, or else you couldn’t compile anything.
  4. Cmake version 3.4 or later, perl version 5.x, and python version 3.6 or later. On systems that are missing these tools or have versions that are too old, you can use spack to build a later version.
  5. Git and curl for downloading files.
  6. (optional) Environment (TCL) or LUA (Lmod) modules, if you want to make HPCToolkit available as a module. Again, spack can install these packages if they are missing from your system.

Hpcviewer and hpctraceviewer require Java 11 or later. Spack can install Java, if needed. On Linux, the viewers also require GTK+ version 3.20 or later. Older versions of the viewer use Java 8 and GTK+ 2.x.

3 Spack Notation

Spack uses a special notation for specifying the version, variants, compilers and dependencies when describing how to build a package. This combination of version, variants, etc is called a ’spec’ and is used both on the command line and in config files.

  1. ’@’ specifies the package version. spack info <package> shows the available versions and variants for a package. In most cases, spaces are optional between elements of a spec. For example:
    boost@1.66.0    dyninst @10.1.0    hpctoolkit @master
  2. ’+’, ’-’, ’~’ specify boolean (on/off) variants. Note: - (dash) and ~ (tilde) both mean ’off’. Use dash after a space and tilde after a non-space. For example:
    elfutils+bzip2~nls    elfutils +bzip2 -nls    elfutils@0.176 +bzip2~nls
  3. ’name=value’ specifies a non-boolean variant, for example:
    dyninst+openmp build_type=RelWithDebInfo    xerces-c@3.2.2 transcoder=iconv
  4. ’%’ specifies the build compiler and its version, for example:
    hpctoolkit@master %gcc@7.3.0
  5. ’cflags’, ’cxxflags’, ’fflags’, ’cppflags’, ’ldflags’ and ’ldlibs’ are special name/value variants for compiler flags. These are normally not needed, but if you do need to add a flag to the build, then one example might be:
    amg2013 cflags='-O2 -mavx512pf'
  6. ’^’ represents a dependency spec. The spec for a dependency package is a full spec and may include its own version, variants, etc. For example:
    hpctoolkit@master ^dyninst@10.1.0+openmp
  7. ’arch’, ’platform’, ’os’ and ’target’ are special options for the system architecture and machine type. Platform is normally ’linux’, or else ’cray’ (or even ’darwin’). OS is the Linux distribution, ’rhel7’, ’sles15’, etc, and target is the machine type, ’x86_64’, ’ppc64le’, etc. Arch is a triple of platform, os and target separated by dashes.

    Normally, a system has only one arch type and you don’t need to specify this. However, for systems with separate front and back-end types, the default is the back end. For example, if you wanted to build python for the front end on Cray, then you might use something like this.

    python@3.7.4 arch=cray-sles15-x86_64

    Also, now that spack has implemented microarchitecture targets (haswell, ivybridge, etc), you can use ’target’ to build for a generic x86_64 or a specific CPU type. For example:

    amg2013 target=x86_64    lulesh target=ivybridge

    You can use spack arch to display the generic, top-level families and the micro-arch targets.

    spack arch --known-targets

When writing a spec (for spack install or other), spack will fully resolve all possible choices for the package and all of its dependencies and create a unique hash value for that exact configuration. This process is called ’concretization.’ To see how spack would concretize a spec, use spack spec.

spack spec hpctoolkit@master ^binutils@2.34 ^boost@1.72.0

4 Clone Spack and HPCToolkit

Spack is available via git clone from GitHub. This includes the core spack machinery and recipes for building over 6,300 packages (and growing). You should also clone HPCToolkit for the packages.yaml file which is used to configure the spack build.

git clone https://github.com/spack/spack.git
git clone https://github.com/hpctoolkit/hpctoolkit.git

After cloning, add the spack/bin directory to your PATH, or else source the spack setup-env script.

(bash)   .  /path/to/spack/share/spack/setup-env.sh
(csh)    setenv SPACK_ROOT /path/to/spack/root
         source $SPACK_ROOT/share/spack/setup-env.csh

It suffices to add spack/bin to your PATH (or even symlink the spack launch script). Sourcing the setup-env script adds extra support for modules built by spack.

5 Config.yaml and Install Roots

config.yaml is the top-level spack config file. This specifies the directory layout for installed files and the top-level spack parameters. There are two fields that you normally want to set, especially if you want to install packages and modules for hpctoolkit in a public directory. For a module to be available, both the install_tree and module_roots directories must be accessible.

By default, spack installs packages inside the spack repository at spack/opt/spack. To use another location, set the root field under install_tree in config.yaml.

    root: /path/to/top-level/install/directory

By default, spack installs module files inside the spack repository at spack/share/spack. The syntax for resetting this is changing. In the new syntax, the paths go in modules.yaml.

      # normally, need only one of these
      tcl:  /path/to/top-level/tcl-module/directory
      lmod: /path/to/top-level/lmod-module/directory
      - tcl  (or lmod)

There are a few other fields that you may want to set for your local system. These are all in config.yaml.

  1. connect_timeout – some download sites, especially sourceforge are often slow to connect. If you find that connections are timing out, then increase this time to 30 or 60 seconds (default is 10 seconds).
  2. url_fetch_method – by default, spack uses a python library (urllib) to fetch source files. If you have trouble downloading files, try changing this to curl.
  3. build_jobs – by default, spack uses all available hardware threads for parallel make, up to a limit of 16. If you want to use a different number, then set this.

There are also parameters for the locations of the build directories, the cache of downloaded tar files, etc, which you may wish to set.

The default config.yaml file is in the spack repository at spack/etc/spack/defaults. The simplest solution is to copy this file one directory up and then edit the copy (don’t edit the default file directly).

cd spack/etc/spack
cp defaults/config.yaml .
vi config.yaml

Alternatively, you could put this file in a separate directory, outside of the spack repository and then use -C dir on the spack command line. (The -C option goes before the spack command name.)

spack -C dir install ...

Note: if you put config.yaml in spack/etc/spack, then it will apply to every spack command for that repository (and you won’t forget). Putting it in a separate directory is more flexible because you can support multiple configurations from the same repository. But then you must use -C dir with every spack command or else you will get inconsistent results.

You can view the current configuration and see where each entry comes from with spack config.

spack [-C dir] config get config
spack [-C dir] config blame config

See the spack docs on ‘Configuration Files’ and ‘Basic Settings’.


6 Packages.yaml

The packages.yaml file specifies the versions and variants for the packages that spack installs and serves as a common reference point for HPCToolkit’s prerequisites. This file also specifies the paths or modules for system build tools (cmake, python, etc) to avoid rebuilding them. Put this file in the same directory as config.yaml. A sample packages.yaml file is available in the spack directory of the hpctoolkit repository.

There are two main sections to packages.yaml. The first specifies the versions and variants for hpctoolkit’s prereqs. By default, spack will choose the latest version of each package (plus any constraints from hpctoolkit’s package.py file). In most cases, this will work, but not always. If you need to specify a different version or variant, then set this in packages.yaml.

Note: the versions and variants specified in hpctoolkit’s package.py file are hard constraints and should not be changed. Variants in packages.yaml are preferences that may be modified for your local system. (But don’t report a bug until you have first tried the versions from packages.yaml that we supply.)

There are at least two packages with a variant that you may need to change depending on your system. But always check the current packages.yaml file to see if any more have been added.

  1. binutils – avoid versions 2.35 and 2.35.1, they have a bug that spews errors from hpcprof. Use 2.34 until 2.36 is available.
  2. intel-tbb – for very old Intel or AMD systems that don’t support transactional memory, change +tm to ~tm. (This option has no effect on non-x86 systems.)

6.1 External Packages

The other sections in packages.yaml specify paths or modules for other packages and system build tools. Building hpctoolkit’s prerequisites requires cmake 3.4 or later, perl 5.x and python 3.5 or later. There are three ways to satisfy these requirements: a system installed version (eg, /usr), a pre-built module or build from scratch.

By default, spack will rebuild these from scratch, even if your local version is perfectly fine. If you already have an installed version and prefer to use that instead, then you can specify this in packages.yaml.

The easiest way to use a pre-built package is to let spack find the package itself. Make sure the program is on your PATH and run spack external. For example, to search for cmake, use:

spack external find cmake

This does not work for every spack package, but it does work with cmake, perl and python. Note: spack puts these entries in packages.yaml in the .spack subdirectory of your home directory.

You can also add these entries manually to packages.yaml. For example, this entry says that cmake 3.7.2 is available from module CMake/3.7.2. buildable: False is optional and means that spack must find a matching external spec or else fail the build.

  - spec: cmake@3.7.2
    - CMake/3.7.2
  buildable: False

This example says that python2 and python3 are both available in /usr/bin. Note that the prefix entry is the parent directory of bin, not the bin directory itself.

  - spec: python@2.7.18
    prefix: /usr
  - spec: python@3.6.8
    prefix: /usr

Note: as a special rule for python, use package name python, even though the program name is python2 or python3.

Warning: It is Ok to use spack externals for build utilities that exist on your system (cmake, perl, python). However, we strongly recommend that you should rebuild all prereq packages that link code into hpctoolkit (dyninst, elfutils, etc).

6.2 Micro-Architecture Targets

Spack implements a hierarchy of micro-architecture targets, where ’target’ is a specific architecture (eg, haswell, ivybridge) instead of a generic family (x86_64). This allows the compiler to optimize code for the specific target.

You will notice this choice in two main places: the ’spack spec’ and the path for the install directory. For example, linux-rhel7-x86_64 might become linux-rhel7-broadwell. You can use spack arch to see the list of generic families and micro-architecture targets.

spack arch --known-targets

If you prefer a generic install, you can use the target option to specify a generic family instead of a micro-architecture target. This would be useful for a shared install that needs to work across multiple machines with different micro-arch types. For example:

spack install hpctoolkit ... target=x86_64

If you want to use this target for all spack builds, then you can specify this in packages.yaml. The generic families are: x86_64, ppc64, ppc64le and aarch64. For example:

    target: [x86_64]

See the spack docs on ’Build Customization’ and ’Specs and Dependencies’.


7 Bootstrapping Clingo

The ’concretizer’ is the part of spack that converts a partial spec into a full spec with values for the version and variants of every package in the spec plus all dependencies. The new concretizer for spack (clingo) is a third-party python library for solving answer set logic problems (eg, satisfiability). Normally, this only needs to be set up once per machine, the first time you run spack.

The easiest way to install clingo it to use spack’s pre-built libraries. These are available for Linux (x86_64, ppc64le, aarch64) and Macos/Darwin (x86_64) for python 3.5 or later. The Macos version also requires Macos 10.13 or later and the Xcode developer package (for python and other programs).

By default, spack will automatically install (bootstrap) clingo the first time you run a command that uses it (eg, spec or solve). However, if this fails or you want to verify the steps yourself, then follow these steps.

In config.yaml, set concretizer to clingo.

  concretizer: clingo

Spack needs at least one compiler configured (see below). If this is your first time running spack on this machine, then use compiler find to detect a compiler. Finally, use spack solve to trigger bootstrapping.

spack compiler list    (to display known compilers)
spack compiler find    (to add a compiler, if needed)
spack solve zlib
==> Bootstrapping clingo from pre-built binaries
zlib@1.2.11%gcc@8.4.1+optimize+pic+shared arch=linux-rhel8-zen

Spack stores the clingo bootstrap files in ~/.spack/bootstrap. You can check on the status of these files or clean (reset) them with the find or clean commands.

spack find -b     (displays the status of the bootstrap files)
spack clean -b    (erases the current bootstrap files)

If the binary bootstrap fails, then try the solve step with debugging turned on.

spack -d solve zlib

If the binary bootstrap fails or if your system is not supported, then you will need to let spack build clingo from source. Reset spack-install to true and rerun spack solve zlib. This requires a compiler with support for C++14 and takes maybe 30-45 minutes to install all the packages.


8 Compilers and compilers.yaml

Building HPCToolkit requires GNU gcc/g++ version 7.3 or later. By default, spack uses the latest available version of gcc, but you can specify a different compiler, if one is available.

Spack uses a separate file, compilers.yaml to store information about available compilers. This file is normally in your home directory at ~/.spack/platform where ‘platform’ is normally ‘linux’ (or else ‘cray’ or ‘bgq’).

The first time you use spack, or after adding a new compiler, you should run spack compiler find to have spack search your system for available compilers. If a compiler is provided as a module, then you should load the module before running find. Normally, you only need to run find once, unless you want to add or delete a compiler. You can also run spack compiler list and spack compiler info to see what compilers spack knows about.

For example, on one power8 system running RedHat 7.3, /usr/bin/gcc is version 4.8.5, but gcc 8.3.0 is available as module GCC/8.3.0.

module load GCC/8.3.0

spack compiler find
==> Added 2 new compilers to /home/krentel/.spack/linux/compilers.yaml
    gcc@8.3.0  gcc@4.8.5
==> Compilers are defined in the following files:

spack compiler list
==> Available compilers
-- gcc rhel7-ppc64le --------------------------------------------
gcc@8.3.0  gcc@4.8.5

spack compiler info gcc@8.3
        cc = /opt/apps/software/Core/GCCcore/8.3.0/bin/gcc
        cxx = /opt/apps/software/Core/GCCcore/8.3.0/bin/g++
        f77 = /opt/apps/software/Core/GCCcore/8.3.0/bin/gfortran
        fc = /opt/apps/software/Core/GCCcore/8.3.0/bin/gfortran
    modules  = ['GCC/8.3.0']
    operating system  = rhel7

Note: for compilers from modules, spack does not fill in the modules: field in the compilers.yaml file. You need to do this manually. In the above example, after running find, I edited compilers.yaml to add GCC/8.3.0 to the modules: field as below. This is important to how spack manipulates the build environment.

- compiler:
    modules: [GCC/8.3.0]
    operating_system: rhel7
    spec: gcc@8.3.0

Spack uses % syntax to specify the build compiler and @ syntax to specify the version. For example, suppose you had gcc versions 7.3.1, 8.3.0 and 10.2.0 available and you wanted to use 8.3.0. You could write this as:

spack install package %gcc@8.3.0

See the spack docs on ‘Compiler Configuration’.


9 Spack Install

First, set up your config.yaml, packages.yaml and compilers.yaml files as above and edit them for your system. You can see how spack will build hpctoolkit with spack spec.

spack spec hpctoolkit

Then, the “one button” method uses spack to install everything.

spack install hpctoolkit

Tip: Spack fetch is somewhat fragile and often has transient problems downloading files. You can use spack fetch -D to pre-fetch all of the tar files and resolve any downloading problems before starting the full install.

spack fetch -D hpctoolkit

10 Manual Install

The manual method uses spack to build hpctoolkit’s prerequisites and then uses the traditional autotools configure && make && make install to install hpctoolkit. This method is primarily for developers who want to compile hpctoolkit, edit the source code and recompile, etc. This method is also useful if you need some configure option that is not available through spack.

First, use spack to build hpctoolkit’s prerequisites as above. You can either build some version of hpctoolkit as before (which will pull in the prerequisites), or else use --only dependencies to avoid building hpctoolkit itself.

spack install --only dependencies hpctoolkit

Then, configure and build hpctoolkit as follows. Hpctoolkit uses automake and so allows for parallel make.

configure  \
   --prefix=/path/to/hpctoolkit/install/prefix  \
   --with-spack=/path/to/spack/install_tree/linux-fedora26-x86_64/gcc-7.3.1  \
make -j <num>
make install

The argument to --with-spack should be the directory containing all of the individual package directories, normally two directories down from the top-level install_tree and named by the platform and compiler. This option replaces the old --with-externals. The following are other options that may be useful. For the full list of options, see configure -h.

  1. --enable-all-static – build hpcprof-mpi statically linked for the compute nodes.
  2. --enable-develop – compile with optimization turned off for debugging.
  3. --with-package=path – specify the install prefix for some prerequisite package, mostly for developers who want to use a custom, non-spack version of some package.
  4. MPICXX=compiler – specify the MPI C++ compiler for hpcprof-mpi, may be a compiler name or full path.

Note: if your spack install tree has multiple versions or variants for the same package, then --with-spack will select the one with the most recent directory time stamp (and issue a warning). If this is not what you want, then you will need to specify the correct version with a --with-package option.

11 Advanced Options

11.1 CUDA

Beginning with the 2020.03.01 version, HPCToolkit now supports profiling CUDA binaries (nVidia only). For best results, use CUDA version 10.1 or later and Dyninst 10.1 or later. Note: in addition to a CUDA installation, you also need the CUDA system drivers installed. This normally requires root access and is outside the scope of spack.

For a spack install with CUDA, use the +cuda variant.

spack install hpctoolkit +cuda

For a manual install, either download and install CUDA or use an existing module, and then use the --with-cuda configure option.

configure  \
   --prefix=/path/to/hpctoolkit/install/prefix  \
   --with-spack=/path/to/spack/install/dir   \
   --with-cuda=/path/to/cuda/install/prefix  \

If you installed CUDA with spack in the same directory as the rest of the prerequisites, then the --with-spack option should find it automatically (but check the summary at the end of the configure output). If you are using CUDA from a separate system module, then you will need the --with-cuda option.

11.2 ROCM

HPCToolkit supports profiling AMD GPU binaries through the HIP/ROCM interface, and beginning with version 2022.04.15, we support building hpctoolkit plus rocm with a fully integrated spack build. We require ROCM 4.5.x or later (prefer 5.x), and the ROCM version should match the version the application uses. This is all very fluid and subject to change.

There are two ways to build HPCToolkit plus ROCM with spack. HPCToolkit uses four ROCM prerequisites (hip, hsa-rocr-dev, roctracer-dev and rocprofiler-dev). If you have AMD’s all-in-one ROCM package installed in /opt, then specify all four prereqs in packages.yaml. For example, if ROCM 5.0.0 is installed at /opt/rocm-5.0.0, then you would use:

    - spec: hip@5.0.0
      prefix: /opt/rocm-5.0.0

    - spec: hsa-rocr-dev@5.0.0
      prefix: /opt/rocm-5.0.0

    - spec: roctracer-dev@5.0.0
      prefix: /opt/rocm-5.0.0

    - spec: rocprofiler-dev@5.0.0
      prefix: /opt/rocm-5.0.0

Currently, with AMD’s directory layout, the hip and hsa-rocr-dev prefixes could be specified either as /opt/rocm-5.0.0 or /opt/rocm-5.0.0/hip (and /opt/rocm-5.0.0/hsa). But roctracer-dev and rocprofiler-dev require /opt/rocm-5.0.0. Also, the rocm packages do not support spack external find. But all this is fluid and subject to change.

Alternatively, if ROCM is not installed in /opt/rocm, or if you want to build a different version, then omit the externals definitions in packages.yaml (but be prepared for spack to build an extra 80-90 packages). In either case, install hpctoolkit with:

spack install hpctoolkit +rocm ...

For developers building with autotools, use the following configure options. If /opt/rocm is available, then use the --with-rocm option. Otherwise, use the other four options.

configure  \
   --with-rocm=/opt/rocm \    (for all-in-one /opt/rocm)

   --with-rocm-hip=/path/to/hip/prefix  \
   --with-rocm-hsa=/path/to/hsa-rocr-dev/prefix  \
   --with-rocm-tracer=/path/to/roctracer-dev/prefix  \
   --with-rocm-profiler=/path/to/rocprofiler-dev/prefix  \

It is allowed to mix the all-in-one option with the individual packages. The rule is that the specific overrides the general.

11.3 MPI

HPCToolkit always supports profiling MPI applications. For hpctoolkit, the spack variant +mpi is for building hpcprof-mpi, the MPI version of hpcprof. If you want to build hpcprof-mpi, then you need to supply an installation of MPI.

Normally, for systems with compute nodes, you should use an existing MPI module that was built for the correct interconnect for your system and add this to packages.yaml. The MPI module should be built with the same version of GNU gcc/g++ used to build hpctoolkit (to keep the C++ libraries in sync).

11.4 PAPI vs Perfmon

HPCToolkit can access the Hardware Performance Counters with either PAPI or Perfmon (libpfm4). By default, the hpctoolkit package uses perfmon. If you want to use PAPI instead, then build hpctoolkit with +papi. However, you can’t use both due to a potential conflict in their header files.

PAPI runs on top of the perfmon library, but PAPI uses its own, internal copy of perfmon. Prior to version 5.6.0, PAPI did not install the perfmon header files, so it was impossible to access the perfmon events through PAPI.

However, starting with version 5.6.0, PAPI now installs both the perfmon library and its header files. Hpctoolkit configure will automatically detect this, so if you build hpctoolkit with a recent enough version of PAPI, then both the PAPI and perfmon interfaces will be available.

12 Platform Specific Notes

12.1 Cray

Cray systems have separate front and back-end architecture types. For example, on theta at ANL, the front-end arch is cray-sles15-haswell (SuSE Linux version 15 for haswell) and the back-end is cray-cnl7-mic_knl (Compute Node Linux for KNL).

Hpctoolkit needs to be built with the GNU Programming Environment and the front-end x86_64 compilers, plus the CC MPI C++ wrapper. Switch to the PrgEnv-gnu module and unload the darshan module. Darshan is a profiling tool that monitors an application’s use of I/O, but it conflicts with hpctoolkit.

module swap PrgEnv-intel PrgEnv-gnu
module unload darshan

By default, spack does not correctly identify the gcc compiler modules as front-end compilers. Start by loading a recent gcc module (eg, 7.x or 8.x) and try running spack compiler find and spack compiler list. For example on theta at ANL,

module load gcc/8.3.0
spack compiler find /opt/gcc/8.3.0

spack compiler list
==> Available compilers
-- gcc cnl7-any -------------------------------------------------
gcc@8.3.0  gcc@8.1.0  gcc@7.2.0  gcc@6.3.0  gcc@5.3.0
gcc@8.2.0  gcc@7.3.0  gcc@7.1.0  gcc@6.1.0  gcc@4.9.3

-- gcc sles15-x86_64 --------------------------------------------
gcc@8.3.0  gcc@4.8

If spack identifies the compiler (as above), then great. Note that we need spack to accept this as a front-end compiler (sles15-x86_64), not just for the back end (cnl7-any). If spack doesn’t find the compiler automatically, then you need to add it manually. Edit ~/.spack/cray/compilers.yaml and add an entry similer to the following (your versions may differ).

- compiler:
    environment: {}
    flags: {}
    - PrgEnv-gnu/6.0.5
    - gcc/8.3.0
    - craype/2.6.1
    - cray-mpich/7.7.10
    operating_system: sles15
      cc:  /opt/gcc/8.3.0/bin/gcc
      cxx: /opt/gcc/8.3.0/bin/g++
      f77: /opt/gcc/8.3.0/bin/gfortran
      fc:  /opt/gcc/8.3.0/bin/gfortran
    spec: gcc@8.3.0
    target: x86_64

Regardless of whether spack created the entry or you added it, you still need to add the modules: field manually (be sure to include all four modules). Although it takes several steps to create the correct entry, the good news is that it should continue to work as long as the underlying module exists.

Next, review your packages.yaml file. On Cray systems with Xeon Phi back-end nodes (KNL, KNH, etc), add ~tm to intel-tbb to disable transactional memory.

Finally, build hpctoolkit with the front-end arch type (eg, cray-sles15-x86_64) and option +cray. Normally, you can use OS type fe (front-end) in place of sles15. As with Blue Gene, the +cray option adds MPI and hpcprof-mpi and supersedes the +mpi variant.

spack install hpctoolkit +cray arch=cray-fe-x86_64

On some systems, spack fails to properly load the compiler modules during the build and fails during hpctoolkit configure with an error about, “MPICXX is not a valid compiler.” In this case, make sure that you have the PrgEnv-gnu and gcc modules loaded and retry the install with --dirty.

spack install --dirty hpctoolkit +cray arch=cray-fe-x86_64

For developers, if you are building hpctoolkit directly (outside of spack) but using spack prerequisites, then use a configure line similar to the following.

configure  \
    --prefix=/path/to/install/prefix  \
    --with-spack=/path/to/cray-sles15-x86_64/gcc-8.3.0  \
    --enable-all-static  \

13 HPCToolkit GUI Interface (Hpcviewer)

Since the 2020.12 release, the HPCToolkit GUI interface provides both profile and trace views in a single application, i.e. hpcviewer. Prior to that, each view was a separate program: hpcviewer to analyze the profile database, and hpctraceviewer to display the traces.

We provide binary distributions for hpcviewer on Linux (x86_64, ppc64le and aarch64), Windows and MacOS (x86_64 and M1). HPCToolkit databases are platform-independent and it is common to run hpcrun on one machine and then view the results on another machine.

Starting with 2021.01, the viewer now requires Java 11 or later, plus GTK+ 3.20 or later on Linux. Older viewers, up through 2020.12, require Java 8 (not 9 or later), plus GTK+ 2.x for Linux.

13.1 Spack Install

The spack install is available on Linux x86_64, little-endian ppc64le (power8 and 9) and aarch64 ARM, and also MacOS on x86_64. This installs hpcviewer and includes the Java prerequisite.

For the current viewers, use openjdk with the most recent version of Java 11 for all platforms. Currently, this is the default, but if not, then you can add an explicit openjdk dependency. You can check this with spack info and spack spec.

spack info openjdk
spack install hpcviewer
spack install hpcviewer ^openjdk @11.0.12_7    (if needed)

For older viewers, up through 2020.12, use the latest verion of openjdk Java 8 (version 1.8). For example (the versions may change):

spack install hpcviewer ^openjdk @1.8.0_265-b01

Note: to run the viewer on Macos, you can either open the Finder and click your way to the hpcviewer.app directory and double-click on the hpcviewer icon, or else use spack load hpcviewer to put hpcviewer on your PATH.

13.2 Manual Install

Binary distributions of the viewers for all supported platforms are available at:


On Linux, download the linux.gtk version of hpcviewer (and also hpctraceviewer for older versions), unpack the tar files and run the install scripts (for both viewers) with the path to the desired install prefix.

./install /path/to/install/directory

On Windows and MacOS, download the win32 or macosx.cocoa versions and unpack the zip or dmg files in the desired directory. Due to Apple’s security precautions, on MacOS, you may need to use curl or wget instead of a web browser.

Note: the manual install uses the existing system version of Java (or one of several versions with modules), whereas the spack install includes the java prerequisite. That is, the spack install is self-contained and does not need to change the system java.

14 Building a New Compiler

Some systems may have compilers that are too old for building HPCToolkit or other packages. For example, RedHat 7.x comes with gcc 4.8.x which is very old. If your system doesn’t already have modules for later compilers, then you may need to build a new compiler yourself.

First, pick a directory in which to install the modules and make subdirectories for the spack packages and module files. In this example, I’m using /opt/spack/Modules as my top-level directory and subdirectories packages and modules. Edit config.yaml and modules.yaml to add these paths.

    root: /opt/spack/Modules/packages

      # just use one of these
        tcl:  /opt/spack/Modules/modules
        lmod: /opt/spack/Modules/modules

Determine if your system uses TCL (environment) or Lmod modules. Normally, the module command is a shell function. TCL modules use modulecmd and Lmod modules eval LMOD_CMD. Edit modules.yaml to enable the module type for your system. Again, you only need one of these (and don’t use dotkit unless you work at LLNL).

     - tcl
     - lmod

Then, choose a version of gcc and use the default compiler (normally /usr/bin/gcc) to build the newer version. Currently, gcc 8.x is a good choice that builds robustly, has enough modern features but is not too new to cause problems for some packages. For example,

spack install gcc@8.4.0

Note: it is not necessary to rebuild the new compiler with itself.

14.1 Using the New Compiler

After building a new compiler, then you need to tell spack how to find it. First, use module use and module load to load the module. For TCL modules, the module files are in a subdirectory of module_roots named after the system architecture. For example,

module use /opt/spack/Modules/modules/linux-rhel7-x86_64
module load gcc-8.4.0-gcc-4.8.5-qemsqrc

For Lmod modules, the module directory is one level below that and the module names are a little different.

module use /opt/spack/Modules/modules/linux-rhel7-x86_64/Core
module load gcc/8.4.0-dan4vbm

For both TCL and Lmod modules, it’s best to put the module use command in your shell’s startup scripts so that module avail and module load will know where to find them. After loading the module, run spack compiler find.

$ spack compiler find
==> Added 1 new compiler to /home/krentel/.spack/linux/compilers.yaml

Finally, always check the new entry in compilers.yaml and add the name of the module to the modules: field.

- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    - gcc-8.4.0-gcc-4.8.5-qemsqrc
    operating_system: rhel7
      cc:  /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gcc
      cxx: /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/g++
      f77: /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gfortran
      fc:  /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gfortran
    spec: gcc@8.4.0
    target: x86_64

Note: as long as the spack packages and modules directories remain intact and you don’t remove the compilers.yaml entry, then this compiler will always be available from within spack. You can also use this compiler outside of spack by using module load. If you want to make this your default compiler for all spack builds, then you can specify this in packages.yaml. For example,

    compiler: [gcc@8.4.0]

Also, when using the compiler from within spack, it doesn’t matter if you have the module loaded or not. Spack will erase your environment and re-add the appropriate modules automatically.


14.2 Bootstrapping Environment Modules

If your system does not support modules, then you will have to add it. If you have root access, the easiest solution is to install a system package for modules. If not, then use spack to install the environment-modules package. Source the bash or csh script in the init directory to add the module function to your environment. For example,

spack install environment-modules
cd /path/to/environment-modules-4.3.1-ism7cdy4xverxywj27jvjstqwk5oxe2v/init
(bash)  . ./bash
(csh)   source ./csh

Again, add the setup command to your shell’s startup scripts.

15 Spack Mirrors

A spack mirror allows you to download and save a source tar file in advance. This is useful if your system is behind a firewall, or if you need to manually agree to a license, or if you just don’t want to keep downloading the same file over and over.

A mirror has a simple directory structure and is easy to set up. Create a top-level directory with subdirectories named after the spack packages and copy the tar files into their package’s subdirectory. For example,

      boost-1.66.0.tar.bz2      (from boost_1_66_0.tar.bz2)
      dyninst-10.1.0.tar.gz     (from git checkout)
      ibm-java-    (from ibm-java-sdk-8.0-5.30-ppc64le-archive.bin)
      intel-xed-2019.03.01.tar.gz    (from git checkout)
      mbuild-2019.03.01.tar.gz       (resource from git checkout)
      jdk-1.8.0_202.tar.gz      (from jdk-8u202-linux-x64.tar.gz)

Note: the names of the files in the spack mirror always follow the same, specific format, regardless of the actual name of the tar file. Version is the spack name for the version (from spack info), and extension is the same extension as the tar file (tar.gz, tar.bz2, etc) or else None for other types of files.

<package-name> - <version> . <extension>

For example, the boost 1.66.0 tar file is actually named boost_1_66_0.tar.bz2 but is stored in the mirror as boost-1.66.0.tar.bz2 and jdk-8u202-linux-x64.tar.gz is renamed to jdk-1.8.0_202.tar.gz.

For packages that use a snapshot from a git repository (tag or commit hash), clone the repository, checkout the desired version, make a tar file and gzip the file. (You should exclude the .git subdirectory.) But note that spack refuses to use a cached file for the head of a branch because it is a moving target.

Finally, after creating the mirror directory, add it to spack with spack mirror add. For example,

spack mirror add my-mirror file:///home/krentel/spack/my-mirror
spack mirror list
my-mirror    file:///home/krentel/spack/mirror

Note: by default, spack stores downloaded files inside the spack repository at spack/var/spack/cache. This directory is a full spack mirror, so instead of creating a separate directory tree, you could just copy the files into the cache directory. This is useful when spack fetch has trouble downloading a file. If you can download the file manually, or copy it from another machine, then just rename the file as above and copy it into the spack file cache.

For more information on mirrors, see:


16 Common Problems

16.1 Unable to fetch tar file

Spack is somewhat fragile for how it downloads tar files and will often fail for transitory network problems. This is especially true for packages with many dependencies. For example:

==> Installing m4
==> Searching for binary cache of m4
==> No binary for m4 found: installing from source
curl: (6) Could not resolve host: ftp.wayne.edu; Name or service not known
==> Fetching https://ftpmirror.gnu.org/m4/m4-1.4.18.tar.gz
==> Fetching from https://ftpmirror.gnu.org/m4/m4-1.4.18.tar.gz failed.
==> Error: FetchError: All fetchers failed for m4-1.4.18-vorbvkcjfac43b7vuswsvnm6xe7w7or5

There are two workarounds. First, assuming the problem is temporary, simply wait 10 minutes or an hour and try again. Second, you could manually download the file(s) by some other means and copy them to spack’s cache directory spack/var/spack/cache/<package> or to a spack mirror.

16.2 New releases break the build

Normally, HPCToolkit should build and work correctly with the latest version for all of its dependencies. But sometimes a new release will change something and break the build. This has happened a couple times where a new release of Boost has broken the build for Dyninst. Or, maybe the latest version of gcc/g++ disallows some usage and breaks the build.

The solution is to use packages.yaml to specify an earlier version until the rest of the code adapts to the change.

16.3 Failure to load modules

Spack is quite aggressive about compiling with a clean environment and will unload modules unless they are specifically required by some config file (compilers.yaml or packages.yaml). This can result in a situation where you think some compiler or build tool is available from your environment but spack removes it during the build.

In this example, I am using modules for GCC/8.3.0 and CMake/3.8.2. Spack finds the gcc 8.3.0 compiler and I added cmake@3.8.2 to packages.yaml. But I failed to add the modules: field for gcc 8.3.0 in compilers.yaml. As a result, the build fails with:

cmake: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by cmake)
cmake: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by cmake)
cmake: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by cmake)
==> Error: ProcessError: Command exited with status 1:

The problem is that cmake 3.8.2 was built with g++ 8.3.0, but spack is running cmake without the GCC/8.3.0 libraries and so the build fails as above. One way to confirm this is to rerun spack install --dirty which then succeeds. The --dirty option tells spack not to unload your modules. Whenever the build fails with a mismatched library as above and especially when --dirty fixes the problem, this is a clear sign that spack is missing a module during the build.

Although --dirty may make the build succeed, there should be no case where this is necessary. The correct solution is to fill in the modules: field in compilers.yaml or some other config file. See the section on Compilers above.

©2000-2021 Rice UniversityRice Computer Science