[ Home | Overview | Publications ] • [ Examples ] • [ Software/Downloads | Documentation/Questions ] • [ People | Acks ]

Current Spack Issues for HPCToolkit

Table of Contents

1 Introduction

Spack is a moving target and receives multiple commits per day. Normally, HPCToolkit will build and run successfully with the latest version of all of its prerequisite packages, but sometimes not. This page covers the current known issues where HPCToolkit fails to build with the latest version of spack.

Report problems to hpctoolkit-forum at rice dot edu. But before reporting a problem, first try the versions recommended in the packages.yaml file in the spack subdirectory of the hpctoolkit repository. And always check the latest version of this file on the hpctoolkit web site.


Last revised: Dec 1, 2019.

2 Current Issues

2.1 (2019-09-25) New micro-architecture targets

Spack recently changed how it treats a system’s architecture and target to allow a hierarchy of fine-grained micro-architectures. The ’target’ is now a specific micro-architecture (eg, haswell, ivybridge) instead of a generic family (x86_64). You will see this change in two main places: in ’spack spec’ and the path for the install directory. For example, linux-rhel7-x86_64 might become linux-rhel7-broadwell. You can use spack arch to see the list of generic families and micro-architecture targets.

spack arch --known-targets

Workaround: Although a fresh install from scratch should work and usually does work, there may be some packages or corner cases that can’t handle the micro-arch specs. If you prefer to avoid this change until the dust settles, then use commit a940ff34d745 from Sept 19, 2019.

git clone https://github.com/spack/spack.git
cd spack
git checkout a940ff34d745

Workaround: A better workaround is to use the current develop head but build for a generic family instead of the micro-arch target. Use the target option to specify the arch family. For example:

spack install hpctoolkit ... target=x86_64

If you want to use this target for all spack builds, then you can specify this in packages.yaml. For example:

    target: ['x86_64']

2.2 (2019-08-28) Cray front-end compilers

Spack compiler find is currently broken for detecting the front-end compilers on Cray that HPCToolkit uses. Normally, you would load a module for gcc and run spack compiler find and spack would add that compiler to compilers.yaml, but this currently does not work.

Workaround: If you have a working compiler: entry for a front-end GNU compiler on Cray, then that will continue to work. If not, then you will have to add one manually. For example, this is an entry for the gcc/7.3.0 module on theta at ANL. Note that the front-end operating_system is something like sles12 (not cnl6), and the front-end target is x86_64 (not mic_knl).

- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    - PrgEnv-gnu/6.0.4
    - gcc/7.3.0
    - cray-mpich/7.7.3
    operating_system: sles12
      cc:  /opt/gcc/7.3.0/bin/gcc
      cxx: /opt/gcc/7.3.0/bin/g++
      f77: /opt/gcc/7.3.0/bin/gfortran
      fc:  /opt/gcc/7.3.0/bin/gfortran
    spec: gcc@7.3.0
    target: x86_64

Workaround: Alternatively, the last working commit was 9c1c50fb7632 on 2019-06-07. You could revert to this commit, run spack compiler find and then return to the current spack head (develop). Be sure to fill in the modules: field.

3 Recently Resolved Issues

3.1 (2019-11-19) External perl breaks libunwind

A recent commit (99dfff447509 on 2019-11-04, but merged on 11-18) modified the treatment of PATH for packages that depend on perl. If you use packages.yaml to specify an external perl from a system directory such as /usr/bin, then this puts /usr/bin at the front of PATH during the build. Unfortunately, this covers up all of the other build dependencies and can break the build.

For example, on systems with an older libtool, this breaks libunwind as follows.

294   libtool: Version mismatch error.  This is libtool 2.4.2, but the
295   libtool: definition of this LT_INIT comes from libtool 2.4.6.
296   libtool: You should recreate aclocal.m4 with macros from libtool 2.4.2
297   libtool: and run autoconf again.

Fixed: This is now fixed in commit cacfc3a6e1c7 on 2019-11-29. Or, you can workaround the problem by not using an external perl and instead having spack build perl itself.

3.2 (2019-10-08) Python 3.x breaks PAPI

Python 2.x is nearing end-of-life and Spack recently changed their launch script to look for and use python 3.x if available. There are incompatibilities and some spack packages (including PAPI) currently break with python 3.x. (The papi recipe hangs while patching files with filter_file.) There are two workarounds, depending on whether python 2.x is available on your system.

Fixed: The filter file problem is now fixed in commit 5cd28847e81b on 2019-10-15.

Workaround: If python 2.x is available on your system, then one solution is to remove the test for python3 in the bin/spack launch script.

# This file is bilingual. The following shell code finds our preferred python.
# Following line is a shell no-op, and starts a multi-line Python comment.
# See https://stackoverflow.com/a/47886254
# prefer python3, then python, then python2
for cmd in python3 python python2; do
   command -v > /dev/null $cmd && exec $cmd $0 "$"

3.3 (2019-08-28) External cuda modules

Sometimes spack misreads the module: entry for an external package in packages.yaml and selects the wrong install directory. For example, spack misreads the cuda/10.1.168 module on cori at NERSC (incorrectly using /usr) and the build for hpctoolkit fails as follows.

==> cuda@10.1.168 : has external module in cuda/10.1.168
==> cuda@10.1.168 : is actually installed in /usr

.../configure --prefix='...' ... '--with-cuda=/usr'
>> 208    configure: error: '/usr/include/cuda.h' not found

Fixed: This is now fixed in commit b1868f35ec91 on 2019-09-11. Or, you can workaround the problem by replacing modules: with a paths: entry.

    cuda@10.1.168:  /usr/common/software/cuda/10.1.168

3.4 (2019-08-19) Build stage not writable

Spack has reorganized the build directories and the value for build_stage in config.yaml may not work.

  - $tempdir/spack-stage

The problem with this value is that the first user to run spack on this machine will create the directory, owned by that user and thus not writable by any other user. For every other user, spack install will fail with:

==> Error: No accessible stage paths in:

Fixed: This is now fixed in commit c141e99e062b on 2019-09-03. Check your build_stage directory and make sure that you have write access. One solution is to add $user to the path.

  - $tempdir/$user/spack-stage

4 General Problems

These are general problems that arise from time to time.

4.1 Unable to fetch

Sometimes spack fails to download the source file(s) for some package and dies with a message similar to this.

==> Fetching from https://ftpmirror.gnu.org/m4/m4-1.4.18.tar.gz failed.
==> Error: FetchError: All fetchers failed for m4-1.4.18-vorbvkcjfac43b7vuswsvnm6xe7w7or5

This problem is usually temporary and the solution is to either wait a few minutes or an hour and try again, or else download the file manually and put it into a spack mirror.

4.2 New version breaks the build

Sometimes the latest version of some package breaks the build. This has happened a couple of times where a new version of Boost has broken the build for Dyninst. The solution is to revert the package to an earlier version until the rest of the code catches up.

4.3 Spack core breaks the build

Sometimes but rarely, something in the spack core will change or break the code in some package.py file. The solution is to look through the spack git log and revert the repository to a recent commit before the breakage.

5 Long Term Issues

5.1 Boost 1.68.0

Avoid boost version 1.68.0, it breaks the build for hpctoolkit. Version 1.70.0 works with the latest version of dyninst (10.1.0), or else 1.66.0 is good and works with all versions of dyninst.

5.2 Elfutils 0.176

Elfutils 0.176 requires glibc 2.16 or later (for aligned_alloc) and won’t work with an older glibc, including RedHat or CentOS 6.x and Blue Gene. On systems with an old glibc, use version 0.175.

©2000-2018 Rice UniversityRice Computer Science