[ Home | Overview | Publications | Software/Downloads ] • [ Documentation/Questions | Training Videos and Slides ] • [ People | Acks ]

Current Spack Issues for HPCToolkit

Table of Contents

1 Introduction

Spack is a moving target and receives multiple commits per day. Normally, HPCToolkit will build and run successfully with the latest version of all of its prerequisite packages, but sometimes not. This page covers the current known issues where HPCToolkit fails to build with the latest version of spack. The main build directions are at:


Report problems to hpctoolkit-forum at rice dot edu. But before reporting a problem, first try the versions recommended in the packages.yaml file in the spack subdirectory of the hpctoolkit repository. And always check the latest version of this file on the hpctoolkit web site.


Last revised: September 26, 2022.

2 Current Issues

2.1 (2022-09-23) Hpcprof-mpi is currently disabled

Currently, on the develop branch and in the most recent release, the hpcprof-mpi program is disabled due to unresolved errors. After these bugs are resolved, then it will be re-enabled.

Workaround: Use hpcprof which is now multi-threaded and should suffice.

Note: For a spack build, do not use +mpi, and for an autotools build, make sure MPICXX is not set in your environment, or else set MPICXX= on the configure command.

2.2 (2022-09-23) Hpctoolkit +cray fails

Suppose spack install hpctoolkit +cray fails as follows.

   161   configure: supplied MPICXX compiler:
   162   configure: Cray MPI wrapper: /opt/cray/pe/craype/2.6.5/bin/CC
   163   configure: MPICXX compiler: g++ (GCC) 9.3.0 20200312 (Cray Inc.)
   164   configure: MPICXX path: /opt/cray/pe/craype/2.6.5/bin/CC
   165   checking if MPICXX is a valid MPI C++ compiler... no
>> 166   configure: error: MPICXX is not a valid compiler: /opt/cray/pe/craype/2.6.5/bin/CC

Building hpctoolkit +cray requires --dirty and building for the front end arch. Make sure you have the PrgEnv-gnu, gcc, craype and cray-mpich modules loaded (and darshan unloaded) and re-install with:

spack install --dirty hpctoolkit +cray arch=cray-fe-x86_64

2.3 (2021-07-11) HPCToolkit build fails in hpcrun-fmt.h

Very rarely, the hpctoolkit build fails in hpcrun-fmt.c or hpcrun-fmt.h with a spew of messages similar to the following.

In file included from hpcrun-fmt.c:84:
hpcrun-fmt.h:1:1: error: expected identifier or '(' before string constant
hpcrun-fmt.c: In function 'hpcrun_fmt_hdr_fwrite':
hpcrun-fmt.c:131:10: error: 'HPCRUN_FMT_Magic' undeclared (first use in this function)
   fwrite(HPCRUN_FMT_Magic,   1, HPCRUN_FMT_MagicLen, fs);

The problem is that git clone resulted in a corrupted copy of hpcrun-fmt.h. This file should be a C/C++ header file beginning with a copyright notice in comments.

// -*-Mode: C++;-*- // technically C99

// * BeginRiceCopyright *****************************************************
// $HeadURL$
// $Id$

The corrupted copy is the same as hpcrun-fmt.txt except that every line begins and ends with double quotes.

"hpcrun binary data format (see "Abbreviation notes" below)\n"
"fmt-hdr {epoch}*\n"
"  \n"

This problem is rare, non-deterministic, not at all understood, and impossible to reproduce on demand. If you can reproduce the problem, even non-deterministically, please report this to hpctoolkit-forum as above.

Workaround: The only workaround is to retry the git clone. If building outside of spack, then delete the source tree and repeat git clone. If building with spack, first remove spack’s downloaded copy with spack clean -d, then retry spack install.

3 Recently Resolved Issues

None at this time.

4 General Problems

These are general problems that arise from time to time.

4.1 Unable to fetch

Sometimes spack fails to download the source file(s) for some package and dies with a message similar to this.

==> Fetching from https://ftpmirror.gnu.org/m4/m4-1.4.18.tar.gz failed.
==> Error: FetchError: All fetchers failed for m4-1.4.18-vorbvkcjfac43b7vuswsvnm6xe7w7or5

This problem is usually temporary and the solution is to either wait a few minutes or an hour and try again, or else download the file manually and put it into a spack mirror.

Workaround: Spack’s default method of fetching is using internal Python libraries (urllib). Sometimes, depending on version, this may have trouble. In that case, you could try resetting this to the external curl program in config.yaml.

  url_fetch_method: curl

4.2 Connection timeout

Another way fetch can fail is with a connection timeout. Some sites, especially sourceforge are often slow to connect. If this happens, then increase the connection timeout in config.yaml to 30 or 60 seconds (default is 10 seconds).

  connect_timeout: 60

4.3 New version breaks the build

Sometimes the latest version of some package breaks the build. This has happened a couple of times where a new version of Boost has broken the build for Dyninst. The solution is to revert the package to an earlier version until the rest of the code catches up.

4.4 Spack core breaks the build

Sometimes but rarely, something in the spack core will change or break the code in some package.py file. The solution is to look through the spack git log and revert the repository to a recent commit before the breakage.

5 Issues with specific versions

5.1 Binutils 2.35

Avoid binutils versions 2.35 and 2.35.1, they contain a bug that causes hpcprof to spew BFD Dwarf errors about “could not find variable specification at offset xxxx.” This is fixed in release 2.35.2 or 2.36 or later.

5.2 Boost 1.68.0

Avoid boost version 1.68.0, it breaks the build for hpctoolkit.

5.3 Elfutils 0.176

Beginning with 0.176, elfutils requires glibc 2.16 or later and won’t work with an older glibc, including RedHat or CentOS 6.x and Blue Gene.

©2000-2021 Rice UniversityRice Computer Science