hpctoolkit

[ Home | Overview | Publications | Software/Downloads ] • [ Documentation/Questions | Training Videos and Slides ] • [ People | Acks ]


Current Spack Issues for HPCToolkit

Table of Contents

1 Introduction

Spack is a moving target and receives multiple commits per day. Normally, HPCToolkit will build and run successfully with the latest version of all of its prerequisite packages, but sometimes not. This page covers the current known issues where HPCToolkit fails to build with the latest version of spack. The main build directions are at:

software-instructions.html

Report problems to hpctoolkit-forum at rice dot edu. But before reporting a problem, first try the versions recommended in the packages.yaml file in the spack subdirectory of the hpctoolkit repository. And always check the latest version of this file on the hpctoolkit web site.

http://hpctoolkit.org/spack-issues.html

Last revised: July 11, 2021.

2 Current Issues

2.1 (2021-07-11) Dyninst cannot find Libiberty libraries

Rarely, on some systems, dyninst cmake is unable to find the libiberty libraries and fails with a message similar to the following.

-- Could NOT find LibIberty (missing: LibIberty_LIBRARIES) 
CMake Error at cmake/LibIberty.cmake:53 (message):
  LibIberty not found and cannot be downloaded because build is sterile.
Call Stack (most recent call first):
  CMakeLists.txt:34 (include)

This problem is infrequent and not well understood. It seems to happen more often on some Debian or Ubuntu systems, but certainly not all such systems. So, if you run into this problem and can reproduce it, please report this to hpctoolkit-forum as above.

Workaround: Apply the following patch to the spack repository, or just hand-edit the dyninst package.py file to add the one line. (Note: the last context line in the patch is a blank line.)

diff --git a/var/spack/repos/builtin/packages/dyninst/package.py b/var/spack/repos/builtin/packages/dyninst/package.py
index 38c664d500..0b71d332a8 100644
--- a/var/spack/repos/builtin/packages/dyninst/package.py
+++ b/var/spack/repos/builtin/packages/dyninst/package.py
@@ -103,6 +103,7 @@ def cmake_args(self):
             '-DBoost_ROOT_DIR=%s'     % spec['boost'].prefix,
             '-DElfUtils_ROOT_DIR=%s'  % spec['elf'].prefix,
             '-DLibIberty_ROOT_DIR=%s' % spec['libiberty'].prefix,
+            '-DLibIberty_LIBRARIES=%s' % spec['libiberty'].libs,
             '-DTBB_ROOT_DIR=%s'       % spec['tbb'].prefix,
         ]

2.2 (2021-07-11) HPCToolkit build fails in hpcrun-fmt.h

Very rarely, the hpctoolkit build fails in hpcrun-fmt.c or hpcrun-fmt.h with a spew of messages similar to the following.

In file included from hpcrun-fmt.c:84:
hpcrun-fmt.h:1:1: error: expected identifier or '(' before string constant
 "$Id$\n"
 ^~~~~~~~
hpcrun-fmt.c: In function 'hpcrun_fmt_hdr_fwrite':
hpcrun-fmt.c:131:10: error: 'HPCRUN_FMT_Magic' undeclared (first use in this function)
   fwrite(HPCRUN_FMT_Magic,   1, HPCRUN_FMT_MagicLen, fs);

The problem is that git clone resulted in a corrupted copy of hpcrun-fmt.h. This file should be a C/C++ header file beginning with a copyright notice in comments.

// -*-Mode: C++;-*- // technically C99

// * BeginRiceCopyright *****************************************************
//
// $HeadURL$
// $Id$
//
...

The corrupted copy is the same as hpcrun-fmt.txt except that every line begins and ends with double quotes.

"$Id$\n"
"\n"
"=============================================================================\n"
"hpcrun binary data format (see "Abbreviation notes" below)\n"
"=============================================================================\n"
"\n"
"fmt-hdr {epoch}*\n"
"  \n"
...

This problem is rare, non-deterministic, not at all understood, and impossible to reproduce on demand. So again, if you can reproduce the problem, even non-deterministically, please report this as above.

Workaround: The only workaround is to retry the git clone. If building outside of spack, the delete the source tree and repeat git clone. If building with spack, first remove spack’s downloaded copy with spack clean -d, then retry spack install.

3 Recently Resolved Issues

3.1 (2020-09-05) Configure fails without -fPIC in dependencies

As of commit b2d3d067ecad on 2020-09-05, the hpctoolkit master branch now requires libunwind and xz (lzma) to be built with variant +pic. This is used to better separate and hide third-party libraries in hpcrun from the application. If not, then configure will fail with an error such as:

configure: libunwind.a static archive: yes
configure: libunwind.a compiled with -fPIC: no
configure: error: libunwind.a must be compiled with -fPIC

Fixed: Build libunwind and xz (lzma) with the +pic variants. For example, in packages.yaml,

libunwind:
  version:  [1.5.0]
  variants: +xz +pic
xz:
  version:  [5.2.5]
  variants: +pic

3.2 (2020-06-29) Cray front-end compilers

Spack compiler find was previously broken for detecting the front-end compilers on Cray that HPCToolkit uses.

Fixed: This is now fixed in commit 789d060ff61b on 2020-06-29.

Note: Remember, you still need to fill in the modules: field with the following four modules. For example, this is an entry for the gcc/8.3.0 module on theta at ANL. Note that the front-end operating_system is something like sles15 (not cnl6), and the front-end target is x86_64 (not mic_knl). Your versions may differ.

- compiler:
    environment: {}
    flags: {}
    modules:
    - PrgEnv-gnu/6.0.7
    - gcc/8.3.0
    - craype/2.6.5
    - cray-mpich/7.7.14
    operating_system: sles15
    paths:
      cc:  /opt/gcc/8.3.0/bin/gcc
      cxx: /opt/gcc/8.3.0/bin/g++
      f77: /opt/gcc/8.3.0/bin/gfortran
      fc:  /opt/gcc/8.3.0/bin/gfortran
    spec: gcc@8.3.0
    target: x86_64

4 General Problems

These are general problems that arise from time to time.

4.1 Unable to fetch

Sometimes spack fails to download the source file(s) for some package and dies with a message similar to this.

==> Fetching from https://ftpmirror.gnu.org/m4/m4-1.4.18.tar.gz failed.
==> Error: FetchError: All fetchers failed for m4-1.4.18-vorbvkcjfac43b7vuswsvnm6xe7w7or5

This problem is usually temporary and the solution is to either wait a few minutes or an hour and try again, or else download the file manually and put it into a spack mirror.

4.2 Connection timeout

Another way fetch can fail is with a connection timeout. Some sites, especially sourceforge are often slow to connect. If this happens, then increase the connection timeout in config.yaml to 30 or 60 seconds (default is 10 seconds).

4.3 New version breaks the build

Sometimes the latest version of some package breaks the build. This has happened a couple of times where a new version of Boost has broken the build for Dyninst. The solution is to revert the package to an earlier version until the rest of the code catches up.

4.4 Spack core breaks the build

Sometimes but rarely, something in the spack core will change or break the code in some package.py file. The solution is to look through the spack git log and revert the repository to a recent commit before the breakage.

4.5 IBM-Java fails to fetch (powerpc)

IBM is fairly aggressive about taking down old versions, so you may find that fetching ibm-java fails while trying to install hpcviewer (powerpc only).

==> Installing ibm-java
==> No binary for ibm-java found: installing from source
==> Error: FetchError: All fetchers failed

If this happens, first compare spack info ibm-java with what versions are available at IBM’s download site. If there is another 8.0.x.y version that spack understands that is still available for download, then use that.

http://public.dhe.ibm.com/ibmdl/export/pub/systems/cloud/runtimes/java

If not, then manually download the latest 8.0.x.y version, compute a sha256sum checksum for the file and then edit the ibm-java/package.py file to add this version. And then report the problem to the spack maintainer.

5 Issues with specific versions

5.1 Binutils 2.35

Avoid binutils versions 2.35 and 2.35.1, they contain a bug that causes hpcprof to spew BFD Dwarf errors about “could not find variable specification at offset xxxx.” This is fixed in release 2.35.2 or 2.36 or later.

5.2 Boost 1.68.0

Avoid boost version 1.68.0, it breaks the build for hpctoolkit.

5.3 Elfutils 0.176

Beginning with 0.176, elfutils requires glibc 2.16 or later and won’t work with an older glibc, including RedHat or CentOS 6.x and Blue Gene.



©2000-2021 Rice UniversityRice Computer Science