©2000-2018 Rice University • Rice Computer Science
[ Home | Overview | Publications ] • [ Examples ] • [ Software/Downloads | Documentation/Questions ] • [ People | Acks ]
These notes describe how to build and install HPCToolkit and hpcviewer and their prerequisites with Spack. HPCToolkit proper (hpcrun, hpcstruct and hpcprof) is used to measure and analyze an application’s performance and then produce a database for hpcviewer. HPCToolkit is supported on the following platforms.
We provide binary distributions for hpcviewer and hpctraceviewer on Linux (x86_64 and powerpc), Windows and MacOS. HPCToolkit databases are platform-independent and it is common to run hpcrun on one machine and then view the results on another machine.
We build HPCToolkit and its prerequisite libraries from source.
HPCToolkit has some 15-20 prerequisites and we now use spack to build
them. It is possible to use spack to install all of hpctoolkit or
build just the prerequisites and then build hpctoolkit with the
traditional configure ; make ; make install method from
autotools. Developers will probably want to run configure and
make manually, but both methods are supported.
Note: the old method of using hpctoolkit-externals to build the prerequisite libraries is now superseded by spack. Although externals will continue to work for a while, we have stopped updating externals and now add newer versions of packages only in spack.
These notes are written mostly from the view of using spack to build hpctoolkit and its dependencies. If you are a more experienced spack user, especially if you want to use spack to build hpctoolkit plus several other packages, then you will want to adapt these directions to your own needs.
Spack documentation is available at:
The current status of using Spack for HPCToolkit is at:
Last revised: Feb 28, 2020.
Building HPCToolkit requires the following prerequisites.
Hpcviewer and hpctraceviewer require Java version 8. On Linux, the viewers also require GTK+ version 2.x.
Spack uses a special notation for specifying the version, variants, compilers and dependencies when describing how to build a package. This combination of version, variants, etc is called a ’spec’ and is used both on the command line and in config files.
spack info <package> shows
the available versions and variants for a package. In most cases,
spaces are optional between elements of a spec. For example:
boost@1.66.0 dyninst @10.1.0 hpctoolkit @master
-
(dash) and ~ (tilde) both mean ’off’. Use dash after a space
and tilde after a non-space. For example:
elfutils+bzip2~nls elfutils +bzip2 -nls elfutils@0.176 +bzip2~nls
dyninst+openmp build_type=RelWithDebInfo xerces-c@3.2.2 transcoder=iconv
hpctoolkit@master %gcc@7.3.0
amg2013 cflags='-O2 -mavx512pf'
hpctoolkit@master ^dyninst@10.1.0+openmp
Normally, a system has only one arch type and you don’t need to specify this. However, for systems with separate front and back-end types, the default is the back end. For example, if you wanted to build python for the front end on Cray, then you might use something like this.
python@3.7.4 arch=cray-sles12-x86_64
Also, now that spack has implemented microarchitecture targets (haswell, ivybridge, etc), you can use ’target’ to build for a generic x86_64 or a specific CPU type. For example:
amg2013 target=x86_64 lulesh target=ivybridge
When writing a spec (for spack install or other), spack will
fully resolve all possible choices for the package and all of its
dependencies and create a unique hash value for that exact
configuration. This process is called ’concretization.’ To see how
spack would concretize a spec, use spack spec <partial-spec>.
Spack is available via git clone from GitHub. This includes the core
spack machinery and recipes for building more than 3,900 packages (and
growing). You should also clone HPCToolkit for the
packages.yaml file which is used to configure the spack build.
git clone https://github.com/spack/spack.git git clone https://github.com/hpctoolkit/hpctoolkit.git
After cloning, add the spack/bin directory to your PATH, or else
source the spack setup-env script.
(bash) . /path/to/spack/share/spack/setup-env.sh
(csh) setenv SPACK_ROOT /path/to/spack/root
source $SPACK_ROOT/share/spack/setup-env.csh
It suffices to add spack/bin to your PATH (or even symlink the
spack launch script). Sourcing the setup-env script adds extra
support for modules built by spack.
config.yaml is the top-level spack config file. This specifies
the directory layout for installed files and the top-level spack
parameters. There are two or three fields in this file that you should
set for your local system.
install_tree – by default, spack installs packages inside the
spack repository at spack/opt/spack. If you want to use another
location, then set this.
module_roots – by default, spack installs module files inside
the spack repository at spack/share/spack. If you want to use
another location, then set this.
build_jobs – by default, spack uses all available hardware
threads for parallel make. For machines with many threads, this can
sometimes overwhelm the system and you should set this to a smaller
number. (There is precedent.)
There are also parameters for the locations of the build directories, the cache of downloaded tar files, etc, which you may wish to set.
The default config.yaml file is in the spack repository at
spack/etc/spack/defaults. The simplest solution is to copy this
file one directory up and then edit the copy (don’t edit the default
file directly).
cd spack/etc/spack cp defaults/config.yaml . vi config.yaml
Alternatively, you could put this file in a separate directory, outside
of the spack repository and then use -C dir on the spack command
line. (The -C option goes before the spack command name.)
spack -C dir install ...
Note: if you put config.yaml in spack/etc/spack, then it
will apply to every spack command for that repository (and you won’t
forget). Putting it in a separate directory is more flexible because
you can support multiple configurations from the same repository. But
then you must use -C dir with every spack command or else you
will get inconsistent results.
You can view the current configuration with spack config.
spack [-C dir] config get config
See the spack docs on ‘Configuration Files’ and ‘Basic Settings’.
https://spack.readthedocs.io/en/latest/configuration.html https://spack.readthedocs.io/en/latest/config_yaml.html
The packages.yaml file specifies the versions and variants for
the packages that spack installs and serves as a common reference
point for HPCToolkit’s prerequisites. This file also specifies the
paths or modules for system build tools (cmake, python, etc) to avoid
rebuilding them. Put this file in the same directory as
config.yaml. A sample packages.yaml file is available
in the spack directory of the hpctoolkit repository.
There are two main sections to packages.yaml. The first
specifies the versions and variants for hpctoolkit’s prereqs. By
default, spack will choose the latest version of each package (plus
any constraints from hpctoolkit’s package.py file). In most
cases, this will work, but not always. If you need to specify a
different version or variant, then set this in packages.yaml.
Note: the versions and variants specified in hpctoolkit’s
package.py file are hard constraints and should not be changed.
Variants in packages.yaml are preferences that may be modified
for your local system. (But don’t report a bug until you have first
tried the versions from packages.yaml that we supply.)
There are at least two packages with a variant that you may need to
change depending on your system. But always check the current
packages.yaml file to see if any more have been added.
intel-tbb – for very old Intel or AMD systems that don’t support
transactional memory, change +tm to ~tm. (This option has
no effect on non-x86 systems.)
libmonitor – on Blue Gene/Q, add +bgq.
The second section in packages.yaml specifies a path or module
for system build tools. Building hpctoolkit’s prerequisites requires
cmake 3.4 or later, perl 5.x and python 2.7 or 3.4 or later. There are
three ways to satisfy these requirements: a system installed version
(eg, /usr), a pre-built module or build from scratch.
By default, spack will rebuild these from scratch, even if your local
version is perfectly fine. If you already have an installed version
and prefer to use that instead, then you can specify this in
packages.yaml. Note that these are only build tools.
Hpctoolkit does not link with any of their libraries.
For example, this entry says that cmake 3.7.2 is available from module
CMake/3.7.2 and that spack should use this instead of building
its own copy.
cmake:
modules:
cmake@3.7.2: CMake/3.7.2
buildable: False
This example says that perl v5.16.3 is installed at
/usr/bin/perl. Note that the paths: entry is the parent
directory of bin, not the bin directory itself (similar to
prefix).
perl:
paths:
perl@5.16.3: /usr
buildable: False
See the spack docs on ’Build Customization’ and ’Specs and Dependencies’.
https://spack.readthedocs.io/en/latest/build_settings.html https://spack.readthedocs.io/en/latest/basic_usage.html#specs-dependencies
Building HPCToolkit requires GNU gcc/g++ version 5.x or later. By default, spack uses the latest available version of gcc, but you can specify a different compiler, if one is available.
Spack uses a separate file, compilers.yaml to store information
about available compilers. This file is normally in your home directory
at ~/.spack/platform where ‘platform’ is normally ‘linux’ (or
else ‘cray’ or ‘bgq’).
The first time you use spack, or after adding a new compiler, you should
run spack compiler find to have spack search your system for
available compilers. If a compiler is provided as a module, then you
should load the module before running find. Normally, you only
need to run find once, unless you want to add or delete a
compiler. You can also run spack compiler list and spack
compiler info to see what compilers spack knows about.
For example, on one power8 system running RedHat 7.3, /usr/bin/gcc is
version 4.8.5, but gcc 6.4.0 is available as module GCC/6.4.0.
module load GCC/6.4.0
spack compiler find
==> Added 2 new compilers to /home/krentel/.spack/linux/compilers.yaml
gcc@6.4.0 gcc@4.8.5
==> Compilers are defined in the following files:
/home/krentel/.spack/linux/compilers.yaml
spack compiler list
==> Available compilers
-- gcc rhel7-ppc64le --------------------------------------------
gcc@6.4.0 gcc@4.8.5
spack compiler info gcc@6.4
gcc@6.4.0:
paths:
cc = /opt/apps/software/Core/GCCcore/6.4.0/bin/gcc
cxx = /opt/apps/software/Core/GCCcore/6.4.0/bin/g++
f77 = /opt/apps/software/Core/GCCcore/6.4.0/bin/gfortran
fc = /opt/apps/software/Core/GCCcore/6.4.0/bin/gfortran
modules = ['GCC/6.4.0']
operating system = rhel7
Note: for compilers from modules, spack does not fill in the
modules: field in the compilers.yaml file. You need to
do this manually. In the above example, after running find, I
edited compilers.yaml to add GCC/6.4.0 to the
modules: field as below. This is important to how spack
manipulates the build environment.
- compiler:
modules: [GCC/6.4.0]
operating_system: rhel7
spec: gcc@6.4.0
...
Spack uses % syntax to specify the build compiler and @
syntax to specify the version. For example, suppose you had gcc
versions 7.3.1, 6.4.0 and 5.4.0 available and you wanted to use 6.4.0.
You could write this as:
spack install package %gcc@6.4.0
See the spack docs on ‘Compiler Configuration’.
First, set up your config.yaml, packages.yaml and
compilers.yaml files as above and edit them for your system.
You can see how spack will build hpctoolkit with spack spec and
spack graph.
spack spec hpctoolkit spack graph hpctoolkit
Then, the “one button” method uses spack to install everything.
spack install hpctoolkit
Tip: Spack fetch is somewhat fragile and often has transient
problems downloading files. You can use spack fetch -D to
pre-fetch all of the tar files and resolve any downloading problems
before starting the full install.
spack fetch -D hpctoolkit
The manual method uses spack to build hpctoolkit’s prerequisites and
then uses the traditional autotools configure && make && make
install to install hpctoolkit. This method is primarily for
developers who want to compile hpctoolkit, edit the source code and
recompile, etc. This method is also useful if you need some configure
option that is not available through spack.
First, use spack to build hpctoolkit’s prerequisites as above. You
can either build some version of hpctoolkit as before (which will pull
in the prerequisites), or else use --only dependencies to avoid
building hpctoolkit itself.
spack install --only dependencies hpctoolkit
Then, configure and build hpctoolkit as follows. Hpctoolkit uses automake and so allows for parallel make.
configure \ --prefix=/path/to/hpctoolkit/install/prefix \ --with-spack=/path/to/spack/install_tree/linux-fedora26-x86_64/gcc-7.3.1 \ ... make -j <num> make install
The argument to --with-spack should be the directory containing
all of the individual package directories, normally two directories
down from the top-level install_tree and named by the platform
and compiler. This option replaces the old --with-externals.
The following are other options that may be useful. For the full list
of options, see configure -h.
--enable-all-static – build hpcprof-mpi statically linked for
the compute nodes.
--enable-bgq – configure for Blue Gene/Q back end.
--enable-develop – compile with optimization turned off for
debugging.
--with-package=path – specify the install prefix for some
prerequisite package, mostly for developers who want to use a custom,
non-spack version of some package.
MPICXX=compiler – specify the MPI C++ compiler for
hpcprof-mpi, may be a compiler name or full path.
Note: if your spack install tree has multiple versions or variants for
the same package, then --with-spack will select the one with
the most recent directory time stamp (and issue a warning). If this
is not what you want, then you will need to specify the correct
version with a --with-package option.
Beginning with the 2020.03.01 version, HPCToolkit now supports profiling CUDA binaries (nVidia only). For best results, use CUDA version 10.1 or later and Dyninst 10.1 or later. Note: in addition to a CUDA installation, you also need the CUDA system drivers installed. This normally requires root access and is outside the scope of spack.
For a spack install with CUDA, use the +cuda variant.
spack install hpctoolkit +cuda
For a manual install, either download and install CUDA or use an
existing module, and then use the --with-cuda configure option.
configure \ --prefix=/path/to/hpctoolkit/install/prefix \ --with-spack=/path/to/spack/install/dir \ --with-cuda=/path/to/cuda/install/prefix \ ...
If you installed CUDA with spack in the same directory as the rest of
the prerequisites, then the --with-spack option should find it
automatically (but check the summary at the end of the configure
output). If you are using CUDA from a separate system module, then you
will need the --with-cuda option.
HPCToolkit always supports profiling MPI applications. For
hpctoolkit, the spack variant +mpi is for building hpcprof-mpi,
the MPI version of hpcprof. If you want to build hpcprof-mpi, then
you need to supply an installation of MPI.
Normally, for systems with compute nodes, you should use an existing
MPI module that was built for the correct interconnect for your system
and add this to packages.yaml. The MPI module should be built
with the same version of GNU gcc/g++ used to build hpctoolkit (to keep
the C++ libraries in sync).
HPCToolkit can access the Hardware Performance Counters with either
PAPI or Perfmon (libpfm4). By default, the hpctoolkit package uses
perfmon. If you want to use PAPI instead, then build hpctoolkit with
+papi. However, you can’t use both due to a potential conflict
in their header files.
PAPI runs on top of the perfmon library, but PAPI uses its own, internal copy of perfmon. Prior to version 5.6.0, PAPI did not install the perfmon header files, so it was impossible to access the perfmon events through PAPI.
However, starting with version 5.6.0, PAPI now installs both the perfmon library and its header files. Hpctoolkit configure will automatically detect this, so if you build hpctoolkit with a recent enough version of PAPI, then both the PAPI and perfmon interfaces will be available.
Support for Blue Gene/Q is now deprecated and versions 2020.03.01 and later will no longer build. If you want to install HPCToolkit on Blue Gene, then use spack version 2019.12.28 or git commit 0d91fb34edb2 with gcc 4.8.x and follow the directions below. These directions will also work (without Blue Gene) for Red Hat 6.x (glibc 2.12 is too old to support current hpctoolkit).
Blue Gene systems are being phased out, without a next generation replacement, but are still supported by hpctoolkit (for now). Blue Gene normally comes with a Red Hat Enterprise Linux (RHEL) 6.x front end and the GNU gcc/g++ 4.4 compilers with are too old to support the latest version of hpctoolkit.
Spack supports two architecture types on Blue Gene. The front-end arch
is bgq-rhel6-ppc64 and the back-end arch is bgq-cnk-ppc64.
Normally, it would matter whether we built for the front end or back
end. But for the GNU compilers on Blue Gene, both compilers are
/usr/bin/gcc. So, we just build for the default back end.
There are two ways to build hpctoolkit for Blue Gene, depending on the
version of the compiler and Dyninst. The simple method uses the default
GCC 4.4 compiler and Dyninst 9.3.2. Edit these two entries in the
packages.yaml file.
9.3.2.
+bgq to the libmonitor variants.
Then, build hpctoolkit with +bgq turned on. The +bgq
variant adds MPI and hpcprof-mpi and supersedes the +mpi
variant.
spack install hpctoolkit +bgq
The advanced method involves building a new compiler (gcc 4.8 or 4.9) but allows using Dyninst 10.x. This method requires TCL or Lmod modules and an MPI C++ compiler built for a compatible version of g++. Mira and vesta at ANL have softenv modules for +mpiwrapper-gcc and +bgqtoolchain-gcc484 but don’t provide TCL or Lmod modules.
First, use spack to build a new compiler. Dyninst 10.x requires
gcc 4.8 or later, but libmonitor requires the true back-end
powerpc64-bgq-linux-gcc compiler which is version 4.4. We
settle on gcc 4.8.5 as a compromise between these two constraints.
Edit config.yaml to set module_roots to a directory in
which to install the modules and then build gcc.
spack install gcc @4.8.5
Assuming you have modules installed, then load the new gcc module and
rerun spack compiler find. This should add the new compiler to
your compilers.yaml file (twice, for front and back end). For
example, on vulcan at LLNL,
module use /path/to/modules/bgq-cnk-ppc64
module load gcc-4.8.5-gcc-4.4.7-nzvjwva
spack compiler find
==> Added 2 new compilers to /g/g21/krentel1/.spack/bgq/compilers.yaml
gcc@4.8.5 gcc@4.8.5
Note: after running spack compiler find, you still need to add
the modules field to the compiler entry in compilers.yaml as in
the section on Compilers.
Note: in the above example, we used gcc 4.4.7 to build gcc 4.8.5 (so the gcc-4.8.5 package is in the gcc-4.4.7 directory). You could then use gcc 4.8.5 to rebuild 4.8.5, if you like, but this is not necessary.
Finally, use the new compiler to build hpctoolkit with the new compiler.
If you have modified the dyninst entry in packages.yaml, then
reset the dyninst version to 10.1.0 (or later).
spack install hpctoolkit +bgq %gcc@4.8.5
If you don’t have modules installed, then use spack bootstrap
to build the environment-modules package. Then, source the bash or
csh script in the Modules/init directory to add the
module function to your environment. For example,
spack bootstrap cd /path/to/environment-modules-4.2.4-rwtvlanuss35fxd3xuyldbfopt2m4ecs/init (bash) . ./bash (csh) source ./csh
Alternatively, you could try adding the module environment variables
(PATH and LD_LIBRARY_PATH) manually, but then you may have to run
spack install with --dirty. The --dirty option tells
spack not to erase LD_LIBRARY_PATH while building packages.
spack install --dirty hpctoolkit +bgq %gcc@4.8.5
For developers, if you are building hpctoolkit directly (outside of spack) but using spack prerequisites, then use a configure line similar to the following.
configure \
--prefix=/path/to/install/prefix \
--with-spack=/path/to/bgq-cnk-ppc64/gcc-4.8.5 \
--enable-bgq \
--enable-all-static \
MPICXX=mpicxx
Cray systems, like Blue Gene, have separate front and back-end
architecture types. For example, on theta at ANL, the front-end arch
is cray-sles12-x86_64 (SuSE Linux version 12 for x86_64) and
the back-end is cray-cnl6-mic_knl (Compute Node Linux for KNL).
Hpctoolkit needs to be built with the GNU Programming Environment and
the front-end x86_64 compilers, plus the CC MPI C++ wrapper.
Switch to the PrgEnv-gnu module and unload the darshan module.
Darshan is a profiling tool that monitors an application’s use of I/O,
but it conflicts with hpctoolkit.
module swap PrgEnv-intel PrgEnv-gnu module unload darshan
By default, spack probably does not recognize the gcc compiler modules
as front-end compilers. You have to tell spack to add them
explicitly. Check that the latest gcc module is loaded (it probably
is from PrgEnv-gnu) and run spack compiler find with the
path to the compiler (one directory up from bin). For example,
on theta at ANL,
module load gcc/7.3.0 spack compiler find /opt/gcc/7.3.0 spack compiler list ==> Available compilers -- gcc cnl6-any ------------------------------------------------- gcc@7.3.0 gcc@7.2.0 gcc@7.1.0 gcc@6.3.0 gcc@6.1.0 gcc@5.3.0 gcc@4.9.3 -- gcc sles12-x86_64 -------------------------------------------- gcc@7.3.0 gcc@4.8
Normally, spack does not fill in the modules: field for the new
compiler, you have to add that manually. And although we mostly use
the front-end environment, we also need the CC wrapper to be
the back-end MPI compiler.
Spack does not support using both front-end and back-end compilers in
the same build. But there are two workarounds for Cray. One option is
to load the cray-mpich module and run spack install with
option --dirty. A better way is to add cray-mpich to the
list of modules for gcc in compilers.yaml. Edit the front-end
gcc entry (operating system sles12 and target x86_64) to add the
following modules (your versions may differ).
- compiler:
modules:
- PrgEnv-gnu/6.0.4
- gcc/7.3.0
- cray-mpich/7.7.3
operating_system: sles12
paths:
cc: /opt/gcc/7.3.0/bin/gcc
cxx: /opt/gcc/7.3.0/bin/g++
f77: /opt/gcc/7.3.0/bin/gfortran
fc: /opt/gcc/7.3.0/bin/gfortran
spec: gcc@7.3.0
target: x86_64
This tells spack to load the above modules when using the gcc 7.3.0
compiler. With the cray-mpich module loaded, the CC wrapper
becomes an MPI C++ compiler, which is what hpctoolkit needs.
Next, review your packages.yaml file. On Cray systems with
Xeon Phi back-end nodes (KNL, KNH, etc), add ~tm to intel-tbb
to disable transactional memory.
Finally, build hpctoolkit with the front-end arch type
(cray-sles12-x86_64) and option +cray. Normally, you can use OS
type fe (front-end) in place of sles12. As with Blue
Gene, the +cray option adds MPI and hpcprof-mpi and supersedes
the +mpi variant.
spack install hpctoolkit +cray arch=cray-fe-x86_64
For developers, if you are building hpctoolkit directly (outside of spack) but using spack prerequisites, then use a configure line similar to the following.
configure \
--prefix=/path/to/install/prefix \
--with-spack=/path/to/cray-sles12-x86_64/gcc-7.3.0 \
--enable-all-static \
MPICXX=CC
We provide binary distributions for hpcviewer and hpctraceviewer on Linux (x86_64 and powerpc), Windows and MacOS. HPCToolkit databases are platform-independent and it is common to run hpcrun on one machine and then view the results on another machine.
All versions of the viewers require Java version 8 (not 9 or later). The Linux versions also require GTK+ version 2.x.
The spack install is available on Linux x86_64, big-endian power7 (ppc64) and little-endian power8 and 9 (ppc64le). This installs both hpcviewer and hpctraceviewer and includes the Java 8 prerequisite.
On x86_64, use either jdk or openjdk. The latest 1.8 jdk package, 1.8.0_202 has trouble downloading due to licensing issues, however 1.8.0_141-b15 seems to work. (Or, you could manually download jdk 1.8.0_202 and copy it to the spack file cache.) For openjdk, use version 1.8.0_202-b08.
spack install hpcviewer ^jdk@1.8.0_141-b15 spack install hpcviewer ^openjdk@1.8.0_202-b08
On powerpc (big and little-endian), use the IBM version of Java.
spack install hpcviewer ^ibm-java
Binary distributions of the viewers for all supported platforms are available at:
On Linux, download the linux.gtk versions of hpcviewer and
hpctraceviewer, unpack the tar files and run the install scripts (for
both viewers) with the path to the desired install prefix.
./install /path/to/install/directory
On Windows and MacOS, download the win32 or macosx.cocoa
versions and unpack the zip files in the desired directory. Due to
Apple’s security precautions, on MacOS, you will need to use curl or
wget instead of a web browser.
Some systems may have compilers that are too old for building HPCToolkit or other packages. For example, RedHat 7.x comes with gcc 4.8.x which is very old. If your system doesn’t already have modules for later compilers, then you may need to build a new compiler yourself.
First, pick a directory in which to install the modules and make
subdirectories for the spack packages and module files. In this
example, I’m using /opt/spack/Modules as my top-level directory
and subdirectories packages and modules. Edit
config.yaml to add these paths.
config:
install_tree: /opt/spack/Modules/packages
# just use one of these
module_roots:
tcl: /opt/spack/Modules/modules
lmod: /opt/spack/Modules/modules
Determine if your system uses TCL (environment) or Lmod modules.
Normally, the module command is a shell function. TCL modules
use modulecmd and Lmod modules eval LMOD_CMD. Edit
modules.yaml to enable the module type for your system. Again,
you only need one of these (and don’t use dotkit unless you work at
LLNL).
modules: enable: - tcl - lmod
Then, choose a version of gcc and use the default compiler (normally
/usr/bin/gcc) to build the newer version. Currently, gcc 7.x
is a good choice that builds robustly, has enough modern features but
is not too new to cause problems for some packages. For example,
spack install gcc@7.4.0
Note: it is not necessary to rebuild the new compiler with itself.
After building a new compiler, then you need to tell spack how to find
it. First, use module use and module load to load the
module. For TCL modules, the module files are in a subdirectory of
module_roots named after the system architecture. For example,
module use /opt/spack/Modules/modules/linux-rhel7-x86_64 module load gcc-7.4.0-gcc-4.8.5-qemsqrc
For Lmod modules, the module directory is one level below that and the module names are a little different.
module use /opt/spack/Modules/modules/linux-rhel7-x86_64/Core module load gcc/7.4.0-dan4vbm
For both TCL and Lmod modules, it’s best to put the module use
command in your shell’s startup scripts so that module avail and
module load will know where to find them. After loading the
module, run spack compiler find.
$ spack compiler find
==> Added 1 new compiler to /home/krentel/.spack/linux/compilers.yaml
gcc@7.4.0
Finally, always check the new entry in compilers.yaml and add the
name of the module to the modules: field.
- compiler:
environment: {}
extra_rpaths: []
flags: {}
modules:
- gcc-7.4.0-gcc-4.8.5-qemsqrc
operating_system: rhel7
paths:
cc: /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gcc
cxx: /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/g++
f77: /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gfortran
fc: /opt/spack/Modules/packages/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.4.0-qemsqrcwkk52f6neef4kg5wvoucsroif/bin/gfortran
spec: gcc@7.4.0
target: x86_64
Note: as long as the spack packages and modules directories remain
intact and you don’t remove the compilers.yaml entry, then this
compiler will always be available from within spack. You can also use
this compiler outside of spack by using module load. If you
want to make this your default compiler for all spack builds, then you
can specify this in packages.yaml. For example,
packages:
all:
compiler: [gcc@7.4.0]
Also, when using the compiler from within spack, it doesn’t matter if you have the module loaded or not. Spack will erase your environment and re-add the appropriate modules automatically.
If your system does not support modules, then you will have to add it.
If you have root access, the easiest solution is to install a system
package for modules. If not, then use spack to install the
environment-modules package. Source the bash or csh script in the
init directory to add the module function to your
environment. For example,
spack install environment-modules cd /path/to/environment-modules-4.3.1-ism7cdy4xverxywj27jvjstqwk5oxe2v/init (bash) . ./bash (csh) source ./csh
Again, add the setup command to your shell’s startup scripts.
A spack mirror allows you to download and save a source tar file in advance. This is useful if your system is behind a firewall, or if you need to manually agree to a license, or if you just don’t want to keep downloading the same file over and over.
A mirror has a simple directory structure and is easy to set up. Create a top-level directory with subdirectories named after the spack packages and copy the tar files into their package’s subdirectory. For example,
my-mirror/
boost/
boost-1.66.0.tar.bz2 (from boost_1_66_0.tar.bz2)
boost-1.70.0.tar.bz2
dyninst/
dyninst-10.1.0.tar.gz (from git checkout)
ibm-java/
ibm-java-8.0.5.30.None (from ibm-java-sdk-8.0-5.30-ppc64le-archive.bin)
intel-xed/
intel-xed-2019.03.01.tar.gz (from git checkout)
mbuild-2019.03.01.tar.gz (resource from git checkout)
jdk/
jdk-1.8.0_202.tar.gz (from jdk-8u202-linux-x64.tar.gz)
Note: the names of the files in the spack mirror always follow the same,
specific format, regardless of the actual name of the tar file. Version
is the spack name for the version (from spack info), and
extension is the same extension as the tar file (tar.gz,
tar.bz2, etc) or else None for other types of files.
<package-name> - <version> . <extension>
For example, the boost 1.66.0 tar file is actually named
boost_1_66_0.tar.bz2 but is stored in the mirror as
boost-1.66.0.tar.bz2 and jdk-8u202-linux-x64.tar.gz is
renamed to jdk-1.8.0_202.tar.gz.
For packages that use a snapshot from a git repository (tag or commit
hash), clone the repository, checkout the desired version, make a tar
file and gzip the file. (You should exclude the .git
subdirectory.) But note that spack refuses to use a cached file for
the head of a branch because it is a moving target.
Finally, after creating the mirror directory, add it to spack with
spack mirror add. For example,
spack mirror add my-mirror file:///home/krentel/spack/my-mirror spack mirror list my-mirror file:///home/krentel/spack/mirror
Note: by default, spack stores downloaded files inside the spack
repository at spack/var/spack/cache. This directory is a full
spack mirror, so instead of creating a separate directory tree, you
could just copy the files into the cache directory. This is
useful when spack fetch has trouble downloading a file. If you
can download the file manually, or copy it from another machine, then
just rename the file as above and copy it into the spack file cache.
For more information on mirrors, see:
Spack is somewhat fragile for how it downloads tar files and will often fail for transitory network problems. This is especially true for packages with many dependencies. For example:
==> Installing m4 ==> Searching for binary cache of m4 ==> No binary for m4 found: installing from source curl: (6) Could not resolve host: ftp.wayne.edu; Name or service not known ==> Fetching https://ftpmirror.gnu.org/m4/m4-1.4.18.tar.gz ==> Fetching from https://ftpmirror.gnu.org/m4/m4-1.4.18.tar.gz failed. ==> Error: FetchError: All fetchers failed for m4-1.4.18-vorbvkcjfac43b7vuswsvnm6xe7w7or5
There are two workarounds. First, assuming the problem is temporary,
simply wait 10 minutes or an hour and try again. Second, you could
manually download the file(s) by some other means and copy them to
spack’s cache directory spack/var/spack/cache/<package> or to a
spack mirror.
Normally, HPCToolkit should build and work correctly with the latest version for all of its dependencies. But sometimes a new release will change something and break the build. This has happened a couple times where a new release of Boost has broken the build for Dyninst. Or, maybe the latest version of gcc/g++ disallows some usage and breaks the build.
The solution is to use packages.yaml to specify an earlier
version until the rest of the code adapts to the change.
Spack is quite aggressive about compiling with a clean environment and
will unload modules unless they are specifically required by some config
file (compilers.yaml or packages.yaml). This can result
in a situation where you think some compiler or build tool is available
from your environment but spack removes it during the build.
In this example, I am using modules for GCC/6.4.0 and
CMake/3.8.2. Spack finds the gcc 6.4.0 compiler and I added
cmake@3.8.2 to packages.yaml. But I failed to add the
modules: field for gcc 6.4.0 in compilers.yaml. As a
result, the build fails with:
cmake: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by cmake) cmake: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by cmake) cmake: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by cmake) ==> Error: ProcessError: Command exited with status 1:
The problem is that cmake 3.8.2 was built with g++ 6.4.0, but spack is
running cmake without the GCC/6.4.0 libraries and so the build fails as
above. One way to confirm this is to rerun spack install --dirty
which then succeeds. The --dirty option tells spack not to
unload your modules. Whenever the build fails with a mismatched library
as above and especially when --dirty fixes the problem, this is a
clear sign that spack is missing a module during the build.
Although --dirty may make the build succeed, there should be no
case where this is necessary. The correct solution is to fill in the
modules: field in compilers.yaml or some other config
file. See the section on Compilers above.