Building and Installing HPCToolkit

Contents

• Prerequisites
• Downloading Source Code
• Building and Installing
• Infrequently Used Build Options

Prerequisites

[Contents]

To reduce our maintenance efforts, we have opted to distribute HPCToolkit primarily in source form. The following prerequisites apply to our default source code distribution.

1. HPCToolkit can be installed on the following platforms:

   Platform (Supported)	Notes
   Linux + x86_64
   Linux + x86 (32-bit)
   Linux + ppc64
   Cray XT/XE/XK/XC (CNL + x86_64)	Compute Node Linux ≥ 2.1
   IBM Blue Gene/Q (BG/Q kernel + ppc64)
   IBM Blue Gene/P (BG/P kernel + ppc32)	BG/P kernel ≥ V1R3M0
   Platform (Legacy and Not Supported)
   Linux + IA64	testing (uses libunwind)
   Linux + MIPS-le (64 and n32)	for SiCortex clusters; little endian MIPS (no longer tested)

2. Compiler: HPCToolkit requires GNU GCC version 4.4 or later. Some vendor compilers that support GNU extensions may also work, but we recommend building with GNU gcc/g++. HPCToolkit also requires GNU make, preferably version 3.81 or later.

3. To use hpcviewer and hpctraceviewer (HPCToolkit's graphical interface) on MacOS, Windows, or Linux, a Java runtime environment (JRE) ≥ 1.6 is required. Use the Oracle or IBM version; we have had problems with OpenJDK.

   To use hpcviewer and hpctraceviewer on Linux, GTK+ is also required.

   Note: Since HPCToolkit's databases are portable, we often install hpcviewer and hpctraceviewer on local laptops/desktops to avoid sending display traffic over relatively slow network connections.

4. Highly recommended: To collect measurements based on hardware performance monitoring unit (PMU) information, PAPI ≥ 4.1 is required. Without PAPI, HPCToolkit can only collect measurements based on an operating system timer. (Exception: On IBM Blue Gene/P, PAPI is not required. On this system, HPCToolkit uses IBM's native Universal Performance Counters directly.)

   There are currently three ways to support PAPI on Linux-based OSs:

   • Use a kernel with perf_events (see also: this), which was introduced with kernel 2.6.32 (née PCL in kernel 2.6.31) and which is now becoming part of standard distributions. With perf_events, use PAPI ≥ 4.1.2.

   • Apply the perfctr kernel patch (x86_64-based systems only). One benefit of perfctr is that it includes out-of-the-box support for many Red Hat kernels. As of this writing, perfctr supports vanilla kernels through 2.6.32.

   • Apply the perfmon2 kernel patch. As of this writing, perfmon2 supports vanilla kernels through 2.6.30. (Unfortunately, development has stopped because of perf_events.)

5. Minor requirement: To use hpcsummary (a relatively minor tool), Python 2.x is required.

6. Minor requirement: To build HPCToolkit's man pages, a variant of Perl 5 is required. We have successfully used versions 5.6.x - 5.10.x.

Downloading Source Code

[Contents]

Download HPCToolkit components as follows:

1. Download hpctoolkit from GitHub.

   git clone https://github.com/HPCToolkit/hpctoolkit.git

2. Download hpctoolkit-externals from GitHub.

   git clone https://github.com/HPCToolkit/hpctoolkit-externals.git

   Think of hpctoolkit-externals as a "package manager" that automatically configures and builds support libraries that hpctoolkit needs. It contains several external libraries. (Some of these libraries, like libelf and libxml2 are commonly found on Linux systems, but not always. Others, like OpenAnalysis and SymtabAPI are almost never available. Finally, a few packages, like GNU binutils have been heavily patched.) In some cases it may be possible to use versions of these packages that are already installed on your system. However, we do not support such configurations.

3. Download binary releases of hpcviewer and hpctraceviewer for the platform on which your are building HPCToolkit.

   http://hpctoolkit.org/download/hpcviewer

   Note: It is perfectly reasonable to install hpcviewer and hpctraceviewer in multiple locations. Since HPCToolkit's databases are portable, we often install hpcviewer and hpctraceviewer on local laptops/desktops to avoid sending display traffic over relatively slow network connections from systems where we collect measurement data.

Building and Installing

[Contents]

The build and install follows autotools conventions (./configure && make && make install). In each case, more options are available by executing ./configure --help. See the README file for more details.

The typical build is as follows. Please note that a list of examples below provides configure commands for selected platforms.

1. Configuring the perf_events sample source:

   HPCToolkit's configure command will automatically check the availability of the Linux perf_events interface on the build system. The test includes checking the existence of /usr/include/linux/perf_event.h and /proc/sys/kernel/perf_event_paranoid files. If the two files do not exist, it assumes the system does not support the perf_events interface.

   To profile on a remote machine that has different system or architecture configuration, one must download and build the libpfm4 library (http://perfmon2.sourceforge.net) and indicate its install directory by specifying the option --with-perfmon=<libpfm4_install_directory> to HPCToolkit's configure command.

2. Build and install HPCToolkit's Externals, which typically does not require any special options; we recommend using the GCC compilers:

   cd hpctoolkit-externals
mkdir BUILD && cd BUILD
../configure [CC=<c-compiler>] [CXX=<c++-compiler>] \
     [--prefix=<hpctoolkit-externals-install>]
   make install
make clean
   

   Externals is not persistent as HPCToolkit will copy everything it needs from the installation. Thus, the installation path <hpctoolkit-externals-install> should be a local path and not something like /usr. If --prefix is not supplied it will automatically be set to ./<Autoconf-host>.

   To specify a C and C++ compiler other than gcc and g++, use CC and CXX.

   Please be sure the Externals build completes before configuring HPCToolkit. (Note: After building Externals once, HPCToolkit can be configured and built several times using the same Externals installation.)

3. Build and install HPCToolkit using commands similar to the following; we recommend using the GCC compilers:

   cd hpctoolkit
mkdir BUILD && cd BUILD
../configure [CC=<c-compiler>] [CXX=<c++-compiler>] [MPICXX=<mpi-c++-compiler>] \
     --prefix=<hpctoolkit-install> \
     --with-externals=<hpctoolkit-externals-install> \
     [--with-papi=<papi-install>]
make install
   

   The above commands will build HPCToolkit and install it into <hpctoolkit-install>.

   To specify a C and C++ compiler other than gcc and g++, use CC and CXX. To enable MPI support, (1) ensure a working mpicxx or mpiCC is in your shell's path; or (2) use MPICXX. (Naturally, CXX and MPICXX should be compatible; typically MPICXX is a wrapper for CXX.) To enable PAPI-based measurements, use the --with-papi option; it takes as an argument the path of your PAPI installation, <papi-install>.

   Because the HPCToolkit installation is self-contained, it may be renamed or moved to another location. All HPCToolkit Externals source code, build files and installations may be removed; all HPCToolkit source code and build files may be removed.

4. Install hpcviewer and hpctraceviewer .

   • To install hpcviewer or hpctraceviewer as a stand-alone package, simply unpack and move the directory tree to the desired location. (This is the only option on Windows or MacOS.)
   • To install hpcviewer or hpctraceviewer in an HPCToolkit installation, unpack the tarball and execute the following commands:
     cd <hpcviewer> && ./install [ -j <path-to-java> ] <hpctoolkit-install>
cd <hpctraceviewer> && ./install [ -j <path-to-java> ] <hpctoolkit-install>
     Here, <hpcviewer> and <hpctraceviewer> are the directories created by unpacking the respective tarballs; and <hpctoolkit-install> is the above HPCToolkit installation directory.
     The optional flag -j <path-to-java> allows the specification of an appropriate java to be used with hpcviewer or hpctraceviewer.

Selected Examples

The list below gives example HPCToolkit Externals and HPCToolkit configure commands (respectively) for selected platforms.

N.B.: These commands are designed to replace the respective commands in the above build instructions.

• Blue Gene/Q:

  Load the '+mpiwrapper-gcc' softenv package (for the MPICXX compiler) and check that the back-end powerpc64-bgq-linux compilers are on your PATH. Use the '--enable-bgq' option for HPCToolkit externals to build for the Blue Gene/Q compute nodes.

  soft add +mpiwrapper-gcc
PATH="/bgsys/drivers/ppcfloor/gnu-linux/bin/:$PATH"

  HPCToolkit's Externals

  ../configure --prefix=`pwd`/../powerpc64-linux \
     --enable-bgq

  HPCToolkit

  ../configure \
     MPICXX=/bgsys/drivers/ppcfloor/comm/gcc/bin/mpicxx \
     HPCPROFMPI_LT_LDFLAGS="-all-static" \
     --prefix=<install-path> \
     --with-externals=<hpctoolkit-externals>/powerpc64-linux \
     --with-papi=/soft/perftools/papi

  Normally on Blue Gene, you want to build HPCToolkit to run on the compute nodes. Configure will auto-detect this and default to the back-end nodes (but issue a warning). The '--enable-bgq' flag adds options specific to the Blue Gene/Q compute nodes and suppresses the warning. If you want to build hpcrun to run on the front-end nodes, then use the option '--disable-bgq'.

  Note: The paths for MPICXX and PAPI may be different on your system.

• Blue Gene/P:

  HPCToolkit's Externals

  ../configure --prefix=`pwd`/../powerpc32-linux

  HPCToolkit

  ../configure \
     HPC_LT_LDFLAGS="-all-static -L../../lib/stubs-gcc_s" \
     HPCPROFMPI_LT_LDFLAGS="-L/bgsys/drivers/ppcfloor/gnu-linux/powerpc-bgp-linux/lib -lstdc++" \
     --prefix=<hpctoolkit-install> \
     --with-externals=<hpctoolkit-externals>/powerpc32-linux

   

  Note: PAPI is not available on Blue Gene/P.

• Cray XT/XE/XK/XC:

  N.B.: Use the GNU programming environment, e.g.:

  module swap PrgEnv-pgi PrgEnv-gnu

  HPCToolkit's Externals

  ../configure --prefix=`pwd`/../x86_64-linux


  HPCToolkit

  ../configure \
     MPICXX="CC" \
     HPCPROFMPI_LT_LDFLAGS="-all-static" \
     --prefix=<hpctoolkit-install> \
     --with-externals=<hpctoolkit-externals>/x86_64-linux \
     --with-papi=/opt/cray/papi/4.3.0.1/perf_events/no-cuda

   

  Note: On systems with AMD CPUs with the new XOP instructions, add the option '--enable-xop'. The AMD Interlagos CPUs (6200 series) use these instructions. Currently, Jaguar at ORNL uses these CPUs, but Hopper at NERSC does not.

• Intel MIC (early and experimental):

  Systems such as Stampede at TACC (Texas Advanced Computing Center) have Intel MIC (Many Integrated Cores) coprocessors on the compute nodes. Although the MIC cores use an extension of the Intel x86_64 instruction set, the architecture type of the MIC binaries (k1om) is different from the type on the main cores (x86_64). Binaries compiled for the MIC will not run on the main cores and vice versa.

  The solution is to build all of externals and hpctoolkit twice: once for the main cores and once for the MIC cores. Install the builds into two separate install trees (prefixes). Both externals and hpctoolkit support VPATH builds, so you only need to checkout one copy of each source tree. Let's say that you use BUILD-x86_64 and BUILD-k1om for the build directories and install-x86_64 and install-k1om for the install directories.

  HPCToolkit Externals for the main cores:

  cd BUILD-x86_64
../configure --prefix=/path/to/externals/install-x86_64 \
     --enable-mic

  HPCToolkit Externals for the MIC cores:

  PATH="/usr/linux-k1om-4.7/bin/:$PATH"
cd BUILD-k1om
../configure --prefix=/path/to/externals/install-k1om \
     --host=x86_64-k1om-linux \
     CC=x86_64-k1om-linux-gcc \
     CXX=x86_64-k1om-linux-g++

  HPCToolkit for the main cores:

  cd BUILD-x86_64
../configure --prefix=/path/to/hpctoolkit/install-x86_64 \
     --with-externals=/path/to/externals/install-x86_64 \
     --with-papi=/path/to/papi-install

  HPCToolkit for the MIC cores:

  PATH="/usr/linux-k1om-4.7/bin/:$PATH"
cd BUILD-k1om
../configure --prefix=/path/to/hpctoolkit/install-k1om \
     --with-externals=/path/to/externals/install-k1om \
     --host=x86_64-k1om-linux \
     CC=x86_64-k1om-linux-gcc \
     CXX=x86_64-k1om-linux-g++

  If PAPI is available for the MIC cores, then add --with-papi=/path/to/papi-k1om to the hpctoolkit configure for MIC. Make sure that this version of PAPI is built for the MIC cores. hpcprof-mpi is untested for the MIC cores.

  Be sure to use the option --enable-mic in the externals's build for the main cores. This option selects a version of XED that runs on the main (x86_64) cores but understands the MIC (k1om) instructions. (This option is ignored for the k1om build.) Without this option, hpcstruct will crash on MIC binaries. Unfortunately, this version of XED is new and currently available only as a shared library, thus making it incompatible with hpclink and the static case (why the default is no). If you need to run static binaries on Stampede with hpclink, then build a second version of hpctoolkit for the main cores without this option.

  You can build a restricted set of tools with the --enable-back-end option for hpctoolkit (experimental). This option builds only the tools, libraries and prerequisites needed for running hpcrun and hpclink and collecting data on the back-end compute nodes. In that case, run hpcstruct and hpcprof on the main cores.

  On Stampede, you can launch a mixed-mode job (symmetric computing) with ibrun.symm and give separate command lines for the main and MIC cores.

  ibrun.symm -c "command line for main cores" \
     -m "command line for MIC cores"

• Linux-x86 (32-bit) on Linux-x86_64:

  HPCToolkit's Externals

  ../configure CC="gcc -m32" CXX="g++ -m32" \
     --host=i686-linux --prefix=`pwd`/../i686-linux \
     --without-libunwind

  HPCToolkit

  ../configure CC="gcc -m32" CXX="g++ -m32" MPICXX="mpicxx -m32" \
     --host=i686-linux --prefix=<hpctoolkit-install> \
     --with-externals=<hpctoolkit-externals>/i686-linux \
     --with-papi=<papi-install> \
     --disable-hpcrun-static

Infrequently Used Build Options

[Contents]

The following options may be useful in exceptional circumstances.

• To enable/disable HPCToolkit features, the following options may be used.

  Option	Meaning
  --enable-mpi	enable/disable build of hpcprof-mpi [default: yes]
  --with-upc=PATH	path to IBM BlueGene/P UPC installation
  --with-upc-lib=LIBRARY	path to IBM BlueGene/P UPC library(s), absolute or relative to WITH_UPC/runtime/SPI
  --enable-develop	enable/disable build for development (sane debugging) [default: no]

• To request that HPCToolkit Externals use an alternate user-supplied installation for a given package, the following options may be used.

  Option	Meaning
  --with-binutils=PATH	path to binutils install directory
  --with-libdwarf=PATH	path to libdwarf install directory
  --with-libelf=PATH	path to libelf install directory
  --with-libmonitor=PATH	path to libmonitor install directory
  --with-libunwind=PATH	path to libunwind install directory
  --with-libxml2=PATH	path to libxml2 install directory
  --with-old-monitor=PATH	path to old monitor install directory
  --with-open-analysis=PATH	path to open analysis install directory
  --with-symtabAPI=PATH	path to symtabAPI install directory
  --with-xed2=PATH	path to xed2 install directory
  --with-xerces=PATH	path to xerces/c install directory

  In all cases, the default is for HPCToolkit externals build the package itself, depending on platform.

• To explicitly request that a package x not be built, use --without-x.

[Page last updated: 2017/01/20]

---

©2000-2018 Rice University • Rice Computer Science