Statistical Profiling

The HPCToolkit Performance Tools


Version 2017.11

hpcrun is a call path profiler based on statistical sampling. It supports multiple sample sources during one execution. hpcrun profiles complex applications (forks, execs, threads and dynamic linking) and may be used in conjunction with parallel process launchers such as MPICH's mpiexec and SLURM's srun.

See hpctoolkit(1) for an overview of HPCToolkit.

Table of Contents


hpcrun [profiling-options] command [command-arguments]

hpcrun [info-options]


hpcrun profiles the execution of an arbitrary command command using statistical sampling (rather than instrumentation). It collects per-thread call path profiles that represent the full calling context of sample points. Sample points may be generated from multiple simultaneous sampling sources. hpcrun profiles complex applications that use forks, execs, threads, and dynamic linking/unlinking; it may be used in conjuction with parallel process launchers such as MPICH's mpiexec and SLURM's srun.

To profile a statically linked executable, make sure to link with hpclink(1) .

To configure hpcrun's sampling sources, specify events and periods using the -e/--event option. For an event e and period p, after every p instances of e, a sample is generated that causes hpcrun to inspect the and record information about the monitored command.

When command terminates, a profile measurement databse will be written to the directory:
where jobid is a PBS or Sun Grid Engine job identifier.

hpcrun enables a user to abort a process and write the partial profiling data to disk by sending the Interrupt signal (INT or Ctrl-C). This can be extremely useful on long-running or misbehaving applications.


The command to profile.
Arguments to the command to profile.

Default values for an option's optional arguments are shown in {}.

Options: Informational

-l, -L, --list-events
List available events. (N.B.: some may not be profilable)

-V, --version
Print version information.

-h, --help
Print help.

Options: Profiling

-e event[@period], --event event[@period]
An event to profile and its corresponding sample period. event may be either a PAPI, native processor event, or WALLCLOCK (microseconds). May pass multiple times as implementations permits. {WALLCLOCK@5000}

-t, --trace
Generate a call path trace (in addition to a call path profile).

-f frac, -fp frac, --process-fraction frac
Measure only a fraction frac of the execution's processes. For each process, enable measurement (of all threads) with probability frac; frac is a real number (0.10) or a fraction (1/10) between 0 and 1. (To minimize perturbations, when a process is disabled, all threads in a process still receive sampling interrupts, but they are ignored.)

-r, --retain-recursion
Normally, hpcrun will collapse (simple) recursive call chains to a single node. This option disables that behavior: all elements of a recursive call chain are recorded NOTE: If the user employs the RETCNT sample source, then this option is enabled: RETCNT implies *all* elements of call chains, including recursive elements, are recorded.

-o outpath, --output outpath
Directory for output data. {hpctoolkit-<command>-measurements[-<jobid>]}

Environment Variables

For most systems, hpcrun requires no special environment variable settings. There are two situations, however, where hpcrun, to function correctly, must refer to environment variables. These environment variables, and corresponding situations are:
[HPCTOOLKIT] To function correctly, hpcrun must know the location of the HPCToolkit top-level installation directory. The hpcrun script uses elements of the installation lib and libexec subdirectories. For most systems, the installation procedure ensures that hpcrun can find the requisite components. Some parallel job launchers, however, will copy the hpcrun script to a different location from the installed base. If your system uses this copying mechanism, you must set the HPCTOOLKIT environment variable to the top-level installation directory.


Assume we wish to profile the application zoo. The following examples lists some useful events for different processor architectures. In each case, the special option -- is used to clearly demarcate the end of hpcrun options.

  1. hpcrun -e WALLCLOCK@5000 zoo

  2. Opteron, (Rev B-F)
    1. hpcrun -e DC_L2_REFILL@1300013 -e PAPI_L2_DCM@510011 -e PAPI_STL_ICY@5300013 -e PAPI_TOT_CYC@13000019 zoo (DC_L2_REFILL is an approximation of L1 D-cache misses).
    2. hpcrun -e PAPI_L2_DCM@510011 -e PAPI_TLB_DM@510013 -e PAPI_STL_ICY@5300013 -e PAPI_TOT_CYC@13000019 zoo


Sample sources

hpcrun uses the PAPI library to provide access to hardware performance counter events. If you have not configured HPCToolkit to use the PAPI library, you will be unable to measure hardware performance counter events.

The PAPI library supports a large collection of hardware counter events. Some events have standard names across all platforms, e.g. PAPI_TOT_CYC, the event that measures total cycles. In addition to events whose names begin with the PAPI_ prefix, platforms also provide access to a set of native events with names that are specific to the platform's processor. A complete list of events supported by the PAPI library for your platform may be obtained by using the --list-events option. Any event whose name begins with the PAPI_ prefix that is listed as "Profilable" can be used as an event in a sampling source --- provided it does not conflict with another event.

The precise rules for selecting good events and periods are complex.

Platform-specific notes

Cray XE and XK
When using dynamically linked binaries on Cray XE and XK systems, you should add the HPCTOOLKIT environment variable to your launch script. Set HPCTOOLKIT to the top-level HPCToolkit install prefix (the directory containing the bin, lib and libexec subdirectories) and export it to the environment. This is only needed for running dynamically linked binaries. For example:

#PBS -l mppwidth=#nodes
#PBS -l walltime=00:30:00

export HPCTOOLKIT=/path/to/hpctoolkit/install/directory

    ...Rest of Script...

If HPCTOOLKIT is not set, you may see errors such as the following in your job's error log.

/var/spool/alps/103526/hpcrun: Unable to find HPCTOOLKIT root directory.
Please set HPCTOOLKIT to the install prefix, either in this script,
or in your environment, and try again.

The problem is that the Cray job launcher copies the hpcrun script to a directory somewhere below /var/spool/alps/ and runs it from there. By moving hpcrun to a different directory, this breaks hpcrun's method for finding its own install directory. The solution is to add HPCTOOLKIT to your environment so that hpcrun can find its install directory.


See Also

hpctoolkit(1) .
hpclink(1) .


Version: 2017.11 of 2014/03/05.

License and Copyright

© 2002-2018, Rice University.
See README.License.


John Mellor-Crummey
Nathan Tallent
Mark Krentel
Mike Fagan
Rice HPCToolkit Research Group
Email: hpctoolkit-forum =at= rice.edu
WWW: http://hpctoolkit.org.