3. Configuration, build and installation

3.1. Configuration of the instrumentation package

There are many options to be applied at configuration time for the instrumentation package. We point out here some of the relevant options, sorted alphabetically. To get the whole list run configure --help. Options can be enabled or disabled. To enable them use --enable-X or --with-X= (depending on which option is available), to disable them use --disable-X or --without-X.

  • --enable-instrument-dynamic-memory

    Allows instrumentation of dynamic memory related calls (such as malloc, free, realloc).

  • --enable-merge-in-trace

    Embed the merging process in the tracing library so the final tracefile can be generated automatically from the application run.

  • --enable-parallel-merge

    Build the parallel mergers (mpimpi2prv/mpimpi2dim) based on MPI.

  • --enable-posix-clock

    Use POSIX clock (clock_gettime call) instead of low level timing routines. Use this option if the system where you install the instrumentation package modifies the frequency of its processors at runtime.

  • --enable-single-mpi-lib

    Produces a single instrumentation library for MPI that contains both Fortran and C wrappers. Applications that call the MPI library from both C and Fortran languages need this flag to be enabled.

  • --enable-sampling

    Enable PAPI sampling support.

  • --enable-pmapi

    Enable PMAPI library to gather CPU performance counters. PMAPI is a base package installed in AIX systems since version 5.2.

  • --enable-openmp

    Enable support for tracing OpenMP on IBM, GNU and Intel runtimes. The IBM runtime instrumentation is only available for Linux/PowerPC systems.

  • --enable-openmp-gnu

    Enable support for tracing OpenMP on GNU runtime.

  • --enable-openmp-intel

    Enable support for tracing OpenMP on Intel runtime.

  • --enable-openmp-ibm

    Enable support for tracing OpenMP IBM runtime. The IBM runtime instrumentation is only available for Linux/PowerPC systems.

  • --enable-openmp-ompt

    Enables support for tracing OpenMP runtimes through the OMPT specification.

    Note

    Enabling this option disables the regular instrumentation system available through --enable-openmp-gnu, --enable-openmp-intel and --enable-openmp-gnu.

  • --enable-smpss

    Enable support for tracing SMP-superscalar.

  • --enable-nanos

    Enable support for tracing Nanos run-time.

  • --enable-online

    Enables the on-line analysis module.

  • --enable-pthread

    Enable support for tracing pthread library calls.

  • --enable-xml

    Enable support for XML configuration (not available on BG/L, BG/P and BG/Q systems).

  • --enable-xmltest

    Do not try to compile and run a test LIBXML program.

  • --enable-doc

    Generates this documentation.

  • --prefix=<PATH>

    Location where the installation will be placed. After issuing make install you will find under DIR the entries lib/, include/, share and bin containing everything needed to run the instrumentation package.

  • --with-bfd=<PATH>

    Specify where to find the Binary File Descriptor package. In conjunction with libiberty, it is used to translate addresses into source code locations.

  • --with-binary-type=<32|64|default>

    Specifies the type of memory address model when compiling (32bit or 64bit).

  • --with-boost=<PATH>

    Specify the location of the BOOST package. This package is required when using the DynInst instrumentation with versions newer than 7.0.1.

  • --with-binutils=<PATH>

    Specify the location for the binutils package. The binutils package is necessary to translate addresses into source code references.

  • --with-clustering=<PATH>

    If the on-line analysis module is enabled (see --enable-online), specify where to find ClusteringSuite libraries and includes. This package enables support for on-line clustering analysis.

  • --with-cuda=<PATH>

    Enable support for tracing CUDA calls on nVidia hardware and needs to point to the CUDA SDK installation path. This instrumentation is only valid in binaries that use the shared version of the CUDA library. Interposition has to be done through the LD_PRELOAD mechanism. It is superseded by --with-cupti which also supports instrmentation for static binaries.

  • --with-cupti=<PATH>

    Specify the location of the CUPTI libraries. CUPTI is used to instrument CUDA calls, and supersedes the --with-cuda, although it still requires it.

  • --with-dyninst=<PATH>

    Specify the installation location for the DynInst package. Extrae also requires the DWARF package --with-dwarf when using DynInst. Also, newer versions of DynInst (versions after 7.0.1) require the BOOST package --with-boost. This flag is mandatory. Requires a working installation of a C++ compiler.

  • --with-fft=<PATH>

    If the spectral analysis module is enabled (see --with-spectral), specify where to find FFT libraries and includes. This library is a dependency of the Spectral libraries.

  • --with-java-jdk=<PATH>

    Specify the location of JAVA development kit (JDK). This is necessary to create the connectors between Extrae and Java applications.

  • --with-java-aspectj=<PATH>

    Specify the location of the AspectJ infrastructure. AspectJ is used to give support to dynamically instrumented Java applications.

  • --with-java-aspectj-weaver=<path/to/aspectjweaver.jar>

    AspectJ includes the aspectweaver.jar file that is responsible for the execution of dynamically instrumented Java applications. If --with-java-aspectj cannot locate this file, use this option to tell Extrae where to find it.

  • --with-liberty=<PATH>

    Specify where to find the libiberty package. In conjunction with Binary File Descriptor, it is used to translate addresses into source code locations.

  • --with-mpi=<PATH>

    Specify the location of an MPI installation to be used for the instrumentation package. This flag is mandatory.

  • --with-mpi-name-mangling=<0u|1u|2u|upcase|auto>

    Choose the Fortran name decoration (0, 1 or 2 underscores) for MPI symbols, or auto to automatically detect the name mangling.

  • --with-synapse=<PATH>

    If the on-line analysis module is enabled (see --enable-online), specify where to find Synapse libraries and includes. This library is a front-end of the MRNet library.

  • --with-opencl=<PATH>

    Specify the location for the OpenCL package, including library and include directories.

  • --with-openshmem=<PATH>

    Specify the location of the OpenSHMEM installation to be used for the instrumentation package.

  • --with-papi=<PATH>

    Specify where to find PAPI libraries and includes. PAPI is used to gather performance counters. This flag is mandatory.

  • --with-spectral=<PATH>

    If the on-line analysis module is enabled (see --enable-online), specify where to find Spectral libraries and includes. This package enables support for on-line spectral analysis.

  • --with-unwind=<PATH>

    Specify where to find Unwind libraries and includes. This library is used to get callstack information on several architectures (including IA64 and Intel x86-64). This flag is mandatory.

3.2. Build

To build the instrumentation package, just issue make after the configuration.

3.3. Check

The Extrae package contains some consistency checks. The aim of such checks is to determine whether a functionality is operative in the target environment and/or check whether the development of Extrae has introduced any misbehavior. To run the checks, just issue make check after the compilation. Please, notice that checks are meant to be run in the machine that the configure script was run, thus the results of the checks on machines with back-end nodes different to front-end nodes (like BG/* systems) are not representative at all.

3.4. Installation

To install the instrumentation package in the directory chosen at configure step (through --prefix option), issue make install.

3.5. Examples of configuration on different machines

All commands given here are given as an example to configure and install the package, you may need to tune them properly (i.e., choose the appropriate directories for packages and so). These examples assume that you are using a sh/bash shell, you must adequate them if you use other shells (like csh/tcsh).

3.5.1. Cray XC 40 - Extrae 3.5.2

Before issuing the configure command, the following modules were loaded:

  • PrgEnv-gnu/5.2.40
  • cray-mpich/7.2.2
  • cudatoolkit6.5/6.5.14-1.0502.9613.6.1
  • libunwind/1.1-CrayGNU-5.2.4

Configuration command:

./configure --prefix=${PREFIX}/extrae/3.5.2
            --with-papi=/opt/cray/papi/5.4.1.1
            --with-mpi=/opt/cray/mpt/7.2.2/gni/mpich2-gnu/48
            --with-unwind=/apps/daint/5.2.UP02/easybuild/software/libunwind/1.1-CrayGNU-5.2.40
            --with-cuda=/opt/nvidia/cudatoolkit6.5/6.5.14-1.0502.9613.6.1
            --enable-sampling
            --without-dyninst
            --with-binary-type=64
            CC=gcc CXX=g++ MPICC=cc

Build and installation commands:

make
make install

3.5.2. Bluegene (L and P variants)

Configuration command:

./configure --prefix=/homec/jzam11/jzam1128/aplic/extrae/3.5.2
            --with-papi=/homec/jzam11/jzam1128/aplic/papi/4.1.2.1
            --with-bfd=/bgsys/local/gcc/gnu-linux_4.3.2/powerpc-linux-gnu/powerpc-bgp-linux
            --with-liberty=/bgsys/local/gcc/gnu-linux_4.3.2/powerpc-bgp-linux
            --with-mpi=/bgsys/drivers/ppcfloor/comm
            --without-unwind
            --without-dyninst

Build and installation commands:

make
make install

3.5.3. BlueGene/Q

To enable parsing the XML configuration file, the libxml2 must be installed. As of the time of writing this user guide, we have been only able to install the static version of the library in a BG/Q machine, so take this into consideration if you install the libxml2 in the system. Similarly, the binutils package (responsible for translating application addresses into source code locations) that is available in the system may not be properly installed and we suggest installing the binutils from the source code using the BG/Q cross-compiler. Regarding the cross-compilers, we have found that using the IBM XL compilers may require using the XL libraries when generating the final application binary with Extrae, so we would suggest using the GNU cross-compilers (/bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-*).

If you want to add libxml2 and binutils support into Extrae, your configuration command may resemble to:

./configure --prefix=/homec/jzam11/jzam1128/aplic/juqueen/extrae/3.5.2
            --with-mpi=/bgsys/drivers/ppcfloor/comm/gcc
            --without-unwind
            --without-dyninst
            --disable-openmp
            --disable-pthread
            --with-libz=/bgsys/local/zlib/v1.2.5
            --with-papi=/usr/local/UNITE/packages/papi/5.0.1
            --with-xml-prefix=/homec/jzam11/jzam1128/aplic/juqueen/libxml2-gcc
            --with-binutils=/homec/jzam11/jzam1128/aplic/juqueen/binutils-gcc
            --enable-merge-in-trace

Otherwise, if you do not want to add support for the libxml2 library, your configuration may look like this:

./configure --prefix=/homec/jzam11/jzam1128/aplic/juqueen/extrae/3.5.2
            --with-mpi=/bgsys/drivers/ppcfloor/comm/gcc
            --without-unwind
            --without-dyninst
            --disable-openmp
            --disable-pthread
            --with-libz=/bgsys/local/zlib/v1.2.5
            --with-papi=/usr/local/UNITE/packages/papi/5.0.1
            --disable-xml

In any situation, the build and installation commands are:

make
make install

3.5.4. AIX

Some extensions of Extrae do not work properly (nanos, SMPss and OpenMP) on AIX. In addition, if using IBM MPI (aka POE) the make will complain when generating the parallel merge if the main compiler is not xlc/xlC. So, you can either change the compiler or disable the parallel merge at compile step. Also, command ar can complain if 64bit binaries are generated. It’s a good idea to run make with OBJECT_MODE=64 set to avoid this.

3.5.4.1. Compiling the 32bit package using the IBM compilers

Configuration command:

CC=xlc
CXX=xlC
./configure --prefix=${PREFIX}/extrae/3.5.2
            --disable-nanos
            --disable-smpss
            --disable-openmp
            --with-binary-type=32
            --without-unwind
            --enable-pmapi
            --without-dyninst
            --with-mpi=/usr/lpp/ppe.poe

Build and installation commands:

make
make install

3.5.4.2. Compiling the 64bit package without the parallel merge

Configuration command:

./configure --prefix=${PREFIX}/extrae/3.5.2
            --disable-nanos
            --disable-smpss
            --disable-openmp
            --disable-parallel-merge
            --with-binary-type=64
            --without-unwind
            --enable-pmapi
            --without-dyninst
            --with-mpi=/usr/lpp/ppe.poe

Build and installation commands:

OBJECT_MODE=64 make
make install

3.5.5. Linux

3.5.5.1. Compiling using default binary type using MPICH, OpenMP and PAPI

Configuration command:

./configure --prefix=${PREFIX}/extrae/3.5.2
            --with-mpi=/home/harald/aplic/mpich/1.2.7
            --with-papi=/usr/local/papi
            --enable-openmp
            --without-dyninst
            --without-unwind

Build and installation commands:

make
make install

3.5.5.2. Compiling 32bit package in a 32/64bit mixed environment

Configuration command:

./configure --prefix=${PREFIX}/extrae/3.5.2
            --with-mpi=/opt/osshpc/mpich-mx
            --with-papi=/gpfs/apps/PAPI/3.6.2-970mp
            --with-binary-type=32
            --with-unwind=${HOME}/aplic/unwind/1.0.1/32
            --with-elf=/usr
            --with-dwarf=/usr
            --with-dyninst=${HOME}/aplic/dyninst/7.0.1/32

Build and installation commands:

make
make install

3.5.5.3. Compiling 64bit package in a 32/64bit mixed environment

Configuration command:

./configure --prefix=${PREFIX}/extrae/3.5.2
            --with-mpi=/opt/osshpc/mpich-mx
            --with-papi=/gpfs/apps/PAPI/3.6.2-970mp
            --with-binary-type=64
            --with-unwind=${HOME}/aplic/unwind/1.0.1/64
            --with-elf=/usr
            --with-dwarf=/usr
            --with-dyninst=${HOME}/aplic/dyninst/7.0.1/64

Build and installation commands:

make
make install

3.5.5.4. Compiling using default binary type, using OpenMPI, DynInst and libunwind

Configuration command:

./configure --prefix=${PREFIX}/extrae/3.5.2
            --with-mpi=/home/harald/aplic/openmpi/1.3.1
            --with-dyninst=/home/harald/dyninst/7.0.1
            --with-dwarf=/usr
            --with-elf=/usr
            --with-unwind=/home/harald/aplic/unwind/1.0.1
            --without-papi

Build and installation commands:

make
make install

3.5.5.5. Compiling on CRAY XT5 for 64bit package and adding sampling

Notice the --disable-xmltest. As backends programs cannot be run in the frontend, we skip running the XML test. Also using a local installation of libunwind.

Configuration command:

CC=cc
CFLAGS='-O3 -g'
LDFLAGS='-O3 -g'
CXX=CC CXXFLAGS='-O3 -g'
./configure --prefix=${PREFIX}/extrae/3.5.2
            --with-mpi=/opt/cray/mpt/4.0.0/xt/seastar/mpich2-gnu
            --with-binary-type=64
            --with-xml-prefix=/sw/xt5/libxml2/2.7.6/sles10.1_gnu4.1.2
            --disable-xmltest
            --with-bfd=/opt/cray/cce/7.1.5/cray-binutils
            --with-liberty=/opt/cray/cce/7.1.5/cray-binutils
            --enable-sampling
            --enable-shared=no
            --with-papi=/opt/xt-tools/papi/3.7.2/v23
            --with-unwind=/ccs/home/user/lib
            --without-dyninst

Build and installation commands:

make
make install

3.5.5.6. Compiling for the Intel MIC accelerator / Xeon Phi

The Intel MIC accelerators (also codenamed KnightsFerry - KNF and KnightsCorner - KNC) or Xeon Phi processors are not binary compatible with the host (even if it is an Intel x86 or x86/64 chip), thus the Extrae package must be compiled specially for the accelerator (twice if you want Extrae for the host). While the host configuration and installation has been shown before, in order to compile Extrae for the accelerator you must configure Extrae like:

Configuration command:

./configure --prefix=/home/Computational/harald/extrae-mic
            --with-mpi=/opt/intel/impi/4.1.0.024/mic
            --without-dyninst
            --without-papi
            --without-unwind
            --disable-xml
            --disable-posix-clock
            --with-libz=/opt/extrae/zlib-mic
            --host=x86_64-suse-linux-gnu
            --enable-mic
            CFLAGS="-O -mmic -I/usr/include"
            CC=icc
            CXX=icpc
            MPICC=/opt/intel/impi/4.1.0.024/mic/bin/mpiicc

To compile it, just issue:

make
make install

3.5.5.7. Compiling on a ARM based processor machine using Linux

If using the GNU toolchain to compile the library, we suggest at least using version 4.6.2 because of its enhaced in this architecture.

Configuration command:

CC=/gpfs/APPS/BIN/GCC-4.6.2/bin/gcc-4.6.2
./configure --prefix=/gpfs/BSCTOOLS/extrae/3.5.2
            --with-unwind=/gpfs/BSCTOOLS/libunwind/1.0.1-git
            --with-papi=/gpfs/BSCTOOLS/papi/4.2.0
            --with-mpi=/usr
            --enable-posix-clock
            --without-dyninst

Build and installation commands:

make
make install

3.5.5.8. Compiling in a Slurm/MOAB environment with support for MPICH2

Configuration command:

export MP_IMPL=anl2
./configure --prefix=${PREFIX}/extrae/3.5.2
            --with-mpi=/gpfs/apps/MPICH2/mx/1.0.8p1..3/32
            --with-papi=/gpfs/apps/PAPI/3.6.2-970mp
            --with-binary-type=64
            --without-dyninst
            --without-unwind

Build and installation commands:

make
make install

3.5.5.9. Compiling in a environment with IBM compilers and POE

Configuration command:

CC=xlc
CXX=xlC
./configure --prefix=${PREFIX}/extrae/3.5.2
            --with-mpi=/opt/ibmhpc/ppe.poe
            --without-dyninst
            --without-unwind
            --without-papi

Build and installation commands:

make
make install

3.5.5.10. Compiling in a environment with GNU compilers and POE

Configuration command:

./configure --prefix=${PREFIX}/extrae/3.5.2
            --with-mpi=/opt/ibmhpc/ppe.poe
            --without-dyninst
            --without--unwind
            --without-papi

Build and installation commands:

MP_COMPILER=gcc make
make install

3.5.5.11. Compiling Extrae 3.5.2 in Hornet / Cray XC40 system

Configuration command, enabling MPI, PAPI and online analysis over MRNet.

./configure --prefix=/zhome/academic/HLRS/xhp/xhpgl/tools/extrae/3.5.2/intel
            --with-mpi=/opt/cray/mpt/7.1.2/gni/mpich2-intel/140
            --with-unwind=/zhome/academic/HLRS/xhp/xhpgl/tools/libunwind
            --without-dyninst
            --with-papi=/opt/cray/papi/5.3.2.1
            --enable-online
            --with-mrnet=/zhome/academic/HLRS/xhp/xhpgl/tools/mrnet/4.1.0
            --with-spectral=/zhome/academic/HLRS/xhp/xhpgl/tools/spectral/3.1
            --with-synapse=/zhome/academic/HLRS/xhp/xhpgl/tools/synapse/2.0

Build and installation commands:

make
make install

3.5.5.12. Compiling Extrae 3.5.2 in Shaheen II / Cray XC40 system

With the following modules loaded:

module swap PrgEnv-XXX/YYY PrgEnv-cray/5.2.40
module load cray-mpich

Configuration command, enabling MPI, PAPI:

./configure --prefix=${PREFIX}/extrae/3.5.2
            --with-mpi=/opt/cray/mpt/7.1.1/gni/mpich2-cray/83
            --with-binary-type=64
            --with-unwind=/home/markomg/lib
            --without-dyninst
            --disable-xmltest
            --with-bfd=/opt/cray/cce/default/cray-binutils
            --with-liberty=/opt/cray/cce/default/cray-binutils
            --enable-sampling
            --enable-shared=no
            --with-papi=/opt/cray/papi/5.3.2.1

Build and installation commands:

make
make install

3.5.6. Knowing how a package was configured

If you are interested on knowing how an Extrae package was configured execute the following command after setting EXTRAE_HOME to the base location of an installation.

${EXTRAE_HOME}/etc/configured.sh

this command will show the configure command itself and the location of some dependencies of the instrumentation package.