14. Frequently Asked Questions

14.2. Execution FAQ

Question:

I executed my application instrumenting with Extrae, even though it appears that Extrae is not intrumenting anything. There is neither any Extrae message nor any Extrae output files (file:set-X/*.mpit).
Answer 1:
Check that environment variables are correctly passed to the application.
Answer 2:
If the code is Fortran, check that the number of underscores used to decorate routines in the instrumentation library matches the number of underscores added by the Fortran compiler you used to compile and link the application. You can use the nm and grep commands to check it.
Answer 3:
If the code is MPI and Fortran, check that you’re using the proper Fortran library for the instrumentation.
Answer 4:
If the code is MPI and you are using LD_PRELOAD, check that the binary is linked against a shared MPI library (you can use the ldd command).

Question:

Why are the environment variables not exported?
Answer:
MPI applications are launched using special programs (like mpirun, poe, mprun, srun, …) that spawn the application for the selected resources. Some of these programs do not export all the environment variables to the spawned processes. Check if the the launching program does have special parameters to do that, or use the approach used on section Examples based on launching scripts instead of MPI applications.

Question:

The instrumentation begins for a single process instead for several processes.
Answer 1:
Check that you place the appropriate parameter to indicate the number of tasks (typically -np).
Answer 2:

Some MPI implementation require the application to receive special MPI parameters to run correctly. For example, MPICH based on CH-P4 device require the binary to receive som paramters. The following example is an sh-script that solves this issue:

#!/bin/sh
EXTRAE_CONFIG_FILE=extrae.xml ./mpi_program $@ real_params

Question:

The application blocks at the beginning.
Answer:
The application may be waiting for all tasks to startup but only some of them are running. Check for the previous question.

Question:

The resulting traces do not contain the routines that have been instrumented.
Answer 1:
Check that the routines have been actually executed.
Answer 2:
Some compilers do automatic inlining of functions at some optimization levels (e.g., Intel Compiler at -O2). When functions are inlined, they do not have entry and exit blocks and cannot be instrumented. Turn off inlining or decrease the optimization level.

Question:

Number of threads = 1?
Answer:
Some MPI launchers (i.e., mpirun, poe, mprun, …)

Question:

When running the instrumented application, the loader complains about: undefined symbol: clock_gettime
Answer:
The instrumentation package was configured using --enable-posix-clock` and on many systems this implies the inclusion of additional libraries (namely, -lrt)

14.3. Performance monitoring counters FAQ

Question:

How do I know the available performance counters on the system?
Answer 1:
If using PAPI, check the papi_avail or papi_native_avail commands found in the PAPI installation directory.
Answer 2:
If using PMAPI (on AIX systems), check for the pmlist command. Specifically, check for the available groups running pmlist -g -1.

Question:

How do I know how many performance counters can I use?
Answer:
The Extrae package can gather up to eight (8) performance counters at the same time. This also depends on the underlying library used to gather them.

Question:

When using PAPI, I cannot read eight performance counters or the specified in papi_avail output.
Answer 1:
There are some performance counters (those listed in papi_avail) that are classified as derived. Such performance counters depend on more than one counter increasing the number of real performance counters used. Check for the derived column within the list to check whether a performance counter is derived or not.
Answer 2:
On some architectures, like the PowerPC, the performance counters are grouped in a such way that choosing a performance counter precludes others from being elected in the same set. A feasible work-around is to create as many sets in the XML file to gather all the required hardware counters and make sure that the sets change from time to time.

14.4. Merging traces FAQ

Question:

The mpi2prv command shows the following messages at the start-up:

PANIC! Trace file TRACE.0000011148000001000000.mpit is 16 bytes too big! PANIC! Trace file TRACE.0000011147000002000000.mpit is 32 bytes too big! PANIC! Trace file TRACE.0000011146000003000000.mpit is 16 bytes too big!

and it dies when parsing the intermediate files.

Answer 1:
The aforementioned messages are typically related with incomplete writes in disk. Check for enough disk space using the quota and df commands.
Answer 2:
If your system supports multiple ABIs (for example, linux x86-64 supports 32 and 64 bits ABIs), check that the ABI of the target application and the ABI of the merger match.

Question:

The resulting Paraver tracefile contains invalid references to the source code.
Answer:
This usually happens when the code has not been compiled and linked with the -g flag. If your code invokes CUDA kernels you may also need to link with the --generate-line-info flag to generate debug symbols for device code. Moreover, some high level optimizations (which includes inlining, interprocedural analysis, and so on) can lead to generate bad references.

Question:

The resulting trace contains information regarding the stack (like callers) but their value does not coincide with the source code.
Answer:
Check that the same binary is used to generate the trace and referenced with the the -e parameter when generating the Paraver tracefile.