[GeraBarna]

Hello everyone!

I am a student relatively new to OpenFOAM writing my thesis, and for that I need to somehow determine the load-balancing of processors while running parallel solvers in OpenFOAM 2.2. My first idea was to use some MPI profiling tool to measure the time each separate process spends either calculating or communicating, but I cannot get any tool running.

Does anybody have some other idea on how to get some measurement of the load balancing on the processors while running solvers, or some idea on what combinations of mpi implementations and profiling tools might work?

So far I have tried to use: valgrind, fpmpi, Tau, MPE, ipm, mpip with OpenMPI 1.6.3 and MPICH 1.1.1p1 and 1.4.1.1p1.
Valgrind does work, but because of the huge overhead it introduces on the calculations it simply is not feasible.

My problem with fpmpi and Tau is that the MPI versions coming with openFoam ( OpenMPI and MPICH ) disable the mpi profiling interfaces and if I re-enable them the the configure scripts break, thus I cannot use them.
MPE: I canot link the libraries, because I get a linker error:

/usr/bin/ld: /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/liblmpe.a(log_mpi_core.o): relocation R_X86_64_32S against `.bss' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/liblmpe.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status

recompiling MPE and the Pstream library with -fPIC didn't solve the problem.

ipm, mpip: the configure script breaks.
I have also tried using googlePerformance tools but they generate segmentation faults at runtime.

My current idea is to use the linux time command to get a measurement of the CPU time from each process, or to put some time measurements into the PStream library, but I don't know how accurate these can be, thus I would like to have a more precise method.

Any new ideas are much appreciated

[akidess]

Did you read this: https://www.hpc.ntnu.no/display/hpc/...filing+Solvers

[GeraBarna]

Thank you for your answer!
Unfortunately, I don't even get that far with ipm, because the configure script breaks, so I cannot compile and install it.

Here is my attempt: ( the script breaks at the same point without the extra options too)

> ./configure --with-compiler=GNU --with-os=LINUX --with-cpu=XEON 2>&1 | tee log.configure
configuring for unknown w/ OS=LINUX ARCH= COMPILER=GNU CPU=XEON
checking whether the C compiler ( gcc ) works...yes
checking whether the C++ compiler ( g++ ) works...yes
determining how to compile HPM (PAPI) C code ...no
determining how to compile HPM (PMAPI) C code ...no
determining how to compile HPM (DISABLED) C code ...yes
determining how to link HPM (DISABLED) code ( -lpapi)...no
determining how to link HPM (DISABLED) code ( -lpapi -lperfctr)...no
determining how to link HPM (DISABLED) code ( -lpapi -lpfm)...no
determining how to link HPM (DISABLED) code (-lpapi)...no
determining how to link HPM (DISABLED) code (-lpapi -lperfctr)...no
determining how to link HPM (DISABLED) code (-lpmapi)...no
determining how to link HPM (DISABLED) code (64 -lpapi -lperfctr)...no
determining how to link HPM (DISABLED) code (64 -lpapi)...no
determining how to link HPM (DISABLED) code (64 -lpapi -lpfm)...no
*** HPM link files not found, HPM disabled ***
Generating ../include/ipm_calls.h
Generating ../include/ipm_fproto.h
Generating libtiny.c
Generating libipm.c
Generating libipm_io.c
checking for working IPM macros in ipm.h...yes
checking for working gettimeofday...yes
checking for working IPM serial timer macro (RTC) in ipm.h...no
checking for working IPM serial timer macro (READREALTIME) in ipm.h...no
checking for working IPM serial timer macro (RTS) in ipm.h...no
checking for working IPM serial timer macro (ITC) in ipm.h...no
checking for working IPM serial timer macro (RDTSC) in ipm.h...yes
checking for working MPI C compiler (cc) ...no
checking for working MPI C compiler (cc -lmpi) ...no
checking for working MPI C compiler (mpcc) ...no
checking for working MPI C compiler (mpicc) ...yes
checking whether the MPI C compiler (mpicc  -DIPM_DISABLE_EXECINFO -I/home/barna/ipm/include     -L/home/barna/ipm/lib   ) works with IPM macros...yes
checking MPI_STATUS_COUNT (val1)...no
checking MPI_STATUS_COUNT (count)...no
checking MPI_STATUS_COUNT (_count)...no
checking MPI_STATUS_COUNT (size)...no
unknown MPI_STATUS_COUNT

[AlmostSurelyRob]

Barna,

you need to look up the configure file. Apparently the code for MPI_STATUS_COUNT doesn't compile. I made it to work by replacing this bit of code

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char *argv[]) {
MPI_Status s;
s.$tag = 0;
return

with this

cat >> ./$TEST_NAME.c <<EOF
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char *argv[]) {
MPI_Status s;
s.MPI_TAG = 0;
return

It's on the line 1252 in my configure.

Now the configure works, but make breaks. Let me know how it works for you.

[AlmostSurelyRob]

I compiled ipm on blackbird if you're interested. There's something strange about OpenMPI and IPM. The code of IPM assumed that MPI_Status structure should contain completely different fields than OpenMPI offers. I consulted mpi.h to see it and after a bit of "fun" I replaced

#define MPI_STATUS_COUNT size

#define MPI_STATUS_COUNT _ucount

in ipm/include/config.h.

Let me know if this works at all. From the comments it seems that these are internal bits of OpenMPI and shouldn't be touched. I compiled sucessfuly, but I have no idea if it works as intended.

[GeraBarna]

Thank you so much!
I have managed to compile and run a solver linked to ipm with these workarounds. I will now try to do some actual profiling, and see if it works properly.

[GeraBarna]

Thank you for the tipps for IPM compiling. Sadly, even though IPM did compile, I couldn't use it. But since then I have figured out how to fix the Profiling interface in the openMPI version that comes with OpenFOAM 2.2

As it turns out, even though it is specified on the openMPI website that it only uses cuda if one explicitly enables it with the -with-cuda= option, it does use it to compile vampire trace. At cuda 5 they changed the interface, thus vampire trace no longer compiles. Thus the solution to enable the profiling interface was to replace the:
--disable-mpi-profile
with:
--disable-vt
also, I had to remove the -j $WM_NCOMPPROCS
from line 109 : make -j $WM_NCOMPPROCS && make install

After this IVP still didn't work, but I could manage to get TAU to work with a reasonable amount of effort.

[Sethu]

Dear Barnabas,

Could you please tell me the procedure that you have used to implement "tau" or "ipm" in your system and to profile openfoam with it.

Im using OF 22x in centOS 6.4 cluster...

Thanks in advance,

[rishi007bansod]

I am getting following error while profiling using IPM

rishikesh@rishikesh-desktop:~/ipm$ make install
cd src; make ipm
make[1]: Entering directory `/home/rishikesh/ipm/src'
/home/rishikesh/ipm/bin/make_wrappers -funderscore_post ../ipm_key
Generating ../include/ipm_calls.h
Generating ../include/ipm_fproto.h
Generating libtiny.c
Generating libipm.c
Generating libipm_io.c
cc -DIPM_DISABLE_EXECINFO -I/home/rishikesh/ipm/include -DWRAP_FORTRAN -I../include -o ../bin/ipm ipm.c
In file included from ipm.c:114:0:
ipm_init.c: In function ‘ipm_init’:
ipm_init.c:39:2: error: ‘region_wtime_init’ undeclared (first use in this function)
region_wtime_init = task.ipm_trc_time_init;
^
ipm_init.c:39:2: note: each undeclared identifier is reported only once for each function it appears in
make[1]: *** [ipm] Error 1
make[1]: Leaving directory `/home/rishikesh/ipm/src'
make: *** [ipm] Error 2

what is cause of this error?

[banji]

Hi ,

I am trying to analyze the performance of a parallel program using IPM. I have successfully installed the IPM software following the guidelines described at https://github.com/nerscadmin/ipm and followed the explanation at https://www.hpc.ntnu.no/display/hpc/...filing+Solvers to setup a custom solver.

The problem is that when I run the program, for example

mpirun -np 4 pisoFoamIPM -parallel

the program just runs without any time analysis report. Am i doing something wrong or how do I get a result printed?

Thanks.

[WhiteW]

Hi banji. To write the output in a file you have to run:

mpirun -np 4 pisoFoamIPM -parallel >logfile.txt

[ Moderator note: This post was copied-modified from here: http://www.cfd-online.com/Forums/ope...tml#post598940 ]

[banji]

Hellow WhiteW,

Thanks for your reply.

Actually, I know how to redirect messages to an output file. The problem is I can't find any related to IPM i.e. maybe how much time is spent on MPI_wait, MPI_recv , e.t.c.

[ Moderator note: Moved from here: http://www.cfd-online.com/Forums/ope...s-cluster.html ]

[wyldckat]

Originally Posted by banji

the program just runs without any time analysis report. Am i doing something wrong or how do I get a result printed?

Quick answer: I'm guessing that this explains how to process the information: http://ipm-hpc.sourceforge.net/userguide.html - read section "Post-processing IPM output".