CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Programming & Development

MPI profiling

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   June 15, 2013, 17:45
Default MPI profiling
  #1
New Member
 
Gera Barnabás
Join Date: Jun 2013
Posts: 4
Rep Power: 4
GeraBarna is on a distinguished road
Hello everyone!

I am a student relatively new to OpenFOAM writing my thesis, and for that I need to somehow determine the load-balancing of processors while running parallel solvers in OpenFOAM 2.2. My first idea was to use some MPI profiling tool to measure the time each separate process spends either calculating or communicating, but I cannot get any tool running.

Does anybody have some other idea on how to get some measurement of the load balancing on the processors while running solvers, or some idea on what combinations of mpi implementations and profiling tools might work?

So far I have tried to use: valgrind, fpmpi, Tau, MPE, ipm, mpip with OpenMPI 1.6.3 and MPICH 1.1.1p1 and 1.4.1.1p1.
Valgrind does work, but because of the huge overhead it introduces on the calculations it simply is not feasible.

My problem with fpmpi and Tau is that the MPI versions coming with openFoam ( OpenMPI and MPICH ) disable the mpi profiling interfaces and if I re-enable them the the configure scripts break, thus I cannot use them.
MPE: I canot link the libraries, because I get a linker error:

/usr/bin/ld: /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/liblmpe.a(log_mpi_core.o): relocation R_X86_64_32S against `.bss' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/liblmpe.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status

recompiling MPE and the Pstream library with -fPIC didn't solve the problem.

ipm, mpip: the configure script breaks.
I have also tried using googlePerformance tools but they generate segmentation faults at runtime.

My current idea is to use the linux time command to get a measurement of the CPU time from each process, or to put some time measurements into the PStream library, but I don't know how accurate these can be, thus I would like to have a more precise method.

Any new ideas are much appreciated
GeraBarna is offline   Reply With Quote

Old   June 17, 2013, 03:17
Default
  #2
Senior Member
 
akidess's Avatar
 
Anton Kidess
Join Date: May 2009
Location: Delft, Netherlands
Posts: 919
Rep Power: 17
akidess will become famous soon enough
Did you read this: https://www.hpc.ntnu.no/display/hpc/...filing+Solvers
__________________
*On twitter @akidTwit
*Spend as much time formulating your questions as you expect people to spend on their answer.
*Help define the OpenFOAM stackexchange Q&A site: http://area51.stackexchange.com/prop...oam-technology
akidess is offline   Reply With Quote

Old   June 17, 2013, 06:59
Default
  #3
New Member
 
Gera Barnabás
Join Date: Jun 2013
Posts: 4
Rep Power: 4
GeraBarna is on a distinguished road
Thank you for your answer!
Unfortunately, I don't even get that far with ipm, because the configure script breaks, so I cannot compile and install it.

Here is my attempt: ( the script breaks at the same point without the extra options too)
Code:
> ./configure --with-compiler=GNU --with-os=LINUX --with-cpu=XEON 2>&1 | tee log.configure
configuring for unknown w/ OS=LINUX ARCH= COMPILER=GNU CPU=XEON
checking whether the C compiler ( gcc ) works...yes
checking whether the C++ compiler ( g++ ) works...yes
determining how to compile HPM (PAPI) C code ...no
determining how to compile HPM (PMAPI) C code ...no
determining how to compile HPM (DISABLED) C code ...yes
determining how to link HPM (DISABLED) code ( -lpapi)...no
determining how to link HPM (DISABLED) code ( -lpapi -lperfctr)...no
determining how to link HPM (DISABLED) code ( -lpapi -lpfm)...no
determining how to link HPM (DISABLED) code (-lpapi)...no
determining how to link HPM (DISABLED) code (-lpapi -lperfctr)...no
determining how to link HPM (DISABLED) code (-lpmapi)...no
determining how to link HPM (DISABLED) code (64 -lpapi -lperfctr)...no
determining how to link HPM (DISABLED) code (64 -lpapi)...no
determining how to link HPM (DISABLED) code (64 -lpapi -lpfm)...no
*** HPM link files not found, HPM disabled ***
Generating ../include/ipm_calls.h
Generating ../include/ipm_fproto.h
Generating libtiny.c
Generating libipm.c
Generating libipm_io.c
checking for working IPM macros in ipm.h...yes
checking for working gettimeofday...yes
checking for working IPM serial timer macro (RTC) in ipm.h...no
checking for working IPM serial timer macro (READREALTIME) in ipm.h...no
checking for working IPM serial timer macro (RTS) in ipm.h...no
checking for working IPM serial timer macro (ITC) in ipm.h...no
checking for working IPM serial timer macro (RDTSC) in ipm.h...yes
checking for working MPI C compiler (cc) ...no
checking for working MPI C compiler (cc -lmpi) ...no
checking for working MPI C compiler (mpcc) ...no
checking for working MPI C compiler (mpicc) ...yes
checking whether the MPI C compiler (mpicc  -DIPM_DISABLE_EXECINFO -I/home/barna/ipm/include     -L/home/barna/ipm/lib   ) works with IPM macros...yes
checking MPI_STATUS_COUNT (val1)...no
checking MPI_STATUS_COUNT (count)...no
checking MPI_STATUS_COUNT (_count)...no
checking MPI_STATUS_COUNT (size)...no
unknown MPI_STATUS_COUNT

Last edited by GeraBarna; June 17, 2013 at 07:25. Reason: (forgot [CODE] tags)
GeraBarna is offline   Reply With Quote

Old   June 17, 2013, 07:27
Default
  #4
Senior Member
 
Robert Sawko
Join Date: Mar 2009
Posts: 116
Rep Power: 13
AlmostSurelyRob will become famous soon enough
Barna,

you need to look up the configure file. Apparently the code for MPI_STATUS_COUNT doesn't compile. I made it to work by replacing this bit of code
Code:
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char *argv[]) {
MPI_Status s;
s.$tag = 0;
return
with this

Code:
cat >> ./$TEST_NAME.c <<EOF
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char *argv[]) {
MPI_Status s;
s.MPI_TAG = 0;
return
It's on the line 1252 in my configure.

Now the configure works, but make breaks. Let me know how it works for you.
AlmostSurelyRob is offline   Reply With Quote

Old   June 17, 2013, 07:49
Default
  #5
Senior Member
 
Robert Sawko
Join Date: Mar 2009
Posts: 116
Rep Power: 13
AlmostSurelyRob will become famous soon enough
I compiled ipm on blackbird if you're interested. There's something strange about OpenMPI and IPM. The code of IPM assumed that MPI_Status structure should contain completely different fields than OpenMPI offers. I consulted mpi.h to see it and after a bit of "fun" I replaced

Code:
#define MPI_STATUS_COUNT size
Code:
#define MPI_STATUS_COUNT _ucount
in ipm/include/config.h.

Let me know if this works at all. From the comments it seems that these are internal bits of OpenMPI and shouldn't be touched. I compiled sucessfuly, but I have no idea if it works as intended.
AlmostSurelyRob is offline   Reply With Quote

Old   June 17, 2013, 08:53
Default
  #6
New Member
 
Gera Barnabás
Join Date: Jun 2013
Posts: 4
Rep Power: 4
GeraBarna is on a distinguished road
Thank you so much!
I have managed to compile and run a solver linked to ipm with these workarounds. I will now try to do some actual profiling, and see if it works properly.
GeraBarna is offline   Reply With Quote

Old   June 20, 2013, 09:34
Default
  #7
New Member
 
Gera Barnabás
Join Date: Jun 2013
Posts: 4
Rep Power: 4
GeraBarna is on a distinguished road
Thank you for the tipps for IPM compiling. Sadly, even though IPM did compile, I couldn't use it. But since then I have figured out how to fix the Profiling interface in the openMPI version that comes with OpenFOAM 2.2

As it turns out, even though it is specified on the openMPI website that it only uses cuda if one explicitly enables it with the -with-cuda= option, it does use it to compile vampire trace. At cuda 5 they changed the interface, thus vampire trace no longer compiles. Thus the solution to enable the profiling interface was to replace the:
--disable-mpi-profile
with:
--disable-vt
also, I had to remove the -j $WM_NCOMPPROCS
from line 109 : make -j $WM_NCOMPPROCS && make install

After this IVP still didn't work, but I could manage to get TAU to work with a reasonable amount of effort.
GeraBarna is offline   Reply With Quote

Old   July 9, 2014, 23:30
Default
  #8
New Member
 
sethu
Join Date: Oct 2011
Posts: 5
Rep Power: 5
Sethu is on a distinguished road
Dear Barnabas,

Could you please tell me the procedure that you have used to implement "tau" or "ipm" in your system and to profile openfoam with it.

Im using OF 22x in centOS 6.4 cluster...

Thanks in advance,
Sethu is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
mpirun, best parameters pablodecastillo Hardware 17 April 27, 2012 13:05
Error using LaunderGibsonRSTM on SGI ALTIX 4700 jaswi OpenFOAM 2 April 29, 2008 10:54
Is Testsuite on the way or not lakeat OpenFOAM Installation 6 April 28, 2008 11:12
MPI profiling OpenFOAM damBreak3D application mellanoxuser OpenFOAM Pre-Processing 0 April 13, 2008 23:15
MPI profiling OpenFOAM damBreak3D application mellanoxuser OpenFOAM Running, Solving & CFD 0 April 13, 2008 23:04


All times are GMT -4. The time now is 16:52.