CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Programming & Development

MPI profiling

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   June 15, 2013, 17:45
Default MPI profiling
  #1
New Member
 
Gera Barnabás
Join Date: Jun 2013
Posts: 4
Rep Power: 5
GeraBarna is on a distinguished road
Hello everyone!

I am a student relatively new to OpenFOAM writing my thesis, and for that I need to somehow determine the load-balancing of processors while running parallel solvers in OpenFOAM 2.2. My first idea was to use some MPI profiling tool to measure the time each separate process spends either calculating or communicating, but I cannot get any tool running.

Does anybody have some other idea on how to get some measurement of the load balancing on the processors while running solvers, or some idea on what combinations of mpi implementations and profiling tools might work?

So far I have tried to use: valgrind, fpmpi, Tau, MPE, ipm, mpip with OpenMPI 1.6.3 and MPICH 1.1.1p1 and 1.4.1.1p1.
Valgrind does work, but because of the huge overhead it introduces on the calculations it simply is not feasible.

My problem with fpmpi and Tau is that the MPI versions coming with openFoam ( OpenMPI and MPICH ) disable the mpi profiling interfaces and if I re-enable them the the configure scripts break, thus I cannot use them.
MPE: I canot link the libraries, because I get a linker error:

/usr/bin/ld: /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/liblmpe.a(log_mpi_core.o): relocation R_X86_64_32S against `.bss' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/liblmpe.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status

recompiling MPE and the Pstream library with -fPIC didn't solve the problem.

ipm, mpip: the configure script breaks.
I have also tried using googlePerformance tools but they generate segmentation faults at runtime.

My current idea is to use the linux time command to get a measurement of the CPU time from each process, or to put some time measurements into the PStream library, but I don't know how accurate these can be, thus I would like to have a more precise method.

Any new ideas are much appreciated
GeraBarna is offline   Reply With Quote

Old   June 17, 2013, 03:17
Default
  #2
Senior Member
 
akidess's Avatar
 
Anton Kidess
Join Date: May 2009
Location: Delft, Netherlands
Posts: 1,133
Rep Power: 20
akidess will become famous soon enough
Did you read this: https://www.hpc.ntnu.no/display/hpc/...filing+Solvers
__________________
*On twitter @akidTwit
*Spend as much time formulating your questions as you expect people to spend on their answer.
*Join the OpenFOAM stackexchange Q&A site: http://area51.stackexchange.com/prop...oHPxcPqde7HtA2
akidess is offline   Reply With Quote

Old   June 17, 2013, 06:59
Default
  #3
New Member
 
Gera Barnabás
Join Date: Jun 2013
Posts: 4
Rep Power: 5
GeraBarna is on a distinguished road
Thank you for your answer!
Unfortunately, I don't even get that far with ipm, because the configure script breaks, so I cannot compile and install it.

Here is my attempt: ( the script breaks at the same point without the extra options too)
Code:
> ./configure --with-compiler=GNU --with-os=LINUX --with-cpu=XEON 2>&1 | tee log.configure
configuring for unknown w/ OS=LINUX ARCH= COMPILER=GNU CPU=XEON
checking whether the C compiler ( gcc ) works...yes
checking whether the C++ compiler ( g++ ) works...yes
determining how to compile HPM (PAPI) C code ...no
determining how to compile HPM (PMAPI) C code ...no
determining how to compile HPM (DISABLED) C code ...yes
determining how to link HPM (DISABLED) code ( -lpapi)...no
determining how to link HPM (DISABLED) code ( -lpapi -lperfctr)...no
determining how to link HPM (DISABLED) code ( -lpapi -lpfm)...no
determining how to link HPM (DISABLED) code (-lpapi)...no
determining how to link HPM (DISABLED) code (-lpapi -lperfctr)...no
determining how to link HPM (DISABLED) code (-lpmapi)...no
determining how to link HPM (DISABLED) code (64 -lpapi -lperfctr)...no
determining how to link HPM (DISABLED) code (64 -lpapi)...no
determining how to link HPM (DISABLED) code (64 -lpapi -lpfm)...no
*** HPM link files not found, HPM disabled ***
Generating ../include/ipm_calls.h
Generating ../include/ipm_fproto.h
Generating libtiny.c
Generating libipm.c
Generating libipm_io.c
checking for working IPM macros in ipm.h...yes
checking for working gettimeofday...yes
checking for working IPM serial timer macro (RTC) in ipm.h...no
checking for working IPM serial timer macro (READREALTIME) in ipm.h...no
checking for working IPM serial timer macro (RTS) in ipm.h...no
checking for working IPM serial timer macro (ITC) in ipm.h...no
checking for working IPM serial timer macro (RDTSC) in ipm.h...yes
checking for working MPI C compiler (cc) ...no
checking for working MPI C compiler (cc -lmpi) ...no
checking for working MPI C compiler (mpcc) ...no
checking for working MPI C compiler (mpicc) ...yes
checking whether the MPI C compiler (mpicc  -DIPM_DISABLE_EXECINFO -I/home/barna/ipm/include     -L/home/barna/ipm/lib   ) works with IPM macros...yes
checking MPI_STATUS_COUNT (val1)...no
checking MPI_STATUS_COUNT (count)...no
checking MPI_STATUS_COUNT (_count)...no
checking MPI_STATUS_COUNT (size)...no
unknown MPI_STATUS_COUNT

Last edited by GeraBarna; June 17, 2013 at 07:25. Reason: (forgot [CODE] tags)
GeraBarna is offline   Reply With Quote

Old   June 17, 2013, 07:27
Default
  #4
Senior Member
 
Robert Sawko
Join Date: Mar 2009
Posts: 117
Rep Power: 14
AlmostSurelyRob will become famous soon enough
Barna,

you need to look up the configure file. Apparently the code for MPI_STATUS_COUNT doesn't compile. I made it to work by replacing this bit of code
Code:
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char *argv[]) {
MPI_Status s;
s.$tag = 0;
return
with this

Code:
cat >> ./$TEST_NAME.c <<EOF
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char *argv[]) {
MPI_Status s;
s.MPI_TAG = 0;
return
It's on the line 1252 in my configure.

Now the configure works, but make breaks. Let me know how it works for you.
AlmostSurelyRob is offline   Reply With Quote

Old   June 17, 2013, 07:49
Default
  #5
Senior Member
 
Robert Sawko
Join Date: Mar 2009
Posts: 117
Rep Power: 14
AlmostSurelyRob will become famous soon enough
I compiled ipm on blackbird if you're interested. There's something strange about OpenMPI and IPM. The code of IPM assumed that MPI_Status structure should contain completely different fields than OpenMPI offers. I consulted mpi.h to see it and after a bit of "fun" I replaced

Code:
#define MPI_STATUS_COUNT size
Code:
#define MPI_STATUS_COUNT _ucount
in ipm/include/config.h.

Let me know if this works at all. From the comments it seems that these are internal bits of OpenMPI and shouldn't be touched. I compiled sucessfuly, but I have no idea if it works as intended.
AlmostSurelyRob is offline   Reply With Quote

Old   June 17, 2013, 08:53
Default
  #6
New Member
 
Gera Barnabás
Join Date: Jun 2013
Posts: 4
Rep Power: 5
GeraBarna is on a distinguished road
Thank you so much!
I have managed to compile and run a solver linked to ipm with these workarounds. I will now try to do some actual profiling, and see if it works properly.
GeraBarna is offline   Reply With Quote

Old   June 20, 2013, 09:34
Default
  #7
New Member
 
Gera Barnabás
Join Date: Jun 2013
Posts: 4
Rep Power: 5
GeraBarna is on a distinguished road
Thank you for the tipps for IPM compiling. Sadly, even though IPM did compile, I couldn't use it. But since then I have figured out how to fix the Profiling interface in the openMPI version that comes with OpenFOAM 2.2

As it turns out, even though it is specified on the openMPI website that it only uses cuda if one explicitly enables it with the -with-cuda= option, it does use it to compile vampire trace. At cuda 5 they changed the interface, thus vampire trace no longer compiles. Thus the solution to enable the profiling interface was to replace the:
--disable-mpi-profile
with:
--disable-vt
also, I had to remove the -j $WM_NCOMPPROCS
from line 109 : make -j $WM_NCOMPPROCS && make install

After this IVP still didn't work, but I could manage to get TAU to work with a reasonable amount of effort.
GeraBarna is offline   Reply With Quote

Old   July 9, 2014, 23:30
Default
  #8
New Member
 
sethu
Join Date: Oct 2011
Posts: 6
Rep Power: 6
Sethu is on a distinguished road
Dear Barnabas,

Could you please tell me the procedure that you have used to implement "tau" or "ipm" in your system and to profile openfoam with it.

Im using OF 22x in centOS 6.4 cluster...

Thanks in advance,
Sethu is offline   Reply With Quote

Old   February 16, 2016, 17:15
Default Getting error while installing IPM for profiling
  #9
New Member
 
Rishi
Join Date: Feb 2016
Posts: 2
Rep Power: 0
rishi007bansod is on a distinguished road
I am getting following error while profiling using IPM

rishikesh@rishikesh-desktop:~/ipm$ make install
cd src; make ipm
make[1]: Entering directory `/home/rishikesh/ipm/src'
/home/rishikesh/ipm/bin/make_wrappers -funderscore_post ../ipm_key
Generating ../include/ipm_calls.h
Generating ../include/ipm_fproto.h
Generating libtiny.c
Generating libipm.c
Generating libipm_io.c
cc -DIPM_DISABLE_EXECINFO -I/home/rishikesh/ipm/include -DWRAP_FORTRAN -I../include -o ../bin/ipm ipm.c
In file included from ipm.c:114:0:
ipm_init.c: In function ‘ipm_init’:
ipm_init.c:39:2: error: ‘region_wtime_init’ undeclared (first use in this function)
region_wtime_init = task.ipm_trc_time_init;
^
ipm_init.c:39:2: note: each undeclared identifier is reported only once for each function it appears in
make[1]: *** [ipm] Error 1
make[1]: Leaving directory `/home/rishikesh/ipm/src'
make: *** [ipm] Error 2


what is cause of this error?
rishi007bansod is offline   Reply With Quote

Old   May 5, 2016, 14:15
Default IPM (Integrated Performance Monitoring profiler) does not output any result file
  #10
New Member
 
Olabanji
Join Date: Jan 2013
Location: U.S.A
Posts: 29
Rep Power: 5
banji is on a distinguished road
Hi ,

I am trying to analyze the performance of a parallel program using IPM. I have successfully installed the IPM software following the guidelines described at https://github.com/nerscadmin/ipm and followed the explanation at https://www.hpc.ntnu.no/display/hpc/...filing+Solvers to setup a custom solver.

The problem is that when I run the program, for example

Quote:
mpirun -np 4 pisoFoamIPM -parallel

the program just runs without any time analysis report. Am i doing something wrong or how do I get a result printed?

Thanks.
banji is offline   Reply With Quote

Old   May 6, 2016, 04:23
Default
  #11
Member
 
Join Date: Dec 2015
Posts: 37
Rep Power: 2
WhiteW is on a distinguished road
Hi banji. To write the output in a file you have to run:

Code:
mpirun -np 4 pisoFoamIPM -parallel >logfile.txt
[ Moderator note: This post was copied-modified from here: Run OpenFoam in 2 nodes of a cluster ]

Last edited by wyldckat; May 8, 2016 at 15:17. Reason: see "Moderator note:"
WhiteW is offline   Reply With Quote

Old   May 6, 2016, 04:30
Default
  #12
New Member
 
Olabanji
Join Date: Jan 2013
Location: U.S.A
Posts: 29
Rep Power: 5
banji is on a distinguished road
Hellow WhiteW,

Thanks for your reply.

Actually, I know how to redirect messages to an output file. The problem is I can't find any related to IPM i.e. maybe how much time is spent on MPI_wait, MPI_recv , e.t.c.

[ Moderator note: Moved from here: Run OpenFoam in 2 nodes of a cluster ]

Last edited by wyldckat; May 8, 2016 at 15:20. Reason: see "Moderator note:"
banji is offline   Reply With Quote

Old   May 8, 2016, 15:28
Default
  #13
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,556
Blog Entries: 39
Rep Power: 97
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Quote:
Originally Posted by banji View Post
the program just runs without any time analysis report. Am i doing something wrong or how do I get a result printed?
Quick answer: I'm guessing that this explains how to process the information: http://ipm-hpc.sourceforge.net/userguide.html - read section "Post-processing IPM output".
wyldckat is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
mpirun, best parameters pablodecastillo Hardware 17 April 27, 2012 13:05
Error using LaunderGibsonRSTM on SGI ALTIX 4700 jaswi OpenFOAM 2 April 29, 2008 10:54
Is Testsuite on the way or not lakeat OpenFOAM Installation 6 April 28, 2008 11:12
MPI profiling OpenFOAM damBreak3D application mellanoxuser OpenFOAM Pre-Processing 0 April 13, 2008 23:15
MPI profiling OpenFOAM damBreak3D application mellanoxuser OpenFOAM Running, Solving & CFD 0 April 13, 2008 23:04


All times are GMT -4. The time now is 23:35.