CFD Online Logo CFD Online URL
Home > Forums > OpenFOAM Running, Solving & CFD

Interesting behavior when using LAM without ib rpi on an infiniband cluster OF 141

Register Blogs Members List Search Today's Posts Mark Forums Read

LinkBack Thread Tools Display Modes
Old   October 2, 2007, 16:31
Default I am running a multphaseInterF
Senior Member
Srinath Madhavan (a.k.a pUl|)
Join Date: Mar 2009
Location: Edmonton, AB, Canada
Posts: 703
Rep Power: 14
msrinath80 is on a distinguished road
I am running a multphaseInterFoam case (approx 1.1 million hex cells) on an infiniband cluster. Thanks to Henry's help I have switched from quartic interfacecompression scheme to quadratic. This seems to keep the volume fractions stable. Anyways, when I tried to use the following procedure to start a parallel run with 8 processors, I get an error message similar to that reported in [1]:

lamboot -v $PBS_NODEFILE
mpirun -ssi rpi ib -np 8 multiphaseInterFoam . three_fluid_test_4 -parallel

I was just about to give up on getting the inifiniband to work, when I came across this webpage[2]. Following their procedure, I used the following commands to start the run:

lamboot -v -b -ssi boot rsh -ssi rsh_agent rsh $PBS_NODEFILE
mpirun -ssi rpi tcp -np 8 multiphaseInterFoam . three_fluid_test_4 -parallel

and the run started without problems. The lamboot initialization parameters suggest that I am not using the ib module? So I expected degraded performance. However, comparing the time after 7 time steps for a serial run with that of a parallel run, I get a speed up of 8.125818891 which is indeed impressive. I cannot explain why this seems to work. Any comments are appreciated.

PS: The cluster is a combination of 256 processors (i.e. 128 dual opteron boxes NOT dual-core) connected with infiniband.

msrinath80 is offline   Reply With Quote


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
HPMPI Infiniband problem carsten OpenFOAM Bugs 2 January 25, 2009 16:36
Infiniband alexander_rudert OpenFOAM 20 January 14, 2009 09:01
OpenFOAM and infiniband mrangitschdowcom OpenFOAM Installation 5 October 30, 2008 08:47
infiniband interconnect help rjh FLUENT 0 February 4, 2008 12:41
OpenFOAM 141 parallel results infiniband vs gigabit vs SMP msrinath80 OpenFOAM Running, Solving & CFD 10 November 30, 2007 19:11

All times are GMT -4. The time now is 10:12.