CFD Online Discussion Forums

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)

- OpenFOAM (https://www.cfd-online.com/Forums/openfoam/)

- - Infiniband (https://www.cfd-online.com/Forums/openfoam/60711-infiniband.html)

Hello! I could use some hel

Hello!

I could use some help with starting Foam/LAM on a linux cluster with infinband interconnect. So far i can only use the ethernet between the nodes. I tried lamboot -v -ssi rpi ib hostfile, but i only get the message ib module not found. Does somebody know this problem?
And another question: How much better is infiniband compared to gigabit ethernet? Has somebody experience with infiniband?

I guess you shouldnt use LAM a

I guess you shouldnt use LAM as mpi library, but you should use mvapich, the mpi implementation for infiniband.
I'll try it as soon as I can, and I'll let you know.
Regarding Infiniband, it's really much better than Gigabit. The transfer rate of infiniband, measured from application, is about 960MB/s (Megabyte/s), and the latency time is very, very low.
You can therefore use a high number of processors in a parallel run.
As an example, my application saturates gigabit with 12 processes, while you can scale up to 48 and more with infiniband, reducing wallclock computational time.
It worths to spend some time in understanding how to use infiniband, really.

I have a similar issue. The la

I have a similar issue. The lam compilation shipped with OpenFOAM does not include support for Myrinet, but I know that lam does have a native Myrinet module. Does anyone know which options you need to invoke to compile these modules into lam?

you have to build LAM from sou

you have to build LAM from source for native support of Myrinet.

you can do this with the option gm:
./configure --prefix=/your/lam/dir --with-gm=/path/to/gm ...

We have recently made a test G

We have recently made a test Gigabit v.s. Infiniband. I will post a short report on that on the discussion page soon. The test does not show a significant improvement when using Infiniband, at least for the case that was used in the test. The test was a simpleFoam computation with 10^6 cells distributed on 1-16 CPUs. In the higher range of CPU numbers there might be some improvement when using Infiniband, and if using even more CPUs Infiniband might be better relative to Gigabit. For 16 CPU's the parallel efficiency is however only 0.45 for Gigabit and 0.51 for Infiniband, which is not really great.

My collaboration partner at Gridcore (www.gridcore.se) did the technical stuff. He just re-compiled pstream.so and liked it to the correct Infiniband mpi-library.

Håkan.

The flag mentioned above is st

The flag mentioned above is still usable but deprecated, please use the "--with-rpi-gm=PATH" flag instead!

For further details have a look at the LAM/MPI Installation Guide.

we hare recently made a test G

we hare recently made a test Gigabit vs Infiniband on an AMD Opteron Cluster with another application (a Navier-Stokes software for engine application with structured grids and moving meshes), written in Fortran77/Fortran95 on an mpi base, and the differences between the networks are really relevant.
Depending on the case, we have saturated Gigabit with 12 processors (speedup<1), while with infiniband the speedup grows till 24 processors used, which was the maximum number of processes we have tested.
With a bigger test case, with 16 processors the time spent was 113 min with gigabit and 75 with infiniband. We couldn't test more than 16 processors with gigabit, but using Infiniband the speedup always grows, till 48 processors, which is the total number of CPUs of the cluster.
With combustion, moreover, the value of the speedup obviously grows, with an efficency of parallelism up to 63% with 30 processors.
I've no experience with OpenFOAM over Infiniband, but with some kind of applications, this kind of network is what really makes the difference.

Dear Håkan, Are you compari

Dear Håkan,

Are you comparing versus a sequential run? Or is this the %cpu utilization? Is the difference due to the number of sweeps being much larger?

Did you try various decompositions? Hierarchical with almost equal number in x,y,z is a good starting point.
Did you try scheduledTransfer on/off?

Dear 'Infiniband Interest Grou

Dear 'Infiniband Interest Group',

You can find a short report on my test of Infiniband here: http://www.cfd-online.com/OpenFOAM_D...hment_icon.gif Infiniband_vs_Gigabit.pdf.gz

I would be very happy to get some comments on how the test was made. I will soon make similar comparisons with InfiniPath, and I would like to make as good a test as possible.

I hope that the document answers the questions from Mattijs. Some additional answers follow:

The number of iterations for p are slightly changed with the decomposition, typically: 229,251,258,261,271 for 1,2,4,8,16 CPU's. The other equations take 1 iteration for all decompositions. Is this what you are referring to 'number of sweeps'? I don't know if the actual linear solver (ICCG) does more sweeps per iteration.

I did not try various decompositions, only load balanced Metis. This test is only a small project on the side of what I should really be doing, and I don't have the time to try all options. There are many ways of decomposing, but I think that the Metis decomposition should be a good starting point.

I did not try scheduledTransfer on/off. Can you please tell me what this will affect, and how to do it? Thank you on advance, Mattijs!

I have asked before about some hints on how to make the comparisons the best way, and that question is still open.

Håkan.

scheduledTransfer makes sure o

scheduledTransfer makes sure one side (process) of a processor patch is in receive mode while the other one is sending. Means you can use mpi_send instead of mpi_bsend. Sometimes benificial (no buffer, no thread?), sometimes detrimental (more chance of everyone waiting for the slowest).

Hi Håkan! I recently got a

Hi Håkan!

I recently got a benchmark paper by Cisco where they compared Fluent on GEthernet with Infiniband. For the case that was similar to yours (850k cells) for the number of processors you had available their results are very similar to yours: the speedup with EN and IB are almost the same. For larger numbers of processors (they had 64 nodes) IB performs better.

The other results (for smaller runs) suggest in my opinion that for Infiniband to be of use you either have to have small runs and/or a large number (>>10) of nodes. I'd say that the major advantage of IB is not the bandwidth but the low latency.

(If you're interested I can send you the PDF)

Hi Infiniband users! I have

Hi Infiniband users!

I have problems compiling lam7.1.1 with Infiniband support. What I did (all what is in Allwmake file in the src directory):

gmake distclean
rm -rf $LAM_ARCH_PATH

./configure \
--prefix=$LAM_ARCH_PATH \
--with-rpi-ib=/usr/ibgd/driver/infinihost \
--enable-shared \
--disable-static \
--without-romio \
--without-mpi2cpp \
--without-profiling \
--without-fc

gmake

So when I compile it I get the following error

gmake[6]: Entering directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi/ib/src'
if /bin/sh ../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID="" -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_ack.lo -MD -MP -MF ".deps/ssi_rpi_ib_ack.Tpo" -c -o ssi_rpi_ib_ack.lo ssi_rpi_ib_ack.c; \
then mv -f ".deps/ssi_rpi_ib_ack.Tpo" ".deps/ssi_rpi_ib_ack.Plo"; else rm -f ".deps/ssi_rpi_ib_ack.Tpo"; exit 1; fi
mkdir .libs
gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID= -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_ack.lo -MD -MP -MF .deps/ssi_rpi_ib_ack.Tpo -c ssi_rpi_ib_ack.c -fPIC -DPIC -o .libs/ssi_rpi_ib_ack.o
if /bin/sh ../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID="" -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_actions.lo -MD -MP -MF ".deps/ssi_rpi_ib_actions.Tpo" -c -o ssi_rpi_ib_actions.lo ssi_rpi_ib_actions.c; \
then mv -f ".deps/ssi_rpi_ib_actions.Tpo" ".deps/ssi_rpi_ib_actions.Plo"; else rm -f ".deps/ssi_rpi_ib_actions.Tpo"; exit 1; fi
gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID= -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_actions.lo -MD -MP -MF .deps/ssi_rpi_ib_actions.Tpo -c ssi_rpi_ib_actions.c -fPIC -DPIC -o .libs/ssi_rpi_ib_actions.o
ssi_rpi_ib_actions.c: In function 'send_peer_fc_info':
ssi_rpi_ib_actions.c:1202: warning: right shift count >= width of type
ssi_rpi_ib_actions.c:1244: warning: left shift count >= width of type
if /bin/sh ../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID="" -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_bitmap.lo -MD -MP -MF ".deps/ssi_rpi_ib_bitmap.Tpo" -c -o ssi_rpi_ib_bitmap.lo ssi_rpi_ib_bitmap.c; \
then mv -f ".deps/ssi_rpi_ib_bitmap.Tpo" ".deps/ssi_rpi_ib_bitmap.Plo"; else rm -f ".deps/ssi_rpi_ib_bitmap.Tpo"; exit 1; fi
gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID= -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_bitmap.lo -MD -MP -MF .deps/ssi_rpi_ib_bitmap.Tpo -c ssi_rpi_ib_bitmap.c -fPIC -DPIC -o .libs/ssi_rpi_ib_bitmap.o
if /bin/sh ../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID="" -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_dreg.lo -MD -MP -MF ".deps/ssi_rpi_ib_dreg.Tpo" -c -o ssi_rpi_ib_dreg.lo ssi_rpi_ib_dreg.c; \
then mv -f ".deps/ssi_rpi_ib_dreg.Tpo" ".deps/ssi_rpi_ib_dreg.Plo"; else rm -f ".deps/ssi_rpi_ib_dreg.Tpo"; exit 1; fi
gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID= -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_dreg.lo -MD -MP -MF .deps/ssi_rpi_ib_dreg.Tpo -c ssi_rpi_ib_dreg.c -fPIC -DPIC -o .libs/ssi_rpi_ib_dreg.o
ssi_rpi_ib_dreg.c:40: error: static declaration of 'lam_ssi_rpi_ib_env_mempool' follows non-static declaration
./rpi_ib_dreg.h:24: error: previous declaration of 'lam_ssi_rpi_ib_env_mempool' was here
gmake[6]: *** [ssi_rpi_ib_dreg.lo] Error 1
gmake[6]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi/ib/src'
gmake[5]: *** [all] Error 2
gmake[5]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi/ib/src'
gmake[4]: *** [all-recursive] Error 1
gmake[4]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi/ib'
gmake[3]: *** [all-recursive] Error 1
gmake[3]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi'
gmake[2]: *** [all-recursive] Error 1
gmake[2]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi'
gmake[1]: *** [all-recursive] Error 1
gmake[1]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share'
gmake: *** [all-recursive] Error 1

Any help is apreciated. Thanks

Jens

Hi Infiniband users! I have

How about using openmpi? This

How about using openmpi? This might have more uptodate infiniband. We include 1.0.2a7.

When I compile openmpi everyth

When I compile openmpi everything works fine.

When I recompile the the Pstream libs I get no errors, but the following warnings

OPwrite.C: In static member function âÂ€Â˜static bool Foam:http://www.cfd-online.com/OpenFOAM_D...part/proud.gifPstream::write(int, const char*, std::streamsize, bool)âÂ€Â™:
OPwrite.C:77: warning: use of old-style cast
OPwrite.C:89: warning: use of old-style cast

1. Are these warnings problematic?

However I ignored them and did:

mpirun -v --mca btl mvapi,self -np 4 --hostfile ompimachinefile -ssh "/home/klosterm/OpenFOAM/OpenFOAM-1.3/applications/bin/linuxAMD64Gcc4DPOpt/inter Foam . dambreak -parallel"
with the following error as result:

[stokes:25010] [0,0,0] ORTE_ERROR_LOG: Not implemented in file rmgr_urm.c at line 177
[stokes:25010] [0,0,0] ORTE_ERROR_LOG: Not implemented in file rmgr_urm.c at line 365
[stokes:25010] mpirun: spawn failed with errno=-7

Thank you for helping!

Jens

Those warnings are not problem

Those warnings are not problematic.

Never ran openmpi+infiniband so cannot help you there. Can you post the solution if you find it out?

So the problem is solved. Open

So the problem is solved. OpenFOAM 1.3 is running with infiniband and openmpi. I use the openmpi version 1.2a1r10111, but I think the one which is shipped with OpenFOAM 1.3 is also working.

The problem was that it was not possible to run different versions of mpirun (lam, mpich or openmpi) for different users at the same time. So we hat a lam job running over ethernet which was somehow blocking the mpirun of openmpi.

Hello Jens, I am very inter

Hello Jens,

I am very interested in having an idea
about your compilation of openmpi with infiniband.

I am trying several mpi on my infiniband (and let already a message about mvapich) but after having read this threat I tried to compile
openmpi included in openfoam distribution and did not succeed. I have some errors put at the end of my present mail.
These errors are independant of the option inifiniband in my configure file.

How does your openmpi version work?

Thanks

Anne

------------------------------------
gmake[5]: *** No hay ninguna regla para construir el objetivo `distclean'. Alto.
gmake[5]: Leaving directory `/afs/ciemat.es/users/u5303/OpenFOAM/OpenFOAM-1.3/src/openmpi-1.0.2a7/ompi/mca/i o/romio/romio'
gmake[4]: *** [distclean-recursive] Error 1
gmake[4]: Leaving directory `/afs/ciemat.es/users/u5303/OpenFOAM/OpenFOAM-1.3/src/openmpi-1.0.2a7/ompi/mca/i o/romio'
gmake[3]: *** [distclean-recursive] Error 1
gmake[3]: Leaving directory `/afs/ciemat.es/users/u5303/OpenFOAM/OpenFOAM-1.3/src/openmpi-1.0.2a7/ompi/mca/i o'
gmake[2]: *** [distclean-recursive] Error 1
gmake[2]: Leaving directory `/afs/ciemat.es/users/u5303/OpenFOAM/OpenFOAM-1.3/src/openmpi-1.0.2a7/ompi/mca'
gmake[1]: *** [distclean-recursive] Error 1
gmake[1]: Leaving directory `/afs/ciemat.es/users/u5303/OpenFOAM/OpenFOAM-1.3/src/openmpi-1.0.2a7/ompi'
gmake: *** [distclean-recursive] Error 1
--------------------------------------------------

Hi Anne, I had a lot of tro

Hi Anne,

I had a lot of troubles setting up OpenFOAM for infiniband communication. So far it works with openmpi for almost a year now. From my experience I suggest to use always the most recent version of openmpi.

Regards Jens

Hi - I'm new to OpenFOAM. I

Hi - I'm new to OpenFOAM.

I ran damBreak3d using openMPI over Infiniband with the "runnproc" scripts 1-8.

Now I want to try 16 processes and more.

My questions is:
1) Should I clone the case = nproc8 and create a "nproc16" with FoamX (FoamX won't start on my system) ?

2) Is it best to create a new case? Is there a way from command line to create a new case?

3) I copied the case "nproc8" and changed "8" to 16 but Foam IO had trouble opening the controlDict file

Hi everyone. Could someone

Hi everyone.

Could someone give the steps to run the last OpenFOAM on a cluster with Infiniband?

Do I have to compile OpenFOAM for this specific purpose?

Thank you for your help.