I am running a multphaseInterF
 

I am running a multphaseInterFoam case (approx 1.1 million hex cells) on an infiniband cluster. Thanks to Henry's help I have switched from quartic interfacecompression scheme to quadratic. This seems to keep the volume fractions stable. Anyways, when I tried to use the following procedure to start a parallel run with 8 processors, I get an error message similar to that reported in [1]:

lamboot -v $PBS_NODEFILE
mpirun -ssi rpi ib -np 8 multiphaseInterFoam . three_fluid_test_4 -parallel

I was just about to give up on getting the inifiniband to work, when I came across this webpage[2]. Following their procedure, I used the following commands to start the run:

lamboot -v -b -ssi boot rsh -ssi rsh_agent rsh $PBS_NODEFILE
mpirun -ssi rpi tcp -np 8 multiphaseInterFoam . three_fluid_test_4 -parallel

and the run started without problems. The lamboot initialization parameters suggest that I am not using the ib module? So I expected degraded performance. However, comparing the time after 7 time steps for a serial run with that of a parallel run, I get a speed up of 8.125818891 which is indeed impressive. I cannot explain why this seems to work. Any comments are appreciated.

PS: The cluster is a combination of 256 processors (i.e. 128 dual opteron boxes NOT dual-core) connected with infiniband.

[1] http://www.cfd-online.com/OpenFOAM_D...tml?1188946223
[2] http://tensor.tamu.edu/docs/lam.html