[tjliang]

Dear Foamers,

I am going to run my 3D simualtion on a cluster. I can run on each node the simulation without problem. But when I want to allocate the simulation into different nodes at the same time using the following pbs script, it turns out to run only in a single node. My openfoam version is 2.4.x.

#!/bin/bash
#PBS -j oe
#PBS -l walltime=10:00:00
#PBS -N 3D-Arc-Welding
#PBS -l nodes=2: ppn=8
#PBS -m abe
#PBS -q default

. ~/.bashrc

cd $PBS_O_WORKDIR

mpirun --mca btl ^openib -np 16 ArgonPlasmaMultiRegionSimpleFoam -parallel>run.\
log

For this cluster this is a sample code similar to my problem, but I don't know how to apply that to my case, itis as follows:

#!/bin/bash
#PBS -l walltime=01:00:00
#PBS -l nice=19
#PBS -q default
#PBS -l nodes=6: ppn=16

mpiexec -n 16 -host n001 hostname : -n 16 -host n002 hostname : -n 16 -host n003 hostname : -n 16 -host n004 hostname : -n 16 -host n005 hostname : -n 16 -host n006 hostname | sort


echo "end"


Can anyone give me suggestions to correct it? Thank you.


Bests,

Peng

[me3840]

If I recall correctly running
--mca btl ^openib
would tell OpenMPI to not use infiniband interfaces, isn't that the opposite of what you want? I would always run --mca btl ^tcp previously, but perhaps something has changed?

The code you have should run fine, can you please post the logfile from the cluster? If it only runs on a single node and you allocate all the MPI ranks to that node, OpenMPI should abort or give you a warning, so I'm not sure how it would run that way.

[tjliang]

Originally Posted by me3840

If I recall correctly running
--mca btl ^openib
would tell OpenMPI to not use infiniband interfaces, isn't that the opposite of what you want? I would always run --mca btl ^tcp previously, but perhaps something has changed?

The code you have should run fine, can you please post the logfile from the cluster? If it only runs on a single node and you allocate all the MPI ranks to that node, OpenMPI should abort or give you a warning, so I'm not sure how it would run that way.

Thank you for the quick reply, I used your --mca btl ^tcp script, but get the error:

did not launch the job. This error was first reported for process
rank 0; it may have occurred for other processes as well.

NOTE: A common cause for this error is misspelling a mpirun command
line parameter option (remember that mpirun interprets the first
unrecognized command line token as the executable).

Node: n001
Executable: ArgonPlasmaMultiRegionSimpleFoam
--------------------------------------------------------------------------
16 total processes failed to start


And when I use my own script, I actually didn't get any errors if I take the line

". $WM_PROJECT_DIR/etc/bashrc" away and it runs as if no problems happened. But actually only in single node.

But if I add that line, I got the error:

"mpirun was unable to find the specified executable file, and therefore
did not launch the job. This error was first reported for process
rank 0; it may have occurred for other processes as well.

NOTE: A common cause for this error is misspelling a mpirun command
line parameter option (remember that mpirun interprets the first
unrecognized command line token as the executable).

Node: n002
Executable: ArgonPlasmaMultiRegionSimpleFoam
--------------------------------------------------------------------------
16 total processes failed to start"