SimpleFoam run in Parallel
 

Hi Foamers

I have two machines and it's host names are 'cfd0' and 'cfd1'.
The cfd0 machine is main node machine and cfd1 is client node machine.
I have install OF on cfd0 and mpich is also install on cfd0. And cfd0 is mounted on cfd1 machine. Both machine are dual core machines.
Now this two machines are in cluster and I can run simple hello world program on this cluster properly using all 4 processor.
When I run simpleFoam case in parallel on one machine with decomposepardict file as :

numberOfSubdomains 4;

method simple;

simpleCoeffs
{
n ( 2 2 1 );
delta 0.001;
}

hierarchicalCoeffs
{
n ( 1 1 1 );
delta 0.001;
order xyz;
}

metisCoeffs
{
processorWeights ( 1 1 1 1 );
}

manualCoeffs
{
dataFile "";
}

distributed no;


roots ( );
/*------------------------------------------------------------------------*/

it works on only one machine.
To run simpleFoam on cluster I need to change

distributed yes;

roots 2(cfd0, cfd1 );

in decomposepardict file.
But it dosen't work.
Please help.
Thank you in advance.

Jay

Hello Jay,

I'm not quite sure but maybe this helps:

set distributed no; and roots ();

create a file in your case-directory which is named machines or something like that. In the machines-file write down the IP-addresses of the machines you would like to compute on, as follows:
192.168.2.1 slots=2 max_slots=2
192.168.2.2 slots=2 max_slots=2 and so on

decompose your case and start simpleFoam using the mpirun-command, e.g. in your case something like:
mpirun -np 4 --hostfile machines /Path/To/OpenFOAM/bin/foamExec simpleFoam -parallel > logfile

I had VMWare running on the machines so mpirun couldn't find the IP-adresses of the machines for some reason so I added the following mca-option after the hostfile-option to the mpirun-command:
--mca btl_tcp_if_include eth0

worked fine for me with 4 PCs and 16 CPUs, so I hope it helps...

Greetings.

Hello Yasar
Thank you for reply.

As you said I create one file in case directory named as machines and added ip addresses of the machines as said.
and after decompose when I run the case :

$ mpirun -np 4 --mca btl_tcp_if_include eth0 machines /home/mpi/OpenFOAM/OpenFOAM-1.6/bin/foamExec simpleFoam -parallel
--------------------------------------------------------------------------
mpirun was unable to launch the specified application as it could not find an executable:

Executable: machines
Node: cfd0

while attempting to start process rank 0.
--------------------------------------------------------------------------
I got the above error. Please help if have any option.
Regards
Jay

Hello Jay,

seems to me that you mixed something up in your command. Try the following and hopefully it'll do:

$ mpirun -np 4 --hostfile machines --mca btl_tcp_if_include eth0 /home/mpi/OpenFOAM/OpenFOAM-1.6/bin/foamExec simpleFoam -parallel

Regards
Yasar

Hello Yasar
Thanks for again reply.
It doesn't work again.
$ mpirun -np 4 --hostfile machines --mca btl_tcp_if_include eth0 /home/mpi/OpenFOAM/OpenFOAM-1.6/bin/foamExec simpleFoam -parallel
bash: orted: command not found
--------------------------------------------------------------------------
A daemon (pid 8078) died unexpectedly with status 127 while attempting
to launch so we are aborting.

There may be more information reported by the environment (see above).

This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.
--------------------------------------------------------------------------
mpirun: clean termination accomplished

Again having some error.

Regards
Jay

Hello Jay,

maybe you should try it without the mca-command:

$ mpirun -np 4 --hostfile machines /home/mpi/OpenFOAM/OpenFOAM-1.6/bin/foamExec simpleFoam -parallel

if this doesn't help I'm afraid I don't know further suggestions...

Regards
Yasar

Hello Jay and Yasar,

Jay, might I suggest that you try using the parallelTest utility - information available on this post: OpenFOAM updates post #19
The parallelTest utility (it's part of OpenFOAM's test utilities) can aid you in sorting out the more basic MPI problems, like communication problems or missing environment settings or libraries not found, without running any particular solver functionalities.

Secondly, since it's MPICH (and by my experience with MPICH 1 and 2), you have to force it to send the environment variables to the targeted slaves. Check the manual on how to force it to send environment variables. Although, the script foamExec should do that for you, but only if you've setup properly the files settings.sh and bashrc in the folder OpenFOAM/OpenFOAM-1.6/etc!

As for:
If I'm not mistaken, orted is also known as mpirun and mpiexec in some MPI libraries. Are you sure that the same MPICH library is installed in both machines? And is it installed as a system library/application? Or is it installed as part of OpenFOAM's Third Party applications in the folder ThirdParty-1.6?
Because if it's not installed as a system library/application, and like I've written above, you should edit the files settings.sh and bashrc in OpenFOAM/OpenFOAM-1.6/etc folder, to define properly the environment variables for using your MPICH version! But even if it's a system MPI, you should edit them just the same, so it won't set-up the default OpenMPI library as the working MPI library to be used!

Best regards,
Bruno

question
 

Hello,
I really need your help, I am stuck with this issue for a while and I
can not figure it out.
I build a cluster using rocks Centos 5.3 and everything seems works
fine accept when I am trying to submit a job to the cluster (try to do
parallel simulation) I always get the message: mpiexec was unable to
launch the specified application as it could not find an executable:
Executable: abinip"

I am using mpich2 in my system.

I am not sure what to do?

I am new to the linux world so any help will be appreciated.

Hello jtalla
First of all I would like to thank Bruno Santos.
Jtalla if you have properly install mpich2 and if you can access other machine without password, then you can try the command which is there in Bruno's link
"Jay, might I suggest that you try using the parallelTest utility - information available on this post: OpenFOAM updates post #19
The parallelTest utility (it's part of OpenFOAM's test utilities) can aid you in sorting out the more basic MPI problems, like communication problems or missing environment settings or libraries not found, without running any particular solver functionalities. "

Because I have sorted out my problems.
Once again Thank you Bruno
Jay

Hey guys,

Great responses.

I'm running some cases and after I've decomposed the file I can run it using this command:


'C:/Program Files/Microsoft MPI/Bin/mpiexec.exe' -np 8 simpleFoam.exe -parallel 2>&1 | tee solution.log

ie I literally paste that into the command line and it works.
Question is how do I get it to do that in a batch script so it's automatic?