CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Running, Solving & CFD

OpenFOAM MPI doesn't work on LAN due to "missing" executables

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree1Likes
  • 1 Post By dhbw

Reply
 
LinkBack Thread Tools Display Modes
Old   February 4, 2016, 12:31
Default OpenFOAM MPI doesn't work on LAN due to "missing" executables
  #1
New Member
 
Join Date: Oct 2015
Location: Germany
Posts: 16
Rep Power: 2
dhbw is on a distinguished road
Hello everybody,

I tried to set up a small OpenFOAM cluster via LAN. MPI works just fine when I run it locally, say on 6 cores. But when I try to run it on LAN it always fails independently from the machine I want to use for parallelization (I have three machines connected to a switch). The error message always says:

mpirun was unable to launch the specified application as it could not find an executable:

Executable: simpleFoam
Node: 192.168.1.2

while attempting to start process rank 6.


But OF is properly installed. I can ssh into all machines without any problems.

Can someone give me a hint or suggestion?

Thanks in advance!

Regards

Last edited by dhbw; February 5, 2016 at 04:07.
dhbw is offline   Reply With Quote

Old   February 6, 2016, 14:39
Default OpenFOAM shows up in task manager but doesn't seem to compute solution
  #2
New Member
 
Join Date: Oct 2015
Location: Germany
Posts: 16
Rep Power: 2
dhbw is on a distinguished road
I managed to export the environment variables, so the run seems to start now without error messages.

But now I recognized that although the processes show up in the task managers on all the used nodes with about 99 % CPU usage, the solution don't seem to be computed. The log file only contains the OpenFOAM header sequence but no convergence history. Furthermore the processes seem to run infinitely.

Hope someone can help me...

Thanks in advance!
dhbw is offline   Reply With Quote

Old   February 6, 2016, 18:26
Default
  #3
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,648
Blog Entries: 39
Rep Power: 99
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Quick answer: Quoting from a blog post of mine: Notes about running OpenFOAM in parallel
Quote:
Is the output from mpirun (Open-MPI) only coming out at the end of the run? Check this post: mpirun openfoam output is buffered, only output at the end post #9
wyldckat is offline   Reply With Quote

Old   February 8, 2016, 05:13
Default
  #4
New Member
 
Join Date: Oct 2015
Location: Germany
Posts: 16
Rep Power: 2
dhbw is on a distinguished road
Quote:
Originally Posted by wyldckat View Post
Quick answer: Quoting from a blog post of mine: Notes about running OpenFOAM in parallel
Thank you very much for your help! Unfortunately your hint did not solve the problem. So here is what I'm trying to do in detail:

I have three nodes on a LAN with 6, 2 and 2 cores. I want to initialize a simplefoam run on 4, 2 an 2 cores at a time using Open MPI 1.6.5. To do so I decomposed the domain using
Code:
decomposePar
into 8 sub domains without error.

The required machine file named "machine" is located within the "system" directory of the OF case and contains the following:

Code:
ihgg-ubuntu cpu=4
cluster_login@192.168.1.2 cpu=2
cfd_cluster@192.168.1.3 cpu=2
The first node is the six-core master machine. The two below are the two-core slaves. The user names are the admistrator logins for each machine. I established passwordless ssh from and to all nodes within the LAN, which works perfectly fine.

The mpirun command only "successfully" initializes simplefoam if i export the OF environment variables:

Code:
mpirun -x LD_LIBRARY_PATH -x PATH -x WM_PROJECT_DIR -x WM_PROJECT_INST_DIR -x WM_OPTIONS -x FOAM_LIBBIN -x FOAM_APPBIN -x MPI_BUFFER_SIZE -machinefile system/machine -np 8 -output-filename openfoam_log 'simpleFoam' -parallel &
From this point on, simplefoam shows up in the task managers of all nodes with the correct number of processes. But when I open the fresh log file, it shows no convergence history but one error message.

Code:
/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  3.0.1                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 3.0.1-119cac7e8750
Exec   : simpleFoam -parallel
Date   : Feb 08 2016
Time   : 10:08:50
Host   : "ihgg-ubuntu"
PID    : 13154
[ihgg-ubuntu][[8881,1],0][btl_tcp_endpoint.c:655:mca_btl_tcp_endpoint_complete_connect] connect() to 10.3.140.137 failed: No route to host (113)
Is the error message caused by the connection attempt to ip 10.3.140.137?

I'm totally stuck.

Thanks in advance for any further suggestions!
dhbw is offline   Reply With Quote

Old   March 15, 2016, 06:15
Default
  #5
New Member
 
Join Date: Oct 2015
Location: Germany
Posts: 16
Rep Power: 2
dhbw is on a distinguished road
I put my problem aside for quite awhile. Now that I got to review the whole issue I solved my problem. The unknown IP ("no route to host") resulted from an active WLAN connection on the remote node (client). OpenMPI tried to use this connection instead of the local ethernet IPs assigned by the
Code:
machinefile
in conjunction with the
Code:
/etc/hosts
file. Deactivating WLAN solved it partially. My entire solution procedure was as follows:
  1. Made sure all nodes are accessible via password-less ssh and the host can also be accessed in the same manner from all the client nodes.
  2. Put the
    Code:
    source /opt/openfoam...
    line into the bashrc-files on all the nodes (OpenFOAM install dir is the same on all nodes).
  3. Use the MPI command in the following manner:
    Code:
    mpirun -hostfile <hostfile> -np <no. of processors> /opt/openfoam30/bin/foamExec icoFoam -parallel
Although this got me way further I was then struggeling with the proper case decomposition over the LAN. Thus, I ended up using "manually" distributed data (run decomposePar nondistributed on the hosting node and distribute the <processorX> directories manually according to the node sequence and cpu numbers in the OpenMPI hostfile). Data access for the client nodes can then conveniently be established e.g. by using the "Connect to Server" (via GUI) functionality of the "Files" application.

I consider this topic as CLOSED.
wyldckat likes this.

Last edited by dhbw; March 21, 2016 at 07:24.
dhbw is offline   Reply With Quote

Old   March 19, 2016, 16:43
Default
  #6
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,648
Blog Entries: 39
Rep Power: 99
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Quick answer: Many thanks for posting your solution!

This reminded me of the following solution for disabling certain network interfaces:
Quote:
Originally Posted by pkr View Post
When using MPI_reduce, the OpenMPI was trying to establish TCP through a different interface. The problem is solved if the following command is used:
Code:
mpirun --mca btl_tcp_if_exclude lo,virbr0  -hostfile machines -np 2  /home/rphull/OpenFOAM/OpenFOAM-1.6/bin/foamExec interFoam -parallel
The above command will restrict MPI to use certain networks (lo, vibro in this case).
wyldckat is offline   Reply With Quote

Old   March 21, 2016, 07:26
Default
  #7
New Member
 
Join Date: Oct 2015
Location: Germany
Posts: 16
Rep Power: 2
dhbw is on a distinguished road
Thanks for your hint wyldckat. Actually I have tried this approach without success. But maybe because of a typo or such XD...
dhbw is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Suggestion for a new sub-forum at OpenFOAM's Forum wyldckat Site Help, Feedback & Discussions 20 October 28, 2014 10:04
Error using LaunderGibsonRSTM on SGI ALTIX 4700 jaswi OpenFOAM 2 April 29, 2008 10:54
Is Testsuite on the way or not lakeat OpenFOAM Installation 6 April 28, 2008 11:12
Adventure of fisrst openfoam installation on Ubuntu 710 jussi OpenFOAM Installation 0 April 24, 2008 14:25
OpenFOAM Debian packaging current status problems and TODOs oseen OpenFOAM Installation 9 August 26, 2007 13:50


All times are GMT -4. The time now is 08:39.