[thoraxe]

I'm helping a friend get mpirun in parallel (multiple hosts) going. Things seem to work just fine on a single host.


I realize that openfoam5 is old, but that's what he's got installed.


openfoam is installed to /opt/openfoam5 on all hosts, and we have NFS mounting the test files/data in the same place on all hosts


I'm trying to run the incompressible/simpleFoam/motorBike tutorial from the installation just to validate that multiple machines work.


passwordless SSH works in all directions.


Here's the machines file:

localhost slots=3
10.0.0.113 slots=3

Here's the mpirun invocation:

Cleaning /home/rob/cases/incompressible/simpleFoam/motorBike case
Running surfaceFeatureExtract on /home/rob/cases/incompressible/simpleFoam/motorBike
Running blockMesh on /home/rob/cases/incompressible/simpleFoam/motorBike
Running decomposePar on /home/rob/cases/incompressible/simpleFoam/motorBike
/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  5.x                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 5.x-68e8507efb72
Exec   : snappyHexMesh -overwrite
Date   : Jan 24 2020
Time   : 14:48:59
Host   : "solver10"
PID    : 61205
I/O    : uncollated
Case   : /home/rob/cases/incompressible/simpleFoam/motorBike
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Read mesh in = 0 s

Overall mesh bounding box  : (-5 -4 0) (15 4 8)
Relative tolerance         : 1e-06
Absolute matching distance : 2.29783e-05

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  5.x                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 5.x-68e8507efb72
Exec   : snappyHexMesh -overwrite
Date   : Jan 24 2020
Time   : 14:48:59
Host   : "solver10"
PID    : 61206
I/O    : uncollated
Case   : /home/rob/cases/incompressible/simpleFoam/motorBike
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Read mesh in = 0 s

Overall mesh bounding box  : (-5 -4 0) (15 4 8)
Relative tolerance         : 1e-06
Absolute matching distance : 2.29783e-05

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  5.x                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 5.x-68e8507efb72
Exec   : snappyHexMesh -overwrite
Date   : Jan 24 2020
Time   : 14:48:59
Host   : "solver10"
PID    : 61207
I/O    : uncollated
Case   : /home/rob/cases/incompressible/simpleFoam/motorBike
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Read mesh in = 0 s

Overall mesh bounding box  : (-5 -4 0) (15 4 8)
Relative tolerance         : 1e-06
Absolute matching distance : 2.29783e-05

--------------------------------------------------------------------------
mpirun was unable to find the specified executable file, and therefore
did not launch the job.  This error was first reported for process
rank 3; it may have occurred for other processes as well.

NOTE: A common cause for this error is misspelling a mpirun command
      line parameter option (remember that mpirun interprets the first
      unrecognized command line token as the executable).

Node:       10.0.0.113
Executable: /opt/openfoam5/platforms/linux64GccDPInt32Opt/bin/snappyHexMesh
--------------------------------------------------------------------------
3 total processes failed to start

I think this has to do with ssh environment sourcing.

No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:    18.04
Codename:    bionic

I am sourcing the correct bashrc file in .bashrc for the user:

source /opt/openfoam5/etc/bashrc

And locally, all the paths and env-vars are working just fine. All the WM stuff and the PATH and etc are there.


If I try to do the following:

ssh rob@10.0.0.113 /opt/openfoam5/platforms/linux64GccDPInt32Opt/bin/snappyHexMesh

I don't get a "file not found" type error, but rather a library missing error, which is undoubtedly due to the missing paths and other environment variables.


So, I'm not entirely sure where to go from here. Any thoughts?


Trying to pass environment variables with PermitUesrEnv / AcceptEnv /SendEnv didn't quite seem to do the trick (couldn't get PATH working right).


Trying to use .profile/.bash_profile/etc to source also didn't seem to work


PermitUserRC and trying to source the bashrc resulted in other weird problems (for example -- the source command couldn't be found)....


I'm kind of out of thoughts here...