Source FE3 bashrc on Parallel Run
 

I've recently compiled foam-extend-3.0 from source and distributed to a cluster where we have different versions of OpenFOAM already installed from the Ubuntu repo. We've been able to run any version of OF with no issue by sourcing the appropriate version's bashrc on the host and the appropriate arguments are passed to load that version on the slave machines when mpirun is called.

However, this is not occuring in foam-extend. I have to manually source the /opt/foam-extend-3.0/etc/bashrc in the slave machines' $HOME/.bashrc on the first line; however, this prevents us from running other versions of OF since every mpirun call automatically sources foam-extend on the slaves (overwriting the mpirun-called version sourcing).

1. Is there a settings file I need to modify in foam-extend-3.0 to source the bashrc upon mpirun call? Or is there a way of passing the source command during the mpirun call?

2. On a semi related note, foamJob does not appear to work with foam-extend-3.0, with an error thrown:

Thank you for any help.

Further information:

I can run both single-core with icoFoam and in parallel on a local machine with

However, when I run with a machines file pointing to an external node

foam-extend-3.0 fails to find the executable:

I've tried installing the pre-compiled foam-extend-3.0 binaries as well, but there is no change in the behaviour. I've tried passing environment variables through the -x argument in mpirun to no avail. I can't piece together the behaviour difference between our openfoam installations and the foam-extend installation to determine why the former run flawlessly and the latter won't run at all in parallel.

Revisiting this issue as it remains unsolved:

Checking the environment variables for OF221, it appears that SYSTEMOPENMPI is set for $WM_MPLIB while for FE3 OPENMPI is set. $MPI_ARCH_PATH points to the ThirdParty package implementation of MPI for FE3. Now, I'm fine with using the newer version of OpenMPI but why will the environment variables not properly source under this version? The file structure, paths, users, hardware -- everything is identical across nodes. The /usr/lib/openmpi implementation in OF211/221/230/etc. works fine...

I cannot fathom me being the only person with this problem, though, and I would appreciate someone with a bit more knowledge helping me out.

Hi

I have the same problem like you !! OF2.3 is working fine with mpi,foamJob over Infinband. But 3.1 isn't.

Br

Christian

I still haven't been able to get it working, unfortunately. Would be nice to get some input from someone with more expertise on OpenMPI or this particular issue. I'll let you know if I ever figure it out, Kojote.

Greetings to all!

I should be able to help, but I need more information, namely:

1. What were the exact installation instructions that were followed by each one of you?
2. @cdm: Any chance you can try installing the latest foam-extend 3.1 instead of 3.0?

Regarding the first post, it seems that foamExec or foamJob in foam-extend 3.0 is broken. Which is why I'm suggesting that you try installing 3.1, which with any luck has this issue already fixed.

In addition to the above, try running with the following command:
Best regards,
Bruno

Hi Bruno,

Thank you for taking an interest in this issue.

I have recently been trying to get foam-extend-3.1 to build using the instructions here; however, no more luck than with fe3. I still can't get parallel to work. I should note I've attempted to compile on multiple machines and multiple methods, including compiling in the local user directory and running from there, compiling in the local user directory and moving the files to /opt/ and changing permissions, and compiling as root in /opt/foam/ directly (all approaches using default ThirdParty OPENMPI settings). None of these methods differed in terms of correcting the mpi issues described above. If you have any ideas, please let me know.


Anyways, I have compiled this morning under root in the /opt/ directory with the following prefs.sh settings (to use SYSTEMOPENMPI instead):

However, when I run:

I get the following:
Followed by the verbose output of the call to settings.sh and aliases.sh.

Using mpirun as you suggest above yields similar results:

I get the error:

Again followed by verbose output of settings.sh and aliases.sh along with a couple more calls to the bashrc.

I tried running ./bin/foamInstallationTest and all was well except a warning regarding conflicting gcc versions, but I'm not sure this is relevant to the issue at hand since everything compiles and runs fine locally.

Hi cdm,

OK, I've finally managed to look into this in detail, report the issue and push a fix and respective merge request:

• Bug report: https://sourceforge.net/p/openfoam-e...ndrelease/252/
• Merge request: https://sourceforge.net/p/openfoam-e...e-requests/11/

Now... how am I going to explain how you can pull this fix?..
OK, the "simplest" way is to rely on git, so please follow these steps:
The first command is an alias command and should jump to the main folder for the foam-extend source code.
This is the quickest and will solve most of the problems.

But there is one more fix that has to be manually fixed:

1. Edit this file:
   
   Code:

   ---

   ThirdParty/packages/openmpi-1.6.5/platforms/linux64GccDPOpt/etc/openmpi-1.6.5.sh

   ---

2. Change these two lines:
   
   Code:

   ---

   export PINC=$OPENMPI_COMPILE_FLAGS
export PLIBS=$OPENMPI_LINK_FLAGS

   ---

3. To this:
   
   Code:

   ---

   export PINC="${OPENMPI_COMPILE_FLAGS}"
export PLIBS="${OPENMPI_LINK_FLAGS}"

   ---

4. Save the modified file.

And that's it! All patched up :D
Both foamExec and foamJob should now work as intended. You won't even have to start a new terminal :D

Best regards,
Bruno

Bruno,

I pulled the fix from your git and modified the file as outlined above, then distributed this copy to two machines. I'm still receiving the following error upon running foamJob or mpirun with a hostfile specified:

I can still run on either machine locally with:

Appears to be similar to the issue mentioned in the ticket you created regarding the shell. Any ideas?

Hi cdm,

It looks to me like that something went very wrong with the "git pull" command :( If you still are getting that error, then that means that the changes done through git were not applied correctly.

Perhaps you only applied the last change from my repository/branch? There are 3 commits needed from there, not just the last one.

If the commands had worked as intended, then running this command should give you the complete change list:
Attached is the full change list that the command above should give you.

If you prefer to apply the attached changes, you can use this command (after unpacking):
If you prefer, you can download the files that need to be fixed from the following locations:

• "bin/foamExec": http://sourceforge.net/u/wyldckat/fo...e/bin/foamExec
• "etc/bashrc": http://sourceforge.net/u/wyldckat/fo...ree/etc/bashrc

Best regards,
Bruno