CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Installation

Source FE3 bashrc on Parallel Run

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree1Likes
  • 1 Post By wyldckat

Reply
 
LinkBack Thread Tools Display Modes
Old   June 17, 2014, 12:37
Default Source FE3 bashrc on Parallel Run
  #1
cdm
Member
 
Join Date: May 2013
Location: Canada
Posts: 32
Rep Power: 5
cdm is on a distinguished road
I've recently compiled foam-extend-3.0 from source and distributed to a cluster where we have different versions of OpenFOAM already installed from the Ubuntu repo. We've been able to run any version of OF with no issue by sourcing the appropriate version's bashrc on the host and the appropriate arguments are passed to load that version on the slave machines when mpirun is called.

However, this is not occuring in foam-extend. I have to manually source the /opt/foam-extend-3.0/etc/bashrc in the slave machines' $HOME/.bashrc on the first line; however, this prevents us from running other versions of OF since every mpirun call automatically sources foam-extend on the slaves (overwriting the mpirun-called version sourcing).

1. Is there a settings file I need to modify in foam-extend-3.0 to source the bashrc upon mpirun call? Or is there a way of passing the source command during the mpirun call?

2. On a semi related note, foamJob does not appear to work with foam-extend-3.0, with an error thrown:

Code:
/opt/foam-extend-3.0/bin/foamExec: 398: /opt/foam-extend-3.0/etc/bashrc: [[: not found
/opt/foam-extend-3.0/bin/foamExec: 21: export: -pthread: bad variable name
Thank you for any help.
cdm is offline   Reply With Quote

Old   June 17, 2014, 16:04
Default
  #2
cdm
Member
 
Join Date: May 2013
Location: Canada
Posts: 32
Rep Power: 5
cdm is on a distinguished road
Further information:

I can run both single-core with icoFoam and in parallel on a local machine with

Code:
mpirun -np 6 icoFoam -parallel
However, when I run with a machines file pointing to an external node

Code:
mpirun --hostfile machines -np 6 icoFoam -parallel
foam-extend-3.0 fails to find the executable:

Code:
--------------------------------------------------------------------------
mpirun was unable to launch the specified application as it could not find an executable:

Executable: icoFoam
Node: cfd16

while attempting to start process rank 0.
--------------------------------------------------------------------------
cdm is offline   Reply With Quote

Old   June 18, 2014, 14:03
Default
  #3
cdm
Member
 
Join Date: May 2013
Location: Canada
Posts: 32
Rep Power: 5
cdm is on a distinguished road
I've tried installing the pre-compiled foam-extend-3.0 binaries as well, but there is no change in the behaviour. I've tried passing environment variables through the -x argument in mpirun to no avail. I can't piece together the behaviour difference between our openfoam installations and the foam-extend installation to determine why the former run flawlessly and the latter won't run at all in parallel.
cdm is offline   Reply With Quote

Old   July 7, 2014, 16:25
Default
  #4
cdm
Member
 
Join Date: May 2013
Location: Canada
Posts: 32
Rep Power: 5
cdm is on a distinguished road
Revisiting this issue as it remains unsolved:

Checking the environment variables for OF221, it appears that SYSTEMOPENMPI is set for $WM_MPLIB while for FE3 OPENMPI is set. $MPI_ARCH_PATH points to the ThirdParty package implementation of MPI for FE3. Now, I'm fine with using the newer version of OpenMPI but why will the environment variables not properly source under this version? The file structure, paths, users, hardware -- everything is identical across nodes. The /usr/lib/openmpi implementation in OF211/221/230/etc. works fine...

I cannot fathom me being the only person with this problem, though, and I would appreciate someone with a bit more knowledge helping me out.
cdm is offline   Reply With Quote

Old   July 25, 2014, 03:55
Default
  #5
New Member
 
Join Date: Aug 2011
Posts: 22
Rep Power: 6
Kojote is on a distinguished road
Quote:
Originally Posted by cdm View Post
Revisiting this issue as it remains unsolved:

Checking the environment variables for OF221, it appears that SYSTEMOPENMPI is set for $WM_MPLIB while for FE3 OPENMPI is set. $MPI_ARCH_PATH points to the ThirdParty package implementation of MPI for FE3. Now, I'm fine with using the newer version of OpenMPI but why will the environment variables not properly source under this version? The file structure, paths, users, hardware -- everything is identical across nodes. The /usr/lib/openmpi implementation in OF211/221/230/etc. works fine...

I cannot fathom me being the only person with this problem, though, and I would appreciate someone with a bit more knowledge helping me out.
Hi

I have the same problem like you !! OF2.3 is working fine with mpi,foamJob over Infinband. But 3.1 isn't.

Br

Christian
Kojote is offline   Reply With Quote

Old   July 25, 2014, 18:41
Default
  #6
cdm
Member
 
Join Date: May 2013
Location: Canada
Posts: 32
Rep Power: 5
cdm is on a distinguished road
I still haven't been able to get it working, unfortunately. Would be nice to get some input from someone with more expertise on OpenMPI or this particular issue. I'll let you know if I ever figure it out, Kojote.
cdm is offline   Reply With Quote

Old   August 5, 2014, 13:21
Default
  #7
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,531
Blog Entries: 39
Rep Power: 97
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings to all!

I should be able to help, but I need more information, namely:
  1. What were the exact installation instructions that were followed by each one of you?
  2. @cdm: Any chance you can try installing the latest foam-extend 3.1 instead of 3.0?
Regarding the first post, it seems that foamExec or foamJob in foam-extend 3.0 is broken. Which is why I'm suggesting that you try installing 3.1, which with any luck has this issue already fixed.

In addition to the above, try running with the following command:
Code:
mpirun --hostfile machines -np 6 /opt/foam-extend-3.0/bin/foamExec icoFoam -parallel
Best regards,
Bruno
__________________
wyldckat is offline   Reply With Quote

Old   August 5, 2014, 13:48
Default
  #8
cdm
Member
 
Join Date: May 2013
Location: Canada
Posts: 32
Rep Power: 5
cdm is on a distinguished road
Hi Bruno,

Thank you for taking an interest in this issue.

I have recently been trying to get foam-extend-3.1 to build using the instructions here; however, no more luck than with fe3. I still can't get parallel to work. I should note I've attempted to compile on multiple machines and multiple methods, including compiling in the local user directory and running from there, compiling in the local user directory and moving the files to /opt/ and changing permissions, and compiling as root in /opt/foam/ directly (all approaches using default ThirdParty OPENMPI settings). None of these methods differed in terms of correcting the mpi issues described above. If you have any ideas, please let me know.


Anyways, I have compiled this morning under root in the /opt/ directory with the following prefs.sh settings (to use SYSTEMOPENMPI instead):

Code:
FOAM_VERBOSE=1
compilerInstall=System
export WM_MPLIB=SYSTEMOPENMPI
export QT_THIRD_PARTY=1
export QT_BIN_DIR=/usr/bin
However, when I run:

Code:
foamJob -p -s icoFoam
I get the following:
Code:
Parallel processing using SYSTEMOPENMPI with 6 processors
Executing: mpirun -np 6 /opt/foam/foam-extend-3.1/bin/foamExec icoFoam -parallel | tee log
Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh
Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh
Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh
Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh
Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh
Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
Sourcing: /opt/foam/foam-extend-3.1/etc/settings.sh
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
Sourcing: /opt/foam/foam-extend-3.1/etc/settings.sh
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
Sourcing: /opt/foam/foam-extend-3.1/etc/settings.sh
Followed by the verbose output of the call to settings.sh and aliases.sh.

Using mpirun as you suggest above yields similar results:

Code:
mpirun --hostfile machines -np 6 /opt/foam/foam-extend-3.1/bin/foamExec icoFoam -parallel
I get the error:

Code:
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
Sourcing: /opt/foam/foam-extend-3.1/etc/settings.sh
Again followed by verbose output of settings.sh and aliases.sh along with a couple more calls to the bashrc.

I tried running ./bin/foamInstallationTest and all was well except a warning regarding conflicting gcc versions, but I'm not sure this is relevant to the issue at hand since everything compiles and runs fine locally.

Code:
Third party software
-------------------------------------------------------------------------------
Software Version   Location 
-------------------------------------------------------------------------------
WARNING: gcc version does not match gcc supplied with this release of foam-extend
         Supplied version: 4.3.3
         User version    : 4.6.3
         Minimum required: 4.3.1

gcc      4.6.3    
WARNING:  Conflicting installations:
          foam-extend settings        : /bin/gcc
          current path             : /usr/bin/gcc
          CRITICAL ERROR
cdm is offline   Reply With Quote

Old   August 11, 2014, 14:52
Default
  #9
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,531
Blog Entries: 39
Rep Power: 97
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Hi cdm,

OK, I've finally managed to look into this in detail, report the issue and push a fix and respective merge request:
Now... how am I going to explain how you can pull this fix?..
OK, the "simplest" way is to rely on git, so please follow these steps:
Code:
foam
git remote add wyldckat git://git.code.sf.net/u/wyldckat/foam-extend-3.1
git pull wyldckat BUGFIX/Issue252
The first command is an alias command and should jump to the main folder for the foam-extend source code.
This is the quickest and will solve most of the problems.

But there is one more fix that has to be manually fixed:
  1. Edit this file:
    Code:
    ThirdParty/packages/openmpi-1.6.5/platforms/linux64GccDPOpt/etc/openmpi-1.6.5.sh
  2. Change these two lines:
    Code:
    export PINC=$OPENMPI_COMPILE_FLAGS
    export PLIBS=$OPENMPI_LINK_FLAGS
  3. To this:
    Code:
    export PINC="${OPENMPI_COMPILE_FLAGS}"
    export PLIBS="${OPENMPI_LINK_FLAGS}"
  4. Save the modified file.
And that's it! All patched up
Both foamExec and foamJob should now work as intended. You won't even have to start a new terminal

Best regards,
Bruno
__________________
wyldckat is offline   Reply With Quote

Old   August 11, 2014, 16:04
Default
  #10
cdm
Member
 
Join Date: May 2013
Location: Canada
Posts: 32
Rep Power: 5
cdm is on a distinguished road
Bruno,

I pulled the fix from your git and modified the file as outlined above, then distributed this copy to two machines. I'm still receiving the following error upon running foamJob or mpirun with a hostfile specified:

Code:
Parallel processing using OPENMPI with 6 processors
Executing: mpirun -np 6 -hostfile machines /opt/foam/foam-extend-3.1/bin/foamExec icoFoam -parallel | tee log
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
I can still run on either machine locally with:

Code:
mpirun -np 6 icoFoam -parallel | tee log
Appears to be similar to the issue mentioned in the ticket you created regarding the shell. Any ideas?

Last edited by cdm; August 11, 2014 at 17:09.
cdm is offline   Reply With Quote

Old   August 12, 2014, 14:35
Default
  #11
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 9,531
Blog Entries: 39
Rep Power: 97
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Hi cdm,

Quote:
Originally Posted by cdm View Post
Code:
Parallel processing using OPENMPI with 6 processors
Executing: mpirun -np 6 -hostfile machines /opt/foam/foam-extend-3.1/bin/foamExec icoFoam -parallel | tee log
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found
[...]

Appears to be similar to the issue mentioned in the ticket you created regarding the shell. Any ideas?
It looks to me like that something went very wrong with the "git pull" command If you still are getting that error, then that means that the changes done through git were not applied correctly.

Perhaps you only applied the last change from my repository/branch? There are 3 commits needed from there, not just the last one.

If the commands had worked as intended, then running this command should give you the complete change list:
Code:
git diff origin/master
Attached is the full change list that the command above should give you.

If you prefer to apply the attached changes, you can use this command (after unpacking):
Code:
patch -p1 < changes.patch
If you prefer, you can download the files that need to be fixed from the following locations:
Best regards,
Bruno
Attached Files
File Type: zip changes.patch.zip (1.4 KB, 3 views)
ashim likes this.
wyldckat is offline   Reply With Quote

Reply

Tags
bashrc, foam-extend-3.0, mpirun, ubuntu 12.04

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
what is swap4foam ?? AB08 OpenFOAM 28 February 2, 2016 02:22
Superlinear speedup in OpenFOAM 13 msrinath80 OpenFOAM Running, Solving & CFD 18 March 3, 2015 06:36
Cannot get refineMesh to run in parallel smschnob OpenFOAM Meshing & Mesh Conversion 2 June 3, 2014 11:20
Help for the small implementation in turbulence model shipman OpenFOAM Programming & Development 25 March 19, 2014 11:08
Version 15 on Mac OS X gschaider OpenFOAM Installation 120 December 2, 2009 11:23


All times are GMT -4. The time now is 08:44.