|
[Sponsors] |
June 17, 2014, 13:37 |
Source FE3 bashrc on Parallel Run
|
#1 |
Member
Join Date: May 2013
Location: Canada
Posts: 32
Rep Power: 13 |
I've recently compiled foam-extend-3.0 from source and distributed to a cluster where we have different versions of OpenFOAM already installed from the Ubuntu repo. We've been able to run any version of OF with no issue by sourcing the appropriate version's bashrc on the host and the appropriate arguments are passed to load that version on the slave machines when mpirun is called.
However, this is not occuring in foam-extend. I have to manually source the /opt/foam-extend-3.0/etc/bashrc in the slave machines' $HOME/.bashrc on the first line; however, this prevents us from running other versions of OF since every mpirun call automatically sources foam-extend on the slaves (overwriting the mpirun-called version sourcing). 1. Is there a settings file I need to modify in foam-extend-3.0 to source the bashrc upon mpirun call? Or is there a way of passing the source command during the mpirun call? 2. On a semi related note, foamJob does not appear to work with foam-extend-3.0, with an error thrown: Code:
/opt/foam-extend-3.0/bin/foamExec: 398: /opt/foam-extend-3.0/etc/bashrc: [[: not found /opt/foam-extend-3.0/bin/foamExec: 21: export: -pthread: bad variable name |
|
June 17, 2014, 17:04 |
|
#2 |
Member
Join Date: May 2013
Location: Canada
Posts: 32
Rep Power: 13 |
Further information:
I can run both single-core with icoFoam and in parallel on a local machine with Code:
mpirun -np 6 icoFoam -parallel Code:
mpirun --hostfile machines -np 6 icoFoam -parallel Code:
-------------------------------------------------------------------------- mpirun was unable to launch the specified application as it could not find an executable: Executable: icoFoam Node: cfd16 while attempting to start process rank 0. -------------------------------------------------------------------------- |
|
June 18, 2014, 15:03 |
|
#3 |
Member
Join Date: May 2013
Location: Canada
Posts: 32
Rep Power: 13 |
I've tried installing the pre-compiled foam-extend-3.0 binaries as well, but there is no change in the behaviour. I've tried passing environment variables through the -x argument in mpirun to no avail. I can't piece together the behaviour difference between our openfoam installations and the foam-extend installation to determine why the former run flawlessly and the latter won't run at all in parallel.
|
|
July 7, 2014, 17:25 |
|
#4 |
Member
Join Date: May 2013
Location: Canada
Posts: 32
Rep Power: 13 |
Revisiting this issue as it remains unsolved:
Checking the environment variables for OF221, it appears that SYSTEMOPENMPI is set for $WM_MPLIB while for FE3 OPENMPI is set. $MPI_ARCH_PATH points to the ThirdParty package implementation of MPI for FE3. Now, I'm fine with using the newer version of OpenMPI but why will the environment variables not properly source under this version? The file structure, paths, users, hardware -- everything is identical across nodes. The /usr/lib/openmpi implementation in OF211/221/230/etc. works fine... I cannot fathom me being the only person with this problem, though, and I would appreciate someone with a bit more knowledge helping me out. |
|
July 25, 2014, 04:55 |
|
#5 | |
Member
Join Date: Aug 2011
Posts: 37
Rep Power: 15 |
Quote:
I have the same problem like you !! OF2.3 is working fine with mpi,foamJob over Infinband. But 3.1 isn't. Br Christian |
||
July 25, 2014, 19:41 |
|
#6 |
Member
Join Date: May 2013
Location: Canada
Posts: 32
Rep Power: 13 |
I still haven't been able to get it working, unfortunately. Would be nice to get some input from someone with more expertise on OpenMPI or this particular issue. I'll let you know if I ever figure it out, Kojote.
|
|
August 5, 2014, 14:21 |
|
#7 |
Retired Super Moderator
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,981
Blog Entries: 45
Rep Power: 128 |
Greetings to all!
I should be able to help, but I need more information, namely:
In addition to the above, try running with the following command: Code:
mpirun --hostfile machines -np 6 /opt/foam-extend-3.0/bin/foamExec icoFoam -parallel Bruno
__________________
|
|
August 5, 2014, 14:48 |
|
#8 |
Member
Join Date: May 2013
Location: Canada
Posts: 32
Rep Power: 13 |
Hi Bruno,
Thank you for taking an interest in this issue. I have recently been trying to get foam-extend-3.1 to build using the instructions here; however, no more luck than with fe3. I still can't get parallel to work. I should note I've attempted to compile on multiple machines and multiple methods, including compiling in the local user directory and running from there, compiling in the local user directory and moving the files to /opt/ and changing permissions, and compiling as root in /opt/foam/ directly (all approaches using default ThirdParty OPENMPI settings). None of these methods differed in terms of correcting the mpi issues described above. If you have any ideas, please let me know. Anyways, I have compiled this morning under root in the /opt/ directory with the following prefs.sh settings (to use SYSTEMOPENMPI instead): Code:
FOAM_VERBOSE=1 compilerInstall=System export WM_MPLIB=SYSTEMOPENMPI export QT_THIRD_PARTY=1 export QT_BIN_DIR=/usr/bin Code:
foamJob -p -s icoFoam Code:
Parallel processing using SYSTEMOPENMPI with 6 processors Executing: mpirun -np 6 /opt/foam/foam-extend-3.1/bin/foamExec icoFoam -parallel | tee log Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh Sourcing: /opt/foam/foam-extend-3.1/etc/prefs.sh /opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found /opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found /opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found Sourcing: /opt/foam/foam-extend-3.1/etc/settings.sh /opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found Sourcing: /opt/foam/foam-extend-3.1/etc/settings.sh /opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found /opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found Sourcing: /opt/foam/foam-extend-3.1/etc/settings.sh Using mpirun as you suggest above yields similar results: Code:
mpirun --hostfile machines -np 6 /opt/foam/foam-extend-3.1/bin/foamExec icoFoam -parallel Code:
/opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found /opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found Sourcing: /opt/foam/foam-extend-3.1/etc/settings.sh I tried running ./bin/foamInstallationTest and all was well except a warning regarding conflicting gcc versions, but I'm not sure this is relevant to the issue at hand since everything compiles and runs fine locally. Code:
Third party software ------------------------------------------------------------------------------- Software Version Location ------------------------------------------------------------------------------- WARNING: gcc version does not match gcc supplied with this release of foam-extend Supplied version: 4.3.3 User version : 4.6.3 Minimum required: 4.3.1 gcc 4.6.3 WARNING: Conflicting installations: foam-extend settings : /bin/gcc current path : /usr/bin/gcc CRITICAL ERROR |
|
August 11, 2014, 15:52 |
|
#9 |
Retired Super Moderator
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,981
Blog Entries: 45
Rep Power: 128 |
Hi cdm,
OK, I've finally managed to look into this in detail, report the issue and push a fix and respective merge request:
OK, the "simplest" way is to rely on git, so please follow these steps: Code:
foam git remote add wyldckat git://git.code.sf.net/u/wyldckat/foam-extend-3.1 git pull wyldckat BUGFIX/Issue252 This is the quickest and will solve most of the problems. But there is one more fix that has to be manually fixed:
Both foamExec and foamJob should now work as intended. You won't even have to start a new terminal Best regards, Bruno
__________________
|
|
August 11, 2014, 17:04 |
|
#10 |
Member
Join Date: May 2013
Location: Canada
Posts: 32
Rep Power: 13 |
Bruno,
I pulled the fix from your git and modified the file as outlined above, then distributed this copy to two machines. I'm still receiving the following error upon running foamJob or mpirun with a hostfile specified: Code:
Parallel processing using OPENMPI with 6 processors Executing: mpirun -np 6 -hostfile machines /opt/foam/foam-extend-3.1/bin/foamExec icoFoam -parallel | tee log /opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found /opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found /opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found /opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found /opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found /opt/foam/foam-extend-3.1/bin/foamExec: 400: /opt/foam/foam-extend-3.1/etc/bashrc: [[: not found Code:
mpirun -np 6 icoFoam -parallel | tee log Last edited by cdm; August 11, 2014 at 18:09. |
|
August 12, 2014, 15:35 |
|
#11 | |
Retired Super Moderator
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,981
Blog Entries: 45
Rep Power: 128 |
Hi cdm,
Quote:
Perhaps you only applied the last change from my repository/branch? There are 3 commits needed from there, not just the last one. If the commands had worked as intended, then running this command should give you the complete change list: Code:
git diff origin/master If you prefer to apply the attached changes, you can use this command (after unpacking): Code:
patch -p1 < changes.patch
Bruno |
||
Tags |
bashrc, foam-extend-3.0, mpirun, ubuntu 12.04 |
Thread Tools | Search this Thread |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
what is swap4foam ?? | AB08 | OpenFOAM | 28 | February 2, 2016 02:22 |
Superlinear speedup in OpenFOAM 13 | msrinath80 | OpenFOAM Running, Solving & CFD | 18 | March 3, 2015 06:36 |
[mesh manipulation] Cannot get refineMesh to run in parallel | smschnob | OpenFOAM Meshing & Mesh Conversion | 2 | June 3, 2014 12:20 |
Help for the small implementation in turbulence model | shipman | OpenFOAM Programming & Development | 25 | March 19, 2014 11:08 |
Version 15 on Mac OS X | gschaider | OpenFOAM Installation | 113 | December 2, 2009 11:23 |