Non-root installation of OpenFOAM 2.4.x, parallel issue
 

Hi Foamers,

I have been trying to install OF 2.4.x on a CentOS 7 cluster. It was fine in serial but when I try to submit a job in parallel I get the following error:

To be indicated, I intend to use system gcc as the compiler. I use the following commands to submit the job:

To have access to OpenMPI, I load a gcc module and OpenMPI is loaded as the result:

I guess that OpenFOAM cannot find the system mpi on the cluster so it complains accordingly. I hope someone had similar experience with this issue and could help me out.

Regards,
Syavash

Quick answer: Given that OpenFOAM's "etc/bashrc" file is being used in the job script, then you also need to load in the job script the module for Open-MPI, before sourcing/activating OpenFOAM's shell environment.

Dear Bruno,

Thanks for your reply. I tried to include the following line in the job script:

The above line was added prior to However, I still get the same error after submitting the job. My question is that am I loading Open-MPI in a correct way? Could you help me to figure it out?

Regards,
Syavash

Hello Ehsan


How exactly did you install OF, using openmpi module in the cluster or the openmpi shipped with OF?


If you want to stick to the cluster's OMPI (which is the right way usually), then be sure that you made the required modifications in the OF's bashrc file. To check if everything is built correctly, you can have a look at the $FOAM_LIBBIN contents as well. Also, I'd suggest ldd tool to check if dependencies can be resolved.

Dear Fatih,

Thanks for your attention. Honestly, I am not so sure, however I intend to use system default gcc/open-mpi modules. I followed the steps described in the following link:

I skipped the first three steps. I also replaced the command in step 8 with the one below:

The above command outputs:

Please let me know if you need further information to diagnose the problem. I am confused now!

Regards,
Syavash

Edit: when I enter the command below I am redirected to the following address:

All right, I think the problem is the module you loaded doesn't actually load openmpi environment. It seems like it is a compiler module to access actual compilers (not just runtime libs) as well as to load the other modules built with that specific gcc version.

To check whether you have ompi in your environment, you can run something like: "which mpirun". Since I don't know how your cluster is configured you might want to check mpi compilers as well with "which mpicc". If you don't have those binaries located somewhere then you need to load the specific openmpi module and re-start the compilation.

If you have them in your environment and also if you're sure that built was completed using correct openmpi modules, then your job script can be wrong as well.

So in general:
== Before initiating the installation load correct modules of gcc and openmpi. Rest is optional (boost, metis etc.). After loading them, make sure they are loaded properly. You can use commands like which, module show etc.
== Check OF path is correctly defined in the etc/bashrc file (e.g. $HOME/OpenFOAM)
== Make sure "export WM_MPLIB=SYSTEMOPENMPI" is defined in OF/etc/bashrc when you use the ompi module
== After the installation check if mpi directories exist (e.g. openmpi-system) under the $FOAM_LIBBIN. So just being able to cd there doesn't mean much.
== To run solvers, meshers etc. load the same modules required for build
== Don't forget to source OF/etc/bashrc in your scripts
== Don't put an alias for it since multi-node jobs need to source it by default

Good luck

// Fatih

Dear Fatih,

Thanks for your elaboration. Well, I used "which mpirun" and "which mpicc" and I got the following outputs:

WM_MPLIB=SYSTEMOPENMPI is OK in etc/bashrc.

Also, inside $FOAM_LIBBIN I have "openmpi-system" directory which contains the following objects:

I am not sure of anything else which might be of importance. Please let me know if you (Bruno as well :)) had any other idea on how to resolve this issue.

Regards,
Syavash

Quick answer @syavash: Why can't you add this line:
before this one in the job script:
Does the case not launch with that?

The other hypothesis is to check if this module will in fact load other modules. If you start a new terminal and run:
what does it give you?

Dear Bruno,

Thanks for your reply.

I had added the following alias for OF which I used to call it before submitting the job:

So I excluded the buildenv-gcc/2018b-eb module in the job script. Nevertheless, adding the line before the line did not make any changes and the error is still there.

Here is the output for :

and the output for :

Any idea? :confused:

Regards,
Syavash

Quick answer: If using "module load" doesn't work within a job, then it's because it's not able to properly load it... or because mpprun needs to be used properly.

If the cluster/supercomputer you are using has a support page, you should consult it and tell us about it if you cannot understand it!
I did a quick search online for mpprun and found this page: https://www.nsc.liu.se/support/tutorials/mpprun/

Apparently the error message:
is because the application itself does not have the tag needed for mpprun to assess which modules to load.

So if you run in the login node this command:
you should get a similar message stating that the tag was not found.

Apparently you must use the "-Nmpi" additional compilation argument, as documented here: https://www.nsc.liu.se/software/buildenv/ and here: https://www.nsc.liu.se/software/mpi-libraries/

Therefore, if my estimates are correct:

1. Edit the file "OpenFOAM-2.4.x/wmake/rules/linux64Gcc/c++Opt".
2. Add the entry "-Nmpi" to "c++OPT", e.g.:
   
   Code:

   ---

   c++OPT      = -O3 -Nmpi

   ---

3. And then you have to rebuild OpenFOAM 2.4.x entirely, because this extra build option has to be added to all object files during compiling...
4. Then again, you could try only rebuilding only pimpleFoam and hope it's enough:
   
   Code:

   ---

   wclean $FOAM_SOLVERS/incompressible/pimpleFoam
wmake $FOAM_SOLVERS/incompressible/pimpleFoam

   ---

5. Then check if it's tagged properly:
   
   Code:

   ---

   dumptag $(which pimpleFoam)

   ---

   which should tell you several details, including a line similar to this one:
   
   Code:

   ---

   Built with MPI:        openmpi 1_6_2_build1

   ---

Beyond this, using "module load" should not be necessary in the job script.

Dear Bruno, I see your point, however I should say that I never came across the support page! I searched the error message in Google only but found little help. I know it might look weird but I was confused and missed the support web page, for that I am sorry.

I should also thank you again as your thorough elaboration made the job run in parallel. As you instructed I did a fresh installation, this time by including -Nmpi flag.

Regards,
Syavash

Quick answer: I'm very glad that it worked!

Mmm... it is possible that Google was bias towards me, given my search profile, even though I wasn't logged in...

Then again, when i searched for "Could not retrieve MPI tag from" without the quotes, it didn't give me anything, but with the quotes, it did give me just one answer: http://www.tfd.chalmers.se/~hani/wik.../_Installation - namely one of Håkan Nilsson's wiki pages, but I didn't understand very well the context either, since his solution was to simply use Intel MPI... I did have to research a bit more and happened to look for "mpprun" and started to read through things a bit more carefully.

But I was also expecting that the cluster/supercomputer you are using, would have an instructions page... but I'm glad that NSC has a fairly complete instructions page and managed to help us out!

Dear Bruno,

You are right about Håkan's page, as I had installed OF 2.3.1 based on his guide and had used Intel as the compiler. However, Intel compiler would give me some issues when I wanted to use mapFields or when trying to extract lines of data by sampleDict. I had previously installed OF 2.3.1 on another cluster with Gcc (though an older version) and I did not have those issues, so I decided to install OF 2.4.x with Gcc this time, hoping that it does not complain with mapFields anymore.
Also, I examined NSC page. As you indicated they have provided a fairly helpful support page so I will check it out before bringing up the issue here! :)

Regards,
Syavash