CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Installation

[OpenFOAM.org] [GUIDE] How to compile OpenFOAM source on a cluster/server with no admin rights

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree3Likes
  • 3 Post By gabrielfelix

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   September 6, 2021, 15:14
Smile [GUIDE] How to compile OpenFOAM source on a cluster/server with no admin rights
  #1
Member
 
Gabriel Felix
Join Date: May 2021
Location: Brazil
Posts: 30
Rep Power: 4
gabrielfelix is on a distinguished road
Hi folks,

I've recently gained access to my university cluster, HPC or server (call it as you prefer) and I wanted to run my OpenFOAM simulations on it.

In these clusters you usually do not have any administrator rights so you can't install precompiled builds of OF as you do in your personal computer. Therefore, you will need to download and compile the OpenFOAM source code on the server.

I took almost two weeks to succesfully compile OF on the cluster. I have read almost every thread about the subject here on CFD-Online comunity and just a few of them were actually useful to apply in my case. I'll try to compile here in this thread all the information I could gather among these threads and I hope that it helps you if you find yourself in the same situation I was.

What I did initially was to follow the OpenFoam.org official instructions constained here. However, my compilation crashed/broke on a mpi.h error.

When I set the ./Allwmake -k flag (according to this thread
[OpenFOAM.com] Compiling v2006, mpicc -- broken), the compilation skipped the mpi.h, and succesfully compiled the other packages. I could actually run a test case such as pitzDaily with simpleFoam, but I could not run it in parallel as it needs the library OpenMPI.

I started then to research about the OpenMPI installation and it seems that it is the error that most people get. The clusters usually have different versions of OpenMPI or it can even not be intalled on the machine. In most of the threads people said to modify the WM_MPLIB flag on etc/bashrc from SYSTEMOPENMPI to OPENMPI and other ones. However you will always get the same error because the MPI might not be running on the server. You will need to discover which version is installed on the server and then you will have to load it.

After loading the OpenMPI module, I stopped getting the mpi.h error and compilation advanced until it crashed again in a ld.gold error, which I managed to solve through steps 1 to 4 of this thread Compiling OpenFoam from git : problem with ld. Compilation finally finished. However, when I tried to decompose the case with decomposePar, I got errors again.

After many tries, I decided to delete this compilation and start a fresh-new one because I have done many modifications in etc/bashrc and Allwmake and these modifications could have introduced some user-induced errors to the process. To delete the compilation you just got to delete the opt/OpenFoam folder and unpack the .tgz file again to the same directory.

Until here I was always using the default "system" compiler and a user in this thread
[OpenFOAM.org] OpenFOAM 2.3.0 on CentOS 6.5 suggested to use the "ThirdParty" compiler and modify the WM_MPLIB to OPENMPI. I loaded the OpenMPI module and made the modifications, however I still got errors. Then I only modified the compiler and not the WM_MPLIB and it worked! There finally were no errors during compilation and it succesfully finished compiling all packages.

So, in order to succesfully compile OF on the server I followed the steps above. I hope that this set up works for you!

1. Download the source code in your client PC and transfer to the server

Code:
https://dl.openfoam.com/source/v2012/OpenFOAM-v2012.tgz
Code:
scp -P <port> ~/OpenFOAM-v2012-windows10.tgz <your-user>@XXX.XX.XX.XX:$HOME/
You might not need to specify the port as I did.

2. Unpack

Code:
tar -xvzf -C OpenFOAM-v2012.tgz $HOME/opt/
3. Identify which OpenMPI version is/are installed on the cluster

Code:
module spider openmpi
If there is any OpenMPI installed in the machine you will probably need to compile it yourself. There are some threads on the forum to help you.

Code:
$ module spider openmpi

--------------------------------------------------------------------------------------------------------------------
  OpenMPI:
--------------------------------------------------------------------------------------------------------------------
    Description:
      The Open MPI Project is an open source MPI-3 implementation.

     Versions:
        OpenMPI/3.1.1-GCC-7.3.0-2.30
        OpenMPI/3.1.3-GCC-8.2.0-2.31.1
        OpenMPI/3.1.4-GCC-8.3.0
        OpenMPI/4.0.5-GCC-10.2.0

--------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "OpenMPI" module (including how to load the modules) use the module's full name.
  For example:

     $ module spider OpenMPI/4.0.5-GCC-10.2.0
--------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------------------------------
  openmpi/3.1.3: openmpi/3.1.3/2019
--------------------------------------------------------------------------------------------------------------------

    This module can be loaded directly: module load openmpi/3.1.3/2019
4. Load the version you want. I chose the latest that was shown in step 3

Copy exactly the name of the version that appears in the message returned in step 3.

Code:
module load OpenMPI/4.0.5-GCC-10.2.0 ||  export PATH=$PATH:/usr/lib64/openmpi/bin
5. Edit etc/bashrc parameters

Code:
WM_COMPILER_TYPE=ThirdParty
Do not change the WM_MPLIB=SYSTEMOPENMPI parameter

6. Source etc/bashrc

Code:
source etc/bashrc
7. Execute compilator

You can run it in parallel with the -j flag. Change the number of processors according to your machine. You don't need to use the maximum number of processors available. Use just a few.

Code:
./Allwmake -j 6
8. Test the installation

Code:
foamInstallationTest
Run a case such as pitzDaily

Code:
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily $HOME/
cd $HOME/pitzDaily
blockMesh
simpleFoam
Run in parallel.
Create a decomposeParDict similar to the other tutorials

Code:
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v2012                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 5;

method          hierarchical;

coeffs
{
    n           (1 5 1);
}
Code:
blockMesh
decomposePar
mpirun -np 5 simpleFoam -parallel
or
mpiexec -np 5 simpleFoam -parallel
reconstructPar
Change the number of processors according to your machine

9. Load OpenMPI and OF automatically

You will need to modify the server bashrc to load the OF and the OpenMPI whenever you login in to it so you won't have to load it manually every time.

Add these lines to the end of ~/.bashrc
Code:
alias of2012='alias of2012='module load OpenMPI/4.0.5-GCC-10.2.0 || export PATH=$PATH:/usr/lib64/openmpi/bin; source /home70/gfelix/opt/OpenFOAM-v2012/etc/bashrc $FOAM_SETTINGS'
When you login again just type
Code:
of2012
to load it.

Now the OF is ready to run your cases. You just gotta transfer your run files with scp as done in step 1.

NOTE: You might want to set up a passwordless login in order to avoid having to type your user and password everytime you login to the server. Just follow this guide How to Setup Passwordless SSH Login

NOTE 2: You can check the information about the cluster's nodes and number of processors with the following commands:
Code:
Number of processors (CPU)
	grep -c ^processor /proc/cpuinfo
Infos about the nodes
	cat /proc/cpuinfo
rmaries, mikulo and MonkeyWithFox like this.

Last edited by gabrielfelix; September 8, 2021 at 09:33.
gabrielfelix is offline   Reply With Quote

Old   July 5, 2022, 06:18
Default
  #2
Member
 
Join Date: Nov 2020
Posts: 51
Rep Power: 4
mikulo is on a distinguished road
Quote:
Originally Posted by gabrielfelix View Post
Hi folks,

I've recently gained access to my university cluster, HPC or server (call it as you prefer) and I wanted to run my OpenFOAM simulations on it.

In these clusters you usually do not have any administrator rights so you can't install precompiled builds of OF as you do in your personal computer. Therefore, you will need to download and compile the OpenFOAM source code on the server.

I took almost two weeks to succesfully compile OF on the cluster. I have read almost every thread about the subject here on CFD-Online comunity and just a few of them were actually useful to apply in my case. I'll try to compile here in this thread all the information I could gather among these threads and I hope that it helps you if you find yourself in the same situation I was.

What I did initially was to follow the OpenFoam.org official instructions constained here. However, my compilation crashed/broke on a mpi.h error.

When I set the ./Allwmake -k flag (according to this thread
[OpenFOAM.com] Compiling v2006, mpicc -- broken), the compilation skipped the mpi.h, and succesfully compiled the other packages. I could actually run a test case such as pitzDaily with simpleFoam, but I could not run it in parallel as it needs the library OpenMPI.

I started then to research about the OpenMPI installation and it seems that it is the error that most people get. The clusters usually have different versions of OpenMPI or it can even not be intalled on the machine. In most of the threads people said to modify the WM_MPLIB flag on etc/bashrc from SYSTEMOPENMPI to OPENMPI and other ones. However you will always get the same error because the MPI might not be running on the server. You will need to discover which version is installed on the server and then you will have to load it.

After loading the OpenMPI module, I stopped getting the mpi.h error and compilation advanced until it crashed again in a ld.gold error, which I managed to solve through steps 1 to 4 of this thread Compiling OpenFoam from git : problem with ld. Compilation finally finished. However, when I tried to decompose the case with decomposePar, I got errors again.

After many tries, I decided to delete this compilation and start a fresh-new one because I have done many modifications in etc/bashrc and Allwmake and these modifications could have introduced some user-induced errors to the process. To delete the compilation you just got to delete the opt/OpenFoam folder and unpack the .tgz file again to the same directory.

Until here I was always using the default "system" compiler and a user in this thread
[OpenFOAM.org] OpenFOAM 2.3.0 on CentOS 6.5 suggested to use the "ThirdParty" compiler and modify the WM_MPLIB to OPENMPI. I loaded the OpenMPI module and made the modifications, however I still got errors. Then I only modified the compiler and not the WM_MPLIB and it worked! There finally were no errors during compilation and it succesfully finished compiling all packages.

So, in order to succesfully compile OF on the server I followed the steps above. I hope that this set up works for you!

1. Download the source code in your client PC and transfer to the server

Code:
https://dl.openfoam.com/source/v2012/OpenFOAM-v2012.tgz
Code:
scp -P <port> ~/OpenFOAM-v2012-windows10.tgz <your-user>@XXX.XX.XX.XX:$HOME/
You might not need to specify the port as I did.

2. Unpack

Code:
tar -xvzf -C OpenFOAM-v2012.tgz $HOME/opt/
3. Identify which OpenMPI version is/are installed on the cluster

Code:
module spider openmpi
If there is any OpenMPI installed in the machine you will probably need to compile it yourself. There are some threads on the forum to help you.

Code:
$ module spider openmpi

--------------------------------------------------------------------------------------------------------------------
  OpenMPI:
--------------------------------------------------------------------------------------------------------------------
    Description:
      The Open MPI Project is an open source MPI-3 implementation.

     Versions:
        OpenMPI/3.1.1-GCC-7.3.0-2.30
        OpenMPI/3.1.3-GCC-8.2.0-2.31.1
        OpenMPI/3.1.4-GCC-8.3.0
        OpenMPI/4.0.5-GCC-10.2.0

--------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "OpenMPI" module (including how to load the modules) use the module's full name.
  For example:

     $ module spider OpenMPI/4.0.5-GCC-10.2.0
--------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------------------------------
  openmpi/3.1.3: openmpi/3.1.3/2019
--------------------------------------------------------------------------------------------------------------------

    This module can be loaded directly: module load openmpi/3.1.3/2019
4. Load the version you want. I chose the latest that was shown in step 3

Copy exactly the name of the version that appears in the message returned in step 3.

Code:
module load OpenMPI/4.0.5-GCC-10.2.0 ||  export PATH=$PATH:/usr/lib64/openmpi/bin
5. Edit etc/bashrc parameters

Code:
WM_COMPILER_TYPE=ThirdParty
Do not change the WM_MPLIB=SYSTEMOPENMPI parameter

6. Source etc/bashrc

Code:
source etc/bashrc
7. Execute compilator

You can run it in parallel with the -j flag. Change the number of processors according to your machine. You don't need to use the maximum number of processors available. Use just a few.

Code:
./Allwmake -j 6
8. Test the installation

Code:
foamInstallationTest
Run a case such as pitzDaily

Code:
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily $HOME/
cd $HOME/pitzDaily
blockMesh
simpleFoam
Run in parallel.
Create a decomposeParDict similar to the other tutorials

Code:
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v2012                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 5;

method          hierarchical;

coeffs
{
    n           (1 5 1);
}
Code:
blockMesh
decomposePar
mpirun -np 5 simpleFoam -parallel
or
mpiexec -np 5 simpleFoam -parallel
reconstructPar
Change the number of processors according to your machine

9. Load OpenMPI and OF automatically

You will need to modify the server bashrc to load the OF and the OpenMPI whenever you login in to it so you won't have to load it manually every time.

Add these lines to the end of ~/.bashrc
Code:
alias of2012='alias of2012='module load OpenMPI/4.0.5-GCC-10.2.0 || export PATH=$PATH:/usr/lib64/openmpi/bin; source /home70/gfelix/opt/OpenFOAM-v2012/etc/bashrc $FOAM_SETTINGS'
When you login again just type
Code:
of2012
to load it.

Now the OF is ready to run your cases. You just gotta transfer your run files with scp as done in step 1.

NOTE: You might want to set up a passwordless login in order to avoid having to type your user and password everytime you login to the server. Just follow this guide How to Setup Passwordless SSH Login

NOTE 2: You can check the information about the cluster's nodes and number of processors with the following commands:
Code:
Number of processors (CPU)
	grep -c ^processor /proc/cpuinfo
Infos about the nodes
	cat /proc/cpuinfo
Hi,

I encounter the same problem as you are. I followed almost the same route as what you did, however, if I'll run parallel processing it seems like all the cores in our cluster are hosting. What I am saying is, for example, if I'll ask 1 node which has 20 cores, then all these cores are hosting when in fact 19 cores must be slaves while one is the master. Have you encountered these issues?

Regards,
Mike
mikulo is offline   Reply With Quote

Reply

Tags
cluster, compile, openfoam, server, source

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
polynomial BC srv537 OpenFOAM Pre-Processing 4 December 3, 2016 10:07
SparceImage v1.7.x Issue on MAC OS X rcarmi OpenFOAM Installation 4 August 14, 2014 07:42
Problem compiling a custom Lagrangian library brbbhatti OpenFOAM Programming & Development 2 July 7, 2014 12:32
[swak4Foam] Swak4FOAM 0.2.3 / OF2.2.x installation error FerdiFuchs OpenFOAM Community Contributions 27 April 16, 2014 16:14
centOS 5.6 : paraFoam not working yossi OpenFOAM Installation 2 October 9, 2013 02:41


All times are GMT -4. The time now is 17:35.