CFD Online Logo CFD Online URL
Home > Forums > OpenFOAM

Parallelizing a code that involves OpenFOAM

Register Blogs Members List Search Today's Posts Mark Forums Read

LinkBack Thread Tools Display Modes
Old   October 10, 2011, 22:30
Default Parallelizing a code that involves OpenFOAM
New Member
Join Date: May 2011
Posts: 4
Rep Power: 6
Nicolas Périnet is on a distinguished road

I am writing a code to compute periodic solutions in fluid dynamics using a arclength continuation method that involves OpenFOAM functions as subroutines. At each iteration of a main loop, OpenFOAM is used to integrate the Navier-Stokes equations. The obtained solution is post-processed and modified and this new result is then used for the next integration by OpenFOAM and so on.

How can I parallelize such a code?
I've read some topics and documents about parallel programming with OpenFOAM and apparently, the software uses some functions decomposeParDict and recomposeParDict that are in general run separately, and a library PStream. Can I use decomposeParDict/recomposeParDict inside the code? Do I have to parallelize everything with MPI without PStream because of the structure of my code?

Thanks in advance for your answers.

Nicolas Périnet
Nicolas Périnet is offline   Reply With Quote

Old   October 11, 2011, 02:55
Senior Member
akidess's Avatar
Anton Kidess
Join Date: May 2009
Location: Delft, Netherlands
Posts: 924
Rep Power: 17
akidess will become famous soon enough
Typically, you use decompose at the start of the simulation, and reconstruct at the end of it. I believe parallelization is done using PStream to scatter and gather data on processor boundary patches, and PStream is built on top of MPI functions. You should be able to use PStream if it makes your life easier (I can't tell if it will). If not, you can surely use the underlying MPI functions directly.
akidess is online now   Reply With Quote

Old   November 23, 2011, 18:59
New Member
Join Date: May 2011
Posts: 4
Rep Power: 6
Nicolas Périnet is on a distinguished road
Thanks a lot Anton for your advises.

I continue on this thread after a long time of wandering.

I've tried to make interact Openfoam with MPI. The useful objects are defined in the file UPStream.H. for instance:
-number of processes : nProcs,
-index of the process [from 0 (masterNo) to nprocs-1]: myProcNo,
-index of master process: masterNo,
typing for instance
I've got a test example based on icoFOAM and, on a single file, that I can post there if somebody finds it useful.

Now, I'm trying to use Openfoam in a loop initialized in a fortran subroutine which is itself part of a main code in C++ involving OpenFOAM functions.
It compiles but when I'm running the code I get the error message

Calling MPI_Init or MPI_Init_thread twice is erroneous.
once I call Openfoam at the first step of my loop. I've dug for a while and found it was just after

#include SetRootCase
because it uses itself UPstream that initializes MPI (setRootCase.H is included in my main program and also in the OpenFOAM function inside the loop). How can I avoid this error?

If I comment the call to setRootCase inside the loop, I also get error messages.

Apparently it is possible to use the function MPI_Initialized to avoid initializing MPI again, but I don't know yet how to use it. I'm afraid there will be a lot of work to do and I'll have to modify a bunch a files to achieve this.

I would be grateful to anybody that could help to get rid of this error.
Nicolas Périnet is offline   Reply With Quote

Old   November 24, 2011, 01:27
New Member
Join Date: May 2011
Posts: 4
Rep Power: 6
Nicolas Périnet is on a distinguished road
after having searched for a while, I've found that mpi_initialize seems to be indeed a good solution since the code is now working better, I don't know yet on what extent, but it doesn't stop at the beginning of the loop with the previous error message.

I replaced in UPstream.C the line

    MPI_Init(&argc, &argv);

    int is_initialized;
        MPI_Init(&argc, &argv);
Now, it checks whether mpi_init has already been performed each time setRootCase is run (and subsequently each time mpi_init was called), avoiding the previous error.

The library containing UPstream.C has to be recompiled.

UPstream.C is in

the library is recompiled by typing

wmake libso
After this it should work fine.
Nicolas Périnet is offline   Reply With Quote


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
Access source code of OpenFOAM nmansouri OpenFOAM 1 October 14, 2010 02:17
OpenFOAM 1.5.x package - CentOS 5.3 x86_64 linnemann OpenFOAM Installation 7 July 30, 2009 03:14
Modified OpenFOAM Forum Structure and New Mailing-List pete Site News & Announcements 0 June 29, 2009 05:56
OpenFOAM Debian packaging current status problems and TODOs oseen OpenFOAM Installation 9 August 26, 2007 13:50
Parallelizing User Code for Fluent 4.5 Greg Perkins FLUENT 1 September 25, 2000 19:51

All times are GMT -4. The time now is 10:53.