CFD Online Discussion Forums - MPI barrier inside OpenFoam

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)

- OpenFOAM Programming & Development (https://www.cfd-online.com/Forums/openfoam-programming-development/)

- - MPI barrier inside OpenFoam (https://www.cfd-online.com/Forums/openfoam-programming-development/79880-mpi-barrier-inside-openfoam.html)

matteoL

September 7, 2010 11:12

MPI barrier inside OpenFoam

Hello,
when running in parallel, for some reasons I have some actions that have to be done only by the master core and I would like the other partitions to wait until the master has finished.
(I call those action doing:
If (Pstream::master()){ ...} )

I think the solution in a general c++ code using MPI would be to add a line like:
MPI_BARRIER(MPI_COMM_WORLD)

Is there an already implemented function in OF available?
OR,
how could i get acces to the MPI class/function/settings?

Thank you very much,
best regards,
Matteo

niklas

September 7, 2010 11:47

does the master node information need to be distibuted to the other processors?
(it usually does)

in which case you can just

Code:

if (master)

   loop i over number of procs except master

       send info to proc i

else

    receive info from master

end if

which would have the same effect

matteoL

September 7, 2010 11:57

Hello Nicklas, thanks a lot for your quick answer.
Not sure I have understood what you meant.

1)Easy case
Let's say that the master is doing some calculations and then writes down those info on a file.

AFTER that the file have been written, I want all procs to read the file and go on ...

How do i do that?
If I simply do:
If (Pstream::master()){
...cacl...
write file
}

read file

then a non-master node will skip the first and will try to read the file BEFORE the file has even been written...

How would you do?

2)Same thing as before, but the data are not written to a file but, after been computed are shared by the master to all other node.
-Once again,I do i tell the other proc to wait for the master to do the calculations?
-how do i send data from the master to all other proc? (using "reduce"?)

thanks again,
Matteo

niklas

September 7, 2010 12:26

First off, I think that communicating to other processors via files is a really really bad idea
and should be abandoned. If you want to have that info on file, sure write it to a file, but dont make the other processors read it in via the file, it will drastically reduce your performance...just think of all the processors asking for i/o access to the disk.
I think you should view your processors as the file instead, what you want to do is stream the info to the other processors directly, it is just as easy as streaming it to a file.

below is a very simple example of a piece of code sending a vector to the main processor.
it should be easy enough to figure out how to do the reverse, i.e send all the info to
the other processors from the master

Code:



    // get the number of processors

    label n=Pstream::nProcs();



    vector localV = vector::zero;



    if (Pstream::myProcNo() == 0)

    {

        Random rnd(0);

        localV = rnd.vector01();

        List<vector> allV(n, vector::zero);



        allV[0] = localV;

        for(label i=1; i<n; i++)

        {

            // create the input stream from processor i

             IPstream vStream(Pstream::blocking, i);

            vStream >> allV[i];

        }

        // print the list of all vectors on the main proc

        Info << allV << endl;

    }

    else

    {

        Random rnd(0);

        localV = rnd.vector01();

        // print the vector on the processor (so we can compare it later when we print it on the main proc)

        Sout << "[" << Pstream::myProcNo() << "] V = " << localV << endl;



        // create the stream to send to the main proc

        OPstream vectorStream

        (

            Pstream::blocking, 0

        );

        vectorStream << localV;

    }

since the streaming is 'blocking' the processor will wait until it has received the information

i didnt really answer any of your questions cause I hope that the above code will help you
solve the problem yourself, otherwise post them again :)

matteoL

September 7, 2010 12:51

I see very well what you mean and I think I may use some of the information you have sent me.
Although in my case I think i am forced to use file for I/O.

Here is the explanation:
Mainly, I am coupling OF with an external structural solver written in Fortran.
Thus I use as OF as main code and the structural solver is written as Fortran functions that are called inside OF thanks to the " extern "C" options".

Since I don't want(can not, at the moment) pass vectors from OF to the Fortran code, I decided to pass the needed informations by file.

Do you see my problem?how would you tackle it?
Thanks again,
matteo

niklas

September 7, 2010 13:05

ah, I see.

If it is possible I would have the master processor read all the info from the file and then send it to each processor. (it will also solve your sync problem)

This should be alooot faster than having each processor reading the file #proc number of times.

N

cliffoi

June 28, 2011 17:30

I know this thread is no longer active but in case anyone is interested in interfacing OpenFOAM and Fortran I felt like adding my own two cents worth: I agree with Niklas, there should be no need to use file IO. I've passed fields of data in OpenFOAM to and from Fortran very happily in the past. Fortran 90 and above is easy. If you're still using Fortran 77, I'd first suggest you seriously think about upgrading, and then write a F90 wrapper around your F77 interface functions. If you're looking at passing fields of any data type to an external Fortran program you simply need to pass the memory address of the start of the field and then tell the Fortran function what data type it should expect.

Code:

SUBROUTINE fortranSub(vf, size)

    USE ISO_C_BINDING, ONLY : C_DOUBLE, C_INT

    IMPLICIT NONE



    REAL(C_DOUBLE), INTENT(INOUT)   :: vf(size)

    INTEGER(C_INT), INTENT(IN)      :: size



    vf(:) = 100D0



END SUBROUTINE

and in your C++ header

Code:

extern"C"

{

    void __fortransub

    (

        double* vf,

        const int& size

    );

}

You can then call it from OpenFOAM like follows by explicitly passing the array address and size.

Code:

scalarField sf(mesh.nCells());



__fortransub(&sf[0], sf.size());



Info << sf << endl;

jherb

August 29, 2013 13:36

Simple blocking mechanism

Just if somebody arrives here from google in this very old thread (I didn't find a newer one): You could use something like this to synchronize your processes:

Code:

    if (Pstream::master())

    {

        /* what ever */

    }

    int dummy;

    Pstream::scatter(dummy, Pstream::blocking);

zhengzh5

September 25, 2013 14:45

Quote:

Originally Posted by jherb (Post 448769)

Just if somebody arrives here from google in this very old thread (I didn't find a newer one): You could use something like this to synchronize your processes:

Code:

    if (Pstream::master())

    {

        /* what ever */

    }

    int dummy;

    Pstream::scatter(dummy, Pstream::blocking);

Hey Joachim,

what version of OpenFOAM are you using (i assume 2.2)? I tried it on OpenFOAM 2.0.x and there's no match for the function. Do you know if there's anyways to sync the processors before a reduce call in 2.0.x?

thanks for your help in advance!

Jace

jherb

September 25, 2013 16:50

Hi Jace,

actually I am using OpenFOAM 2.2.x but it should not matter. The source files for Pstream::scatter are also in the repository of OpenFOAM 2.0.x:
https://github.com/OpenFOAM/OpenFOAM...eams/Pstream.H
https://github.com/OpenFOAM/OpenFOAM...atherScatter.C

So what are the error message(s) during compiling/running? Perhaps you have to add the correct header file (probably Pstream.H) and perhaps you have to specify additional libraries for linking.

Joachim

Quote:

Originally Posted by zhengzh5 (Post 453580)

zhengzh5

September 25, 2013 17:07

Quote:

Originally Posted by jherb (Post 453591)

Hi, the following are the error message:

In file included from spray/sprayInject.C:124:0:
spray/findInjectorCell.H: In member function ‘void Foam::spray::inject()’:
spray/findInjectorCell.H:23:45: error: no matching function for call to ‘Foam::Pstream::scatter(int&, Foam::UPstream::commsTypes)’
spray/findInjectorCell.H:23:45: note: candidates are:
/home/gandalf/OpenFOAM/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/Pstream.H:109:25: note: template<class T> static void Foam::Pstream::scatter(const Foam::List<Foam::UPstream::commsStruct>&, T&)
/home/gandalf/OpenFOAM/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/Pstream.H:113:25: note: template<class T> static void Foam::Pstream::scatter(T&)
make: *** [Make/linuxGccDPOpt/sprayInject.o] Error 1

I am able to access other functions belonging to Pstream, such as Pstream::nProcs, Pstream::master(). To me, it seems like the foam 2.0.x version can't take in the blocking argument. what do you think?

thanks for quick response!

Jace

jherb

September 25, 2013 19:08

I guess you are right: This function declaration is new:
https://github.com/OpenFOAM/OpenFOAM...source=cc#L124

The source code is probably one of these:
https://github.com/OpenFOAM/OpenFOAM...Scatter.C#L193
https://github.com/OpenFOAM/OpenFOAM...Scatter.C#L218

Perhaps you can backport it?

Quote:

Originally Posted by zhengzh5 (Post 453592)

klausb

September 29, 2018 12:10

How to gather distributed data?

I want to gather the distributed upperAddr(), lowerAddr(), psi.size() and number of matrix coefficients on the master thread to do some checks on the global matrix - something like:

Code:

if (Pstream::master())

{

  // gather the parts of upperAddr(), lowerAddr(), psi.size(), n_coefficients

   global_upperAddr() = ?; // OF or c++ list or vector



   global_upperAddr() = ?; // OF or c++ list or vector



   global_n = gSum(psi.size()) ?; // matrix size



   global_n_coefficients = gSum(matrix().lower().size() + matrix().upper().size() + matrix().diag().size()) ?;



}

How can I gather the distributed data?

All times are GMT -4. The time now is 11:18.