CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Programming & Development

MPI barrier inside OpenFoam

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree14Likes
  • 1 Post By niklas
  • 8 Post By niklas
  • 4 Post By jherb
  • 1 Post By zhengzh5

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   September 7, 2010, 12:12
Default MPI barrier inside OpenFoam
  #1
Member
 
matteo lombardi
Join Date: Apr 2009
Posts: 67
Rep Power: 16
matteoL is on a distinguished road
Hello,
when running in parallel, for some reasons I have some actions that have to be done only by the master core and I would like the other partitions to wait until the master has finished.
(I call those action doing:
If (Pstream::master()){ ...} )


I think the solution in a general c++ code using MPI would be to add a line like:
MPI_BARRIER(MPI_COMM_WORLD)

Is there an already implemented function in OF available?
OR,
how could i get acces to the MPI class/function/settings?

Thank you very much,
best regards,
Matteo
matteoL is offline   Reply With Quote

Old   September 7, 2010, 12:47
Default
  #2
Super Moderator
 
niklas's Avatar
 
Niklas Nordin
Join Date: Mar 2009
Location: Stockholm, Sweden
Posts: 693
Rep Power: 29
niklas will become famous soon enoughniklas will become famous soon enough
does the master node information need to be distibuted to the other processors?
(it usually does)

in which case you can just
Code:
if (master)
   loop i over number of procs except master
       send info to proc i
else
    receive info from master
end if
which would have the same effect
the_ichthyologist likes this.
niklas is offline   Reply With Quote

Old   September 7, 2010, 12:57
Default
  #3
Member
 
matteo lombardi
Join Date: Apr 2009
Posts: 67
Rep Power: 16
matteoL is on a distinguished road
Hello Nicklas, thanks a lot for your quick answer.
Not sure I have understood what you meant.

1)Easy case
Let's say that the master is doing some calculations and then writes down those info on a file.

AFTER that the file have been written, I want all procs to read the file and go on ...

How do i do that?
If I simply do:
If (Pstream::master()){
...cacl...
write file
}

read file

then a non-master node will skip the first and will try to read the file BEFORE the file has even been written...

How would you do?

2)Same thing as before, but the data are not written to a file but, after been computed are shared by the master to all other node.
-Once again,I do i tell the other proc to wait for the master to do the calculations?
-how do i send data from the master to all other proc? (using "reduce"?)

thanks again,
Matteo
matteoL is offline   Reply With Quote

Old   September 7, 2010, 13:26
Default
  #4
Super Moderator
 
niklas's Avatar
 
Niklas Nordin
Join Date: Mar 2009
Location: Stockholm, Sweden
Posts: 693
Rep Power: 29
niklas will become famous soon enoughniklas will become famous soon enough
First off, I think that communicating to other processors via files is a really really bad idea
and should be abandoned. If you want to have that info on file, sure write it to a file, but dont make the other processors read it in via the file, it will drastically reduce your performance...just think of all the processors asking for i/o access to the disk.
I think you should view your processors as the file instead, what you want to do is stream the info to the other processors directly, it is just as easy as streaming it to a file.

below is a very simple example of a piece of code sending a vector to the main processor.
it should be easy enough to figure out how to do the reverse, i.e send all the info to
the other processors from the master

Code:
    // get the number of processors
    label n=Pstream::nProcs();

    vector localV = vector::zero;

    if (Pstream::myProcNo() == 0)
    {
        Random rnd(0);
        localV = rnd.vector01();
        List<vector> allV(n, vector::zero);

        allV[0] = localV;
        for(label i=1; i<n; i++)
	{
            // create the input stream from processor i
 	    IPstream vStream(Pstream::blocking, i);
            vStream >> allV[i];
	}
        // print the list of all vectors on the main proc
        Info << allV << endl;
    }
    else
    {
        Random rnd(0);
        localV = rnd.vector01();
        // print the vector on the processor (so we can compare it later when we print it on the main proc)
        Sout << "[" << Pstream::myProcNo() << "] V = " << localV << endl;

        // create the stream to send to the main proc
        OPstream vectorStream
	(
	    Pstream::blocking, 0
        );
        vectorStream << localV;
    }
since the streaming is 'blocking' the processor will wait until it has received the information

i didnt really answer any of your questions cause I hope that the above code will help you
solve the problem yourself, otherwise post them again
niklas is offline   Reply With Quote

Old   September 7, 2010, 13:51
Default
  #5
Member
 
matteo lombardi
Join Date: Apr 2009
Posts: 67
Rep Power: 16
matteoL is on a distinguished road
I see very well what you mean and I think I may use some of the information you have sent me.
Although in my case I think i am forced to use file for I/O.

Here is the explanation:
Mainly, I am coupling OF with an external structural solver written in Fortran.
Thus I use as OF as main code and the structural solver is written as Fortran functions that are called inside OF thanks to the " extern "C" options".

Since I don't want(can not, at the moment) pass vectors from OF to the Fortran code, I decided to pass the needed informations by file.

Do you see my problem?how would you tackle it?
Thanks again,
matteo
matteoL is offline   Reply With Quote

Old   September 7, 2010, 14:05
Default
  #6
Super Moderator
 
niklas's Avatar
 
Niklas Nordin
Join Date: Mar 2009
Location: Stockholm, Sweden
Posts: 693
Rep Power: 29
niklas will become famous soon enoughniklas will become famous soon enough
ah, I see.

If it is possible I would have the master processor read all the info from the file and then send it to each processor. (it will also solve your sync problem)

This should be alooot faster than having each processor reading the file #proc number of times.

N
niklas is offline   Reply With Quote

Old   June 28, 2011, 18:30
Default
  #7
Member
 
Ivor Clifford
Join Date: Mar 2009
Location: Switzerland
Posts: 94
Rep Power: 17
cliffoi is on a distinguished road
I know this thread is no longer active but in case anyone is interested in interfacing OpenFOAM and Fortran I felt like adding my own two cents worth: I agree with Niklas, there should be no need to use file IO. I've passed fields of data in OpenFOAM to and from Fortran very happily in the past. Fortran 90 and above is easy. If you're still using Fortran 77, I'd first suggest you seriously think about upgrading, and then write a F90 wrapper around your F77 interface functions. If you're looking at passing fields of any data type to an external Fortran program you simply need to pass the memory address of the start of the field and then tell the Fortran function what data type it should expect.

Code:
SUBROUTINE fortranSub(vf, size)
    USE ISO_C_BINDING, ONLY : C_DOUBLE, C_INT
    IMPLICIT NONE

    REAL(C_DOUBLE), INTENT(INOUT)   :: vf(size)
    INTEGER(C_INT), INTENT(IN)      :: size

    vf(:) = 100D0

END SUBROUTINE
and in your C++ header

Code:
extern"C"
{
    void __fortransub
    (
        double* vf,
        const int& size
    );
}
You can then call it from OpenFOAM like follows by explicitly passing the array address and size.

Code:
scalarField sf(mesh.nCells());

__fortransub(&sf[0], sf.size());

Info << sf << endl;
cliffoi is offline   Reply With Quote

Old   August 29, 2013, 14:36
Default Simple blocking mechanism
  #8
Senior Member
 
Joachim Herb
Join Date: Sep 2010
Posts: 650
Rep Power: 21
jherb is on a distinguished road
Just if somebody arrives here from google in this very old thread (I didn't find a newer one): You could use something like this to synchronize your processes:
Code:
    if (Pstream::master())
    {
        /* what ever */
    }
    int dummy;
    Pstream::scatter(dummy, Pstream::blocking);
See also http://www.cfd-online.com/Forums/ope...tml#post440958
jherb is offline   Reply With Quote

Old   September 25, 2013, 15:45
Default
  #9
Member
 
Jace
Join Date: Oct 2012
Posts: 77
Rep Power: 15
zhengzh5 is on a distinguished road
Quote:
Originally Posted by jherb View Post
Just if somebody arrives here from google in this very old thread (I didn't find a newer one): You could use something like this to synchronize your processes:
Code:
    if (Pstream::master())
    {
        /* what ever */
    }
    int dummy;
    Pstream::scatter(dummy, Pstream::blocking);
See also http://www.cfd-online.com/Forums/ope...tml#post440958
Hey Joachim,

what version of OpenFOAM are you using (i assume 2.2)? I tried it on OpenFOAM 2.0.x and there's no match for the function. Do you know if there's anyways to sync the processors before a reduce call in 2.0.x?

thanks for your help in advance!

Jace
mbookin likes this.
zhengzh5 is offline   Reply With Quote

Old   September 25, 2013, 17:50
Default
  #10
Senior Member
 
Joachim Herb
Join Date: Sep 2010
Posts: 650
Rep Power: 21
jherb is on a distinguished road
Hi Jace,

actually I am using OpenFOAM 2.2.x but it should not matter. The source files for Pstream::scatter are also in the repository of OpenFOAM 2.0.x:
https://github.com/OpenFOAM/OpenFOAM...eams/Pstream.H
https://github.com/OpenFOAM/OpenFOAM...atherScatter.C

So what are the error message(s) during compiling/running? Perhaps you have to add the correct header file (probably Pstream.H) and perhaps you have to specify additional libraries for linking.

Joachim

Quote:
Originally Posted by zhengzh5 View Post
Hey Joachim,

what version of OpenFOAM are you using (i assume 2.2)? I tried it on OpenFOAM 2.0.x and there's no match for the function. Do you know if there's anyways to sync the processors before a reduce call in 2.0.x?

thanks for your help in advance!

Jace
jherb is offline   Reply With Quote

Old   September 25, 2013, 18:07
Default
  #11
Member
 
Jace
Join Date: Oct 2012
Posts: 77
Rep Power: 15
zhengzh5 is on a distinguished road
Quote:
Originally Posted by jherb View Post
Hi Jace,

actually I am using OpenFOAM 2.2.x but it should not matter. The source files for Pstream::scatter are also in the repository of OpenFOAM 2.0.x:
https://github.com/OpenFOAM/OpenFOAM...eams/Pstream.H
https://github.com/OpenFOAM/OpenFOAM...atherScatter.C

So what are the error message(s) during compiling/running? Perhaps you have to add the correct header file (probably Pstream.H) and perhaps you have to specify additional libraries for linking.

Joachim
Hi, the following are the error message:

In file included from spray/sprayInject.C:124:0:
spray/findInjectorCell.H: In member function ‘void Foam::spray::inject()’:
spray/findInjectorCell.H:23:45: error: no matching function for call to ‘Foam::Pstream::scatter(int&, Foam::UPstream::commsTypes)’
spray/findInjectorCell.H:23:45: note: candidates are:
/home/gandalf/OpenFOAM/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/Pstream.H:109:25: note: template<class T> static void Foam::Pstream::scatter(const Foam::List<Foam::UPstream::commsStruct>&, T&)
/home/gandalf/OpenFOAM/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/Pstream.H:113:25: note: template<class T> static void Foam::Pstream::scatter(T&)
make: *** [Make/linuxGccDPOpt/sprayInject.o] Error 1

I am able to access other functions belonging to Pstream, such as Pstream::nProcs, Pstream::master(). To me, it seems like the foam 2.0.x version can't take in the blocking argument. what do you think?

thanks for quick response!

Jace
zhengzh5 is offline   Reply With Quote

Old   September 25, 2013, 20:08
Default
  #12
Senior Member
 
Joachim Herb
Join Date: Sep 2010
Posts: 650
Rep Power: 21
jherb is on a distinguished road
I guess you are right: This function declaration is new:
https://github.com/OpenFOAM/OpenFOAM...source=cc#L124

The source code is probably one of these:
https://github.com/OpenFOAM/OpenFOAM...Scatter.C#L193
https://github.com/OpenFOAM/OpenFOAM...Scatter.C#L218

Perhaps you can backport it?

Quote:
Originally Posted by zhengzh5 View Post
Hi, the following are the error message:

In file included from spray/sprayInject.C:124:0:
spray/findInjectorCell.H: In member function ‘void Foam::spray::inject()’:
spray/findInjectorCell.H:23:45: error: no matching function for call to ‘Foam::Pstream::scatter(int&, Foam::UPstream::commsTypes)’
spray/findInjectorCell.H:23:45: note: candidates are:
/home/gandalf/OpenFOAM/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/Pstream.H:109:25: note: template<class T> static void Foam::Pstream::scatter(const Foam::List<Foam::UPstream::commsStruct>&, T&)
/home/gandalf/OpenFOAM/OpenFOAM-2.0.x/src/OpenFOAM/lnInclude/Pstream.H:113:25: note: template<class T> static void Foam::Pstream::scatter(T&)
make: *** [Make/linuxGccDPOpt/sprayInject.o] Error 1

I am able to access other functions belonging to Pstream, such as Pstream::nProcs, Pstream::master(). To me, it seems like the foam 2.0.x version can't take in the blocking argument. what do you think?

thanks for quick response!

Jace
jherb is offline   Reply With Quote

Old   September 29, 2018, 13:10
Default How to gather distributed data?
  #13
Senior Member
 
Klaus
Join Date: Mar 2009
Posts: 250
Rep Power: 22
klausb will become famous soon enough
I want to gather the distributed upperAddr(), lowerAddr(), psi.size() and number of matrix coefficients on the master thread to do some checks on the global matrix - something like:

Code:
if (Pstream::master())
{
  // gather the parts of upperAddr(), lowerAddr(), psi.size(), n_coefficients
   global_upperAddr() = ?; // OF or c++ list or vector

   global_upperAddr() = ?; // OF or c++ list or vector

   global_n = gSum(psi.size()) ?; // matrix size

   global_n_coefficients = gSum(matrix().lower().size() + matrix().upper().size() + matrix().diag().size()) ?;

}
How can I gather the distributed data?
klausb is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Cross-compiling OpenFOAM 1.7.0 on Linux for Windows 32 and 64bits with Mingw-w64 wyldckat OpenFOAM Announcements from Other Sources 3 September 8, 2010 07:25
Error using LaunderGibsonRSTM on SGI ALTIX 4700 jaswi OpenFOAM 2 April 29, 2008 11:54
Is Testsuite on the way or not lakeat OpenFOAM Installation 6 April 28, 2008 12:12
Intelbs MPI and performance tools in OpenFOAM hplum OpenFOAM 16 December 16, 2007 15:58
meshing F1 front wing Steve FLUENT 0 April 17, 2003 13:37


All times are GMT -4. The time now is 08:00.