CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Programming & Development

parallel programming issues

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   January 23, 2015, 09:39
Default parallel programming issues
  #1
New Member
 
Theodore S. Richfield
Join Date: Jan 2015
Location: germany
Posts: 3
Rep Power: 11
mrRichField is on a distinguished road
Dear All,

in the course of a university project we are trying to adapt the basic simpleFoam solver to our needs by adding some lines of code. Everything works fine performing single core, but as soon as we go parallel, we encountered problems.

Right before the simple algorithm we perform a loop over all cells, where the elements of vector A are initialized with the average of vector B, using something like...
Code:
forAll(mesh.cells(), i)
{
    A[i] = B.average().value();
    ...
}
In parallel we receive an MPI_ERROR, but we soon found, that we can proceed using
Code:
temp=B.average().value();
forAll(mesh.cells(), i)
{
    A[i] = tmp;
    ...
}
but unfortunately we are not aware of the reason for this behaviour!??

Furthermore, we thought, that as with 'conventional' MPI parallelization we know from pure C programming, we used mpi_reduce statements to sum up and count, and calculate an average of a quantity over all workers. Please, could someone point us in the right direction in the context of C++ & OpenFOAM programming, since we are "just beginners"

We are already struggling with getting some debug information from each worker, i.e. how can we put something like
Code:
Info << "worker no. " << i << " | value = " << v << endl;
in the code, in order to get a feedback from each worker?

We've checked the manuals and tutorials available throughout the internet, but ... Is there any good programming tutorial focusing on parallel?

We would be grateful for your feedback,
Kind Regards,

George
mrRichField is offline   Reply With Quote

Old   January 24, 2015, 09:38
Default
  #2
Senior Member
 
floquation's Avatar
 
Kevin van As
Join Date: Sep 2014
Location: TU Delft, The Netherlands
Posts: 252
Rep Power: 20
floquation will become famous soon enough
A very incomplete answer, but this is what I know:

The following statement I use for parallel debugging (that way, each processor shows its own message with the processor ID appended):
Code:
Sout << "[" << Pstream::myProcNo() << "] " << "Debug message" << nl;

Last edited by floquation; January 24, 2015 at 14:05.
floquation is offline   Reply With Quote

Old   January 25, 2015, 09:48
Lightbulb
  #3
New Member
 
Theodore S. Richfield
Join Date: Jan 2015
Location: germany
Posts: 3
Rep Power: 11
mrRichField is on a distinguished road
Thanks a lot floquation!

In the meantime I also found that
Code:
Sout
is the output stream which provides an answer from each worker. Furthermore I found that the
Code:
.average()
method uses
Code:
gaverage
(something I think of as 'global' average), which ensures that the respective
Code:
mpi_reduce
formalism is performed. Ok, thus I can use the average method on a field in a straight forward way for single proc or parallel applications, respectively.

However, I still don't understand, what is the difference between using the average method inside the 'forall' scope or outside? Well, my code does it's job right now, but I'm really interested in what is going on behind the scenes!? Please, if anyone can put a light on this, I would be grateful!

Cheers,
George

BTW: Could anyone provide me a link on a more comprehensive programming guideline (sorry, even though I do programming for a while, I'm an engineer and not an IT specialist) that goes a bit further than the official OF programming guide, and which provides a clarifying picture of the underlying structure and hierarchy of OF?
mrRichField is offline   Reply With Quote

Old   January 29, 2015, 05:39
Default
  #4
Senior Member
 
Joachim Herb
Join Date: Sep 2010
Posts: 650
Rep Power: 21
jherb is on a distinguished road
Why do you call B.average() inside the loop? It looks like it is not depending on i (or the cell), so you are calculating the same value again and again. Can't you put it into a temporary variable defined before the loop? (Ok, the compiler might be optimizing it this way, but who knows?)
Quote:
Originally Posted by mrRichField View Post
Code:
forAll(mesh.cells(), i)
{
    A[i] = B.average().value();
    ...
}
jherb is offline   Reply With Quote

Old   January 29, 2015, 05:58
Default
  #5
New Member
 
Theodore S. Richfield
Join Date: Jan 2015
Location: germany
Posts: 3
Rep Power: 11
mrRichField is on a distinguished road
Hi jherb,

indeed, you#re right, and that's what I already did (cf. my first post). But, even though it will be optimized by either the compiler (if...) or by developer (if she is aware of it...), it is not clear for me, why we receive an mpi_error in the first case (average in loop)...

So finally, it is clear, that one should keep the calculation of a non-changing value outside of the loop with respect to performance (regardless of compiler optimization), but with regards to efficiently do parallel programming in openFoam, we'd like to know more about the source of the problem.

Kind Regards,
George
mrRichField is offline   Reply With Quote

Old   January 29, 2015, 19:08
Default
  #6
Senior Member
 
Joachim Herb
Join Date: Sep 2010
Posts: 650
Rep Power: 21
jherb is on a distinguished road
The gaverage function (see also e.g. the gSum function) automatically reduce the results from all CPUs, i.e. collect the results from all CPUs and does the averaging. Then the result is sent to all CPUs.

Just speculating, but I think the call to such a reducing function has to be called on all CPUs at the same point in the code. But the loop of the mesh cells is for different cells on the different CPUs. So if there are different number of cells on different CPUs this sounds like trouble
jherb is offline   Reply With Quote

Reply

Tags
mpi, parallel

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
simpleFoam parallel AndrewMortimer OpenFOAM Running, Solving & CFD 12 August 7, 2015 19:45
FE-3.1 Parallel issues with BCs (gMax, reduce...) Phicau OpenFOAM Bugs 2 August 1, 2014 07:22
How can i make a parallel programming in OpenFOAM? jignesh_thaker2007 OpenFOAM Running, Solving & CFD 3 July 2, 2014 10:37
Installation issues for parallel computation Akash C SU2 Installation 1 June 21, 2013 06:26
PROBLEM IN PARALLEL PROGRAMMING WITH MPI Niavarani Main CFD Forum 1 April 20, 2004 07:51


All times are GMT -4. The time now is 13:42.