CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)
-   OpenFOAM Programming & Development (http://www.cfd-online.com/Forums/openfoam-programming-development/)
-   -   Code parallelization using Pstream: simultaneous send/receive deadlocking (http://www.cfd-online.com/Forums/openfoam-programming-development/104411-code-parallelization-using-pstream-simultaneous-send-receive-deadlocking.html)

tomislav_maric July 9, 2012 06:14

Code parallelization using Pstream: simultaneous send/receive deadlocking
 
Hi everyone,

I am trying to write the simplest parallel code that involves simultaneous communication accross processes:

Proc1: Send(0),Receive(0)

Proc2: Send(1),Receive(1)

Of course, such code will experience a deadlock. There are multiple solutions for this beginner problem: exchanging the order of Send/Receive based on the process rank, using non-blocking communication, using buffered communication (bSend, bRecv), etc.

Both the problem and possible solutions are explained here (Note: if the links don't work by clicking, copy/paste them):

http://www.ncsa.illinois.edu/UserInf...IDeadlock.html

and here:

http://www.cs.ucsb.edu/~hnielsen/cs140/mpi-deadlocks.html

So. I have tried to use these solutions in OpenFOAM, sending/receiving a scalarField around, and I keep getting errors. I linked the application source below, where you can uncomment parts of the code to get specific solutions (described in the links above). What I find strange is:

  • the code deadlocks when I properly combine blocking send/receive and receive/send
  • the code experiences FoamFatalError when I use nonBlocking communication: UPstream::waitRequests() is an empty function, Pstream::nRequests() returns 0 regardless of the number of outstanding nonBlocking requests in the program


I am not sure if I am missing something obvious, but it seems strange that every option I try, fails.


Also: I've switched on the Pstream debugSwitch in etc/controlDict, but I still don't see the debug info comming from the UPstream.C in the Pstream/mpi folder. This looks like my application is using the dummy implementation of UPstream and its children, instead of linking with stuff from Pstream/mpi.



Download link (OpenFOAM 2.0.x):


http://dl.dropbox.com/u/3316983/pStr...leExchange.tgz


Tomislav

tomislav_maric July 13, 2012 04:47

(bump)

O.K. , let me rephrase the question: there are two processes available.

How do I use Pstream to exchange the number Pstream::myProcNo() between the processes without deadlocking?

wyldckat July 13, 2012 05:14

Greetings Tomislav,

Have you looked at the source code for the OpenFOAM application Test-parallel? It's located at "applications/test/parallel".
It's a very decent example of what you seem to be trying to do!

Browse the other applications in the test folder as well, because I remember there are other very sweet examples of parallel manipulations!

Good luck!
Bruno

tomislav_maric July 13, 2012 05:16

Thanks, I'm on it! :D

akidess July 13, 2012 05:20

What went wrong with the ucsb solution? It seems brilliantly simple...

Code:

if (rank % 2) {
    MPI_Send(data, (rank + 1) % size);
    MPI_Recv(newdata, (rank - 1) % size);
} else {
    MPI_Recv(newdata, (rank - 1) % size);
    MPI_Send(data, (rank + 1) % size);
}


tomislav_maric July 13, 2012 05:22

Quote:

Originally Posted by akidess (Post 371266)
What went wrong with the ucsb solution? It seems brilliantly simple...

Code:

if (rank % 2) {
    MPI_Send(data, (rank + 1) % size);
    MPI_Recv(newdata, (rank - 1) % size);
} else {
    MPI_Recv(newdata, (rank - 1) % size);
    MPI_Send(data, (rank + 1) % size);
}


Well, when you try this with Pstream, the thing just hangs in the air... If you don't believe me, uncomment SOLUTION 0 in the source I linked below... it's just weird. :)

Added: I wrote it for just 2 processes: 0 is sending/receiving to/from 1, and 1 is sending/receiving to/from 0.


All times are GMT -4. The time now is 17:10.