[tomislav_maric]

Hi everyone,

I am trying to write the simplest parallel code that involves simultaneous communication accross processes:

Proc1: Send(0),Receive(0)

Proc2: Send(1),Receive(1)

Of course, such code will experience a deadlock. There are multiple solutions for this beginner problem: exchanging the order of Send/Receive based on the process rank, using non-blocking communication, using buffered communication (bSend, bRecv), etc.

Both the problem and possible solutions are explained here (Note: if the links don't work by clicking, copy/paste them):

http://www.ncsa.illinois.edu/UserInf...IDeadlock.html

and here:

http://www.cs.ucsb.edu/~hnielsen/cs140/mpi-deadlocks.html

So. I have tried to use these solutions in OpenFOAM, sending/receiving a scalarField around, and I keep getting errors. I linked the application source below, where you can uncomment parts of the code to get specific solutions (described in the links above). What I find strange is:

• the code deadlocks when I properly combine blocking send/receive and receive/send
• the code experiences FoamFatalError when I use nonBlocking communication: UPstream::waitRequests() is an empty function, Pstream::nRequests() returns 0 regardless of the number of outstanding nonBlocking requests in the program

I am not sure if I am missing something obvious, but it seems strange that every option I try, fails.


Also: I've switched on the Pstream debugSwitch in etc/controlDict, but I still don't see the debug info comming from the UPstream.C in the Pstream/mpi folder. This looks like my application is using the dummy implementation of UPstream and its children, instead of linking with stuff from Pstream/mpi.



Download link (OpenFOAM 2.0.x):


http://dl.dropbox.com/u/3316983/pStr...leExchange.tgz


Tomislav