[tomislav_maric]

Hi everyone,

I am trying to write the simplest parallel code that involves simultaneous communication accross processes:

Proc1: Send(0),Receive(0)

Proc2: Send(1),Receive(1)

Of course, such code will experience a deadlock. There are multiple solutions for this beginner problem: exchanging the order of Send/Receive based on the process rank, using non-blocking communication, using buffered communication (bSend, bRecv), etc.

Both the problem and possible solutions are explained here (Note: if the links don't work by clicking, copy/paste them):

http://www.ncsa.illinois.edu/UserInf...IDeadlock.html

and here:

http://www.cs.ucsb.edu/~hnielsen/cs140/mpi-deadlocks.html

So. I have tried to use these solutions in OpenFOAM, sending/receiving a scalarField around, and I keep getting errors. I linked the application source below, where you can uncomment parts of the code to get specific solutions (described in the links above). What I find strange is:

• the code deadlocks when I properly combine blocking send/receive and receive/send
• the code experiences FoamFatalError when I use nonBlocking communication: UPstream::waitRequests() is an empty function, Pstream::nRequests() returns 0 regardless of the number of outstanding nonBlocking requests in the program

I am not sure if I am missing something obvious, but it seems strange that every option I try, fails.


Also: I've switched on the Pstream debugSwitch in etc/controlDict, but I still don't see the debug info comming from the UPstream.C in the Pstream/mpi folder. This looks like my application is using the dummy implementation of UPstream and its children, instead of linking with stuff from Pstream/mpi.



Download link (OpenFOAM 2.0.x):


http://dl.dropbox.com/u/3316983/pStr...leExchange.tgz


Tomislav

[tomislav_maric]

(bump)

O.K. , let me rephrase the question: there are two processes available.

How do I use Pstream to exchange the number Pstream::myProcNo() between the processes without deadlocking?

[wyldckat]

Greetings Tomislav,

Have you looked at the source code for the OpenFOAM application Test-parallel? It's located at "applications/test/parallel".
It's a very decent example of what you seem to be trying to do!

Browse the other applications in the test folder as well, because I remember there are other very sweet examples of parallel manipulations!

Good luck!
Bruno

[tomislav_maric]

Thanks, I'm on it!

[akidess]

What went wrong with the ucsb solution? It seems brilliantly simple...

if (rank % 2) {
    MPI_Send(data, (rank + 1) % size);
    MPI_Recv(newdata, (rank - 1) % size);
} else {
    MPI_Recv(newdata, (rank - 1) % size);
    MPI_Send(data, (rank + 1) % size);
}

[tomislav_maric]

Originally Posted by akidess

What went wrong with the ucsb solution? It seems brilliantly simple...

if (rank % 2) {
    MPI_Send(data, (rank + 1) % size);
    MPI_Recv(newdata, (rank - 1) % size);
} else {
    MPI_Recv(newdata, (rank - 1) % size);
    MPI_Send(data, (rank + 1) % size);
}

Well, when you try this with Pstream, the thing just hangs in the air... If you don't believe me, uncomment SOLUTION 0 in the source I linked below... it's just weird.

Added: I wrote it for just 2 processes: 0 is sending/receiving to/from 1, and 1 is sending/receiving to/from 0.

[sturmk]

Originally Posted by tomislav_maric

Well, when you try this with Pstream, the thing just hangs in the air... If you don't believe me, uncomment SOLUTION 0 in the source I linked below... it's just weird.

Added: I wrote it for just 2 processes: 0 is sending/receiving to/from 1, and 1 is sending/receiving to/from 0.

Sorry to dig this post up one decade later, but the problem still persists today (OpenFoam v9).

I hope this reaches some of the OF devs by bringing it up again...

[sturmk]

Since the abovementioned might be a roadblocker, here's a workaround:

if(isPrimaryNode()) {
     IPstream *recv = new IPstream(Pstream::commsTypes::blocking,
                                                   secondaryNodeNo());
     *recv >> remote_data;
     delete recv;
     OPstream *send = new OPstream(Pstream::commsTypes::blocking,
                                                   secondaryNodeNo());
     *send << local_data;
     delete send;
}
if(isSecondaryNode()) {
     OPstream *send = new OPstream(Pstream::commsTypes::blocking,
                                                   primaryNodeNo());
     *send << local_data;
     delete send;
     IPstream *recv = new IPstream(Pstream::commsTypes::blocking,
                                                   primaryNodeNo());
     *recv >> remote_data;
     delete recv;
  }

FYI: I also tried to implement a "proper" solution by allocating a separate communicator and using Pstream::exchange(). Unfortunately, UPstream::allocateCommunicator() is the bad call and blocks indefinitely.

[olesen]

If it is contiguous data, and you know the dimensions, a combination of non-blocking UIPstream::read and UOPstream::write, followed by a UPstream::waitRequests should work.


If you have non-contiguous data and want a whatever to whatever type of exchange. consider using PstreamBuffers. Setup all of your sends on primary/secondary node, mark the sends as finished and then followup with the receives. It is slight overkill since it will use the personalised exchange (involving an all-to-all for distributing sizes) but if not called within a tight loop and with fewer than say 8k ranks, should be OK.