|
[Sponsors] |
Code parallelization using Pstream: simultaneous send/receive deadlocking |
|
LinkBack | Thread Tools | Search this Thread | Display Modes |
July 9, 2012, 06:14 |
Code parallelization using Pstream: simultaneous send/receive deadlocking
|
#1 |
Senior Member
Tomislav Maric
Join Date: Mar 2009
Location: Darmstadt, Germany
Posts: 284
Blog Entries: 5
Rep Power: 21 |
Hi everyone,
I am trying to write the simplest parallel code that involves simultaneous communication accross processes: Proc1: Send(0),Receive(0) Proc2: Send(1),Receive(1) Of course, such code will experience a deadlock. There are multiple solutions for this beginner problem: exchanging the order of Send/Receive based on the process rank, using non-blocking communication, using buffered communication (bSend, bRecv), etc. Both the problem and possible solutions are explained here (Note: if the links don't work by clicking, copy/paste them): http://www.ncsa.illinois.edu/UserInf...IDeadlock.html and here: http://www.cs.ucsb.edu/~hnielsen/cs140/mpi-deadlocks.html So. I have tried to use these solutions in OpenFOAM, sending/receiving a scalarField around, and I keep getting errors. I linked the application source below, where you can uncomment parts of the code to get specific solutions (described in the links above). What I find strange is:
I am not sure if I am missing something obvious, but it seems strange that every option I try, fails. Also: I've switched on the Pstream debugSwitch in etc/controlDict, but I still don't see the debug info comming from the UPstream.C in the Pstream/mpi folder. This looks like my application is using the dummy implementation of UPstream and its children, instead of linking with stuff from Pstream/mpi. Download link (OpenFOAM 2.0.x): http://dl.dropbox.com/u/3316983/pStr...leExchange.tgz Tomislav Last edited by tomislav_maric; July 9, 2012 at 06:31. |
|
July 13, 2012, 04:47 |
|
#2 |
Senior Member
Tomislav Maric
Join Date: Mar 2009
Location: Darmstadt, Germany
Posts: 284
Blog Entries: 5
Rep Power: 21 |
(bump)
O.K. , let me rephrase the question: there are two processes available. How do I use Pstream to exchange the number Pstream::myProcNo() between the processes without deadlocking? |
|
July 13, 2012, 05:14 |
|
#3 |
Retired Super Moderator
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,975
Blog Entries: 45
Rep Power: 128 |
Greetings Tomislav,
Have you looked at the source code for the OpenFOAM application Test-parallel? It's located at "applications/test/parallel". It's a very decent example of what you seem to be trying to do! Browse the other applications in the test folder as well, because I remember there are other very sweet examples of parallel manipulations! Good luck! Bruno
__________________
|
|
July 13, 2012, 05:20 |
|
#5 |
Senior Member
Anton Kidess
Join Date: May 2009
Location: Germany
Posts: 1,377
Rep Power: 30 |
What went wrong with the ucsb solution? It seems brilliantly simple...
Code:
if (rank % 2) { MPI_Send(data, (rank + 1) % size); MPI_Recv(newdata, (rank - 1) % size); } else { MPI_Recv(newdata, (rank - 1) % size); MPI_Send(data, (rank + 1) % size); }
__________________
*On twitter @akidTwit *Spend as much time formulating your questions as you expect people to spend on their answer. |
|
July 13, 2012, 05:22 |
|
#6 | |
Senior Member
Tomislav Maric
Join Date: Mar 2009
Location: Darmstadt, Germany
Posts: 284
Blog Entries: 5
Rep Power: 21 |
Quote:
Added: I wrote it for just 2 processes: 0 is sending/receiving to/from 1, and 1 is sending/receiving to/from 0. |
||
April 8, 2023, 09:06 |
|
#7 | |
New Member
Karl Sturm
Join Date: Apr 2023
Posts: 2
Rep Power: 0 |
Quote:
Sorry to dig this post up one decade later, but the problem still persists today (OpenFoam v9). I hope this reaches some of the OF devs by bringing it up again... |
||
April 8, 2023, 10:26 |
|
#8 |
New Member
Karl Sturm
Join Date: Apr 2023
Posts: 2
Rep Power: 0 |
Since the abovementioned might be a roadblocker, here's a workaround:
Code:
if(isPrimaryNode()) { IPstream *recv = new IPstream(Pstream::commsTypes::blocking, secondaryNodeNo()); *recv >> remote_data; delete recv; OPstream *send = new OPstream(Pstream::commsTypes::blocking, secondaryNodeNo()); *send << local_data; delete send; } if(isSecondaryNode()) { OPstream *send = new OPstream(Pstream::commsTypes::blocking, primaryNodeNo()); *send << local_data; delete send; IPstream *recv = new IPstream(Pstream::commsTypes::blocking, primaryNodeNo()); *recv >> remote_data; delete recv; } |
|
April 14, 2023, 04:31 |
|
#9 |
Senior Member
Mark Olesen
Join Date: Mar 2009
Location: https://olesenm.github.io/
Posts: 1,694
Rep Power: 40 |
If it is contiguous data, and you know the dimensions, a combination of non-blocking UIPstream::read and UOPstream::write, followed by a UPstream::waitRequests should work.
If you have non-contiguous data and want a whatever to whatever type of exchange. consider using PstreamBuffers. Setup all of your sends on primary/secondary node, mark the sends as finished and then followup with the receives. It is slight overkill since it will use the personalised exchange (involving an all-to-all for distributing sizes) but if not called within a tight loop and with fewer than say 8k ranks, should be OK. |
|
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
What is the Better Way to Do CFD? | John C. Chien | Main CFD Forum | 54 | April 23, 2001 08:10 |
Open source CFD code development, possible? | Dr. Yazid Bindar | Main CFD Forum | 27 | July 18, 2000 00:18 |
State of the art in CFD technology | Juan Carlos GARCIA SALAS | Main CFD Forum | 39 | November 1, 1999 14:34 |
public CFD Code development | Heinz Wilkening | Main CFD Forum | 38 | March 5, 1999 11:44 |
What kind of Cmmercial CFD code you feel well? | Lans | Main CFD Forum | 13 | October 27, 1998 10:20 |