CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Programming & Development

Code parallelization using Pstream: simultaneous send/receive deadlocking

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree1Likes
  • 1 Post By wyldckat

Reply
 
LinkBack Thread Tools Display Modes
Old   July 9, 2012, 06:14
Question Code parallelization using Pstream: simultaneous send/receive deadlocking
  #1
Senior Member
 
Tomislav Maric
Join Date: Mar 2009
Location: Darmstadt, Germany
Posts: 260
Blog Entries: 5
Rep Power: 11
tomislav_maric is on a distinguished road
Hi everyone,

I am trying to write the simplest parallel code that involves simultaneous communication accross processes:

Proc1: Send(0),Receive(0)

Proc2: Send(1),Receive(1)

Of course, such code will experience a deadlock. There are multiple solutions for this beginner problem: exchanging the order of Send/Receive based on the process rank, using non-blocking communication, using buffered communication (bSend, bRecv), etc.

Both the problem and possible solutions are explained here (Note: if the links don't work by clicking, copy/paste them):

http://www.ncsa.illinois.edu/UserInf...IDeadlock.html

and here:

http://www.cs.ucsb.edu/~hnielsen/cs140/mpi-deadlocks.html

So. I have tried to use these solutions in OpenFOAM, sending/receiving a scalarField around, and I keep getting errors. I linked the application source below, where you can uncomment parts of the code to get specific solutions (described in the links above). What I find strange is:

  • the code deadlocks when I properly combine blocking send/receive and receive/send
  • the code experiences FoamFatalError when I use nonBlocking communication: UPstream::waitRequests() is an empty function, Pstream::nRequests() returns 0 regardless of the number of outstanding nonBlocking requests in the program


I am not sure if I am missing something obvious, but it seems strange that every option I try, fails.


Also: I've switched on the Pstream debugSwitch in etc/controlDict, but I still don't see the debug info comming from the UPstream.C in the Pstream/mpi folder. This looks like my application is using the dummy implementation of UPstream and its children, instead of linking with stuff from Pstream/mpi.



Download link (OpenFOAM 2.0.x):


http://dl.dropbox.com/u/3316983/pStr...leExchange.tgz


Tomislav

Last edited by tomislav_maric; July 9, 2012 at 06:31.
tomislav_maric is offline   Reply With Quote

Old   July 13, 2012, 04:47
Default
  #2
Senior Member
 
Tomislav Maric
Join Date: Mar 2009
Location: Darmstadt, Germany
Posts: 260
Blog Entries: 5
Rep Power: 11
tomislav_maric is on a distinguished road
(bump)

O.K. , let me rephrase the question: there are two processes available.

How do I use Pstream to exchange the number Pstream::myProcNo() between the processes without deadlocking?
tomislav_maric is offline   Reply With Quote

Old   July 13, 2012, 05:14
Default
  #3
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,488
Blog Entries: 34
Rep Power: 86
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings Tomislav,

Have you looked at the source code for the OpenFOAM application Test-parallel? It's located at "applications/test/parallel".
It's a very decent example of what you seem to be trying to do!

Browse the other applications in the test folder as well, because I remember there are other very sweet examples of parallel manipulations!

Good luck!
Bruno
tomislav_maric likes this.
wyldckat is offline   Reply With Quote

Old   July 13, 2012, 05:16
Default
  #4
Senior Member
 
Tomislav Maric
Join Date: Mar 2009
Location: Darmstadt, Germany
Posts: 260
Blog Entries: 5
Rep Power: 11
tomislav_maric is on a distinguished road
Thanks, I'm on it!
tomislav_maric is offline   Reply With Quote

Old   July 13, 2012, 05:20
Default
  #5
Senior Member
 
akidess's Avatar
 
Anton Kidess
Join Date: May 2009
Location: Delft, Netherlands
Posts: 919
Rep Power: 17
akidess will become famous soon enough
What went wrong with the ucsb solution? It seems brilliantly simple...

Code:
if (rank % 2) {
    MPI_Send(data, (rank + 1) % size);
    MPI_Recv(newdata, (rank - 1) % size);
} else {
    MPI_Recv(newdata, (rank - 1) % size);
    MPI_Send(data, (rank + 1) % size);
}
__________________
*On twitter @akidTwit
*Spend as much time formulating your questions as you expect people to spend on their answer.
*Help define the OpenFOAM stackexchange Q&A site: http://area51.stackexchange.com/prop...oam-technology
akidess is offline   Reply With Quote

Old   July 13, 2012, 05:22
Default
  #6
Senior Member
 
Tomislav Maric
Join Date: Mar 2009
Location: Darmstadt, Germany
Posts: 260
Blog Entries: 5
Rep Power: 11
tomislav_maric is on a distinguished road
Quote:
Originally Posted by akidess View Post
What went wrong with the ucsb solution? It seems brilliantly simple...

Code:
if (rank % 2) {
    MPI_Send(data, (rank + 1) % size);
    MPI_Recv(newdata, (rank - 1) % size);
} else {
    MPI_Recv(newdata, (rank - 1) % size);
    MPI_Send(data, (rank + 1) % size);
}
Well, when you try this with Pstream, the thing just hangs in the air... If you don't believe me, uncomment SOLUTION 0 in the source I linked below... it's just weird.

Added: I wrote it for just 2 processes: 0 is sending/receiving to/from 1, and 1 is sending/receiving to/from 0.
tomislav_maric is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
What is the Better Way to Do CFD? John C. Chien Main CFD Forum 54 April 23, 2001 08:10
Open source CFD code development, possible? Dr. Yazid Bindar Main CFD Forum 27 July 18, 2000 00:18
State of the art in CFD technology Juan Carlos GARCIA SALAS Main CFD Forum 39 November 1, 1999 15:34
public CFD Code development Heinz Wilkening Main CFD Forum 38 March 5, 1999 12:44
What kind of Cmmercial CFD code you feel well? Lans Main CFD Forum 13 October 27, 1998 11:20


All times are GMT -4. The time now is 18:53.