CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Bugs

HPMPI Infiniband problem

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   January 23, 2009, 09:11
Default Hi there, I'm not sure if t
  #1
Member
 
Carsten Thorenz
Join Date: Mar 2009
Location: Germany
Posts: 34
Rep Power: 8
carsten is on a distinguished road
Hi there,

I'm not sure if this is a bug, but maybe...

When running Openfoam on our cluster (HP-Mpi with Infiniband interconnects) I have the problem that only _small_ cases work correctly. For larger cases the job fails immediately:

[snipped ...]
cn112.13384
cn112.13385
)

Pstream initialized with:
floatTransfer : 0
nProcsSimpleSum : 0
commsType : nonBlocking

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

[29]
[29]
[29] IOstream::check(const char* operation) : error in IOstream "IOstream" for operation operator>>(Istream&, List<t>&) : reading first tok
en
[29]
[29] file: IOstream at line 0.
[29]
[29] From function IOstream::fatalCheck(const char* operation) const
[29] in file db/IOstreams/IOstreams/IOcheck.C at line 73.
[29]
FOAM parallel run exiting


This behaviour was previously reported when porting to HPMPI was not yet finished (http://www.cfd-online.com/OpenFOAM_D...es/1/5302.html). Strangely, for me the problem suddenly appeared for a version of Openfoam that I compiled some time ago and that worked flawlessly. Thus I assume it has something to do with changes of the environment on the cluster on which Openfoam reacts, as the code itself was not changed. On the other hand, all other software on the cluster behaves normally, so there is probably no problem with the machine itself.

To complicate matters further, this problem only occurs if the Infiniband-stack is selected for mpi-communication. If I switch to TCPIP it works nicely, albeit slow.

Any help is appreciated

Carsten
carsten is offline   Reply With Quote

Old   January 23, 2009, 12:30
Default Your could try increasing MPI_
  #2
Super Moderator
 
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,416
Rep Power: 16
mattijs is on a distinguished road
Your could try increasing MPI_BUFFER_SIZE. HPMPI might use the buffer space differently.
mattijs is offline   Reply With Quote

Old   January 25, 2009, 16:36
Default Thanks Mattijs. It works ag
  #3
Member
 
Carsten Thorenz
Join Date: Mar 2009
Location: Germany
Posts: 34
Rep Power: 8
carsten is on a distinguished road
Thanks Mattijs.

It works again now. But not due to MPI_BUFFER_SIZE (HPMPI reports explicitly if it is too small), but due to some other event I don't know about. It must be the phase of the moon or the like, because suddenly all versions run again, both for me and for a colleague. To be honest, I could puke. I spent three days hunting a ghost and still don't know what happened. Hope this won't happen again

Many thanks for your time,

Carsten
carsten is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Installation v15 using HPMPI isaac1115 OpenFOAM Installation 3 February 12, 2009 01:29
Infiniband alexander_rudert OpenFOAM 20 January 14, 2009 09:01
HPMPI Compilation problem still carsten OpenFOAM Bugs 3 December 16, 2008 13:23
Case HPMPI missing from etcsettingscsh in OpenFOAM15xgit ruusvuor OpenFOAM Bugs 1 November 26, 2008 08:10
infiniband interconnect help rjh FLUENT 0 February 4, 2008 12:41


All times are GMT -4. The time now is 20:40.