CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > ANSYS > CFX

simulation crashes after hundres of iterations

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree2Likes
  • 1 Post By ghorrocks
  • 1 Post By Opaque

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   May 21, 2021, 22:12
Default simulation crashes after hundres of iterations
  #1
Member
 
phd
Join Date: Oct 2013
Posts: 76
Rep Power: 10
lostking18 is on a distinguished road
Hi, there

Recently while I ran the simulation in CFX, the simulation always crashes with the following warning message:

###############################################
+--------------------------------------------------------------------+
| ERROR #001100279 has occurred in subroutine ErrAction.
| Message:
| Failed to allocate dynamic memory for non-blocking CNL communication.
+--------------------------------------------------------------------+
| ERROR #001100279 has occurred in subroutine ErrAction.
| Message:
| iif_create_empty: error allocating new index: malloc of 2560256 bytes failed
+--------------------------------------------------------------------+
| ERROR #001100279 has occurred in subroutine ErrAction.
| Message:
| iocnt: open the primary file failed
###############################################

Iím running the simulation with work station(Intel I9-18 core, 3.0GHz frequency, 128G Ram). The intel MPI local parallel with 20 partitions is used. This crash comes out almost randomly. Sometimes it works, sometime the simulation will crash after hundreds of iterations. Could you kindly tell me how can I improve my workstation or change the set-up of the solves to eliminate the sort of crash?

Thank you in advance
lostking18 is offline   Reply With Quote

Old   May 22, 2021, 06:10
Default
  #2
Super Moderator
 
Glenn Horrocks
Join Date: Mar 2009
Location: Sydney, Australia
Posts: 17,170
Rep Power: 134
ghorrocks is just really niceghorrocks is just really niceghorrocks is just really niceghorrocks is just really nice
I have no idea what is happening there, I have never seen it before. It is a CFX error message, so CFX is not allocating enough memory for something, but I do not know what CNL communication is. It is definitely something unusual, so there is something unusual about your simulation to trigger it.

All I can suggest is to do some basic debugging. Does it happen if you do the same simulation but with a smaller mesh? Have you tried all the memory options? eg Large problem solver and partitioner, memory allocation factors, and all the other options (both CEL and command line).

Does the problem occur when you run the same simulation but with less cores? (I note you have an 18 core machine and am running 20 partitions - this does not sound wise).
aero_head likes this.
__________________
Note: I do not answer CFD questions by PM. CFD questions should be posted on the forum.
ghorrocks is offline   Reply With Quote

Old   May 22, 2021, 08:48
Default
  #3
Member
 
phd
Join Date: Oct 2013
Posts: 76
Rep Power: 10
lostking18 is on a distinguished road
Hi, Ghorrocks, thank you very much for your always helpful answers.

I already enact the "Large Problem solver". I'm running a turbine stage steady simulation with 15 cooling holes. The total number of mesh nodes is around 10million. There are plenty of interfaces between the cooling injection and the main flow domain.

Sorry that I'm not familiar with partition allocation. Do you suggest I should use 18 partitions instead of 20?



Quote:
Originally Posted by ghorrocks View Post
I have no idea what is happening there, I have never seen it before. It is a CFX error message, so CFX is not allocating enough memory for something, but I do not know what CNL communication is. It is definitely something unusual, so there is something unusual about your simulation to trigger it.

All I can suggest is to do some basic debugging. Does it happen if you do the same simulation but with a smaller mesh? Have you tried all the memory options? eg Large problem solver and partitioner, memory allocation factors, and all the other options (both CEL and command line).

Does the problem occur when you run the same simulation but with less cores? (I note you have an 18 core machine and am running 20 partitions - this does not sound wise).
lostking18 is offline   Reply With Quote

Old   May 22, 2021, 08:51
Default
  #4
Super Moderator
 
Glenn Horrocks
Join Date: Mar 2009
Location: Sydney, Australia
Posts: 17,170
Rep Power: 134
ghorrocks is just really niceghorrocks is just really niceghorrocks is just really niceghorrocks is just really nice
Yes, try 18 partitions.

How many domains are in your simulation?
__________________
Note: I do not answer CFD questions by PM. CFD questions should be posted on the forum.
ghorrocks is offline   Reply With Quote

Old   May 22, 2021, 21:44
Default
  #5
Member
 
phd
Join Date: Oct 2013
Posts: 76
Rep Power: 10
lostking18 is on a distinguished road
Quote:
Originally Posted by ghorrocks View Post
Yes, try 18 partitions.

How many domains are in your simulation?
17domains: 1 for stator, 1 for rotor, the rest 15 for the cooling blocks
lostking18 is offline   Reply With Quote

Old   May 23, 2021, 06:26
Default
  #6
Super Moderator
 
Glenn Horrocks
Join Date: Mar 2009
Location: Sydney, Australia
Posts: 17,170
Rep Power: 134
ghorrocks is just really niceghorrocks is just really niceghorrocks is just really niceghorrocks is just really nice
OK, that is not a huge amount of domains. So what have you tried so far?

Also, if you could post the output file that might help. No guarantees, this is weird one.
__________________
Note: I do not answer CFD questions by PM. CFD questions should be posted on the forum.
ghorrocks is offline   Reply With Quote

Old   May 24, 2021, 07:36
Default
  #7
Senior Member
 
Join Date: Jun 2009
Posts: 1,575
Rep Power: 28
Opaque will become famous soon enough
Would you mind sharing which version you are using?

Very unusual error indeed. Could you also share when it did happen in the overall solution algorithm? Say, what is it when solving one of the equations set? Mass+Momentum, or Energy, or Turbulence?

Could you also post the memory summary in the output file? it should indicate how much of the machine memory is being used by the simulation alone, which will hint at how much memory is left for dynamic allocation later on.

By the way, you said you were running 20 partitions on 18 cores, correct? What was the goal of doing such partitioning?
AtoHM likes this.
__________________
Note: I do not answer CFD questions by PM. CFD questions should be posted on the forum.
Opaque is offline   Reply With Quote

Reply

Tags
crash

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Floating point exception error lpz_michele OpenFOAM Running, Solving & CFD 53 October 19, 2015 02:50
Unstabil Simulation with chtMultiRegionFoam mbay101 OpenFOAM Running, Solving & CFD 13 December 28, 2013 13:12
Micro Scale Pore, icoFoam gooya_kabir OpenFOAM Running, Solving & CFD 2 November 2, 2013 13:58
SLTS+rhoPisoFoam: what is rDeltaT??? nileshjrane OpenFOAM Running, Solving & CFD 4 February 25, 2013 04:13
Error while running rhoPisoFoam.. nileshjrane OpenFOAM Running, Solving & CFD 8 August 26, 2010 12:50


All times are GMT -4. The time now is 15:53.