HELLLLP: Simulation abort with error #001100279 In a Cluster
 

Hi to all,

I am using a Cluster to run a CFX simulation. The simulation could run correctly at first, including Partitioning, initialization. It can simulate for about 3000+ timesteps and an error would occur.

The last simulation calculated for 3495 timesteps and the same error occur.
Hope someone can help out:D
================================================== ====================
TIME STEP = 3495 SIMULATION TIME = 2.0970E-02 CPU SECONDS = 6.303E+07
----------------------------------------------------------------------
Domain: Rotor
Rotated in this time step [degrees] = 5.1480E-01
Rotated up to now [degrees] = 7.2051E+03
Rotated number of pitches up to now = 20.014
----------------------------------------------------------------------
Domain interface: In_R
Non-overlap area fraction on side 1 = 1.30E-07
Non-overlap area fraction on side 2 = 3.87E-04
Domain interface: R_S
Non-overlap area fraction on side 1 = 9.03E-06
Non-overlap area fraction on side 2 = 2.93E-05
----------------------------------------------------------------------
| SOLVING : Wall Scale |
----------------------------------------------------------------------
| Equation | Rate | RMS Res | Max Res | Linear Solution |
+----------------------+------+---------+---------+------------------+
| Wallscale | 1.00 | 4.5E-05 | 3.5E-03 | 6.0 7.0E-02 OK|
+----------------------+------+---------+---------+------------------+
| Wallscale | 0.25 | 1.1E-05 | 8.5E-04 | 11.0 4.3E-02 OK|
+----------------------+------+---------+---------+------------------+
| Wallscale | 0.26 | 3.0E-06 | 2.1E-04 | 11.0 9.3E-02 OK|
+----------------------+------+---------+---------+------------------+
| Wallscale | 0.28 | 8.4E-07 | 5.3E-05 | 21.0 8.1E-02 OK|
+----------------------+------+---------+---------+------------------+
| Wallscale | 0.30 | 2.5E-07 | 1.4E-05 | 16.0 9.9E-02 OK|
+----------------------+------+---------+---------+------------------+
| Wallscale | 0.37 | 9.4E-08 | 3.7E-06 | 11.0 9.0E-02 OK|
+----------------------+------+---------+---------+------------------+
| Wallscale | 0.50 | 4.7E-08 | 2.3E-06 | 6.0 9.2E-02 OK|
+----------------------+------+---------+---------+------------------+
| Wallscale | 0.70 | 3.3E-08 | 1.9E-06 | 6.0 7.7E-02 OK|
+----------------------+------+---------+---------+------------------+
| Wallscale | 0.75 | 2.5E-08 | 1.5E-06 | 6.0 6.5E-02 OK|
+----------------------+------+---------+---------+------------------+
| Wallscale | 0.77 | 1.9E-08 | 1.2E-06 | 6.0 5.6E-02 OK|
+----------------------+------+---------+---------+------------------+
----------------------------------------------------------------------
COEFFICIENT LOOP ITERATION = 1 CPU SECONDS = 6.303E+07
----------------------------------------------------------------------
| Equation | Rate | RMS Res | Max Res | Linear Solution |
+----------------------+------+---------+---------+------------------+
| U-Mom | 1.00 | 4.9E-04 | 3.5E-02 | 1.1E-02 OK|
| V-Mom | 1.00 | 5.0E-04 | 3.7E-02 | 1.0E-02 OK|
| W-Mom | 1.00 | 7.3E-04 | 4.9E-02 | 1.0E-02 OK|
| P-Mass | 1.01 | 9.8E-05 | 1.0E-02 | 5.5 3.7E-02 OK|
+----------------------+------+---------+---------+------------------+
| H-Energy | 0.87 | 9.7E-04 | 4.5E-02 | 5.9 1.3E-02 OK|
+----------------------+------+---------+---------+------------------+
| K-TurbKE | 0.80 | 3.8E-04 | 4.1E-02 | 5.9 4.6E-03 OK|
| O-TurbFreq | 1.05 | 3.5E-04 | 1.5E-01 | 11.6 2.2E-02 OK|
+----------------------+------+---------+---------+------------------+
----------------------------------------------------------------------
COEFFICIENT LOOP ITERATION = 2 CPU SECONDS = 6.303E+07
----------------------------------------------------------------------
| Equation | Rate | RMS Res | Max Res | Linear Solution |
+----------------------+------+---------+---------+------------------+
| U-Mom | 0.29 | 1.4E-04 | 5.2E-02 | 7.1E-02 OK|
| V-Mom | 0.29 | 1.4E-04 | 3.9E-02 | 6.3E-02 OK|
| W-Mom | 0.18 | 1.4E-04 | 2.2E-02 | 7.9E-02 OK|
| P-Mass | 0.18 | 1.8E-05 | 1.5E-03 | 5.5 5.2E-02 OK|
+----------------------+------+---------+---------+------------------+
| H-Energy | 0.27 | 2.6E-04 | 2.9E-02 | 5.9 1.4E-02 OK|
+----------------------+------+---------+---------+------------------+
| K-TurbKE | 2.50 | 9.4E-04 | 7.7E-01 | 5.9 2.4E-02 OK|
| O-TurbFreq | 0.97 | 3.4E-04 | 1.9E-01 | 11.6 8.4E-04 OK|
+----------------------+------+---------+---------+------------------+
----------------------------------------------------------------------
COEFFICIENT LOOP ITERATION = 3 CPU SECONDS = 6.303E+07
----------------------------------------------------------------------
| Equation | Rate | RMS Res | Max Res | Linear Solution |
+----------------------+------+---------+---------+------------------+
| U-Mom | 0.54 | 7.6E-05 | 2.7E-02 | 8.4E-02 OK|
| V-Mom | 0.52 | 7.4E-05 | 2.1E-02 | 7.3E-02 OK|
| W-Mom | 0.52 | 7.0E-05 | 1.2E-02 | 9.3E-02 OK|
| P-Mass | 0.40 | 7.2E-06 | 9.2E-04 | 9.9 6.0E-02 OK|
+----------------------+------+---------+---------+------------------+
| H-Energy | 0.85 | 2.3E-04 | 5.0E-02 | 5.9 5.6E-02 OK|
+----------------------+------+---------+---------+------------------+
| K-TurbKE | 0.10 | 9.7E-05 | 1.2E-01 | 5.9 6.3E-02 OK|
| O-TurbFreq | 0.45 | 1.5E-04 | 9.7E-02 | 11.6 4.0E-04 OK|
+----------------------+------+---------+---------+------------------+
----------------------------------------------------------------------
COEFFICIENT LOOP ITERATION = 4 CPU SECONDS = 6.303E+07
----------------------------------------------------------------------
| Equation | Rate | RMS Res | Max Res | Linear Solution |
+----------------------+------+---------+---------+------------------+
| U-Mom | 0.77 | 5.9E-05 | 1.8E-02 | 7.9E-02 OK|
| V-Mom | 0.86 | 6.3E-05 | 2.3E-02 | 6.7E-02 OK|
| W-Mom | 0.83 | 5.8E-05 | 9.1E-03 | 8.4E-02 OK|
| P-Mass | 0.57 | 4.1E-06 | 7.1E-04 | 9.9 8.3E-02 OK|
+----------------------+------+---------+---------+------------------+
| H-Energy | 0.53 | 1.2E-04 | 3.0E-02 | 5.9 8.3E-02 OK|
+----------------------+------+---------+---------+------------------+
| K-TurbKE | 0.77 | 7.5E-05 | 6.0E-02 | 5.9 8.4E-02 OK|
| O-TurbFreq | 0.57 | 8.6E-05 | 7.3E-02 | 11.6 3.8E-04 OK|
+----------------------+------+---------+---------+------------------+
----------------------------------------------------------------------
COEFFICIENT LOOP ITERATION = 5 CPU SECONDS = 6.304E+07
----------------------------------------------------------------------
| Equation | Rate | RMS Res | Max Res | Linear Solution |
+----------------------+------+---------+---------+------------------+
| U-Mom | 0.52 | 3.1E-05 | 1.5E-02 | 5.8E-02 OK|
| V-Mom | 0.54 | 3.4E-05 | 2.1E-02 | 4.9E-02 OK|
| W-Mom | 0.49 | 2.8E-05 | 5.1E-03 | 5.8E-02 OK|
| P-Mass | 0.67 | 2.8E-06 | 7.6E-04 | 14.4 6.9E-02 OK|
+----------------------+------+---------+---------+------------------+
| H-Energy | 0.60 | 7.1E-05 | 1.5E-02 | 5.9 9.5E-02 OK|
+----------------------+------+---------+---------+------------------+
| K-TurbKE | 0.62 | 4.7E-05 | 4.0E-02 | 10.9 5.4E-02 OK|
| O-TurbFreq | 0.60 | 5.1E-05 | 3.3E-02 | 11.6 3.6E-04 OK|
+----------------------+------+---------+---------+------------------+
----------------------------------------------------------------------
COEFFICIENT LOOP ITERATION = 6 CPU SECONDS = 6.304E+07
----------------------------------------------------------------------
| Equation | Rate | RMS Res | Max Res | Linear Solution |
================================================== ====================
| Timestepping Information |
----------------------------------------------------------------------
| Timestep | Courant Number | Acoustic Courant Number |
+----------------+-------------------------+-------------------------+
| | RMS MAX | RMS MAX |
+----------------+-------------------------+-------------------------+
| 6.0000E-06 | 22.88 982.67 | 332.02 999.99 |
----------------------------------------------------------------------

+--------------------------------------------------------------------+
| ERROR #001100279 has occurred in subroutine ErrAction. |
| Message: |
| abort file found. Aborting run. |
| |
| |
| |
| |
| |
+--------------------------------------------------------------------+

+--------------------------------------------------------------------+
| ERROR #001100279 has occurred in subroutine ErrAction. |
| Message: |
| Signal caught: Abnormal termination |
| |
| |
| |
| |
| |
+--------------------------------------------------------------------+

+--------------------------------------------------------------------+
| ERROR #001100279 has occurred in subroutine ErrAction. |
| Message: |
| Stopped in routine FPX: SIG_HANDLER |
| |
| |
| |
| |
| |
+--------------------------------------------------------------------+

+--------------------------------------------------------------------+
| An error has occurred in cfx5solve: |
| |
| The ANSYS CFX solver exited with return code 1. No results file |
| has been created. |
+--------------------------------------------------------------------+

End of solution stage.

+--------------------------------------------------------------------+
| The following transient and backup files written by the ANSYS CFX |
| solver have been saved in the directory |
| /public/SimulationData/120Degree_0.2DC_65/120Degree_Distortion_65- |
| _Remeshed_Throttle_3dot4_001: |
| |
| 950_full.trn, 900_full.trn, 850_full.trn, 800_full.trn, |
| 750_full.trn, 700_full.trn, 650_full.trn, 600_full.trn, |
| 550_full.trn, 50_full.trn, 500_full.trn, 450_full.trn, |
| 400_full.trn, 350_full.trn, 3450_full.trn, 3400_full.trn, |
| 3350_full.trn, 3300_full.trn, 3250_full.trn, 3200_full.trn, |
| 3150_full.trn, 3100_full.trn, 3050_full.trn, 300_full.trn, |
| 3000_full.trn, 2950_full.trn, 2900_full.trn, 2850_full.trn, |
| 2800_full.trn, 2750_full.trn, 2700_full.trn, 2650_full.trn, |
| 2600_full.trn, 2550_full.trn, 250_full.trn, 2500_full.trn, |
| 2450_full.trn, 2400_full.trn, 2350_full.trn, 2300_full.trn, |
| 2250_full.trn, 2200_full.trn, 2150_full.trn, 2100_full.trn, |
| 2050_full.trn, 200_full.trn, 2000_full.trn, 1950_full.trn, |
| 1900_full.trn, 1850_full.trn, 1800_full.trn, 1750_full.trn, |
| 1700_full.trn, 1650_full.trn, 1600_full.trn, 1550_full.trn, |
| 150_full.trn, 1500_full.trn, 1450_full.trn, 1400_full.trn, |
| 1350_full.trn, 1300_full.trn, 1250_full.trn, 1200_full.trn, |
| 1150_full.trn, 1100_full.trn, 1050_full.trn, 100_full.trn, |
| 1000_full.trn, 0_full.trn |
+--------------------------------------------------------------------+


+--------------------------------------------------------------------+
| The following user files have been saved in the directory |
| /public/SimulationData/120Degree_0.2DC_65/120Degree_Distortion_65- |
| _Remeshed_Throttle_3dot4_001: |
| |
| trace, mon, pids |
+--------------------------------------------------------------------+


+--------------------------------------------------------------------+
| Warning! |
| |
| After waiting for 60 seconds, 1 solver manager process(es) appear |
| not to have noticed that this run has ended. You may get errors |
| removing some files if they are still open in the solver manager. |
+--------------------------------------------------------------------+


This run of the ANSYS CFX Solver has finished.

Try a smaller time step (your acoustic Courant Number is very large)

Also: double precision numerics, improve mesh quality, tighter convergence criteria.

Thank you for your reply, Glenn. I noticed the courant number in my simulation is very large, and I will try to refine follow your advice. However, the error I referred in this question are independent of the courant number because I tried to run the simulation on a windows system computer and no error occurred. So, I suspect that this error might relate to some settings in Linux System. I am totally new guy to the Linux System. Hopefully, Some Linux guy can help me out :D

In that case it suggests that the simulation is borderline numerically stable, and small differences (eg details of how floating point numbers are handled over a cluster between Windows and Linux) have made a simulation go from borderline stable to borderline unstable.

The fix is the same: Smaller time step, double precision numerics, improve mesh quality, tighter convergence criteria.

That make sense. Thank you again! I am going to work through the suggestions.:D

What release version are you running? I recall this error some time ago (@2021), and I think it has been fixed in later releases.

I recall a workaround at the time, but it may create other problems like parallel locking if something else goes wrong.

Advice: upgrade your installation to a more recent version.

I am using AYSYS 2021 R1, exactly the time you found this problem. I will try to contact IT stuff to see if they can upgrade to the last version.
Thank you very much! :D