[aerosayan]

Hello everyone,

I'm trying to automate some kind of regression testing mechanism for ensuring that the code produces same results for same test cases every time...

Checking the solution (i.e conserved values) of the mesh, would be accurate, but somewhat difficult to implement.

I just want to do it in the laziest way possible for now. Would checking the progression of the residuals be accurate enough?

For steady state simulations the residual plots obviously follow the same pattern every time. But for unsteady simulations (i didn't code them yet) the residual plots is very erratic and unstable.

I'm thinking of only checking the final residual values at convergence, instead of checking every point on the residual plots, to ensure the same residual plot is generated every time.

Is this correct?

Can we just check the residuals at the final convergence, and say that our simulation is correct, or do we need to check and verify that the same residual plot is generated every time?


That is, Is the graph shown below important, or only the final residual results at the last iteration?

[sbaffini]

Originally Posted by aerosayan

Hello everyone,

I'm trying to automate some kind of regression testing mechanism for ensuring that the code produces same results for same test cases every time...

Checking the solution (i.e conserved values) of the mesh, would be accurate, but somewhat difficult to implement.

I just want to do it in the laziest way possible for now. Would checking the progression of the residuals be accurate enough?

For steady state simulations the residual plots obviously follow the same pattern every time. But for unsteady simulations (i didn't code them yet) the residual plots is very erratic and unstable.

I'm thinking of only checking the final residual values at convergence, instead of checking every point on the residual plots, to ensure the same residual plot is generated every time.

Is this correct?

Can we just check the residuals at the final convergence, and say that our simulation is correct, or do we need to check and verify that the same residual plot is generated every time?


That is, Is the graph shown below important, or only the final residual results at the last iteration?

Let's supersede the fact that NO, it is not ok for the unsteady residuals to show unpredictable behavior. Even if you were performing DNS, as long as done with the same code on the same machine and boundary/initial conditions, for meaningfully short times you should have identical results if your code has a deterministic behavior.

The general question you ask really has no single answer. Roughly speaking, my point of view is that it is more than ok, but with a caveat. I am fully aware of the fact that I am fully exposed to some problems.

Typically, one either wants to check that, after a modification, the code performs exactly the same, bit to bit, or that performances (speed and/or accuracy) have improved. In the first case, I personally only trust pure diffs on binary outputs, nothing less. In the latter case, however, things are less obvious. Are you willing to accept more speed at lower (absolute) accuracy? What about the opposite? Do you expect, with your modification, to achieve both?

What I want to convey here is the fact that a specific decision on any given metric is only meaningful after a specific modification is taken into account so, chasing a perfect approach is elusive. So the next question is, do I really understand and/or am I really able to generalize any metric based on the full convergence graph which is different from the residual at the end of the prescribed number of iterations? If an RMS of the residual was available as a measure, would I really understand it when comparing multiple cases?

In my case all the above questions had a no as an answer, which lead me to the following: I use the final residual as a measure, but I know it is something just for the sake of it. In the end, I use it more for the cases where identical binary results are expected than the others. Yet, it really is sufficently powerful to be useful. But:

1) It is isn't really any robust

2) Going beyond it doesn't give any meaningful further advantage in my case, so I don't

[Eifoehn4]

Regression testing is meant to check if a recent code change has adversely affected existing features. The test won't tell you if your code works correct.

If you want to check many features, the final iteration step might become expensive in the future. Generally a binary check after a few time steps should be sufficient.

[aerosayan]

Originally Posted by Eifoehn4

Generally a binary check after a few time steps should be sufficient.

Do you mean, we should write the solution state (conserved values of each cell) into a binary file, and match it with a similar binary file that has been previously manually verified?

[Eifoehn4]

Yes or if your IO uses HDF5 you can apply a h5diff on your state file. Here you also can set tolerances if needed.

[noah6273]

hey good one