[Santhosh91]

Hello All,


I am encountering an issue with running interFoam solver in Compute Canada clusters.

Using an adpatative mesh refinement (AMR) along with refined mesh cells near the walls. I have a situation that requires a lot of computational resources. So I am trying to parallelize as much as I can.


However, as I tried to run the simulation on 64 CPUs for instance ( I always make sure, I am using one whole node). The simulation diverges as it seen in the attached picture (pressure residuals blows up).
The decomposition on 4 processors on my computer works fine and it also works with a decomposition on 16 processors in Compute Canada! I don't understand why increasing the number of processors to 32, 64 make the simulation diverges. I tried to contact the support but they couldn't figure out the issue...
I must mention that all simulations using uniform meshing works well on any type of decomposition (16,32,64,...). Meaning it must be linked to the refinement and AMR process but I can't figure out the issue.



Has someone already encountered this type of problem ? Or have any suggestions ?


Thanks for your time and help.


Sincerely,
Santhosh

[jmt_HX]

Interesting problem. I worked with OpenFOAM AMR for a little bit but am no expert.

Does your OpenFOAM AMR implementation have the ability for dynamic load balancing?

What decomposition method did you use?

[Santhosh91]

Hello jmt,


I didn't try the dynamic load balancing, it seems that there is possibility since some people manage to successfully implemented it on OpenFOAM. It might be the answer.

I am using a simple decomposition.

[jmt_HX]

Cool. Can you send me your full log file (i imagine it is small since the crash happens iteration 1)? And your fvSolution file?

Thanks.

[Santhosh91]

Sure! Thanks for the interest. You can find the files as attached files.

[AtoHM]

Trying with dynamic load balancing seems to be a good idea (have not tried this myself yet though). Another hint: I have seen threads in the CFX subforum here, where undesirable locations for the processor boundaries caused divergence in multiphase flows. If the decomposition interface coincides with a high volume fraction gradient, instabilities may result. Another decomposition might help to improve this? Just an idea.

[jmt]

Originally Posted by AtoHM

Trying with dynamic load balancing seems to be a good idea (have not tried this myself yet though). Another hint: I have seen threads in the CFX subforum here, where undesirable locations for the processor boundaries caused divergence in multiphase flows. If the decomposition interface coincides with a high volume fraction gradient, instabilities may result. Another decomposition might help to improve this? Just an idea.

This is a great suggestion--it does seem that based on your description, the problem has something to do with the decomposition.

As AtoHM suggested, could you use some fraction of the node just to test? Decompose the domain for 32, 16, 8 ranks etc and run a few iterations on the cluster?

Thanks for the input and log file. I'll take a look today.

[Santhosh91]

Hello Ato, jmt


Thanks for the inputs! Those help me to get a clearer idea of what might be the issue. I will try to relaunch with another decomposition to see if it works. Otherwise,I'll give a shot to the dynamic load balancing.

[Santhosh91]

Hello all,


Just a quick update. I switched the decomposition method from Simple to Scotch and it is working!



Thanks again for the help