DecomposePar

veeturi · May 13, 2020, 17:58

Hello Foamers,

I have recently started using openFOAM and I use an HPC cluster to run my simulations. Since I am running the job on parallel cores, I am using the decomposePar file in which I specify the method of distribution (Simple, Hierarchical or scotch). Currently I am using scotch however, I see that the reconstruction of the distributed domain takes a lot of time.

In my specific case, I have a mesh of 5 million elements and I run this on 6 nodes with 40 cores each (a total of 240 cores). The simulation itself (running a pimpleFoam model) takes about 10 hours which is good however, the reconstructPar takes 5 hours! which is 30% of the total time. Am I missing something? Will it make a difference if I use Simple or the hierarchical method? I'm not exactly sure why the reconstruction takes so long.
Experts, please correct any misconception I may have on my part.

Thank you.

HPE · May 14, 2020, 01:58

Hi

- reconstructPar is a serial execution utility.
- please try to use 'mpirun -np X redistributePar -reconstruct -parallel -latestTime', to see if it helps.

veeturi · May 14, 2020, 19:07

Yes. That helped. Actually I'm running a field average as well on the simulation hence, all the results I need are in the last time step.

Thank you!

May 13, 2020, 17:58	DecomposePar	#1
veeturi New Member Sricharan S Veeturi Join Date: Jun 2016 Posts: 5 Rep Power: 9	Hello Foamers, I have recently started using openFOAM and I use an HPC cluster to run my simulations. Since I am running the job on parallel cores, I am using the decomposePar file in which I specify the method of distribution (Simple, Hierarchical or scotch). Currently I am using scotch however, I see that the reconstruction of the distributed domain takes a lot of time. In my specific case, I have a mesh of 5 million elements and I run this on 6 nodes with 40 cores each (a total of 240 cores). The simulation itself (running a pimpleFoam model) takes about 10 hours which is good however, the reconstructPar takes 5 hours! which is 30% of the total time. Am I missing something? Will it make a difference if I use Simple or the hierarchical method? I'm not exactly sure why the reconstruction takes so long. Experts, please correct any misconception I may have on my part. Thank you. Last edited by veeturi; May 14, 2020 at 19:05.

May 14, 2020, 01:58		#2
HPE Senior Member Herpes Free Engineer Join Date: Sep 2019 Location: The Home Under The Ground with the Lost Boys Posts: 932 Rep Power: 12	Hi - reconstructPar is a serial execution utility. - please try to use 'mpirun -np X redistributePar -reconstruct -parallel -latestTime', to see if it helps. cryabroad likes this. __________________ The OpenFOAM community is the biggest contributor to OpenFOAM: User guide/Wiki-1/Wiki-2/Code guide/Code Wiki/Journal Nilsson/Guerrero/Holzinger/Holzmann/Nagy/Santos/Nozaki/Jasak/Primer Governance Bugs/Features: OpenFOAM (ESI-OpenCFD-Trademark) Bugs/Features: FOAM-Extend (Wikki-FSB) Bugs: OpenFOAM.org How to create a MWE New: Forkable OpenFOAM mirror

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Problem with decomposePar (and mapFields) for large problem	quarkz	OpenFOAM Pre-Processing	2	February 21, 2019 09:51
decomposePar problem: Cell 0contains face labels out of range (Again))	limonegiallo	OpenFOAM Pre-Processing	4	August 28, 2017 05:18
decomposePar error	chia87	OpenFOAM Pre-Processing	1	May 28, 2017 15:23
decomposePar 4-core warning/error?	Boloar	OpenFOAM Bugs	23	April 8, 2014 08:57
decomposePar gives errors	of_user_	OpenFOAM	1	July 4, 2011 05:27

May 14, 2020, 19:07		#3
veeturi New Member Sricharan S Veeturi Join Date: Jun 2016 Posts: 5 Rep Power: 9	Yes. That helped. Actually I'm running a field average as well on the simulation hence, all the results I need are in the last time step. Thank you!