CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Meshing & Mesh Conversion

[snappyHexMesh] decomposotion options - please help

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   February 4, 2020, 12:40
Default decomposotion options - please help
  #1
New Member
 
Rob
Join Date: Apr 2018
Posts: 17
Rep Power: 3
Morlind is on a distinguished road
I currently run cases on separate servers but I am attempting to restructure into a proper cluster. Unfortunately my scaling has been poor and I suspect decomposition is partly to blame.



I have previously attached 2 servers together and achieved a ~1.2:1 speed up which I was very happy with. When clustering 6x machines I only complete a solve in roughly half the time of a single machine. When utilizing 3x machines I see iteration times of only slightly faster than a single machine (31s/iteration vs 36s/iteration).



I would like to try different methods but my case is not uniform in cell distribution so the Simple and Hierarchical arrangements don't seem like they will work well at first glance.



Also, minimizing cell interfaces is great, but what I really need is to minimize interfaces between the larger portions of each machine to reduce network utilization. Is there a method for this?
Morlind is offline   Reply With Quote

Old   February 5, 2020, 00:58
Default
  #2
Senior Member
 
Join Date: Aug 2013
Posts: 387
Rep Power: 10
Antimony is on a distinguished road
Hi,

Have you tried scotch as your decomposition method? That is supposed to give you a very load balanced mesh distribution...

Cheers,
Antimony
Antimony is offline   Reply With Quote

Old   February 14, 2020, 12:10
Default
  #3
New Member
 
Rob
Join Date: Apr 2018
Posts: 17
Rep Power: 3
Morlind is on a distinguished road
As an update:
Yes, absolutely scotch was where I started and all I have previously used.

I attempted various hierarchical methods and all were slower.

I decided to try using the 1gb network rather than the 10gb network (with the idea to see if it was slower, by how much, and if the same speed then my 10gb net just isn't functioning at peak speed). Surprise, the 1gb network was faster, achieving a reasonable speed-up (ie. 30% s/iteration reduction using 3x nodes). However, this is not consistent. It seems I have some network issues, I believe with the OS or drivers, that need fixing so I am approaching it from there.
Morlind is offline   Reply With Quote

Old   February 14, 2020, 16:32
Default
  #4
HPE
Senior Member
 
Herpes Free Engineer
Join Date: Sep 2019
Posts: 298
Rep Power: 3
HPE is on a distinguished road
What is the order of cells per processor? OF is known to scale well (up to 1.5 billion cells with virtually ideal weak scaling). If you have very few cells per proc, you very likely end up speed decrease at the expense of communication overhead.

You may run 'mpirun -np <procNo> renumberMesh -overwrite -parallel' to gain some speed.

Also, turn off the run time modification, and if possible all FOs. Selecting parallel-consistent solvers, particularly for pressure, is very important as well.

Might compilation with the highest level of optimisation can help.
__________________
Kind regards,
HPE

The OpenFOAM community is the biggest contributor to OpenFOAM; not person A or B. Let's report bugs or issue feature requests in GitLab-OpenFOAM.

Please `design` your questions well. Such effort always pays off well for inquirers and `helpmate`s:
How to create a Minimal, Reproducible Example,
How to give enough info to get help.
HPE is offline   Reply With Quote

Old   February 16, 2020, 16:59
Default
  #5
Senior Member
 
Join Date: Nov 2010
Location: USA
Posts: 1,204
Rep Power: 20
me3840 is on a distinguished road
Quote:
Originally Posted by HPE View Post
Selecting parallel-consistent solvers, particularly for pressure, is very important as well.

Just curious as to what exactly you mean by this?
me3840 is offline   Reply With Quote

Old   February 18, 2020, 17:11
Default
  #6
New Member
 
Rob
Join Date: Apr 2018
Posts: 17
Rep Power: 3
Morlind is on a distinguished road
I am getting between 60m - 80m cell meshes. I have previously seen much better scaling than I currently am, hence the real solution:


- I was getting very inconsistent iteration times when using multiple nodes so I tinkered with which NIC was being used. Using the 1gb (not the 10gb) I actually got faster times. However, this was not consistently repeatable. In some cases it was better if I switched which subnet was being used twice, coming back to the original to find it working faster than before. A friend commented "eh, probably a driver issue"


- On rebooting some machines the NIC was no longer accessible. The firmware was missing. This caused me to head down a variety of paths but in the end I was able to manually download and provide it the correct firmware which Ubuntu was already looking for (It had updated the firmware entry but not placed the actual firmware file in the location directed to).



- Yay, now my NIC's work. Hey, let's try a case. Surprise: Everything is working as intended and scaling well. Using 80 cores (2x nodes) my iteration time is almost exactly half of a 40 core solve. (23 s/iter for 79m cells vs 42 s/iter for 78m cells).



I realize this no longer really fits in this sub forum but this was the solution to the real issue which caused me to investigate decomposition options.
Morlind is offline   Reply With Quote

Old   February 18, 2020, 17:21
Default
  #7
HPE
Senior Member
 
Herpes Free Engineer
Join Date: Sep 2019
Posts: 298
Rep Power: 3
HPE is on a distinguished road
So, problem solved? No longer related to OpenFOAM?

May be an idiot was using the parent node without batching simulations and slowing down the entire system at the same time you do run your simulations in child nodes?
__________________
Kind regards,
HPE

The OpenFOAM community is the biggest contributor to OpenFOAM; not person A or B. Let's report bugs or issue feature requests in GitLab-OpenFOAM.

Please `design` your questions well. Such effort always pays off well for inquirers and `helpmate`s:
How to create a Minimal, Reproducible Example,
How to give enough info to get help.
HPE is offline   Reply With Quote

Old   February 18, 2020, 17:26
Default
  #8
HPE
Senior Member
 
Herpes Free Engineer
Join Date: Sep 2019
Posts: 298
Rep Power: 3
HPE is on a distinguished road
Hi me3840,

e.g. GAMG is inherently not parallel consistent, but PCG. Please do have a look at their diffs, since I usually try not to answer any engineering judgement questions, but only OpenFOAM-usage related questions.

Hope you understand.
__________________
Kind regards,
HPE

The OpenFOAM community is the biggest contributor to OpenFOAM; not person A or B. Let's report bugs or issue feature requests in GitLab-OpenFOAM.

Please `design` your questions well. Such effort always pays off well for inquirers and `helpmate`s:
How to create a Minimal, Reproducible Example,
How to give enough info to get help.
HPE is offline   Reply With Quote

Old   February 18, 2020, 17:26
Default
  #9
New Member
 
Rob
Join Date: Apr 2018
Posts: 17
Rep Power: 3
Morlind is on a distinguished road
Quote:
Originally Posted by HPE View Post
So, problem solved? No longer related to OpenFOAM?

May be an idiot was using the parent node without batching simulations and slowing down the entire system at the same time you do run your simulations in child nodes?

HA! The only idiot on this system is me... I do plenty of dumb things but the nodes are definitely only used for the cases at hand. I use TSP to keep things in line.



I'm reasonably sure this is exclusively related to the NIC drivers and was never related to decomposition in any way.
Morlind is offline   Reply With Quote

Old   February 18, 2020, 17:28
Default
  #10
HPE
Senior Member
 
Herpes Free Engineer
Join Date: Sep 2019
Posts: 298
Rep Power: 3
HPE is on a distinguished road
ahaha

Sorry!

You dont learn fast if you dont do mistakes, do you? Thanks for the info.
__________________
Kind regards,
HPE

The OpenFOAM community is the biggest contributor to OpenFOAM; not person A or B. Let's report bugs or issue feature requests in GitLab-OpenFOAM.

Please `design` your questions well. Such effort always pays off well for inquirers and `helpmate`s:
How to create a Minimal, Reproducible Example,
How to give enough info to get help.
HPE is offline   Reply With Quote

Old   February 18, 2020, 17:30
Default
  #11
New Member
 
Rob
Join Date: Apr 2018
Posts: 17
Rep Power: 3
Morlind is on a distinguished road
Thanks for the solver tip! I will investigate this further.
Morlind is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
fluent text command doesnt show all options! m2montazari FLUENT 3 March 13, 2017 00:56
About Mesh Interface options scarecrow FLUENT 0 July 13, 2016 05:34
correction of Grub after installing Windows XP and 8 immortality Lounge 20 January 5, 2014 18:41
Moving mesh analysis Work-flow and options pici OpenFOAM Programming & Development 9 June 18, 2013 14:19
Recommended Compiler Options for OpenFOAM on a HPC JLight OpenFOAM Installation 1 December 12, 2012 10:15


All times are GMT -4. The time now is 10:22.