CFD Online Logo CFD Online URL
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Running, Solving & CFD

My simulations don't scale when I use multiple nodes -help

Register Blogs Members List Search Today's Posts Mark Forums Read

LinkBack Thread Tools Search this Thread Display Modes
Old   February 11, 2020, 18:47
Default My simulations don't scale when I use multiple nodes -help
Join Date: May 2016
Posts: 66
Rep Power: 5
KTG is on a distinguished road
Hey all,

I am having some sort of issue where I am observing a speedup in my simulations as I add more cores within one node, but as soon as I go to two nodes, there is no increase. I can't find any info on this - I tried the -bind-to core mpi flag, but no luck. Has anyone observed this behavior of openFoam on a cluster, or have any ideas about how to fix it? I am using openMPI and SLURM, so I call the solver this way:

mpirun -np $SLURM_NTASKS pimpleFoam -parallel > log

I am using a cluster running centos with these specs:

  • 144 Dual Socket-Six Core Nodes
  • 2.8 GHz Intel Xeon (Westmere X5660) processors
  • RAM:
    • 24 Gbytes memory per node (2 Gbytes per processor core) on most nodes
  • Mellanox QDR Infiniband interconnect
  • Gigabit Ethernet interconnect for management
I was thinking the issue had to do with a bottleneck? Is there a way to check that its actually using the Infiniband ? Any ideas welcome!

KTG is offline   Reply With Quote

Old   February 12, 2020, 00:09
Senior Member
Herpes Free Engineer
Join Date: Sep 2019
Posts: 298
Rep Power: 3
HPE is on a distinguished road
how many cells - how many cores?
Kind regards,

The OpenFOAM community is the biggest contributor to OpenFOAM; not person A or B. Let's report bugs or issue feature requests in GitLab-OpenFOAM.

Please `design` your questions well. Such effort always pays off well for inquirers and `helpmate`s:
How to create a Minimal, Reproducible Example,
How to give enough info to get help.
HPE is offline   Reply With Quote

Old   February 18, 2020, 22:00
Join Date: May 2016
Posts: 66
Rep Power: 5
KTG is on a distinguished road

sorry for the delay! It seems that CFD forums doesn't send me e-mails anymore...

Last time I was playing with it, I used over 5 million cells, with 64 cores vs. 32, and got was appears to be the same result. Before I though it was a bottle neck issue, but for all I know the first node is doing all the work. I also notice that the averaging function object is taking an insanely long time, which may or may not be related.

Thanks for the reply!
KTG is offline   Reply With Quote


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
Launching multiple CFD simulations in parallel on cluster jthornton232 OpenFOAM Programming & Development 0 August 14, 2019 21:53
Fine/Marine parallel computation on multiple nodes inarus NUMECA 2 February 24, 2015 08:38
MPI code on multiple nodes, scalability and best practice t.teschner Hardware 0 October 7, 2014 06:07
Can I scale up from 1 to 2 nodes with mixed CPU? pongup Hardware 2 April 14, 2014 16:50
Running in parallel on multiple nodes kalyangoparaju OpenFOAM 5 January 18, 2012 12:36

All times are GMT -4. The time now is 10:11.