CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Running, Solving & CFD

Openfoam running extremely slowly using multiple nodes on HPC

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   July 20, 2022, 22:39
Default Openfoam running extremely slowly using multiple nodes on HPC
  #1
Member
 
Dongxu Wang
Join Date: Sep 2018
Location: China
Posts: 33
Rep Power: 7
wdx_cfd is on a distinguished road
Hi Foamers,

I recently complied OpenMPI and OF2006 under my own account of our HPC. The program was successfully installed and can be run in parallel mode. However, I found that the simulation is extremely slow when running the case using multiple nodes. The problem didn't appear when the case is run locally or using only one node.

For example, when I run the dam-break case using 4 processors, I found that if only one node is used (namely, #SBATCH --nodes=1, #SBATCH --ntasks-per-node=4), the time used is 47.34s; if two nodes are used (namely, #SBATCH --nodes=2, #SBATCH --ntasks-per-node=2), the time used is 381.22s. The time increased by approximately 8 times.

It can be easily deduced that the problem is due to the communication of nodes. The administrater suggested I use interMPI but I haven't tried until now because the HPC is temporarily down due to some reason. However, I am not sure if using another mpi can address this problem. Could someone give me some advice?

The node information is as follows:
Code:
u01
    state = free
    np = 56
    ntype = cluster
    status = rectime=1657244447,state=free,slurmstate=idle,size=0kb:0kb,ncpus=56,boards=1,sockets=2,cores=28,threads=1,availmem=380000mb,opsys=linux 3.10.0-862.el7.x86_64
The running command is as follows:
Code:
mpirun --mca btl_tcp_if_include "ip address“ --mca btl '^openib' -np 4 interIsoFoam -parallel 2>&1 > log.interIsoFoam
where "ip address“ is obtained using "ip addr" command.

Thanks!
wdx
wdx_cfd is offline   Reply With Quote

Old   July 22, 2022, 05:09
Default
  #2
Member
 
Dongxu Wang
Join Date: Sep 2018
Location: China
Posts: 33
Rep Power: 7
wdx_cfd is on a distinguished road
OK, I solve the problem.

It is indeed caused by the use of OpenMPI. I installed intelMPI and found that the time required decreased to 73s when 2 nodes are used. Although this time is still longer than that on one node, it has been improved significantly.
wdx_cfd is offline   Reply With Quote

Old   September 17, 2023, 11:34
Default
  #3
New Member
 
Join Date: Mar 2022
Posts: 7
Rep Power: 4
Mid_Night is on a distinguished road
Got the same problem, did you manage to solve it?
Mid_Night is offline   Reply With Quote

Old   September 19, 2023, 05:17
Default
  #4
Senior Member
 
Domenico Lahaye
Join Date: Dec 2013
Posts: 751
Blog Entries: 1
Rep Power: 17
dlahaye is on a distinguished road
Pls try larger cell count (i.e. finer mesh) to avoid interprocessor communication (i.e. data transfer between processor during the linear solve) to dominate computation on each processor (i.e. local linear solves).
dlahaye is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Map of the OpenFOAM Forum - Understanding where to post your questions! wyldckat OpenFOAM 10 September 2, 2021 05:29
Issues with OpenFOAM on an HPC with GCC+MVAPICH2. Prakriti OpenFOAM Programming & Development 0 October 18, 2020 15:48
Problem with OpenFoam 6 -parallel on remote HPC nodes mrishi OpenFOAM Running, Solving & CFD 10 September 6, 2019 08:46
Running Multiple OpenFOAM Cases at Once on HPC papamilad Hardware 5 August 2, 2019 12:20
OpenFOAM Running error with multiple nodes dokeun OpenFOAM Running, Solving & CFD 1 June 28, 2019 00:04


All times are GMT -4. The time now is 12:36.