CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > General Forums > Hardware

OpenFOAM benchmarks on various hardware

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree341Likes

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   April 20, 2018, 00:04
Default
  #61
Member
 
Join Date: Nov 2014
Posts: 92
Rep Power: 10
hokhay is on a distinguished road
I would like to contribute to the chart with my i9-7900X
i9-7900X, 4x8GB DDR4 2133MHz
OpenFOAM 5.X, Ubuntu 16.04, Linux ITHW0115-UB 4.13.0-38-generic

Code:
# cores   Wall time (s):
     1         945.35
     2         471.4
     4         272.85
     6         220.38
     8         194.74
    10         185.35
hokhay is offline   Reply With Quote

Old   April 20, 2018, 10:09
Default
  #62
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,210
Rep Power: 44
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
It is a bit sad to see a 900$ CPU choked by slow memory
hokhay, tin_tad, erik87 and 1 others like this.
flotus1 is offline   Reply With Quote

Old   April 25, 2018, 14:54
Default
  #63
New Member
 
Rob
Join Date: Apr 2018
Posts: 18
Rep Power: 6
Morlind is on a distinguished road
I have a combination of results. The single-core result seems too good to be true, while the others failed in error and I'm not sure why.

* edit - the error was related to the streamlines so I deleted them from the controlDict to get the 2 core. I don't know whether this is an unfair advantage so please don't take these numbers as exactly representative against others.

Intel i7 - 7700 @ 3.6 ghz w/ 32 gb RAM (4x pieces)
Results:

# cores Wall time (s):
------------------------
1 660.52
2 422.81
4
Morlind is offline   Reply With Quote

Old   April 26, 2018, 06:56
Default
  #64
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,210
Rep Power: 44
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
Given the results for an I7-8700k with fast memory posted earlier, your results kind of make sense.
Code:
# cores   Wall time (s):
------------------------
1 531.44
2 312.15
4 249.55
6 247.83
If your CPU is really running at only 3.6GHz (did you deactivate turbo modes? why?) and memory is DDR4-2400, this is well within expectations for your setup.
flotus1 is offline   Reply With Quote

Old   May 9, 2018, 14:05
Default
  #65
New Member
 
Rob
Join Date: Apr 2018
Posts: 18
Rep Power: 6
Morlind is on a distinguished road
Does this case not function well in OpenFoam5? I can't get anything other than the single core to run (this is the second separate system with this issue). It thros errors around the streamlines and some other things. I should add that the single core shows this error but ran, the others didn't run at all. Here is the start of the error string:


--> FOAM Warning :
From function bool Foam::functionObjectList::read()
in file db/functionObjects/functionObjectList/functionObjectList.C at line 675
Caught FatalError
--> FOAM FATAL ERROR:
Attempt to return primitive entry ITstream : /home/rob/Downloads/bench_template/run_1/system/controlDict.functions.streamLines.seedSampleSet, line 45, IOstream: Version 2.0, format ASCII, line 0, OPENED, GOOD
primitiveEntry 'seedSampleSet' comprises
on line 45 the word 'uniform'
as a sub-dictionary

From function virtual const Foam::dictionary& Foam:rimitiveEntry::dict() const
in file db/dictionary/primitiveEntry/primitiveEntry.C at line 189.

nearWallFields near: Sampling 1 fields
--> FOAM Warning :
From function void Foam:article::locate(const vector&, const vector*, Foam::label, bool, Foam::string)
in file particle/particle.C at line 507
"Particle initialised with a location outside of the mesh."
Morlind is offline   Reply With Quote

Old   May 9, 2018, 14:18
Default
  #66
Member
 
Knut Erik T. Giljarhus
Join Date: Mar 2009
Location: Norway
Posts: 35
Rep Power: 20
eric will become famous soon enough
Just open the file basecase/system/controlDict and remove the lines
Code:
#include streamlines
#include wallBoundedStreamlines
That should take care of it. You should also delete all the run_* folders before rerunning the run.sh script.

By the way, I have updated the results on the benchmarks page with the latest contributions, thank you all for sharing!
eric is offline   Reply With Quote

Old   May 9, 2018, 15:28
Default
  #67
New Member
 
Rob
Join Date: Apr 2018
Posts: 18
Rep Power: 6
Morlind is on a distinguished road
That's exactly what I did. Here are the results:

2x E5-2680 v2
128 gb RAM (1333)
These results were running from the SSD:

# cores Wall time (s):
------------------------
1 845.34
2 487.52
4 237.65
6 171.12
8 147.41
12 126.89
16 116.38
20 118.79

FYI the initial single core was 848.79 so the streamlines were worth a tiny bit.

Here are the results run from the HDD location. Disk interface appears to matter a touch.

# cores Wall time (s):
------------------------
1 846.06
2 469.11
4 235.33
6 172.6
8 147.98
12 128.04
16 119.72
20 127.1
Morlind is offline   Reply With Quote

Old   May 27, 2018, 14:40
Default
  #68
New Member
 
Thanasis
Join Date: Mar 2010
Location: Greece
Posts: 9
Rep Power: 14
thanz is on a distinguished road
Quote:
Originally Posted by wyldckat View Post
That and build OpenFOAM from source code, along with dedicated flags for your CPU model-type... don't use pre-built packages.

Which flags do you recommend for the EPYC 7351 for the g++ ?


Thanks in advance.
thanz is offline   Reply With Quote

Old   May 27, 2018, 20:17
Default
  #69
Retired Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,969
Blog Entries: 45
Rep Power: 126
wyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to all
Quick answer:
Quote:
Originally Posted by thanz View Post
Which flags do you recommend for the EPYC 7351 for the g++ ?
I've never tested it myself, so @flotus1 probably knows it better than I do

According to this page: https://www.phoronix.com/scan.php?pa...lers-nov&num=1
Quote:
CFLAGS/CXXFLAGS were set to "-O3 -march=znver1"
In OpenFOAM, you should configure in these files:
  • wmake/rules/linux64Gcc/c++Opt
  • wmake/rules/linux64Gcc/cOpt
and add it on the line for "c++OPT".

This is true as long as "WM_COMPILER" is set to "Gcc", otherwise revise the folder name accordingly.
emjay likes this.
wyldckat is offline   Reply With Quote

Old   May 30, 2018, 10:44
Default
  #70
New Member
 
Join Date: May 2018
Posts: 2
Rep Power: 0
kuhngmbh is on a distinguished road
2x Intel Xeon X5550, 6x 8GB DDR3 1333 MHz, OpenFOAM 5.0, Host OS: Vmware Esxi 6.0 VM OS: Ubuntu 16.04


I have removed the following lines under basecase/system/controlDict
Code:
#include streamlines
#include wallBoundedStreamlines

# cores Wall time (s):
------------------------
1: 1558.47
2: 907.58
4: 603.29
8: 352.56


2x Intel Xeon Gold 6134, 12x 32GB DDR4 2666 MHz, OpenFOAM 5.0, Host OS: Vmware Esxi 6.5 VM OS: Ubuntu 16.04

# cores Wall time (s):
------------------------
1: 874.24
2: 434.65
4: 222.26
8: 135.39

because there are other machines on this host I have tested only on 8 of the 16 cores
kuhngmbh is offline   Reply With Quote

Old   May 30, 2018, 10:54
Default
  #71
New Member
 
Rob
Join Date: Apr 2018
Posts: 18
Rep Power: 6
Morlind is on a distinguished road
kuhngmbh, I'm surprised by your results. I would have expected the Gold to benchmark quite a bit higher. I wonder if it's the VM setup effecting the performance? I'm sure it will scale better than my machine but I expected it to be substantially faster across the range.
Morlind is offline   Reply With Quote

Old   May 30, 2018, 10:58
Default
  #72
New Member
 
Join Date: May 2018
Posts: 2
Rep Power: 0
kuhngmbh is on a distinguished road
I think it's because some other VMs are running on this machine.
If I have time on one of the following weekends, I could test the machine only with the openfoam vm and with 16 cores.
kuhngmbh is offline   Reply With Quote

Old   May 30, 2018, 11:00
Default
  #73
New Member
 
Rob
Join Date: Apr 2018
Posts: 18
Rep Power: 6
Morlind is on a distinguished road
Ah, of course. If others are utilizing the hardware and you only have partial access then the results make sense.
Morlind is offline   Reply With Quote

Old   June 1, 2018, 10:52
Default My results
  #74
Member
 
Join Date: Jan 2014
Posts: 32
Rep Power: 11
spaceprop is on a distinguished road
Hi,

I've been going through some homelab cluster upgrades, so I had a chance to test a ton of different processor/RAM combinations. Here are my all-core iteration/s results.

The processor(s) and RAM are called out in the graph. All OF+v1712, OpenMPI 3.1.0, CentOS 7.5. I applied the following BIOS settings to all: HT disabled (if applicable), anything related to maximum performance on. The "slight OC" for the 2x E5-4627v3's comes from the Asus Z10PE-D8's Ai overclock and "xeon turbo" BIOS settings. If RAM faster than the fastest RAM speed supported by the processor was used, I confirmed that it downclocked. An exception to this is the I7-5960X which was perfectly happy with 2400MHz RAM. I confirmed the RAM speed of all other tests was operating at its design frequency (dmidecode -t memory).



I think these pretty much fall in line with what we've been seeing from the rest of the benchmarks.

Cheers
flotus1 and zyzycomcn like this.

Last edited by spaceprop; June 1, 2018 at 12:03. Reason: adding description of RAM speed
spaceprop is offline   Reply With Quote

Old   June 1, 2018, 11:58
Default
  #75
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,210
Rep Power: 44
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
Did you actually get 2x Xeon E5-2690 running with PC3-14900? Or is the memory running at DDR3-1600? Which motherboard?
flotus1 is offline   Reply With Quote

Old   June 1, 2018, 12:04
Default
  #76
Member
 
Join Date: Jan 2014
Posts: 32
Rep Power: 11
spaceprop is on a distinguished road
No, I did not. It downclocked to 1600MHz. Same with the PC4-19200 (2400MHz); it downclocked to 2133MHz with the e5 v3's.

I edited the post to clarify that.
spaceprop is offline   Reply With Quote

Old   June 26, 2018, 03:59
Default
  #77
Senior Member
 
Join Date: Oct 2011
Posts: 199
Rep Power: 13
naffrancois is on a distinguished road
Hello, here is my contribution on OpenFoam v1712 linux binaries (with controlDict modified as suggested):

Laptop Intel Core i7-7500U CPU @ 2.70GHz
# cores Wall time (s):
------------------------
1 896.86
2 553.51

2xIntel Xeon(R) CPU E5-2687W v2 @ 3.40GHz
# cores Wall time (s):
------------------------
1 786.29
2 465.3
4 215.12
8 137.39
16 118.12

2xAMD EPYC 7501 32-Core Processor +16x8Go 2666MHz + 2To SSD
# cores Wall time (s):
------------------------
1 1021.56
2 570.31
4 255.25
8 137.12
16 66.39
32 44.13
64 32.21
naffrancois is offline   Reply With Quote

Old   June 27, 2018, 10:31
Default
  #78
New Member
 
Join Date: Oct 2014
Location: Lafayette, LA
Posts: 18
Rep Power: 10
gigliagarf is on a distinguished road
Quote:
Originally Posted by naffrancois View Post
Hello, here is my contribution on OpenFoam v1712 linux binaries (with controlDict modified as suggested):

Laptop Intel Core i7-7500U CPU @ 2.70GHz
# cores Wall time (s):
------------------------
1 896.86
2 553.51

2xIntel Xeon(R) CPU E5-2687W v2 @ 3.40GHz
# cores Wall time (s):
------------------------
1 786.29
2 465.3
4 215.12
8 137.39
16 118.12

2xAMD EPYC 7501 32-Core Processor +16x8Go 2666MHz + 2To SSD
# cores Wall time (s):
------------------------
1 1021.56
2 570.31
4 255.25
8 137.12
16 66.39
32 44.13
64 32.21

Jeez thats quick. I'm over here running on a 8 year old laptop, you guys are way better funded than me.
gigliagarf is offline   Reply With Quote

Old   July 12, 2018, 07:34
Default
  #79
Member
 
Join Date: Jan 2014
Posts: 32
Rep Power: 11
spaceprop is on a distinguished road
Quote:
Originally Posted by naffrancois View Post

2xAMD EPYC 7501 32-Core Processor +16x8Go 2666MHz + 2To SSD
# cores Wall time (s):
------------------------
1 1021.56
2 570.31
4 255.25
8 137.12
16 66.39
32 44.13
64 32.21
That's faster than the dual 7601's in the main chart (data from havref posted earlier in this thread). Can you give some more info about your setup?
spaceprop is offline   Reply With Quote

Old   July 12, 2018, 07:48
Default
  #80
Member
 
Join Date: Jan 2014
Posts: 32
Rep Power: 11
spaceprop is on a distinguished road
Updating my earlier table with more results.

X-axis is iter/s. The processor(s) and RAM are called out in the graph. All OF+v1712, OpenMPI 3.1.0, CentOS 7.5. I applied the following BIOS settings to all: HT disabled (if applicable), anything related to maximum performance on. This is slightly cleaned up from the last one and has the E5-2690v2 results added.



This one includes the results from my homelab 5-node cluster:



19.09s (5.24 ips) . Infiniband <3
flotus1 likes this.

Last edited by spaceprop; July 12, 2018 at 07:49. Reason: fix image link
spaceprop is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to contribute to the community of OpenFOAM users and to the OpenFOAM technology wyldckat OpenFOAM 17 November 10, 2017 16:54
UNIGE February 13th-17th - 2107. OpenFOAM advaced training days joegi.geo OpenFOAM Announcements from Other Sources 0 October 1, 2016 20:20
OpenFOAM Training Beijing 22-26 Aug 2016 cfd.direct OpenFOAM Announcements from Other Sources 0 May 3, 2016 05:57
New OpenFOAM Forum Structure jola OpenFOAM 2 October 19, 2011 07:55
Hardware for OpenFOAM LES LijieNPIC Hardware 0 November 8, 2010 10:54


All times are GMT -4. The time now is 08:48.