CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > General Forums > Hardware

Updating CFD server E5-2697AV4 to something faster (x2/x3 the speed)

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree16Likes

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   August 9, 2024, 06:37
Default
  #21
Senior Member
 
Tobias
Join Date: May 2016
Location: Germany
Posts: 295
Rep Power: 11
MFGT is on a distinguished road
Quote:
Originally Posted by flotus1 View Post
If you have a budget for hardware worth 7000€, I would not even bother with older generation dual-socket Epyc.
Current-gen Epyc gets you the generational improvements, making them closer to Ryzen Desktop CPU in single-core performance. While still matching the older dual-socket Epyc in terms of memory bandwidth.
Budget is still 5000€, I dont see Gen4 EPYC there, including Rack, RAM etc. necessary Updates.


So I am looking at the DUAL 7F52 (542€ each). Also for Dual CPU, 16 DIMMS are preferred over 8, right?
MFGT is offline   Reply With Quote

Old   August 9, 2024, 10:35
Default
  #22
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,427
Rep Power: 49
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
No point in getting two CPUs, if you don't populate all their memory channels. So yes, 16 DIMMs for 1st-3rd gen dual-socket Epyc.

I took the 7000€ figure from one of your later posts.
From the history and title of this thread, and the benchmarks posted so far, I concluded that a newer generation Epyc CPU would be the best fit. Much better single-core performance than older generations, similar peak performance as 2nd gen dual-socket setup, and simpler NUMA topology.
5000€ is tight, but you can make it work with maybe a few hundred Euros extra.
Mainboard: Supermicro H13SSL-N retail https://geizhals.de/supermicro-h13ss...loc=at&hloc=de 694€
Memory: 12x 16GB DDR5-4800 reg ECC 1200€
CPU: 2500€
That leaves 600€ for the remaining stuff. I don't know what you need here.
MFGT likes this.
flotus1 is offline   Reply With Quote

Old   August 9, 2024, 11:49
Default
  #23
Senior Member
 
Tobias
Join Date: May 2016
Location: Germany
Posts: 295
Rep Power: 11
MFGT is on a distinguished road
Thanks for your valuable input Alex.

I have to add that we dont build it by our own. For our company we can only buy complete systems and then its difficult to stay within the budget, when the base price for a server rack with mainboard is already 2k. Adding processors (1k), some further RAM, cooling, power supply and 2year warranty etc quickly adds up to 4.5k€ with the Dual EPYC 7F52.

source: www.serverschmiede.com

For new 4th gen solutions we quickly exceed the budget:
soruce e.g.: https://www.primeline-solutions.com/...u_series=30022

Based on my experience with the CFD software I dont want to go over 2x16 cores.


So one may argue between Dual 16Core Gen2 vs. some Single Core Gen4, but than we cant even use the 128GB DDR4 RAM we already have.
MFGT is offline   Reply With Quote

Old   August 10, 2024, 12:10
Default
  #24
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,427
Rep Power: 49
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
My apologies, I misinterpreted a few things here.
From "We plan to buy used/refurbished hardware", I concluded that we are mostly talking about budget for the hardware itself. And that it can be bought in parts, from any source.

Let's go back to the start. Title of this thread is "Updating CFD server E5-2697AV4 to something faster (x2/x3 the speed)".
From the benchmarks shown so far, we need a good mix of single- and multi-core performance, because the code doesn't scale too well. For whatever reason that may be.
The ONLY way to achieve the stated goal is with latest-gen server CPUs.
Current desktop CPUs have the single-core performance, but lack multi-core and/or memory bandwidth.
Older generation server CPUs have the memory bandwidth, at least with a dual-socket setup. But they lack the single-core performance. And the complicated NUMA topology may also be an issue.

Long story short, I don't see any other way to achieve the stated goal.
2x 2nd gen Epyc will probably still be an upgrade compared to the current system. I just don't think that's an uplift worth 7000€.
If you don't mind, a more general comment: engineering time is expensive, especially in Germany. So are software licenses. Using both more efficiently, by using state-of-the-art computer hardware, usually pays off quickly.
flotus1 is offline   Reply With Quote

Old   August 12, 2024, 04:13
Default
  #25
Senior Member
 
Tobias
Join Date: May 2016
Location: Germany
Posts: 295
Rep Power: 11
MFGT is on a distinguished road
Quote:
Originally Posted by flotus1 View Post
If you don't mind, a more general comment: engineering time is expensive, especially in Germany. So are software licenses. Using both more efficiently, by using state-of-the-art computer hardware, usually pays off quickly.
You dont have to tell me, I am just the user but cant argue about the invest limits.
Btw, the 5000€ budget is also considered incl. tax...

So yeah, I think we will go for the Dual EPYC 7F52, as newer Gen Hardware is not affordable and Desktop CPU won't match the old performance (with 8 core and license costs saving).

Last edited by MFGT; August 13, 2024 at 02:53.
MFGT is offline   Reply With Quote

Old   August 13, 2024, 02:01
Default
  #26
Senior Member
 
Will Kernkamp
Join Date: Jun 2014
Posts: 372
Rep Power: 14
wkernkamp is on a distinguished road
Quote:
Originally Posted by MFGT View Post
Thanks a lot you for your input Will.


So the 8 Core 7800X3D would be rougly 70% of performance, right?


Regarding CFD tasks, I do flowbench simulations (20min per case) up to full cycle engine simulations with runtimes between 18-36 hours.
yes, 70% of your dual E5-2697A.
MFGT likes this.
wkernkamp is offline   Reply With Quote

Old   August 13, 2024, 02:54
Default
  #27
Senior Member
 
Tobias
Join Date: May 2016
Location: Germany
Posts: 295
Rep Power: 11
MFGT is on a distinguished road
We will do a test with 7800X3D today, stay tuned.
MFGT is offline   Reply With Quote

Old   August 13, 2024, 05:19
Default
  #28
Senior Member
 
Tobias
Join Date: May 2016
Location: Germany
Posts: 295
Rep Power: 11
MFGT is on a distinguished road
We see approx. 65-75% performance with 8 cores 7800X3D compared to 32 cores Dual-E5-2697Av4.


edit: Interesting though, 8 cores with or without hyperhtreadding was nearly the same. 16 cores was only slightly faster.
flotus1 likes this.

Last edited by MFGT; August 13, 2024 at 08:21.
MFGT is offline   Reply With Quote

Old   August 13, 2024, 08:55
Default
  #29
Senior Member
 
Tobias
Join Date: May 2016
Location: Germany
Posts: 295
Rep Power: 11
MFGT is on a distinguished road
So hopefully last questions:

I have the feeling that the CONVERGE code isn't that hungry for Memory and the higher bandwidth or L3 cache may not be so important as for other applications (I am running "fine" with 40MB L3 cache so far). The Dual 7F52 CPUs cost 1084€, while the Dual 7452 CPUs cost 1242€, only 158€ more. In the passmark score its a plus of 18%, twice the cores but lower clock frequenzy and 128MB L3 cache only.

I am curious if that could be a better solution here? Is it worth the risk or should I simply stick to the 2x16 cores with higher single core performance?
MFGT is offline   Reply With Quote

Old   August 13, 2024, 14:06
Default
  #30
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,427
Rep Power: 49
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
From everything I have seen so far, I have no doubt that the 7F52 is the better option.
MFGT likes this.
flotus1 is offline   Reply With Quote

Old   August 14, 2024, 07:32
Default
  #31
Senior Member
 
Tobias
Join Date: May 2016
Location: Germany
Posts: 295
Rep Power: 11
MFGT is on a distinguished road
Thanks Alex, which power suppyl would you suggest?

2 x HP 800W PSU Netzteil DL360 DL380 ML350 Gen10+ Gen11 80 Plus Platinum PLC
or
2 x HP 1000W PSU Netzteil DL360 DL380 ML350 Gen10+ Gen11 80 Plus Titanium PLC

Is a redundant power supply necessary at all?
MFGT is offline   Reply With Quote

Old   August 14, 2024, 08:25
Default
  #32
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,427
Rep Power: 49
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
Do you have a choice? If so, is there a price difference?
flotus1 is offline   Reply With Quote

Old   August 14, 2024, 09:02
Default
  #33
Senior Member
 
Tobias
Join Date: May 2016
Location: Germany
Posts: 295
Rep Power: 11
MFGT is on a distinguished road
Yes, we have the choice and its 192€ vs 564€, as redundant variant
So the difference is 7.4% of the budget
MFGT is offline   Reply With Quote

Old   August 15, 2024, 15:08
Default
  #34
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,427
Rep Power: 49
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
Redundant power supplies are nice, when you don't have access to the server all the time.
Or when downtime would mean a significant disruption to your operation.
The part itself is certainly not worth 564€, even if it was new. But at this point, in for a penny, in for a pound?
flotus1 is offline   Reply With Quote

Old   August 19, 2024, 10:23
Default
  #35
Senior Member
 
Tobias
Join Date: May 2016
Location: Germany
Posts: 295
Rep Power: 11
MFGT is on a distinguished road
Quote:
Originally Posted by flotus1 View Post
But at this point, in for a penny, in for a pound?
well, if its not necessary I would rather spend the money for something useful such as RAM or SSD
MFGT is offline   Reply With Quote

Old   August 19, 2024, 12:00
Default
  #36
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,427
Rep Power: 49
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
Let me take a wild guess, tell me if I'm wrong.

You were the one who suggested buying used hardware, to make the most of the limited budget?
If there is ever something wrong with the server, like downtime caused by a broken PSU, you will never hear the end of it.
flotus1 is offline   Reply With Quote

Old   August 26, 2024, 06:35
Default
  #37
Senior Member
 
Tobias
Join Date: May 2016
Location: Germany
Posts: 295
Rep Power: 11
MFGT is on a distinguished road
Quote:
Originally Posted by flotus1 View Post
Let me take a wild guess, tell me if I'm wrong.

You were the one who suggested buying used hardware, to make the most of the limited budget?
If there is ever something wrong with the server, like downtime caused by a broken PSU, you will never hear the end of it.
for sure, no way to get something usefull for 5k in new.


btw, available (redundant) power supplies are:

HP 500W PSU Netzteil DL360 DL380 ML350 Gen10+ Gen11 80 Plus Platinum PLC 178.00 €
HP 800W PSU Netzteil DL360 DL380 ML350 Gen10+ Gen11 80 Plus Platinum PLC 192.00 €
HP 1000W PSU Netzteil DL360 DL380 ML350 Gen10+ Gen11 80 Plus Titanium PLC 564.00 €
HP 1400W PSU Netzteil DL360 DL380 ML350 Gen10+ Gen11 80 Plus Platinum PLC 284.00 €
HP 1600W PSU Netzteil DL360 DL380 ML350 Gen10+ Gen11 80 Plus Platinum PLC 442.00 €



I wonder why the 1000W is so expensive (Titanium?). If the 800W is too small, we could still chose 1400W and save 280€ for new 3200MHz RAM. Currently we use 2133MHz.
MFGT is offline   Reply With Quote

Old   August 27, 2024, 08:01
Default
  #38
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,427
Rep Power: 49
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
No idea about their price structure. I can't think of any technical reason why the 1000W option is so expensive.

An 800W power supply is enough, as long as you don't plan to add GPUs to the node.
Making enough room in the budget for DDR4-3200 is definitely the right choice.
flotus1 is offline   Reply With Quote

Old   September 20, 2024, 06:57
Default
  #39
Senior Member
 
Tobias
Join Date: May 2016
Location: Germany
Posts: 295
Rep Power: 11
MFGT is on a distinguished road
So, we packed everthing together and bought the Dual 7F52 with 800W power supply. RAM and HDD will be carry over from the old server.
We received the server yesterday, implemented the server rack into our struckture and will add HDD and RAM of the old server begin of next week (IT department is short-staffed at the moment)


Then we will see what happens.
If everything works we consider upgrading to SSD and 3200MHz RAM.
wkernkamp likes this.
MFGT is offline   Reply With Quote

Old   October 11, 2024, 02:59
Default
  #40
Senior Member
 
Tobias
Join Date: May 2016
Location: Germany
Posts: 295
Rep Power: 11
MFGT is on a distinguished road
So, 3 weeks later we have the server running. We needed to buy some small stuff, install windows and servises, solver software etc and vacaines played a role as well. That were the good news.

However, I think something still isnt right. Based on the benchmark score, it should be 92% faster but it is not. More or less the same speed of the original unaffected old system.

Currently, HT is still activated.
When I use 32 cores for a calculation, the active cores run on 3500MHz, the idle are at 2500MHz.

The old server ran with 2600Mhz clock speed maximum, which is 25% less. So I dont understand its the same performance. We use the old 8x16 GB 2133MHz RAM and according to our IT, it is placed in the correct sockets. Furher carry over is the data HDD.
MFGT is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
On the CFD market and trends sbaffini Main CFD Forum 14 June 13, 2017 12:48
CFD Online Celebrates 20 Years Online jola Site News & Announcements 22 January 31, 2015 01:30
how to solve the diverage of high speed centrifugal compressor, CFD code is STAR CCM layth STAR-CCM+ 3 May 21, 2012 06:48
Which is better to develop in-house CFD code or to buy a available CFD package. Tareq Al-shaalan Main CFD Forum 10 June 13, 1999 00:27
public CFD Code development Heinz Wilkening Main CFD Forum 38 March 5, 1999 12:44


All times are GMT -4. The time now is 15:35.