CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > General Forums > Hardware

Looking for some help with EPYC CPUs

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   January 9, 2023, 09:11
Default Looking for some help with EPYC CPUs
  #1
New Member
 
Chris
Join Date: Nov 2022
Posts: 5
Rep Power: 2
cegan09 is on a distinguished road
I'm looking to build a budget setup to run openFOAM, and I'm getting number blind looking at different CPU models. I'm hoping someone can demystify the specs I should be focusing on.

I need to run the same simulation many times with the object rotated a little between each run. My hope was that I could build something that would let me run a couple of those in parallel, either by setting up a few virtual machines each running one simulation, or just running several studies on the same one (I'm still learning which approach is better, feel free to tell me if there is a better way).

Still learning the lingo, but I believe when describing the size of simulations it's the max local cells setting of the mesh that gets used? Assuming I've got that right, I've been running 20M, but have been experimenting with less to see where results start to diverge too much. If I can build something that lets me run the larger simulations that'd be great however.

Right now I'm using a server that's basically 10 years old with two E5-2670 8 core CPUs and 48gigs of RAM. It's setup with a free ESXI license which limits me to 8 processors per VM. I do have a valid license of windows server 2016 I can switch to, or something like proxmox, or just run the OS direct on the hardware.



Budget and starting points.
I'd like to stay at or under $3k. My initial research says that puts me in early gen EPYC territory. My plan was to use a supermicro H11DSi V2 and start with Naples generation (7xx1) CPUs, with the option to upgrade to Rome in a year or two as those come down in price. Where I'm stuck is trying to make sure I pick the right CPU for the application. I'm just number blind at this point. I'd been looking at 32-core variants with the idea that I can setup 4 simulations at once running 16 cores each. But I'm open to being educated on if that's not the best way.

I know that memory bandwidth is just as or more important than the CPU, so I want to make sure I get that right as well.



Hopefully I got all the info in here that's required. Please tell me if I missed some needed info, or if I just blatantly missed a thread with these answers already. Thank you.
cegan09 is offline   Reply With Quote

Old   January 9, 2023, 10:15
Default
  #2
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,263
Rep Power: 44
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
First things first: forget about virtualization. Especially the idea of spinning up several VMs, each for a single job. That's totally unnecessary, and even if you do it right performance-wise, you still have some overhead from not running bare metal on Linux.

It's much easier, you basically have two options:
1) Just run the simulations one by one, on all cores. Unless they are tiny (not the case with 20M cells) or include something weird that limits parallel efficiency beyond the usual memory bottleneck, you get pretty much maximum throughput this way.
2) If you have one of the two issues mentioned above, you can still run several simulations simultaneously, with lower thread count for each of them. OpenFOAM is MPI parallel, which means core binding options are readily available. Imagine we have 32 cores total, and run 8 simulations simultaneously, each on 4 threads. Then you simply bind the first simulation to cores 0-3, the second simulation to cores 4-7...
This of course requires enough memory for all simulations at the same time.

Earlier Epyc generations aren't a bad choice in terms of price/performance. I'm slightly out of the loop, but I found Epyc 7532 starting at 600€ on ebay. Someone correct me if I am wrong, but that seems like a fairly good deal. In my opinion, 1st gen Epyc is no longer worth it unless you need it really cheap. And then you still have the issue that the motherboard would cost more than both CPUs combined.
Pair that with 16x16GB DDR4-3200 reg ECC (preferably 2Rx8), which can be had for less than 1000€.
Or if you want to start smaller, just get one CPU and half the memory. You can always upgrade to a second CPU later.
flotus1 is offline   Reply With Quote

Old   January 9, 2023, 10:30
Default
  #3
New Member
 
Chris
Join Date: Nov 2022
Posts: 5
Rep Power: 2
cegan09 is on a distinguished road
Fair enough. I hadn't realized that you could bind to specific CPU cores, or at least hadn't gotten to that point in the documentation yet. That's helpful. I dove down virtualization because that's how this server was setup before I started getting into CFD work.

I was looking at the 7xx1 series because there are some motherboard + CPU combos on ebay that get me started in my price bracket, with a dual CPU setup. If I can get going with that, and then upgrade the CPUs to Rome generation in a year, that helps me get going for now. I'm sure someone will tell me to wait until I can afford that second gen version to start, but I need to get this going sooner rather than later.

But regardless of which gen, what are the important specs for the CPU? I think I was reading in other threads that cache amounts are impactful on performance? Does that help guide me on picking a specific CPU?
cegan09 is offline   Reply With Quote

Old   January 9, 2023, 10:46
Default
  #4
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,263
Rep Power: 44
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
Have a look at this: OpenFOAM benchmarks on various hardware
There I compared what is pretty much the fastest 1st gen Epyc setup to a system with two 7532. It's not quite a 2x difference, but close. And there is still the IPC and clock speed uplift, for tasks that don't scale like the solver. Like meshing.
Which means you could start with a single 2nd gen CPU right away, and not waste any money on 1st gen which you can't recuperate when selling.

Let's break down the costs:
H11DSi rev.2: 400
2x7532: 2x600
CPU coolers: 120
16x16GB DDR4-3200 reg ECC: ~950
case: 100
PSU: 100
SSD: 120
graphics card: 150

That's bang on 3000/$
Could you post a link to one of those CPU+motherboard bundles you are looking at?
flotus1 is offline   Reply With Quote

Old   January 9, 2023, 17:47
Default
  #5
New Member
 
Chris
Join Date: Nov 2022
Posts: 5
Rep Power: 2
cegan09 is on a distinguished road
This was one of them, there are tons more from overseas that are just motherboard + CPUs, without ram or heatsinks.



https://www.ebay.com/itm/195489143211?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=euOjwGUQTI2&sssrc=2047675&ssuid=TXJAxy7rRy 6&widget_ver=artemis&media=COPY

if it's really close to a 2x jump from 7xx1 to 7xx2, that might be compelling enough to persuade ignoring the first gen chips.
cegan09 is offline   Reply With Quote

Old   January 9, 2023, 18:33
Default
  #6
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,263
Rep Power: 44
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
That's not worth anywhere near 2000$.
And it comes with really slow memory, even 1st gen Epyc supports DDR4-2666. This will hold back the 1st gen CPUs, and the 2nd gen even more.
Plus you will curse the whiny CPU coolers if you try to use this as a workstation.
I can't stress this enough, don't waste your money on this or similar bundles.
flotus1 is offline   Reply With Quote

Old   January 9, 2023, 21:07
Default
  #7
New Member
 
Chris
Join Date: Nov 2022
Posts: 5
Rep Power: 2
cegan09 is on a distinguished road
Well I appreciate the feedback on that one. Compute level hardware is new to me, and the requirements are much different from normal computers I've built previously. Thank you for pointing me in the right direction. I guess I'll focus on second gen stuff to get me started.

The one point that doesn't matter to me is noise on this one. I plan on putting this machine in the basement with the rest of my network gear. It can be as noisy as it wants down there, won't bother me much. But I guess if I'm picking all the pieces individually I can purposely get quieter stuff.
cegan09 is offline   Reply With Quote

Old   January 10, 2023, 05:57
Default
  #8
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,263
Rep Power: 44
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
With memory as slow as in this bundle, we are definitely at a 2x performance difference between it, and a single 7532 with DDR4-3200. I.e. it's cheaper to get the same performance with a single CPU from the Rome generation.
Without any of the baggage like CPUs that won't sell later, and slow memory that should ideally be replaced.
flotus1 is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
AMD Epyc 9004 "Genoa" buyers guide for CFD flotus1 Hardware 8 January 16, 2023 05:23
CPU for Flow3d mik_urb Hardware 4 December 4, 2022 22:06
General recommendations for CFD hardware [WIP] flotus1 Hardware 15 March 21, 2022 12:11
Memory for AMD Epyc CPUs linuxguy123 Hardware 28 February 1, 2022 16:40
Finalizing Parts List for First PC Build, Seeking Advice Deedledot Hardware 7 November 21, 2021 17:20


All times are GMT -4. The time now is 00:26.