[cegan09]

I'm looking to build a budget setup to run openFOAM, and I'm getting number blind looking at different CPU models. I'm hoping someone can demystify the specs I should be focusing on.

I need to run the same simulation many times with the object rotated a little between each run. My hope was that I could build something that would let me run a couple of those in parallel, either by setting up a few virtual machines each running one simulation, or just running several studies on the same one (I'm still learning which approach is better, feel free to tell me if there is a better way).

Still learning the lingo, but I believe when describing the size of simulations it's the max local cells setting of the mesh that gets used? Assuming I've got that right, I've been running 20M, but have been experimenting with less to see where results start to diverge too much. If I can build something that lets me run the larger simulations that'd be great however.

Right now I'm using a server that's basically 10 years old with two E5-2670 8 core CPUs and 48gigs of RAM. It's setup with a free ESXI license which limits me to 8 processors per VM. I do have a valid license of windows server 2016 I can switch to, or something like proxmox, or just run the OS direct on the hardware.



Budget and starting points.
I'd like to stay at or under $3k. My initial research says that puts me in early gen EPYC territory. My plan was to use a supermicro H11DSi V2 and start with Naples generation (7xx1) CPUs, with the option to upgrade to Rome in a year or two as those come down in price. Where I'm stuck is trying to make sure I pick the right CPU for the application. I'm just number blind at this point. I'd been looking at 32-core variants with the idea that I can setup 4 simulations at once running 16 cores each. But I'm open to being educated on if that's not the best way.

I know that memory bandwidth is just as or more important than the CPU, so I want to make sure I get that right as well.



Hopefully I got all the info in here that's required. Please tell me if I missed some needed info, or if I just blatantly missed a thread with these answers already. Thank you.

[flotus1]

First things first: forget about virtualization. Especially the idea of spinning up several VMs, each for a single job. That's totally unnecessary, and even if you do it right performance-wise, you still have some overhead from not running bare metal on Linux.

It's much easier, you basically have two options:
1) Just run the simulations one by one, on all cores. Unless they are tiny (not the case with 20M cells) or include something weird that limits parallel efficiency beyond the usual memory bottleneck, you get pretty much maximum throughput this way.
2) If you have one of the two issues mentioned above, you can still run several simulations simultaneously, with lower thread count for each of them. OpenFOAM is MPI parallel, which means core binding options are readily available. Imagine we have 32 cores total, and run 8 simulations simultaneously, each on 4 threads. Then you simply bind the first simulation to cores 0-3, the second simulation to cores 4-7...
This of course requires enough memory for all simulations at the same time.

Earlier Epyc generations aren't a bad choice in terms of price/performance. I'm slightly out of the loop, but I found Epyc 7532 starting at 600€ on ebay. Someone correct me if I am wrong, but that seems like a fairly good deal. In my opinion, 1st gen Epyc is no longer worth it unless you need it really cheap. And then you still have the issue that the motherboard would cost more than both CPUs combined.
Pair that with 16x16GB DDR4-3200 reg ECC (preferably 2Rx8), which can be had for less than 1000€.
Or if you want to start smaller, just get one CPU and half the memory. You can always upgrade to a second CPU later.

[cegan09]

Fair enough. I hadn't realized that you could bind to specific CPU cores, or at least hadn't gotten to that point in the documentation yet. That's helpful. I dove down virtualization because that's how this server was setup before I started getting into CFD work.

I was looking at the 7xx1 series because there are some motherboard + CPU combos on ebay that get me started in my price bracket, with a dual CPU setup. If I can get going with that, and then upgrade the CPUs to Rome generation in a year, that helps me get going for now. I'm sure someone will tell me to wait until I can afford that second gen version to start, but I need to get this going sooner rather than later.

But regardless of which gen, what are the important specs for the CPU? I think I was reading in other threads that cache amounts are impactful on performance? Does that help guide me on picking a specific CPU?

[flotus1]

Have a look at this: OpenFOAM benchmarks on various hardware
There I compared what is pretty much the fastest 1st gen Epyc setup to a system with two 7532. It's not quite a 2x difference, but close. And there is still the IPC and clock speed uplift, for tasks that don't scale like the solver. Like meshing.
Which means you could start with a single 2nd gen CPU right away, and not waste any money on 1st gen which you can't recuperate when selling.

Let's break down the costs:
H11DSi rev.2: 400
2x7532: 2x600
CPU coolers: 120
16x16GB DDR4-3200 reg ECC: ~950
case: 100
PSU: 100
SSD: 120
graphics card: 150

That's bang on 3000€/$
Could you post a link to one of those CPU+motherboard bundles you are looking at?

[cegan09]

This was one of them, there are tons more from overseas that are just motherboard + CPUs, without ram or heatsinks.



https://www.ebay.com/itm/195489143211?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=euOjwGUQTI2&sssrc=2047675&ssuid=TXJAxy7rRy 6&widget_ver=artemis&media=COPY

if it's really close to a 2x jump from 7xx1 to 7xx2, that might be compelling enough to persuade ignoring the first gen chips.

[flotus1]

That's not worth anywhere near 2000$.
And it comes with really slow memory, even 1st gen Epyc supports DDR4-2666. This will hold back the 1st gen CPUs, and the 2nd gen even more.
Plus you will curse the whiny CPU coolers if you try to use this as a workstation.
I can't stress this enough, don't waste your money on this or similar bundles.

[cegan09]

Well I appreciate the feedback on that one. Compute level hardware is new to me, and the requirements are much different from normal computers I've built previously. Thank you for pointing me in the right direction. I guess I'll focus on second gen stuff to get me started.

The one point that doesn't matter to me is noise on this one. I plan on putting this machine in the basement with the rest of my network gear. It can be as noisy as it wants down there, won't bother me much. But I guess if I'm picking all the pieces individually I can purposely get quieter stuff.

[flotus1]

With memory as slow as in this bundle, we are definitely at a 2x performance difference between it, and a single 7532 with DDR4-3200. I.e. it's cheaper to get the same performance with a single CPU from the Rome generation.
Without any of the baggage like CPUs that won't sell later, and slow memory that should ideally be replaced.

[cegan09]

I know this is 6 months later, but I wanted to say thank you again for your guidance.

I finally got the pieces together to boot my machine and I am just blown away with how much faster it is than anything else I've run these simulations on.

I wound up with a dual socket motherboard and 7532 CPUs. 128gigs DDR4-3200 per CPU. I only have 1 socket populated right now, I wanted to get it running and tested before buying the second one, but even with 1 this thing is fast. On my old dual E5-2670 machine I was seeing it take 30+ seconds per second of simulation. On this new machine I'm down to 5 and I haven't done any experimenting on optimal configurations.

I look forward to turning electricity into heat and numbers much faster now.

[wkernkamp]

You might try to run the OpenFOAM benchmark and post the result in that thread.

[cegan09]

It's on my list to do for sure. I have a couple simulations queued up to run first, then I will. I'd like to benchmark it right before and after installing the second CPU so I can have that comparison.

[JBeilke]

I never expected that the 7532 processor is so cheap now. This mean, that I can simply swap out my 2x7302 and get the double amount of cores and 4 times the cache for about 1000 Euro.

Is there anything to consider for such an operation. Can I do it myself with two left hands?

[cegan09]

As long as you have the ability to torque the socket correctly swapping them is just like swapping any CPU.

If you need a 1.5nM torque wrench and can't find a threadripper one for sale, slice engineering sells one for 3d printer nozzles that is the right torque. That's the one I bought.

[JBeilke]

Thanks a lot.