[Ry_z]

I am wondering if my planned setup will work for CFD, amongst other things including lasting me a long time. I am specifically curious if the graphics card needs to be "beefed up" because I enjoy playing video games on Saturday nights with my long time friends if I finish working on my projects at a reasonable time in the night as a secondary source of entertainment.

My specs will include:

CPU: Ryzen Threadripper 3960x
Motherboard: MSI TRX40 PRO wi-fi
RAM: 64gb ddr4-3200 MHz
SSD: 2-Tb Western digital blue M.2
HDD: Seagate Barracuda 4-Tb 5400 RPM
GPU: Nvidia Quadro RTX 4000

Would it be worth it to consider moving up to a Quadro RTX 5000 or a Titan RTX for the increased number of Tensor cores and cuda cores. s you can probably tell, I am not extremely well versed in CFD, but I am extremely interested in getting into it. Especially since I am building this work computer. As a final note, I am not 100% sure if I am supposed to make a new thread to post these questions, as this is my second time ever posting on a forum. Hopefully someone can shed some light on my situation.

[flotus1]

Would it be worth it to consider moving up to a Quadro RTX 5000 or a Titan RTX for the increased number of Tensor cores and cuda cores. s you can probably tell, I am not extremely well versed in CFD, but I am extremely interested in getting into it. Especially since I am building this work computer. As a final note, I am not 100% sure if I am supposed to make a new thread to post these questions, as this is my second time ever posting on a forum. Hopefully someone can shed some light on my situation.

Posting your own thread was the right move.
Unless you need certified drivers for some software under Windows, a Quadro GPU is probably not the best choice.
Tensor cores are pretty much useless outside of AI/machine learning. And in case you want do some GPU acceleration for other workloads, a Geforce card for half the price is just as good as the Quadro RTX lineup. They all lack double precision compute performance, and only excel at single precision.
Since you want to game on this system anyway, I would say get a faster Geforce graphics card for less money.

A Threadripper CPU is debatable. If most of your work consists of CFD, an Epyc CPU might be the better choice. But due to lower clock speeds, it can't achieve the same FPS in CPU-bound scenarios in games as a Threadripper CPU. So if you play competitive games at lower image quality settings to get the highest possible FPS, TR would be better.
It's a tradeoff you need to make based on your priorities.

[Ry_z]

Originally Posted by flotus1

Posting your own thread was the right move.
Unless you need certified drivers for some software under Windows, a Quadro GPU is probably not the best choice.
Tensor cores are pretty much useless outside of AI/machine learning. And in case you want do some GPU acceleration for other workloads, a Geforce card for half the price is just as good as the Quadro RTX lineup. They all lack double precision compute performance, and only excel at single precision.
Since you want to game on this system anyway, I would say get a faster Geforce graphics card for less money.

A Threadripper CPU is debatable. If most of your work consists of CFD, an Epyc CPU might be the better choice. But due to lower clock speeds, it can't achieve the same FPS in CPU-bound scenarios in games as a Threadripper CPU. So if you play competitive games at lower image quality settings to get the highest possible FPS, TR would be better.
It's a tradeoff you need to make based on your priorities.

Thank you for the response. I’ll definitely take a look at the Epyc CPUs and the GeForce cards.

[Ry_z]

Originally Posted by flotus1

Posting your own thread was the right move.
Unless you need certified drivers for some software under Windows, a Quadro GPU is probably not the best choice.
Tensor cores are pretty much useless outside of AI/machine learning. And in case you want do some GPU acceleration for other workloads, a Geforce card for half the price is just as good as the Quadro RTX lineup. They all lack double precision compute performance, and only excel at single precision.
Since you want to game on this system anyway, I would say get a faster Geforce graphics card for less money.

A Threadripper CPU is debatable. If most of your work consists of CFD, an Epyc CPU might be the better choice. But due to lower clock speeds, it can't achieve the same FPS in CPU-bound scenarios in games as a Threadripper CPU. So if you play competitive games at lower image quality settings to get the highest possible FPS, TR would be better.
It's a tradeoff you need to make based on your priorities.

After looking at the Epyc CPU's, I am a bit lost in the variations. As I am new to this, is there a specific Epyc CPU you would recommend. Also, if you have the time, would you be so kind as to elaborate on why the Epyc CPU's are better. I am genuinely curious as I am an armature at the hardware stuff.

[flotus1]

The most relevant advantage of Epyc vs Threadripper in the context of parallel CFD is 8 vs. 4 memory channels.
Memory bandwidth is an important metric for parallel CFD. As a rule of thumb: many CFD codes stop scaling somewhere between 2-4 cores per memory channel. This means that a large portion of those 24 Threadripper cores would be useless. You can see it here:
CFD workstation configuration calling for help
More examples here:
OpenFOAM benchmarks on various hardware

So while single-core, Epycs are slower than their TR counterpart due to lower clock speeds, they more than make up for it with much better scaling while solving CFD cases in parallel. A 24-core Epyc 7402p will end up almost twice as fast as a TR 3960x. Memory can be tuned heavily on Threadripper CPUs, but not nearly enough to close this gap.
Which is already my recommendation: Epyc 7402p. It also has 24 cores, and costs about the same. Under an all-core load, it will boost to around 3.3 GHz. As for how many FPS in games you might leave on the table compared to TR: they are mostly the same CPU, so you can simply extrapolate using clock speed. Maybe subtract another 10% for the overall higher latency on Epyc. But again, this is only relevant for CPU-limited scenarios. With higher resolutions and quality settings, you will often be GPU-bound.
Keep in mind that you will need 8 DIMMs to fill all 8 memory channels on an Epyc CPU. And they only use reg ECC memory. So for example 8x8GB or 8x16GB DDR4-3200 reg ECC. In order to squeeze out the last few percent of performance, those 16GB DIMMs should be dual-rank.

[Ry_z]

Oh ok, so the 8 Ram slots on the TRX motherboard can only be fully used by the Epyc? and Using more ram slots, even if it is for the same amount of ram makes the cpu faster? So if I am fully focused on CFD because a buddy of mine offered me a second Geforce GPU, then would I want to put that extra cash into more ram or a more powerful Epyc CPU? Sorry about all of the questions, but you seem to be a really knowledgeable source, and I feel a bit lost haha.

[flotus1]

You will also need a different motherboard for Epyc. The socket is called SP3. The CPUs have the same package and number of pins, but they are not mutually compatible. And even if Epyc Rome CPUs would run on TRX40 boards, you would still be limited to 4 memory channels.
A memory channel is not the same as a DIMM slot. The memory controller, which determines max #memory channels, resides within the CPU package nowadays. Memory traces on TRX40 boards can only connect to 4 memory channels, with 2 DIMMs per channel.
So no, using more DIMM slots does not make things faster. Using all available memory channels -and having more of those to begin with- does.

There are a few options for single-socket SP3 boards. ASock Rack EPYCD8(-2T) for example. Most of these options currently do not provide PCIe 4.0, which is a bit of a shame. But not the end of the world in my opinion. Notable exception: Gigabyte MZ32-AR0, which comes in an SSI-EEB form factor.

The next higher tier CPU would be the 32-core Epyc 7502P. The price increase is pretty steep though, and you will not fully benefit from the additional cores. memory bandwidth bottleneck strikes again.
What I like to recommend is getting a dual-socket board instead. You can run these with only one CPU at first, and just drop in a second CPU with another 8 DIMMs if you feel the need to upgrade. This won't work with Epyc CPUs with a "p" suffix. These can only run single-socket. So you would have to get a similar CPU without a "p" at the end.

[Arnolm]

Looks like someone has just the same idea like me and i was just about to open a thread with the very same question ... I'm also looking into building a CFD-workstation with a side affiliation to gaming. As in this thread already discussed there are two options:
1) Epyc Rome CPU 7402p with advantage of the high number memory lanes. E.g. this build:
https://geizhals.de/?cat=WL-1649374
2) Threadripper 3960x with advantage of higher cpu and ram frequency and official Win10 support. E.g. this build:
https://geizhals.de/?cat=WL-1659456
Both builds have the same peripheral (1TB pcie4 m.2 ssd, geforce 2070s and enough power and cooling) and only differ in cpu, ram and mainboard. Both builds endup with the same 3.1k price tag. The epyc build has the option to go down by 0.5k with a 7302p and thus reduced to 16 cores.

So my questions are:
a) The epyc rome mainboard has officially no support for Windows 10 (none other has either) only Windows server, the geforce has only support for win10. So what are the issues if I install Win10 to the server board anyway? For all the single components i could find win10 drivers by the chip manufacturers, but not by the mainboard manufacturer. The cpu is supported by the newer releases of win10.
b) Comparing the epyc romes, at how many cores is the memory wall for this cpus. Is the +500 for the 24core worth compared to the 16core, or do i hit the wall al e.g 18 active cores. If the wall is beyond the 24, the 7402p is propably worth it.
c) The threadripper will be slower with lower memory channels, but how much slower? Are we talking about 10%, 20%, 50% considering the option for faster memory access? The threadripper can be pushed much harder in this context considering the numbers on the datasheets. Did anyone made a benchmark yet with the threadripper. Didn't find this one in the long benchmark list in the other thread.

Regards
Matthias

p.S. Sorry for intruding this discussion with my own, but the question is SO close match ...

Edit:
Forgot to mention for clarification, the Win10 will be only for games, office, cad, etc. The main OS will be Ubuntu for Openfoam, Python, computations, etc. So it will be dualboot.

[flotus1]

a) The epyc rome mainboard has officially no support for Windows 10

From my experience and that of other people, it just works with the generic chipset drivers that Windows 10 provides. There might be some issues with more advanced features, but probably nothing that will stop you from running games on it.

b) Comparing the epyc romes, at how many cores is the memory wall for this cpus. Is the +500 for the 24core worth compared to the 16core, or do i hit the wall al e.g 18 active cores. If the wall is beyond the 24, the 7402p is propably worth it.

The usual non-answer is: it depends.
For real software, better not think of memory bandwidth limit as a "wall", but as an asymptotic limit. You usually still get some more performance out of it, even if scaling stops being linear. Considering the overall cost of this machine, I don't think saving a few hundred bucks by getting 16 instead of 24 cores is worth it.

c) The threadripper will be slower with lower memory channels, but how much slower? Are we talking about 10%, 20%, 50% considering the option for faster memory access? The threadripper can be pushed much harder in this context considering the numbers on the datasheets. Did anyone made a benchmark yet with the threadripper. Didn't find this one in the long benchmark list in the other thread.

Baseline with default memory settings will be very close to a 2x performance advantage for Epyc. With meticulous memory tuning, you can probably increase TR performance by around 20% over stock.

d) Will the Asus KRPA-U16 fit inside the BeQuiet Silent base 801?

Probably not. And even if you can make it fit, you will loose all 3.5" hard drive bays. I recommend Phanteks Enthoo Pro instead, or any other case that explicitly states compatibility with SSI-EEB motherboards. Remember, E-ATX is not the same as SSI-EEB. The former is a marketing term meaning "anything wider than ATX", the latter is an actual form factor.
As far as I am aware, BeQuiet currently does not sell any case for SSI-EEB without huge compromises.

e) is that Power supply a good choice

BeQuiets Pure Power series is rather entry-level stuff, and lacks a second EPS 12V connector. Maybe you might have no issues leaving that second connector off, but I would rather not risk it and use something like Seasonic Focus PX instead.

[Ry_z]

Originally Posted by flotus1

From my experience and that of other people, it just works with the generic chipset drivers that Windows 10 provides. There might be some issues with more advanced features, but probably nothing that will stop you from running games on it.


The usual non-answer is: it depends.
For real software, better not think of memory bandwidth limit as a "wall", but as an asymptotic limit. You usually still get some more performance out of it, even if scaling stops being linear. Considering the overall cost of this machine, I don't think saving a few hundred bucks by getting 16 instead of 24 cores is worth it.


Baseline with default memory settings will be very close to a 2x performance advantage for Epyc. With meticulous memory tuning, you can probably increase TR performance by around 20% over stock.


Probably not. And even if you can make it fit, you will loose all 3.5" hard drive bays. I recommend Phanteks Enthoo Pro instead, or any other case that explicitly states compatibility with SSI-EEB motherboards. Remember, E-ATX is not the same as SSI-EEB. The former is a marketing term meaning "anything wider than ATX", the latter is an actual form factor.
As far as I am aware, BeQuiet currently does not sell any case for SSI-EEB without huge compromises.


BeQuiets Pure Power series is rather entry-level stuff, and lacks a second EPS 12V connector. Maybe you might have no issues leaving that second connector off, but I would rather not risk it and use something like Seasonic Focus PX instead.

I will most likely be defining my Geometries in Solidworks. Even if I use another platform for the geometry, I will still need Solidworks for a multitude of other reasons. Because Solidworks only supports windows OS, I cannot use Linux if I get the Epyc 7402. That leaves Windows server 2019, but that appears to get expensive very quickly. I was wondering what OS you use and what you would recommend that I do here?

[flotus1]

A lack of official support does not mean things won't run.
Apart from some trouble during installation (IOMMU had to be enabled), Windows 10 Pro worked without any issues for me on AMD Epyc CPUs. However, my personal experience with Windows 10 on these platforms is pretty limited. I only used it to fire up some popular benchmarks, just for shits 'n giggles.

[Arnolm]

Thanks for your advices. Really much appreciated.

Windows 10 Pro worked without any issues for me on AMD Epyc CPUs

That's really good news. This takes the biggest advantage away from the threadripper.

For real software, better not think of memory bandwidth limit as a "wall", but as an asymptotic limit.

Sure, there will be always some kind of convergent behavior, but in the benchmark thread, there are some cpus which get not a single second acceleration after some point. That's what i was refering to with the "wall". But let's not debate on wording. I just wanted to make sure, that I don't pay +50% on the cpu for a +5-10% on performance.
My initial goal was to build a system in the range of 1.5-2k. But well to match the numbers of one component, the quality of another is increased, leading to some some excess in performance on the latter, requiring higher quality on the first component ... and the things are escalating quite fast. Let's hope my bank account and the last tax refund work together to give a "go" in the end ...

With meticulous memory tuning, you can probably increase TR performance by around 20% over stock

ok, that is the second advantage (the higher memory frequency) of the threadripper gone. So it has only the advantage of the single core performance left on the cost of a significantly reduced multicore performance for my planned application. I guess that's it for the threadripper and the money goes on the epyc.

Remember, E-ATX is not the same as SSI-EEB

BeQuiets Pure Power series is rather entry-level stuff, and lacks a second EPS 12V connector

Ok, those two topics i missed. I understood the E-ATX to be some defined formfactor, but looking onto the numbers, I have to agree that will not fit. :-( The HDD cage is not an issue for me, as a don't plan to use any HDDs. I have a central NAS for mass storage. What got me on the bequiet case is the integrated noise cancellation and the included fans, which i know from experience to be quite silent. So a good cost-perfomance-ratio in my eyes, but it seams to be non matching. I will restart my search for a good casing. To be honest, the Phantek is not that pleasing to my eyes, but that is a really really subjective point and should not be the only reason.
Same for the power supply. I had those in the past and never had an issue with noise, stability, etc. But correct, it is better to do the full power cables. I will search for a suitable one and check the seasonics.


So my next step is searching a better matching case and power supply and redefine the epyc build. Ones that is done, the mentioned discussion with the account starts. ;-)


Thanks again for your advices. They saved me from several nasty issues during assembly.

[flotus1]

About that case though...
I occasionally scour the current market and new releases. And I have yet to find a better case for SSI-EEB than the Phanteks Enthoo Pro. I would be willing to spend some serious cash for a case that ticks all my boxes. So far, no winner. What disqualifies many options for me are window side panels. Without that requirement, maybe you have better luck, because nearly everyone wants window side panels these days.

Phanteks Enthoo Pro 2 came out just recently. Maybe you like that one better. It is more expensive though, and comes without fans. Not necessarily a bad thing, because the included fans in its predecessor were nothing to write home about.
Fractal Design Define XL was another case I used in the past. While SSI-EEB boards do fit, the motherboard tray does not have all standoff positions for this form factor. So some DIY might be required. There is a newer version of this case, that also fits SSI-EEB if you don't need 3.5" hard drives.
Same issue (standoff positions) for Nanoxia Deep silence 2, another case I used in the past. It used to be one of the cheapest options for SSI-EEB. But even back then, the overall quality was pretty mediocre. Now that it costs almost 100€, it is a definite no-go for me.

[Ry_z]

How much M.2 and/or HDD storage should I acquire?

[flotus1]

That's entirely up to you.
Only you know how much storage you need, and if you are willing to spend extra to avoid spinning storage entirely.

[Arnolm]

Hi,

I owe you an update on the above topic and how my workstation plans ended up. Based on the discussion, i decided to go for the 7402P on an Asrock ROMEd8-2T with 8x16GB 128GB Kingston Server RAM ECC with 3200MHz-CL22. Well, that didn't go as expected. Runing the Benchmark case on ubuntu with openfoam 8 resulted in the following values:

# cores Wall time (s):
------------------------
1 690.39
2 382.7
4 168.52
6 113.5
8 85.47
12 75.41
16 60.95
20 61.51
24 57.2

The first half of those values were not unexpected, as they match the values for the 7302P CPU. However, the 20 and 24 core steps were a negative surprise. I hoped for an further increase in the performance, but it seems that the 7402P gives only a marginal performance boost over the 7302P. So from that perspective, the 7402P was not worth the extra costs.
And then it got even worse. I tried to install Windows parallel to linux and it failed hardly. Whatever I tried, whatever version of windows I tried to install, whatever settings in BIOS I updated, the installation failed after exactly 12seconds. So I had a CPU not bringing the performance and not allowing the required dualboot ...
Thus, I decided that I have to break with my principles and call myself lucky that there is a 14day return on online orders in our country. So I returned the EPYC, Motherboard, etc.

But I didn't give up. I switched to the Threadripper 3960x (same core number as 7402P, but only half the memory channels) on a Aorus Pro Wifi board with 8x8GBR1 Ripjaws V 3600-CL16 RAM. Testing with the Benchmark again resulted in a negative surprise as i got the following numbers:

# cores Wall time (s):
------------------------
1 717.59
2 487.72
4 290.03
6 239.13
8 211.03
10 212.04
12 213.94
14 213.85
16 216.55
18 217.46
20 220.98
22 222.83
24 227.63
(I increased the resolution in the upper part to see the limitations better)

So, the Threadripper showed a much worse performance in my test than the values "promised" in the benchmark thread. I turned out, one of the 8Dimms was Dead on Arrival, so with the four memory channels, only 6Dimms were operational. So replacing the Dimm gave the following numbers now with 8Dimms:

# cores Wall time (s):
------------------------
1 629.07
2 377.79
4 191.89
6 143.04
8 117.81
10 115.6
12 116.7
14 110.59
16 109.73
18 109.36
20 109.7
22 107.54
24 107.57

Much better and close to the reference results. However, not good compared to the EPYC. So I went on with checking and tuning. For this first test, the Memory was running on default setting with 2133MHz. Turning on the XMP-Profile increased the Memory clock to the 3600MHz, the Dimms are nominally capable of. This value is slightly higher than what the Threadripper is promised to deliver, however literally hundreds of reviews show, that the 3600MHz is a stable possibility for the 3960x. With the XMP setting I'm now at the following values:

# cores Wall time (s):
------------------------
1 495.91
2 277.07
4 135.05
6 98.47
8 80.66
10 83.64
12 77.94
14 73.51
16 72.51
18 72.46
20 72.4
22 72.33
24 71.8

Quite nice I think, as the benchmark thread showed a minimum benchmark time of approx. 90s. So I'm getting closer. But one more step I want to share with you. After installing Windows (this time with much less difficulties), I noticed during other Benchmarks, that the CPU goes up to approx 93°C on the Prime95 Test with a Dark Rock Pro TR4 aircooler. Thermal throttling starts for the CPU at 95°C, so that is in my eyes dangerously close. The Openfoam benchmark pushed it only to approx 75°C, so that is ok, but the prime95 was a warning sign for me. So I went on with tuning. Instead of increasing the performance, I limited the power of the CPU from the nominal PPT=280W down to 180W. On the Cinebench R20 Benchmark this step cost me 7.5% performance, but brought the Prime95 temperatures down to approx 75°C. So, -7.5% on the Cinebench but how much loss for the Openfoam benchmark following are the number:

# cores Wall time (s):
------------------------
1 496.09
2 276.84
4 131.87
6 97.89
8 80.63
10 79.44
12 75.14
14 73.55
16 72.43
18 75
20 73.11
22 71.81
24 71.91

I was quite confused on the results on the first view and first checked all settings triple. But also the power measurements confirmed, the CPU was running at 180W. So power to the CPU reduced by more than 1/3 and yet almost no impact on the Openfoam results? Well, after thinking about, it is logic. The high power is required for the Threadripper to achieve its maximum boost frequency, but the peak frequency is less relevant for Openfoam. Thus, the computation is slower, yes, but the time for the memory access is still slow enough to cover this up.

So my conclusion for now is:
- The Epyc 7402P is not worth compared to the 7302P.
- The Epyc 7402P performed up to 16 cores as expected, but is single core much slower than the Threadripper.

- The Threadripper can make up the memory bandwidth penalty of 4 Channels with the increased frequency and reduced latency
- The Threadripper fulfils my requirements better
- Running the Threadripper with limited power makes sense regarding efficiency and performance drop.


Next step would be checking, whether I can kick the memory a little bit harder regarding frequency or latency. Aim would be to close the gap to the EPYC Rome. However, I'm not sure, whether the additional work and risk to stability is worth those last 15% performance gap. Also, I guess that a further investigation of that power topic might be worth to find the limit.

regards
Matthias