[EternalSeekerX]

Hello Everyone,

I hope everyone is staying safe and things are going well during this pandemic. So to start things off, about a few months ago (might even be a year ago?) I posted a thread elsewhere about my Kraken X61 AIO pump being broken. Due to being out of warranty I never replaced it and been using my Asus Strix Laptop (i7 7700HQ+GTX 1060). Fast forward to now, a couple of my friends surprised me with the newish Kraken X63 AIO cooler. So that means I get to use my desktop again (i7 Rampage 5+5960X+GTX 970+ 16GB DDR4 RAM 2133Mhz). I build this rig when I was in uni in 2014 for gaming and engineering capstone projects. However now with the pandemic I am taking my free time to improve my skillset so I have been working with 3D CAD, Rendering (Blender) and running validation CFD models. Currently I am using my laptop with windows host (for solid works and such) and a Linux guest (running Ansys, openfoam, su2 cfd with cases ranging from 500,000 to 35 million cell cases). And I am also trying to expand into learning StarCCM+, XFlow, and other 3D Cad software in linux as well. In terms of gaming I haven’t been gaming as much, however I have been playing more ps2/wiiu/3ds emulators, bf4, warframe, pubg. I am hoping I can play more of the newest titles as well.

Now comes the part where Idk how to balance my rig now with my newish workloads. Gaming, streaming and CAD favor higher clocks. Simulations favor core, memory channel and then clock speed (there is a rule of thumb where you want at a minimum of 2*N cores where N is the amount of memory channels, as well there is a rule of thumb where the sweet spot for ram is for every 1GB of ram you can fit 2 million mesh cells, of course there are cases where its different and you can get more performance for absurd amount of cores and ram size). That’s why number crunching is mostly done on multi-socket systems as they offer multiple cores and memory channel. So going down to a consumer cpu with 12+ cores might actually decrease performance since memory bottleneck. Currently I have been distro hoping and I found I got most performance in rpm based distros (plus ANSYS and commercial codes officially support centos/rhel), and even then there are tutorial cases that would take weeks to run (specifically the SurfaceMountedCube in OpenFOAM v1912/v2006). Since I was running in a vm and on a laptop with dual channel memory I am hoping my desktop could cut it down by a lot. And of course blender renders prefer more cores/thread as well. I also would love to get back into streaming and editing and hopefully be able to game at higher res+refresh rate (so say 2K240, or 4K60), but not a big requirement. Also I am hoping to dual boot windows 10 (for solidworks, windows specific apps, gaming, etc) and most probably CentOS 7 (blender render, paraview, simulations).

So I was thinking maybe I can overclock the 5960x, upgrade ram (go from 16 to either 32 or 64GB) and storage (currently the desktop has 1TB HDD, 512GB sata SSD for data, and 128GB sata SSD for windows boot) now and maybe when market settles down, buy a new gpu? Maybe play with gpu passthrough too? Can a overclock 5960x handle these types of requirements in 2020 (I think it can!)? Will I be bottlenecked by my cpu?

Thanks

[flotus1]

X99 still relevant in 2020: Yes, especially when you already have most of the parts.
Upgrade to 4x16GB of RAM, and you are good to go for CFD. Don't buy less than DDR4-3200 CL16, prices don't really go down beyond that point. That should cost you somewhere around 220€.

As for your other requirements: X99 never has been the best choice for maximum FPS in games, but it can still hold up quite well. Whether that's enough for you... nobody can tell. Depending on the game, settings and graphics card used, it can be head-to-head with the latest and greatest consumer chips. Or it can fall behind by 40% or more.

[Simbelmynë]

If you get ddr4 3200 dual rank, then you might not be able to run it at XMP speeds, if your IMC is poor (silicon lottery). But you should probably be able to run 3000 MT/s speeds.


Gaming performance is only CPU bottlenecked

1. You are a competitive gamer.
2. If you play on 1080p or lower resolutions (or really low in-game settings)
3. If you have a high end GPU (the 970 will most likely not be bottlenecked by the CPU).


Also, in order to display high frame-rates you need a monitor that can support it, naturally there is no need investing in a CPU+GPU combo for super high frame-rates if you cannot display it.


Finally, it is not easy to tell the difference in frame-rates as you go higher and higher. Between 20-30, sure. Between 100-144, well.. most would probably not be able to tell the difference. Which again means that purchasing a CPU+GPU combo to play (old) games with extreme frame-rates may not be very useful except if you like high numbers on paper.


A lot about gaming, from a person that has gaming as an ambition to relax the mind from work, but very seldom manages to play, due to work.


tl; dr: I think your CPU is a reasonable choice for your needs.

[EternalSeekerX]

Originally Posted by flotus1

X99 still relevant in 2020: Yes, especially when you already have most of the parts.
Upgrade to 4x16GB of RAM, and you are good to go for CFD. Don't buy less than DDR4-3200 CL16, prices don't really go down beyond that point. That should cost you somewhere around 220€.

As for your other requirements: X99 never has been the best choice for maximum FPS in games, but it can still hold up quite well. Whether that's enough for you... nobody can tell. Depending on the game, settings and graphics card used, it can be head-to-head with the latest and greatest consumer chips. Or it can fall behind by 40% or more.

Yes an excellent reason to stick with X99. I was looking for some validation. I still have a few things I can upgrade before x99 become obsolete. Installed the new cooler and voila I get 28c idle temps and at load I'm between 40 and 50c which seems good for cfd runs. And I'm mighty impressed at the performance i get on the desktop thanks to quad channel 16gb. My motorbike run time went down from 680 seconds to 211 seconds.

For a 1 million cell fluent case (SIMPLE aerodynamic)the Windows machine took 4min while the Linux one took 3 min and 7 seconds (even with the vm overhead) on a 5960x using 6 physical cores.

For a 4.5 million cell cfx case (Transient) Windows machine took 2hrs and 16 min and 40 seconds while the Linux machine took 2hrs and 16 min and 53 seconds. Given that vm has some overhead, a 14 second delta could be an outlier. Maybe on a bare bone Linux install it will match or be quicker? It's way quicker than the full 7hour runtime on my laptop.

I assume a ram upgrade might speed it up? Also what sort of effects am I getting if I fill all the ram slots up versus only filling up 4 (curious as to why go 4x16 instead of 8x8? Price?). Also how much does ram frequency effect performance? My 16gb kit is rated for 2133mhz but my mobo seems to run it at 1566mHz?

Originally Posted by Simbelmynë

If you get ddr4 3200 dual rank, then you might not be able to run it at XMP speeds, if your IMC is poor (silicon lottery). But you should probably be able to run 3000 MT/s speeds.


Gaming performance is only CPU bottlenecked

1. You are a competitive gamer.
2. If you play on 1080p or lower resolutions (or really low in-game settings)
3. If you have a high end GPU (the 970 will most likely not be bottlenecked by the CPU).


Also, in order to display high frame-rates you need a monitor that can support it, naturally there is no need investing in a CPU+GPU combo for super high frame-rates if you cannot display it.


Finally, it is not easy to tell the difference in frame-rates as you go higher and higher. Between 20-30, sure. Between 100-144, well.. most would probably not be able to tell the difference. Which again means that purchasing a CPU+GPU combo to play (old) games with extreme frame-rates may not be very useful except if you like high numbers on paper.


A lot about gaming, from a person that has gaming as an ambition to relax the mind from work, but very seldom manages to play, due to work.


tl; dr: I think your CPU is a reasonable choice for your needs.

Great response, I am thinking of adding things to the pc as I go. I think I'll invest in new gpu before a full platform upgrade. So that's why I asked about cpu bottleneck. I run most games at 144hz+ on 1080p as my monitor supports it. I am thinking I would def upgrade monitor if I can play at 1440p high refresh or 4k60.

Thoughts?

[flotus1]

I assume a ram upgrade might speed it up?

It depends...
If that 4.5 million cell benchmark did not run into memory limitations (including OS overhead), using more memory won't help much. My recommendation for 4x16GB was triggered by you hinting at much larger cases with 35 million cells.

Also what sort of effects am I getting if I fill all the ram slots up versus only filling up 4 (curious as to why go 4x16 instead of 8x8? Price?)

On platforms with more than one DIMM slot per memory channel, there is no good reason to fill them up entirely, when you can get the same amount of RAM while using only one DIMM per channel. But there are many reasons against it.
Upgrade path: 35 million cells with Ansys CFX might or might not fit into 64GB of RAM. That entirely depends on the solver settings. When you already filled up all slots with small modules, you can't just do an upgrade to 128GB, but you would have to replace all of the memory.
Memory overclocking, which you will be doing when running memory faster than DDR4-2133: apart from the usual factors like IMC quality, motherboard and DIMM binning, the maximum stable memory frequency depends on the amount of ranks per memory channel. More ranks cause higher load on the IMC, leading to lower frequency. Now technically, most DDR4 UDIMMs with 8GB have a single rank these days, while most 16GB UDIMMs have two ranks. So the amount of ranks per channel is most likely the same for 8x8GB and 4x16GB. But thanks to the simpler topology of one DIMM per channel vs two, the former will still come out on top in most cases, everything else being equal.
Cooling: also ties in with the previous point about OC. Maximum stable memory overclocks can be tied to DIMM temperature. 4 modules will produce less heat than 8. And having them spaced one slot apart instead of crammed together will further reduce temperatures.

Also how much does ram frequency effect performance? My 16gb kit is rated for 2133mhz but my mobo seems to run it at 1566mHz?

Memory frequency matters a lot, for the same reason why X99 is still relevant in 2020, thanks to 4 memory channels vs 2 on consumer platforms: it increases maximum memory bandwidth.
Observing your memory clocking at 1566MHz could mean one of two things:
1) You are already overclocking your memory. Tools like CPU-Z on Windows show "real" memory frequency, which is half the transfer rate for DDR. I.e. DDR4-2133 clocks at 1066MHz, which is what the tools are showing.
2) Somehow memory is running much slower than it should
You should definitely check which one it is in bios/UEFI.

[EternalSeekerX]

Originally Posted by flotus1

It depends...
If that 4.5 million cell benchmark did not run into memory limitations (including OS overhead), using more memory won't help much. My recommendation for 4x16GB was triggered by you hinting at much larger cases with 35 million cells.


On platforms with more than one DIMM slot per memory channel, there is no good reason to fill them up entirely, when you can get the same amount of RAM while using only one DIMM per channel. But there are many reasons against it.
Upgrade path: 35 million cells with Ansys CFX might or might not fit into 64GB of RAM. That entirely depends on the solver settings. When you already filled up all slots with small modules, you can't just do an upgrade to 128GB, but you would have to replace all of the memory.
Memory overclocking, which you will be doing when running memory faster than DDR4-2133: apart from the usual factors like IMC quality, motherboard and DIMM binning, the maximum stable memory frequency depends on the amount of ranks per memory channel. More ranks cause higher load on the IMC, leading to lower frequency. Now technically, most DDR4 UDIMMs with 8GB have a single rank these days, while most 16GB UDIMMs have two ranks. So the amount of ranks per channel is most likely the same for 8x8GB and 4x16GB. But thanks to the simpler topology of one DIMM per channel vs two, the former will still come out on top in most cases, everything else being equal.
Cooling: also ties in with the previous point about OC. Maximum stable memory overclocks can be tied to DIMM temperature. 4 modules will produce less heat than 8. And having them spaced one slot apart instead of crammed together will further reduce temperatures.


Memory frequency matters a lot, for the same reason why X99 is still relevant in 2020, thanks to 4 memory channels vs 2 on consumer platforms: it increases maximum memory bandwidth.
Observing your memory clocking at 1566MHz could mean one of two things:
1) You are already overclocking your memory. Tools like CPU-Z on Windows show "real" memory frequency, which is half the transfer rate for DDR. I.e. DDR4-2133 clocks at 1066MHz, which is what the tools are showing.
2) Somehow memory is running much slower than it should
You should definitely check which one it is in bios/UEFI.

Thanks for the response. The smaller 4.5 million cell case was a mini benchmark for me to test windows and Linux (albeit in a vm) performance for ansys. I was only able to run the case with 4 cores as anything above 6 would reach memory limit. There are a few cases im currently working on, like a SAS Turbulence model of a drone in CFX and an openfoam case for supersonic su-35 jet that seemingly always slow down during meshing due to hitting max ram, so that's where the 35 million cell number came from.

Also I didn't realize the cooling and upgradability aspect of using only 4 dimms. Thanks for the clarification on that. I should check cpu-z then, because in the bios it shows memory speed as 1566mHz and the memory speed is set to Auto in bios.

Also is there any benefit for cpu overclock? Or is cpu overclock a no go for cfd?

Thanks.

[flotus1]

Also is there any benefit for cpu overclock? Or is cpu overclock a no go for cfd?

With your particular setup, there is some potential for CPU overclocking. Not only the CPU cores, but also the "uncore" part of the CPU, which affects L3 cache and memory controller.
Note that I am only referring to stable overclocks, not the ones that can just barely pass a short benchmark to get a high score. No point in getting 30% faster solver times, when your 2 week long simulation crashes halfway through.

[EternalSeekerX]

Originally Posted by flotus1

With your particular setup, there is some potential for CPU overclocking. Not only the CPU cores, but also the "uncore" part of the CPU, which affects L3 cache and memory controller.
Note that I am only referring to stable overclocks, not the ones that can just barely pass a short benchmark to get a high score. No point in getting 30% faster solver times, when your 2 week long simulation crashes halfway through.

Thank you again for the response. So when you mean the 'uncore" part, are you referring to the actual multiplier called uncore in the bios? I have seen it before, never knew what it does. I would obviously try to get a stable overclock (what are you thinking in terms frequency should i target? Have you any overclocking experience with Haswell-E?) I wouldn't be using a overclock that would crash a simulation that takes 2 weeks or even weeks/months (though I am not at that level yet , are there even cfd jobs that take that long?) Just enough to give me a free boost in wall time.

Thanks

[flotus1]

From my limited experience with single-socket X99 (sample size=2), default uncore frequency varies between different motherboards. So depending on your luck of the draw, you can increase that by quite a bit. Somewhere around 4GHz should be possible with some tinkering. But be aware that this increases power draw by a lot. Also, rule of thumb is that setting uncore frequency higher than core frequency is rather pointless. Due to the very steep increase in power draw, maybe keep it 300-400 MHz lower than all-core CPU frequency.
For core frequency, thy sky is the limit
But for 24/7 operation, I would probably not go above 4.2 GHz all core.

[EternalSeekerX]

Originally Posted by flotus1

From my limited experience with single-socket X99 (sample size=2), default uncore frequency varies between different motherboards. So depending on your luck of the draw, you can increase that by quite a bit. Somewhere around 4GHz should be possible with some tinkering. But be aware that this increases power draw by a lot. Also, rule of thumb is that setting uncore frequency higher than core frequency is rather pointless. Due to the very steep increase in power draw, maybe keep it 300-400 MHz lower than all-core CPU frequency.
For core frequency, thy sky is the limit
But for 24/7 operation, I would probably not go above 4.2 GHz all core.

4.2Ghz is a decent overclock, however if higher clocks are stable i think i can go for those? If the oc is stable during stress test does it mean it be stable for long cfd runs to? Because I was already worried that a almost 40+hrs surface mounted cube openfoam case would crash my laptop when I ran it there and I would hate it if it actually crashed on my overclocked desktop. Would be worse for longer runs (can't imagine how salty I'd be if I had to rerun a weeks/months long simulation if it did crash 60% of its way through (though I bet those complex simulation would be cool to see, maybe not at that level yet, have you worked with long wall time cases before on a desktop?). Also I'm gonna probably keep uncore near 4ghz if I can. I'm also gonna keep an eye out for a sale for the ram, im in Canada so a 4x16gb would be near $400CDN.

Thanks again

[flotus1]

It's about diminishing returns.
Running on all 8 cores, CPU frequency won't translate 1:1 into faster solver times.
And you will need increasingly higher voltage for even higher frequency. Which causes a dramatic increase in power draw. Doubling your electricity bill for 5% faster solver times...it's up to you.
Also, degradation is a real thing. Not in the way that some people think about it, with hardware getting slower with age. That's not how it works. CPUs usually last longer than they are useful. But if you hit a 5960X with 4.6GHz for weeks or months, it will no longer be able to run this frequency at some point.

[EternalSeekerX]

Originally Posted by flotus1

It's about diminishing returns.
Running on all 8 cores, CPU frequency won't translate 1:1 into faster solver times.
And you will need increasingly higher voltage for even higher frequency. Which causes a dramatic increase in power draw. Doubling your electricity bill for 5% faster solver times...it's up to you.
Also, degradation is a real thing. Not in the way that some people think about it, with hardware getting slower with age. That's not how it works. CPUs usually last longer than they are useful. But if you hit a 5960X with 4.6GHz for weeks or months, it will no longer be able to run this frequency at some point.

I see, alright then I'll try balance around stable voltages to not degrade cpu as much. Just want a decent speed up for all work loads.