[MangoNrFive]

Hi,

we want to build a CFD-Workstation.

Usage: OpenFOAM, StarCCM+ (core-unlimited)
Models used: stationary, instationary, incompressible, moving-mesh, conjugate-heat-transfer
Mesh-sizes: 5-20Mio up to 60Mio
Operating-system: Linux (preferably Ubuntu)
Location: Munich, Germany
Budget: ~15.000€

This is what I came up with:

• 2x AMD EPYC 9554, 256MB, 64@3,1-3,75GHz, 360W (8203€)
• Gigabyte MZ73-LM1 (1430€)
• 24x Kingston FURY Renegade Pro RDIMM 16GB, DDR5-4800, CL36-38-38, reg ECC, on-die ECC (1977€)
• GIGABYTE Radeon RX 7800 XT Gaming OC 16G, 16GB GDDR6 (565€)
• Samsung SSD 990 PRO 4TB, M.2 (300€)
• 2x XE360-SP5 (AiO) (~400€??)
• be quiet! Dark Power Pro 13 1600W, 80 Plus Titanium (405€)
• Phanteks Enthoo Pro 2 Server Edition, Closed Panel (150€)

Total: 13030€

Some considerations / questions:

• This is my first ever PC-build, so I´m thankfull for any hints even if they may be obvious to others. I tried to get as far as I can with my own research, but there are sure still some things that I´m missing
• I still have room in the Budget, but I don´t see where it would make sense to spend more, better graphics-card and more storage isn´t really necessary I think and the next best CPU is a lot more expensive
• I have only one SSD, as all the persistant storage, simulation results etc. is done on a remote server and/or a local synology. The mainboard has 2 x 10Gb/s BASE-T LAN ports, that is plenty, or would it make sense to get an extra network-card?
• The Power-supply is probably a bit overdimensioned, but the lower wattage one isn´t that much cheaper and this leaves room for possible upgrades in the future
• The PC will be located in the office directly, so noise is an important consideration
• I couldn´t find yet where to buy the XE360-SP5 AiO CPU water-cooler, any ideas? I would like to avoid a custom loop if possible, and this is the only AiO for this socket I could find

[flotus1]

That's an ambitious project for your first PC ever. Maybe you know someone with some experience building PCs, to assist you during assembly? But hats off to you, you got pretty far in the research stage already

I still have room in the Budget, but I don´t see where it would make sense to spend more, better graphics-card and more storage isn´t really necessary I think and the next best CPU is a lot more expensive

The CPUs at that price point are hard to beat. I'd say good choice.

The mainboard has 2 x 10Gb/s BASE-T LAN ports, that is plenty, or would it make sense to get an extra network-card?

That is indeed plenty, no need for additional networking gear.

The Power-supply is probably a bit overdimensioned, but the lower wattage one isn´t that much cheaper and this leaves room for possible upgrades in the future

Yeah, this PC won't draw more than 1000W peak, so there is plenty of room to grow. Though GPUs are the only components that could significantly increase power draw.

I couldn´t find yet where to buy the XE360-SP5 AiO CPU water-cooler, any ideas? I would like to avoid a custom loop if possible, and this is the only AiO for this socket I could find

Tough choice. AIO water coolers for AMDs server sockets mostly exist because there are workstation variants of those chips. I.e. Threadripper. Since we don't have TR for SP5 yet, I am surprised that there even is an AIO for that socket.
The XE360-SP5 has been announced just this month. Maybe we just need to wait a bit for availability. Rumor has it that Genoa Threadripper will be released in Q4 2023. That should improve things on the AIO front.
My pipe dream of a new dual-SP5 workstation includes a custom loop, so I can't really tell you more

RAM:
These "overclocked" DDR5 reg ECC kits were specifically developed for Intels Xeon-W Sapphire Rapids. I'm not saying it won't work on Epyc Genoa, but it is definitely a risk. And I am not even sure if you get the lower timings on any other platform. If you don't need the last 5% of performance from the better timings, maybe stick to the regular reg ECC memory modules.

[MangoNrFive]

Thank you for your reply Alex!

That's an ambitious project for your first PC ever. Maybe you know someone with some experience building PCs, to assist you during assembly?

Indeed I have some help in my office. With that, all the information that is out there nowadays and a little patience I'm sure we'll manage .

But hats off to you, you got pretty far in the research stage already

Hats of to you, for writing a great hardware-guide that helped me a lot in this pursuit.

These "overclocked" DDR5 reg ECC kits were specifically developed for Intels Xeon-W Sapphire Rapids. I'm not saying it won't work on Epyc Genoa, but it is definitely a risk. And I am not even sure if you get the lower timings on any other platform. If you don't need the last 5% of performance from the better timings, maybe stick to the regular reg ECC memory modules.

I don't want to take any risks here for a little uncertain performance gain, so thanks a lot for pointing this out. Can you recommend any/all of these? Or if not, what should I look for then?

Kingston Server Premier RDIMM 16GB, DDR5-4800, CL40-39-39, reg ECC, on-die ECC
Micron RDIMM 16GB, DDR5-4800, CL40-39-39, reg ECC, on-die ECC
Samsung RDIMM 16GB, DDR5-4800, CL40-40-40-77, reg ECC, on-die ECC

[flotus1]

That's what i had in mind. You can also check the memory QVL for your motherboard, maybe one of those even made it to the list.
Don't forget to add some case fans. AIOs for the CPUs are nice, but there are still other components in a server-y PC that need plenty of airflow. And the case does not include any fans.

[JBeilke]

I don't know, why you choose this graphics card. It burns a lot of energy, which makes cooling and noise difficult and does not help for GPU computing, since ccm+ requires a NVIDIA card for this purpose.

I would probably use something like a Quadro P2000 or P4000 in this machine. They are more than enough for typical CFD work and use much less energy.

Since ccm+ works very good in client-server-mode, you can consider a second machine - maybe with a Ryzen 7800X3d - with a "better" graphics. This will make your noise problem much easier.

https://www.serverschmiede.com/de/nv...dp-1024-cuda-1

https://www.serverschmiede.com/de/nv...te-gpu-4x-dp-1

[MangoNrFive]

Hi Joern,

I heard in several places now, that a Quadro isn´t really necessary, as it´s main advantage is better driver support. But we aren´t pushing the boundaries of GPU-capabilities so the standard consumer drivers should be good. For a Quadro with similar VRAM and processing power we´d need to pay a lot more.

Now why I chose exactly this graphics card is, that it has the ports that I want, good performance and plenty of VRAM. But there are a lot of other cards that would fit this criteria as well.

I didn´t really have GPGPU in mind, as we need moving-mesh sometimes which isn´t supported in StarCCM+, so we rather invest our money in a powerful CPU.

@Alex, indeed the Kingston Server Premier RAM was in the QVL, so I´ll be choosing that. I saw, that there is a category for SSDs on the QVL as well, but the SSDs on there are all reaaally slow, especially for writing (3 times slower). Do you think that the Samsung SSD 990 PRO 4TB, M.2 should be good, even though it´s not on the QVL? I think the other ones have a higher reliability, but that is not super important to us as we´ll just use it as a temporary storage for the simulation and have all the data saved on the synology.

[JBeilke]

The reason I suggested the Quadro P2000 is, that it burns only 75 Watt and the card is more than enough for ccm+ pre and postprocessing. 300 Euros are not a big amount of money.

Your main problem is to build a quiet workstation for under the desk with 2 processors, where each on will burn up to 360 Watt.

I recently upgraded my Epycs from 7302 to 7532. Now they burn 2x 200 Watt maximum instead of 2x 150 Watt. I only use more than 40 cores, when I'm not in the room :-)

[flotus1]

These old single-slot Quadro cards aren't exactly what you want for quiet operation.
A modern "gaming" card with a decent cooler will be quieter, even under full load. And it's not like the 7800XT will run at full blast all the time. So I don't understand the concerns about power consumption.
The fact that Star-CCM+ uses CUDA for its GPU acceleration does not really make old Quadros viable either. They are too slow, and don't have enough memory to do any heavy lifting. Not to mention that GPU acceleration with CCM+ requires an entirely different -and more expensive- license.
And quite frankly, nobody should be paying 300€ for used Quadro P2000 in 2023.

Whether the GUI stuff of CCM+ works better with Nvidia cards+drivers in general might be up for debate. I don't have any first-hand experience here, I only ever used CCM+ on systems with Nvidia graphics cards.

[JBeilke]

DeltaComputer is back with a water-cooled 2 processor workstation:

https://www.deltacomputer.com/d22z-wk-zg-2xgpu.html

This would be the easiest way to get such a machine. The pictures there show, how the cooling is done.

If you click on the GPU section, you find some more (and recent) sub 100 Watt cards :-)

When you have a server-room it's likely easier to use a server and put not such a beast under your desk, especially when the temperature in your office is already a problem in the summer.

[MangoNrFive]

Hi Joern,

thanks a lot for the link!

But I think we´ll go for the custom-build I have now spend a lot of time researching PC-Hardware and refining the setup. I´m very confident now that I can build a nice workstation myself for a significantly better price. I have now shifted to a custom water-loop as well. With the research I´ve done that doesn´t seem that scary anymore either.

For anyone that is interested I appended a list of all the parts I´m going to order.
The two CPU-air coolers are used as a backup for the custom water-loop (e.g. if some parts are missing or faulty) and so I can test all the electric components before installing the water-cooling loop and without frying the CPUs.

[JBeilke]

Hi Philipp,

Good luck with your machine. I was a big fan of water cooling with my i7-3960x back in 2012. It worked very well for a few years until it started leaking. I then replaced it with several new models, but the noise from the pump was always annoying.

The Noctuas were a huge step forward, but unfortunately the cooling performance is no longer sufficient for today's Genoa processors, especially with acceptable noise levels.

In any case, such a computer is a good study object for aeroacoustics and thermodynamics.

[flotus1]

You might want to spend 40€ extra for a leak tester like this https://shop.alphacool.com/shop/modd...ool-incl.-pump
Doesn't have to be this one exactly, many brands sell similar items.
When doing your first custom loop on hardware worth more than 10000€, that extra peace of mind is definitely worth it

[sharonyue]

I dont come to hardware quite often. I am searching some LES filter algorithm then I happened to run into this. I am not sure if it is proper for me to say the following. But I cannot help to say it. flotus is quite helpful and kind to help the others on this. But the CPU's price is not good. I am sorry flotus I have to say it. I have been doing clusters/workstations for 3 years. Compared with the others its not a long career period. But I have MUCH MUCH better price, almost everything you listed I have better price. CPU? we have brandnew CPUs. RAM? brandnew. MB? brandnew. Everything brand new with much lower price. Even h11dsi we have brandnew MB and we got them from Taiwan. I am not trying to persuade you to buy it from me. But I know the price. I do not provide products for overseas customers because I dont know how to do it. After I see the price you listed, I am really thinking its the right time for me to expand my overseas business. My company focuses on cfd consulting, like programming in OpenFOAM. Cluster/workstation is not our main business. We do CFD research. But, now, I am really thinking to do overseas workstation business. To be honest, if you are in China, you budget can buy two workstations like this. You may also notice actually they import the workstation components from China.

One thing to be EXTREMELY careful. mz73 is slightly slightly bigger than h12dsi, some case it has screw in it and it breaks the motherboard. Please see the image. If it happens, either only one CPU shows up, or its totally damaged.

PS. I double checked the price you listed. Look at this: a thermal paste 24 euro???

[flotus1]

Sure, we could start nit-picking price vs. quality for most of the components here. I stopped doing that a while ago, since I don't see the point for 10000€ workstations.
It only drags out these hardware recommendation threads unnecessarily. And it always runs the risk of alienating people who already put a lot of effort into researching the components they need, and have readily available. 24€ for thermal paste is the perfect example here. It may not be the cheapest you can go, but who cares about that in a 10000€ workstation. Not to mention that it is a 10g tube of pretty good stuff, that will last for many more applications in the future. I have the exact same tube in my toolbox, and it has lasted for many different builds and re-pastes.
I try to only nitpick the other way round: e.g. when a questionable quality power supply is paired with high-end components.

You can find better price/performance with used components? Yeah, go figure
Used is not always an option due to outside influences. And if you need / can pay for the absolute performance that the latest Genoa CPUs offer, there is just no point in buying used.
My handy little questionnaire includes a question about used vs new vs OEM

Regional prices are a thing. We might not have the same prices as in your region. Assuming we are even talking about the same level of components, not ES/QS CPUs.
And let's not forget the additional logistics of importing parts from China into Europe. For starters, we need to add taxes to the price tag. 19% in Germany for example. Shop prices in Germany already include taxes, unless you are looking at B2B. And in case something breaks, the warranty process is on a whole other level of tedious, when I bought the part on the other side of the globe. If it is possible at all.
And from my personal experience, an item being described as "new", can mean different things to different folks.

This is a bit like the debate about pre-built/OEM PCs vs building it yourself. We all know which is better. But if building it yourself just isn't an option, there is no point in arguing.

So yeah, you are right, there would be various ways to increase price/performance here.
Whether the methods are worth it, everyone can decide for themselves. And I stand by my original verdict: the EPYC 9554 at around 4000€ is the best option, with the boundary conditions we have here.

[CFDfan]

Originally Posted by sharonyue

I dont come to hardware quite often. I am searching some LES filter algorithm then I happened to run into this. I am not sure if it is proper for me to say the following. But I cannot help to say it. flotus is quite helpful and kind to help the others on this. But the CPU's price is not good. I am sorry flotus I have to say it. I have been doing clusters/workstations for 3 years. Compared with the others its not a long career period. But I have MUCH MUCH better price, almost everything you listed I have better price. CPU? we have brandnew CPUs. RAM? brandnew. MB? brandnew. Everything brand new with much lower price. Even h11dsi we have brandnew MB and we got them from Taiwan. I am not trying to persuade you to buy it from me. But I know the price. I do not provide products for overseas customers because I dont know how to do it. After I see the price you listed, I am really thinking its the right time for me to expand my overseas business. My company focuses on cfd consulting, like programming in OpenFOAM. Cluster/workstation is not our main business. We do CFD research. But, now, I am really thinking to do overseas workstation business. To be honest, if you are in China, you budget can buy two workstations like this. You may also notice actually they import the workstation components from China.

One thing to be EXTREMELY careful. mz73 is slightly slightly bigger than h12dsi, some case it has screw in it and it breaks the motherboard. Please see the image. If it happens, either only one CPU shows up, or its totally damaged.

PS. I double checked the price you listed. Look at this: a thermal paste 24 euro???

Thank you for your offer and your comments - yes you are right, the prices in China are lower, but the quality and the customer support sucks. I've been an Aliexpress user for more than 10years and bought hundreds of items from there and my firm conclusion is - NO MISSION CRITICAL COMPONENTS FROM CHINA. Buying something that is more than $100 is a waste since it is just like a playing a lottery - might work by the spec, but if not you better just throw it into the garbage bin. The customer support is a joke - once you pay they loose any interest in you and if something is faulty, or out of spec, your chances to get a refund, or a replacement are about 5% (and that is only if you pay the return postage). If you are very lucky you could get a good deal with used (and working by the spec) components, but it would be an exception. High end, brand new, production grade CPUs you cannot get cheaper from China simply because China buys them from the US. The good deals are only with older generation, used parts. Sorry for saying that, you company might be different, but that's the opinion of somebody who dealt with China for a very long time.
P.S. About the thermal paste. Some of them are very expensive because they have micronized diamond particles improving greatly the thermal conductivity. The thermal paste deposited on the air coolers coming from China has always been of low quality and I just wipe it out and use the proper paste.

[jarjarbinks]

Originally Posted by MangoNrFive

Hi,
[*]I couldn´t find yet where to buy the XE360-SP5 AiO CPU water-cooler, any ideas? I would like to avoid a custom loop if possible, and this is the only AiO for this socket I could find[/LIST]

Hey, I'm planning to go for quite a similar build but with a single 9374f for now with a possible second cpu addition later down the line. Considering all these are super expensive, is there a risk of aio or custom loop leaking and destroying your setup? Why didn't you go with rack type chassis? I'm trying to decide between the two and maybe you can help

[flotus1]

Chassis type depends a bit on how you want to use the computer.
If it's supposed to sit in a server rack, of course you pick a rack-mountable chassis.
"Regular" PC cases are better suited for workstations that reside in an office. Mostly because you can fit larger (=quieter) coolers and fans. And the fans don't need to be of the high pressure variety.
And if you really want to mount two 360mm AIO water coolers, large desktop cases usually are a better fit.

Failure modes for water cooling tend to be more severe than with air cooling. There is just no way around that. If you want the benefits of water cooling, this is the tradeoff.
You can plan ahead to minimize the risks. A few rules of thumb for maintenance-free water cooling:
1) Don't mix materials. Especially not copper with aluminium.
2) Use Norprene tubes. Also called ZMT by EK. 16/10 is my preferred size, pretty much impossible to kink.
3) Keep it simple with the cooling fluid: Aqua computer Double Protect Ultra. No fancy additives for color or UV effects.
4) You can use two pumps for added redundancy.
5) The more expensive your hardware, to more thoroughly you should test for leaks during assembly
6) Flush your radiators before use! There can always be some residue or or even dirt from the manufacturing process.

If you get an AIO instead, you are at the mercy of the manufacturer.
Water cooling gets part of its bad reputation from the early days, when there were barely any dedicated parts for it. It is becoming more mainstream by the day. Even in proper servers, water cooling is starting to see wider adoption.

[JBeilke]

If you have an extra server room, there is really no reason to put a 2 processor Genoa workstation under your desk. Just buy a server and you'll be rid of all the noise and cooling problems.

[jarjarbinks]

Originally Posted by flotus1 View Post
Chassis type depends a bit on how you want to use the computer.
If it's supposed to sit in a server rack, of course you pick a rack-mountable chassis.
"Regular" PC cases are better suited for workstations that reside in an office. Mostly because you can fit larger (=quieter) coolers and fans. And the fans don't need to be of the high pressure variety.
And if you really want to mount two 360mm AIO water coolers, large desktop cases usually are a better fit.

The problem is I don't have any experience assembling liquid cooling rigs and going the aio route seems a lot safer as long as I can find a reputed one (there are hardly any in the market for sp5 sockets right now). Also, where can I buy new epyc cpu? Everywhere I look only whole server builds are being sold. Any reputed seller of server grade cpu that you people are aware of? I'm in canada btw.

I can go the server route, the noise is also under 60dB from a 2u chassis, according to lenovo's website for a higher spec build. Which is bearable and not bad at all. The problem there is sellers don't seem to have 8gb ram sticks (looking for 12x8gb) anything higher would be a waste of money

[JBeilke]

I bought my new used Epycs from here:


https://www.ebay.de/itm/115852114752...3ABFBM3seoxaVi


Maybe they also have what you want.