|
[Sponsors] |
October 18, 2023, 04:22 |
EPYC Genoa CFD-Workstation ~13.000€
|
#1 |
Member
Philipp Wiedemer
Join Date: Dec 2016
Location: Munich, Germany
Posts: 42
Rep Power: 9 |
Hi,
we want to build a CFD-Workstation. Usage: OpenFOAM, StarCCM+ (core-unlimited) Models used: stationary, instationary, incompressible, moving-mesh, conjugate-heat-transfer Mesh-sizes: 5-20Mio up to 60Mio Operating-system: Linux (preferably Ubuntu) Location: Munich, Germany Budget: ~15.000€ This is what I came up with:
Total: 13030€ Some considerations / questions:
|
|
October 18, 2023, 09:56 |
|
#2 | ||||
Super Moderator
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,406
Rep Power: 47 |
That's an ambitious project for your first PC ever. Maybe you know someone with some experience building PCs, to assist you during assembly? But hats off to you, you got pretty far in the research stage already
Quote:
Quote:
Quote:
Quote:
The XE360-SP5 has been announced just this month. Maybe we just need to wait a bit for availability. Rumor has it that Genoa Threadripper will be released in Q4 2023. That should improve things on the AIO front. My pipe dream of a new dual-SP5 workstation includes a custom loop, so I can't really tell you more RAM: These "overclocked" DDR5 reg ECC kits were specifically developed for Intels Xeon-W Sapphire Rapids. I'm not saying it won't work on Epyc Genoa, but it is definitely a risk. And I am not even sure if you get the lower timings on any other platform. If you don't need the last 5% of performance from the better timings, maybe stick to the regular reg ECC memory modules. |
|||||
October 18, 2023, 16:45 |
|
#3 | |||
Member
Philipp Wiedemer
Join Date: Dec 2016
Location: Munich, Germany
Posts: 42
Rep Power: 9 |
Thank you for your reply Alex!
Quote:
Quote:
Quote:
Kingston Server Premier RDIMM 16GB, DDR5-4800, CL40-39-39, reg ECC, on-die ECC Micron RDIMM 16GB, DDR5-4800, CL40-39-39, reg ECC, on-die ECC Samsung RDIMM 16GB, DDR5-4800, CL40-40-40-77, reg ECC, on-die ECC |
||||
October 19, 2023, 02:04 |
|
#4 |
Super Moderator
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,406
Rep Power: 47 |
That's what i had in mind. You can also check the memory QVL for your motherboard, maybe one of those even made it to the list.
Don't forget to add some case fans. AIOs for the CPUs are nice, but there are still other components in a server-y PC that need plenty of airflow. And the case does not include any fans. |
|
October 20, 2023, 02:53 |
|
#5 |
Senior Member
Joern Beilke
Join Date: Mar 2009
Location: Dresden
Posts: 502
Rep Power: 20 |
I don't know, why you choose this graphics card. It burns a lot of energy, which makes cooling and noise difficult and does not help for GPU computing, since ccm+ requires a NVIDIA card for this purpose.
I would probably use something like a Quadro P2000 or P4000 in this machine. They are more than enough for typical CFD work and use much less energy. Since ccm+ works very good in client-server-mode, you can consider a second machine - maybe with a Ryzen 7800X3d - with a "better" graphics. This will make your noise problem much easier. https://www.serverschmiede.com/de/nv...dp-1024-cuda-1 https://www.serverschmiede.com/de/nv...te-gpu-4x-dp-1 |
|
October 20, 2023, 09:00 |
|
#6 |
Member
Philipp Wiedemer
Join Date: Dec 2016
Location: Munich, Germany
Posts: 42
Rep Power: 9 |
Hi Joern,
I heard in several places now, that a Quadro isn´t really necessary, as it´s main advantage is better driver support. But we aren´t pushing the boundaries of GPU-capabilities so the standard consumer drivers should be good. For a Quadro with similar VRAM and processing power we´d need to pay a lot more. Now why I chose exactly this graphics card is, that it has the ports that I want, good performance and plenty of VRAM. But there are a lot of other cards that would fit this criteria as well. I didn´t really have GPGPU in mind, as we need moving-mesh sometimes which isn´t supported in StarCCM+, so we rather invest our money in a powerful CPU. @Alex, indeed the Kingston Server Premier RAM was in the QVL, so I´ll be choosing that. I saw, that there is a category for SSDs on the QVL as well, but the SSDs on there are all reaaally slow, especially for writing (3 times slower). Do you think that the Samsung SSD 990 PRO 4TB, M.2 should be good, even though it´s not on the QVL? I think the other ones have a higher reliability, but that is not super important to us as we´ll just use it as a temporary storage for the simulation and have all the data saved on the synology. |
|
October 20, 2023, 14:27 |
|
#7 |
Senior Member
Joern Beilke
Join Date: Mar 2009
Location: Dresden
Posts: 502
Rep Power: 20 |
The reason I suggested the Quadro P2000 is, that it burns only 75 Watt and the card is more than enough for ccm+ pre and postprocessing. 300 Euros are not a big amount of money.
Your main problem is to build a quiet workstation for under the desk with 2 processors, where each on will burn up to 360 Watt. I recently upgraded my Epycs from 7302 to 7532. Now they burn 2x 200 Watt maximum instead of 2x 150 Watt. I only use more than 40 cores, when I'm not in the room :-) |
|
October 21, 2023, 11:56 |
|
#8 |
Super Moderator
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,406
Rep Power: 47 |
These old single-slot Quadro cards aren't exactly what you want for quiet operation.
A modern "gaming" card with a decent cooler will be quieter, even under full load. And it's not like the 7800XT will run at full blast all the time. So I don't understand the concerns about power consumption. The fact that Star-CCM+ uses CUDA for its GPU acceleration does not really make old Quadros viable either. They are too slow, and don't have enough memory to do any heavy lifting. Not to mention that GPU acceleration with CCM+ requires an entirely different -and more expensive- license. And quite frankly, nobody should be paying 300€ for used Quadro P2000 in 2023. Whether the GUI stuff of CCM+ works better with Nvidia cards+drivers in general might be up for debate. I don't have any first-hand experience here, I only ever used CCM+ on systems with Nvidia graphics cards. Last edited by flotus1; October 23, 2023 at 04:57. |
|
October 25, 2023, 02:44 |
|
#9 |
Senior Member
Joern Beilke
Join Date: Mar 2009
Location: Dresden
Posts: 502
Rep Power: 20 |
DeltaComputer is back with a water-cooled 2 processor workstation:
https://www.deltacomputer.com/d22z-wk-zg-2xgpu.html This would be the easiest way to get such a machine. The pictures there show, how the cooling is done. If you click on the GPU section, you find some more (and recent) sub 100 Watt cards :-) When you have a server-room it's likely easier to use a server and put not such a beast under your desk, especially when the temperature in your office is already a problem in the summer. |
|
October 25, 2023, 05:13 |
|
#10 |
Member
Philipp Wiedemer
Join Date: Dec 2016
Location: Munich, Germany
Posts: 42
Rep Power: 9 |
Hi Joern,
thanks a lot for the link! But I think we´ll go for the custom-build I have now spend a lot of time researching PC-Hardware and refining the setup. I´m very confident now that I can build a nice workstation myself for a significantly better price. I have now shifted to a custom water-loop as well. With the research I´ve done that doesn´t seem that scary anymore either. For anyone that is interested I appended a list of all the parts I´m going to order. The two CPU-air coolers are used as a backup for the custom water-loop (e.g. if some parts are missing or faulty) and so I can test all the electric components before installing the water-cooling loop and without frying the CPUs. |
|
October 25, 2023, 06:02 |
|
#11 |
Senior Member
Joern Beilke
Join Date: Mar 2009
Location: Dresden
Posts: 502
Rep Power: 20 |
Hi Philipp,
Good luck with your machine. I was a big fan of water cooling with my i7-3960x back in 2012. It worked very well for a few years until it started leaking. I then replaced it with several new models, but the noise from the pump was always annoying. The Noctuas were a huge step forward, but unfortunately the cooling performance is no longer sufficient for today's Genoa processors, especially with acceptable noise levels. In any case, such a computer is a good study object for aeroacoustics and thermodynamics. |
|
October 25, 2023, 07:02 |
|
#12 |
Super Moderator
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,406
Rep Power: 47 |
You might want to spend 40€ extra for a leak tester like this https://shop.alphacool.com/shop/modd...ool-incl.-pump
Doesn't have to be this one exactly, many brands sell similar items. When doing your first custom loop on hardware worth more than 10000€, that extra peace of mind is definitely worth it |
|
November 3, 2023, 04:37 |
|
#13 |
Senior Member
Dongyue Li
Join Date: Jun 2012
Location: Beijing, China
Posts: 841
Rep Power: 18 |
I dont come to hardware quite often. I am searching some LES filter algorithm then I happened to run into this. I am not sure if it is proper for me to say the following. But I cannot help to say it. flotus is quite helpful and kind to help the others on this. But the CPU's price is not good. I am sorry flotus I have to say it. I have been doing clusters/workstations for 3 years. Compared with the others its not a long career period. But I have MUCH MUCH better price, almost everything you listed I have better price. CPU? we have brandnew CPUs. RAM? brandnew. MB? brandnew. Everything brand new with much lower price. Even h11dsi we have brandnew MB and we got them from Taiwan. I am not trying to persuade you to buy it from me. But I know the price. I do not provide products for overseas customers because I dont know how to do it. After I see the price you listed, I am really thinking its the right time for me to expand my overseas business. My company focuses on cfd consulting, like programming in OpenFOAM. Cluster/workstation is not our main business. We do CFD research. But, now, I am really thinking to do overseas workstation business. To be honest, if you are in China, you budget can buy two workstations like this. You may also notice actually they import the workstation components from China.
One thing to be EXTREMELY careful. mz73 is slightly slightly bigger than h12dsi, some case it has screw in it and it breaks the motherboard. Please see the image. If it happens, either only one CPU shows up, or its totally damaged. PS. I double checked the price you listed. Look at this: a thermal paste 24 euro???
__________________
My OpenFOAM algorithm website: http://dyfluid.com By far the largest Chinese CFD-based forum: http://www.cfd-china.com/category/6/openfoam We provide lots of clusters to Chinese customers, and we are considering to do business overseas: http://dyfluid.com/DMCmodel.html |
|
November 3, 2023, 06:01 |
|
#14 |
Super Moderator
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,406
Rep Power: 47 |
Sure, we could start nit-picking price vs. quality for most of the components here. I stopped doing that a while ago, since I don't see the point for 10000€ workstations.
It only drags out these hardware recommendation threads unnecessarily. And it always runs the risk of alienating people who already put a lot of effort into researching the components they need, and have readily available. 24€ for thermal paste is the perfect example here. It may not be the cheapest you can go, but who cares about that in a 10000€ workstation. Not to mention that it is a 10g tube of pretty good stuff, that will last for many more applications in the future. I have the exact same tube in my toolbox, and it has lasted for many different builds and re-pastes. I try to only nitpick the other way round: e.g. when a questionable quality power supply is paired with high-end components. You can find better price/performance with used components? Yeah, go figure Used is not always an option due to outside influences. And if you need / can pay for the absolute performance that the latest Genoa CPUs offer, there is just no point in buying used. My handy little questionnaire includes a question about used vs new vs OEM Regional prices are a thing. We might not have the same prices as in your region. Assuming we are even talking about the same level of components, not ES/QS CPUs. And let's not forget the additional logistics of importing parts from China into Europe. For starters, we need to add taxes to the price tag. 19% in Germany for example. Shop prices in Germany already include taxes, unless you are looking at B2B. And in case something breaks, the warranty process is on a whole other level of tedious, when I bought the part on the other side of the globe. If it is possible at all. And from my personal experience, an item being described as "new", can mean different things to different folks. This is a bit like the debate about pre-built/OEM PCs vs building it yourself. We all know which is better. But if building it yourself just isn't an option, there is no point in arguing. So yeah, you are right, there would be various ways to increase price/performance here. Whether the methods are worth it, everyone can decide for themselves. And I stand by my original verdict: the EPYC 9554 at around 4000€ is the best option, with the boundary conditions we have here. |
|
November 3, 2023, 23:17 |
|
#15 | |
Senior Member
Join Date: Jun 2011
Posts: 202
Rep Power: 15 |
Quote:
P.S. About the thermal paste. Some of them are very expensive because they have micronized diamond particles improving greatly the thermal conductivity. The thermal paste deposited on the air coolers coming from China has always been of low quality and I just wipe it out and use the proper paste. |
||
November 14, 2023, 08:05 |
|
#16 |
New Member
Anirudh
Join Date: Feb 2021
Posts: 8
Rep Power: 5 |
Hey, I'm planning to go for quite a similar build but with a single 9374f for now with a possible second cpu addition later down the line. Considering all these are super expensive, is there a risk of aio or custom loop leaking and destroying your setup? Why didn't you go with rack type chassis? I'm trying to decide between the two and maybe you can help
|
|
November 14, 2023, 09:24 |
|
#17 |
Super Moderator
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,406
Rep Power: 47 |
Chassis type depends a bit on how you want to use the computer.
If it's supposed to sit in a server rack, of course you pick a rack-mountable chassis. "Regular" PC cases are better suited for workstations that reside in an office. Mostly because you can fit larger (=quieter) coolers and fans. And the fans don't need to be of the high pressure variety. And if you really want to mount two 360mm AIO water coolers, large desktop cases usually are a better fit. Failure modes for water cooling tend to be more severe than with air cooling. There is just no way around that. If you want the benefits of water cooling, this is the tradeoff. You can plan ahead to minimize the risks. A few rules of thumb for maintenance-free water cooling: 1) Don't mix materials. Especially not copper with aluminium. 2) Use Norprene tubes. Also called ZMT by EK. 16/10 is my preferred size, pretty much impossible to kink. 3) Keep it simple with the cooling fluid: Aqua computer Double Protect Ultra. No fancy additives for color or UV effects. 4) You can use two pumps for added redundancy. 5) The more expensive your hardware, to more thoroughly you should test for leaks during assembly 6) Flush your radiators before use! There can always be some residue or or even dirt from the manufacturing process. If you get an AIO instead, you are at the mercy of the manufacturer. Water cooling gets part of its bad reputation from the early days, when there were barely any dedicated parts for it. It is becoming more mainstream by the day. Even in proper servers, water cooling is starting to see wider adoption. |
|
November 14, 2023, 10:06 |
|
#18 |
Senior Member
Joern Beilke
Join Date: Mar 2009
Location: Dresden
Posts: 502
Rep Power: 20 |
If you have an extra server room, there is really no reason to put a 2 processor Genoa workstation under your desk. Just buy a server and you'll be rid of all the noise and cooling problems.
|
|
November 14, 2023, 17:12 |
|
#19 | |
New Member
Anirudh
Join Date: Feb 2021
Posts: 8
Rep Power: 5 |
Quote:
I can go the server route, the noise is also under 60dB from a 2u chassis, according to lenovo's website for a higher spec build. Which is bearable and not bad at all. The problem there is sellers don't seem to have 8gb ram sticks (looking for 12x8gb) anything higher would be a waste of money |
||
November 15, 2023, 00:54 |
|
#20 |
Senior Member
Joern Beilke
Join Date: Mar 2009
Location: Dresden
Posts: 502
Rep Power: 20 |
I bought my new used Epycs from here:
https://www.ebay.de/itm/115852114752...3ABFBM3seoxaVi Maybe they also have what you want. |
|
Tags |
amd epyc for cfd, water cooling, workstation desktop |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
AMD Epyc 9004 "Genoa" buyers guide for CFD | flotus1 | Hardware | 8 | January 16, 2023 05:23 |
CPU for Flow3d | mik_urb | Hardware | 4 | December 4, 2022 22:06 |
Workstation build for CFD | trans(sonic)_pride | Hardware | 6 | July 21, 2022 20:14 |
Workstation fot research position in CFD | Laerte | Hardware | 9 | June 21, 2020 15:17 |
EPYC based workstation for OpenFOAM simulations | roenby | Hardware | 21 | May 30, 2018 09:29 |