[Tensian]

Dear all,

As many people in this forum, I have doubts about a good set-up for a new workstation.

I have a budget of around 5000 euros. I work mainly with OpenFOAM and openLB /Palabos, so mostly of my projects are related to CFD (maybe eventually a structural project using ANSYS MEchanical).

I work with aerodynamics, multiphase flow, tubulent flows, heat transfer and fluid-structure interaction. More complex physics are not expected yet (like combustion for example).

I will have to deal with steady and unsteady computations, with meshes around 10-15 million elements.

So I am wondering which set up would be a good trade-off between cost and performance (I wouldn't like to wait for weeks for an unsteady computation). That's why I am here asking for help.

I have some set-ups here that maybe with a bit of help can be refined to get the best price/performance relationship.

__________________________________________________ _________________________
1.
Case: Black Phanteks Enthoo Pro Tempered Glass Side Panel Full Tower Computer Case
Fans: Noctua Redux - Low Noise High-Performance Fans - Rear, Front & Top
CPU: 2x AMD EPYC Rome 7302 3.0GHz (3.3GHz Turbo) 155W 128MB Cache (32 Cores / 64 Threads)
CPU Cooling: 2x Noctua NH-U12S TR4-SP3 Premium-grade 120mm CPU cooler
Motherboard: Supermicro H11DSI-B - Dual AMD EPYC Processors Motherboard
Memory: 64GB (16 x 4GB) DDR4 SDRAM 2.66GHz ECC Memory
Power: 850W - 80 PLUS GOLD Certified Full Modular Power Supply
Video Card: NVIDIA Quadro RTX 4000 8GB GDDR6 Workstation Video Card
Sound Card: Creative Sound Blaster AUDIGY FX 5.1 PCIe 24-bit 192KHz Sound Card
STORAGE
M.2 NVMe Drive: 250GB M.2 NVMe PCIe 3.0 x4 Solid State Drive - read/write 2100/1300 MBps
SSD: SAMSUNG 860 EVO 2.5" 1TB SATA III TLC Solid State Drive (SSD)
CD/DVD: LG 16x BlueRay DVD CD Blu-ray Burner,3D Play Back

__________________________________________________ ______________________________
2.
AMD EPYC ROME 7282 2.80GHz (3.2GHz Boost) 16 Core Processor
ISV certified AMD Radeon Pro WX 9100 Graphics
8 Channel 64GB ECC DDR4 2666MHz Memory
1TB NVMe M.2 SSD + 4TB HDD
Support for up to 6 displays (10-bit)
Built with noise dampening components
__________________________________________________ _____________________
3.
AMD EPYC ROME 7452 2.35GHz 32 Core Processor
NVIDIA Quadro P620 Graphics
128GB ECC DDR4 2666MHz Memory
O/S drive: 1TB Encrypted M.2 NVME SSD
4x 4TB Enterprise-class HDD
Tableau Universal Forensic Bridge
__________________________________________________ _________________________
4.

AMD EPYC 7351
16 cores per socket/32 sockets per node
256 Dual rank DDR4-2666
Storage OS: 256 GB NVMe
Storage data: 1 TB NVMe
Mellanox ConnectX-5 EDR 100 GBInfiniband, 16 PCIe
__________________________________________________ _____________
5.

1 CPU - Intel Xeon E5-16XX, 8 cores, 64 GB ram, GPU, 2x HDD
_________________________________________
6.
2 CPU - Intel Xeon E5-26XX, 24 CPU, 128 GB ram, GPU, 2x HDD
__________________________________________________ ______________
7.
DELL T7820 2-CPU (6 memory slots/CPU):
DUAL INTEL XEON SILVER (2.1GHz, 3.0GHz Turbo, 8 Cores, 11MB Cache)
NVIDIA video card P4000 (8GB, 4DP)
2 2TB-SSDs
RAM size: either 96GB
P4000 Graphic card (8GB GDDR-VRAM)
__________________________________________________ _______
8.
Logic Case SC-416A 4U Rackmount Chassis

Supermicro H11DSI-O, 2x 1GbE NICs

2x AMD EPYC™ 7282, S SP3, 7nm, Infinity / Zen 2, 16 Core, 32 Thread, 2.8GHz, 3.2GHz Turbo, 64MB, 120W, CPU, OEM

2x Noctua NH-U9 TR4-SP3 Compact Single Tower CPU Cooler, 6 Heatpipes, 2x92mm PWM Fans, for AMD Ryzen Threadripper/Epyc ONLY


128GB (16x 8GB) ECC Reg DDR4 2933MHz


2GB NVIDIA Quadro P620, 512 Cores, Supports 4 Displays


1000W Corsair RMx, Modular, Silent, 80PLUS Gold

1TB Samsung 970 EVO PLUS NVMe PCIe 3.0, 3500MB/s Read, 3300MB/s Write, 600K IOPS

4TB Seagate ST4000DM004 BarraCuda, 3.5" HDD, SATA III - 6Gb/s, 5400rpm, 256MB Cache, OEM NCQ

__________________________________________________ _______

8.

Base Components WS-E1320A-G4 - Base
Processor 1x AMD EPYC 7452 - 2.35GHz Base, 3.35GHz Turbo, 128MB Cache, 155W TDP, 32 Cores/64 Threads per CPU
Memory 128GB (8x16GB) 2666MHz Quad Channel ECC Registered DDR4 Memory
Graphics 1 AMD Radeon Pro WX 4100 - 4GB Workstation Graphics Card (1024 Stream Processors)
Primary Hard Drive*1TB Samsung 970 EVO Plus M.2 PCIe 3.0 x4 NVMe SSD - 3,500MBps Read, 3,300MBps Write, 600K/550K IOPs
Hard Drive 24TB 3.5" Desktop Hard Drive - 7,200rpm, 128MB Cache, SATA3
Hard Drive 3 Optical1 x DVD/RW +/- Optical Drive - Black (5.25" SATA)
Power Supply 1500W ATX 80-Plus Gold Certified Power Supply
__________________________________________________ __________________________________________________ ___________

These are some configurations I found on the web and also in the forum, but still not sure which one I should buy. DO ou guys know any shop in Europe that can provide a good set-up for CFD if I told them the requirements of my projects (of course a shop you trust...).

Which one you recommend? What would you modify and why? Do I have budget enough for my requierements?

Any help will be really appreciated.

Best,

T

[trampoCFD]

Hi Tensian,

We're updating the memory on all our workstations to DDR4 3.2GHz ECC RAM and the OS drive to Nvme SSD 500GB 3430/2600 MBps.

Our 64 Workstation now comes with 128 GB RAM (16x8GB) in order to maximise memory bandwidth for CFD.

https://trampocfd.com/collections/wo...es-workstation

We previously had 16x4GB 2.6GHz RAM on our 64 cores workstation in order to create a relatively cheap 64 cores workstation, that could easily be upgraded by just changing the RAM and processors. We made the change based on feedback and parts availability.

We can ship to Europe.

Please contact us through PM or contact form on our home page, and we’ll send you a questionnaire to make sure we address all your needs thoroughly.

Best regards,
Gui

[flotus1]

I have a nitpick here: marketing a workstation with two 16-core CPUs (Epyc 7302) as a "64 cores CFD workstation" is misleading at best. It's 32 cores, and 64 threads if you decide to leave SMT on. I assume this is just an honest mistake, because the other options with 96 cores and 128 cores are labelled correctly.

[Tensian]

First of all thanks for your replies.

This WS looks interesting. Do you guys consider that it can fit my numerical requierements? I don't know if this will work with my mesh sizes and types of analysis...and also considering I am not using STAR but OpenFOAM...

Any suggestion will be really appreciated.

Best,

T

[trampoCFD]

@ Alex
Oops, we originally had 64 cores on our smallest workstation, then we changed it to 32 Cores to match demand and create a workstation that could easily be upgraded with 4x mores cores by simply changing the procs.
But I forgot to update its product name...
Thank you for flagging this, we fixed it.

Also, all our workstations come with 3.2GHz RAM now.

@James
You need to find out how many GB ram per 1M cells your codes need for meshing and for the physics you need to run. Sorry I can't help you with that.
Check with your software vendors / users groups.

FEA can use A LOT of RAM. You might find that you need more RAM for FEA than CFD.

[CRI_CFD]

Hi!


Tensian, trampo and flotus, what do you think about this WS?




Fractal Define 7 XL [Dark Tempered Glass]

Supermicro H11DSI-O, 2x 1GbE NICs
2x AMD EPYC™ 7352, S SP3, 7nm, Infinity / Zen 2, 24 Core, 48 Thread, 2.3GHz, 3.2GHz Turbo, 128MB, 155W, CPU, OEM


2x Noctua NH-U9 TR4-SP3 Compact Single Tower CPU Cooler, 6 Heatpipes, 2x92mm PWM Fans, for AMD Ryzen Threadripper/Epyc ONLY


128GB (16x 8GB) ECC Reg DDR4 2933MHz


2GB NVIDIA Quadro P620, 512 Cores, Supports 4 Displays

1000W Corsair RMx, Modular, Silent, 80PLUS Gold


1TB Samsung 970 EVO PLUS NVMe PCIe 3.0, 3500MB/s Read, 3300MB/s Write, 600K IOPS


4TB Seagate ST4000DM004 BarraCuda, 3.5" HDD, SATA III - 6Gb/s, 5400rpm, 256MB Cache, OEM NCQ



This is a total budget of £6105.11 including VAT. Dont know if it is worth to go for 256 GB RAM, it is about £170 more...

[flotus1]

128Gb more RAM for only 170£ extra? That's a no-brainer if I ever saw one. Regardless if you think you might need it or not.
A Quadro P620 on the other hand is pretty low-end stuff. Depending on what you need the workstation for, this should be beefed up. Not necessarily with a different Quadro or Nvidia card, but that depends on your OS and applications.
That seagate drive might be SMR judging by the large cache. Better check with the seller, and swap it for a non-SMR hard drive.

[CRI_CFD]

Hi flotus,


Nice comments. I know, the difference in RAM is incredible


Regarding the use, I recommended this set up for the specifications Tensian mentioned (OpenFOAM...) and considering the use of graphic card only for postprocessing. Do you consider it won't be enough? What do you suggest?


I realize about the SMR, but considering a SSD to run simulations and the Seagate for storage, I considered it a good option considering also the budget mentioned...Maybe there are better choices...


Cheers,


CRI

[flotus1]

For post-processing with ParaView under Linux, nothing beats a Radeon RX570 8GB in price/performance. Which would be my recommendation in this case. 4 times the VRAM compared to a Quadro P620 make this the much more capable card, not to mention much higher tessellation performance. Potential Quadro driver optimizations don't count with these requirements.

I just checked it myself, the ST4000DM004 uses SMR. What's the big deal with SMR?
*rant mode on*
After being caught sneaking SMR into some of their drives, manufacturers like WD tried to come up with some explanations why SMR benefits the customers. Leaving aside the loss of trust thanks to this move, let's have a look at them:
1) Larger capacity. It's easy to see why this is bs. The largest drives still use CMR, SMR is mostly a thing with low- to medium capacity.
2) Lower cost. Maybe for the manufacturers. But as a customer, I haven't sen much change in the cost per TB over the last few years. I can only conclude that those cost savings went towards the margins of HDD manufacturers.
3) No impact for most consumer workloads. Overlooking the fact that SMR-HDDs marketed for usage in NAS have been demonstrated to be a terrible idea: there is some truth to that. Most consumers would not notice a difference in their desktp PC. But let's face it: a 6000$/€/£ CFD workstation does not represent the average consumer solution. I would not want to wait hours to days while moving the remains of a finished project from the NVMe SSD to the HDD.
Taking all of this together, and adding the overall poor management of the PR debacle that was sneaking SMR into well-established products: you won't see me recommending any SMR HDD in the foreseeable future. And for me personally, WD is blacklisted.
*rant mode off*
That, and HDDs with CMR still exist, without too much of a price increase. 4TB can be had for less than 100€
https://geizhals.eu/?cat=hde7s&sort=...MR)%7E958_4000

[Tensian]

Hi guys,


Many thanks for your comments.


I have a couple of questions more:


1. What processor will work better for my CFD purposes: AMD EPYC 7302 or AMS EPYC 7352?



2. I have checked the graphic card suggested for Paraview, but it seems it is not available. I have available:

• 4GB AMD Radeon Pro WX 3100, 4GB AMD Radeon Pro WX 4100,8GB AMD Radeon Pro WX 5100,8GB AMD Radeon Pro WX 7100, 2GB AMD Radeon Pro WX 2100
• 4GB EVGA GeForce GTX 1650 SUPER SC ULTRA, 6GB EVGA GeForce GTX 1660 SUPER BLACK GAMING, 6GB EVGA GeForce GTX 1660 Ti BLACK EDITION

Which one would you recommend for a similar budget of the QUADRO CRI mentioned? There are also other QUADRO, but much more expensive...


Thanks guys for seeding light into this.



Best,


T

[flotus1]

1. What processor will work better for my CFD purposes: AMD EPYC 7302 or AMS EPYC 7352?

The latter, because it has 50% more cores.

2. I have checked the graphic card suggested for Paraview, but it seems it is not available. I have available:

Using the search function in that shop, I find two 8GB RX 570 graphics cards in stock.
And many more RX 580 for just a bit more money.
https://www.scan.co.uk/search?q=rx+570+8gb
https://www.scan.co.uk/search?q=rx+580+8gb

[trampoCFD]

Hi
I agree with Flotus on the RAM and the Graphics Card.

Regarding HDD's, unless you get paid 5$ per hour and never run large or transient simulations, I don't understand why an engineer would bother with an HDD on a 10,000$ workstation when large capacity SSDs are available at reasonable prices. There is nothing more painful (and expensive for your company) than having to wait hours/days because you've run out of space on your main SSD. Saving simulations directly on a secondary SSD can also be a life saver on weekend runs that would crash because of lack of space on the main drive. These are some reasons why the workstations we sell always come with a "cheap" SSD as a second drive.

To some extent that thinking could apply to the graphics card: why would you want to take the risk to have issues with a CAD software down the track on a 10,000$ workstation?

Also, the AMD epyc Rome support 3.2GHZ RAM, which is roughly a 10% speed increase, going toward memory bandwidth, which is the definite bottleneck in CFD.

We've already optimised all of this on the CFD workstations we sell.
When you've finalised your hardware selection, let me know if you would like a quote from us. Support fellow CFD engineers ;-)

[Tensian]

Hi,


Thanks for your support.


Just to mention: I don't know where is written here that I wanted to buy a 10000 dollars WS (it will be great! btw). I am considering budgets about 5-6 K. So...of course better components mean better performance, but it also implies costs...


The HDD can be easily solved, but going for expensive QUADRO cards if (correct me if I am wrong) if they are not involved in OF compoutations and doesn't mean a difference for postprocessing doesn't look like the best option. That's the reason I was considering to save some money in graphic card and try to improve RAM for example.


flotus suggested a RADEON for a good post using Paraview. Do you guys consider that this card will work well with CAD software like Salome or similar?


Thanks for the support!

[flotus1]

I don't understand why an engineer would bother with an HDD on a 10,000$ workstation when large capacity SSDs are available at reasonable prices.

Neither do I, my workstation is equipped with a 7.68TB NVMe SSD, and I would not want to miss it. But it still has spinning hard drives. The reasons are simple: Flash storage with this kind of capacity is still expensive, and the benefit would be minimal. It is used for not-so-frequently accessed data.
Now that's my hardware, picked and paid by myself. The reality of the situation at my day job is kind of different. You would be surprised (or maybe not, since you sell workstations to engineers) how tight money is when it comes to storage capacity and speed. Convincing people to spend 10 times more per TB can be hard.
Anyway, I think we are getting side-tracked here. If his current projects fit onto the SSD he picked, I think it is an acceptable trade-off to get the rest of the storage in HDDs, in order to keep the budget under control. Installing a larger SSD later is an easy upgrade.

To some extent that thinking could apply to the graphics card: why would you want to take the risk to have issues with a CAD software down the track on a 10,000$ workstation?

My take on this is a bit different, particularly with Linux as the main OS. In my daily routine, the vast majority of Issues (real issues like black screen on boot, not minor inconveniences) were ultimately linked to Nvidias closed source graphics drivers. So in my experience, spending more money on a Quadro GPU it is actually detrimental to productivity. At least when mostly working in Linux with open source software. Radeon GPUs for consumers with the open source driver gave me the best user experience so far.
I have zero experience with AMDs professional GPU lineup. But given my past experience with Nvidias closed source driver (last time I checked, Radeon Pro drivers were also closed source), I don't feel too confident about them either.
Side-note here, before someone mentions "reliability" as a selling point for Quadro cards: I wonder which percentage of Quadro 4000 cards lived beyond their third birthday, if they were actually utilized. It is true: on average, failure rates on Quadro cards are lower than on their Geforce counterparts. But there is a bias in those statistics: many Quadro cards are bought "just in case", and rarely pushed to the limit. While the majority of Geforce cards regularly see full utilization while rendering games.

For Salome, all I could find was a seriously outdated FAQ. Since It's open source, I highly doubt that you could improve the user experience with a Quadro GPU.
If you should ever come across a software that particularly benefits from Quadro driver optimizations, it will most likely be on Windows. If that should ever happen, you can still buy a midrange Quadro card, and pass it through to a windows VM. But again, from the requirements you mentioned so far, this seems highly unlikely.

[trampoCFD]

For sure, there were some issues on NVIDIA Quadro drivers on Linux, like the 169.XX series drivers. There was even some issues in Windows. However, as far as I know, these were fixed in drivers update in the supported OS.

I operate in the commercial CAD and CFD world, were software/hardware are expected to work out of the box, or get fixed by support. So my experience is quite different from Alex.
CAD and Engineering software official support will be far better for a Quadro Card, than for a gamers card.

As far as I am concerned, there is enough hacking to do to get complex multiphysics simulations working properly. The last thing I feel like doing when modelling a 2way coupled CFD-DEM with DFBI simulation is to hack my way through graphics card drivers...

Choose your battles wisely :-)

[flotus1]

Originally Posted by trampoCFD

Choose your battles wisely :-)

My thoughts exactly when I made the switch to AMD consumer GPUs. It just works

[CRI_CFD]

I will put my 2 cents.

As far as I know, Tensian talked about OpenFOAM (eventually ANSYS Mechanical). I found this:

https://www.simutechgroup.com/suppor...rdware-support

So it seems AMD and QUADRO are supported for graphics and the mentioned QUADRO P620 is not providing any kind of GPU acceleration in your computations. A different thing is going for more expensive QUADRO cards. But it seems that AMD Radeon suggested here has not being tested:

https://www.ansys.com/-/media/ansys/...-192.pdf?la=en

I also found this:

https://technical.city/es/video/Rade...vs-Quadro-P620

So in general terms, performance seems to be improved with RADEON Rx 580 compared with the QUADRO P620. The only issue seems to be the higher electricity consumption.

Maybe this can help as well:

https://openbenchmarking.org/test/pts/paraview

Nice debate here!

[Simbelmynë]

Originally Posted by CRI_CFD

Maybe this can help as well:

https://openbenchmarking.org/test/pts/paraview

Nice debate here!

Those benchmarks seems a bit off. They do not state what drivers were used or which operating system.


Using the correct driver under Linux gives huge differences. If Noveau is used for Nvidia or if an old AMDGPU is used for a relatively new AMD card.


Anyway, my take is that Nvidia can be quite easy to work with if you do not plan to do any major kernel updates. On Ubuntu based systems it is more or less a one-button-install and it never causes any issues at least that is my experience from the last few years.


If you opt for CentOS then it is a bit more complicated, but when you have managed to install it then it works without much problems as long as you keep the original kernel (not talking about minor revisions / security updates - those are not a problem and most users will never install anything else).


The least success I've had with Nvidia drivers is with openSUSE.


All this being said - on our workstations I usually opt for AMD consumer cards just to be safe.