Desktop - CFD and Machine Learning Workstation

Sebs · October 27, 2020, 08:06

Hi,
I am a doctoral student in a lab, that in the past has done work mostly in numerical simulations (Matlab/Fortran/C/Python).
Recently, we have been shifting more and more in the direction of machine learning, so I build some computers with just regular hardware (Ryzen 3700x and some beefy GPUs for tensorflow), which work well. Now we also want to do more in the direction of CFD - namely StarCCM, Fluent and OpenFoam. We have the budget to build a workstation that will mainly be used for CFD, but might sometimes also be used to run some ML and or our Matlab models.
I am fairly well versed in hardware, but am not so familiar with the Epyc CPUs and what the best bang for the buck would be for the following requirements:

Budget:USD5-10k
Usage: Mainly OpenFoam and Fluent (maybe also StarCCM and ML and Matlab simulations)
Special considerations: The workstation needs to be in the office, since we don't have a server closet available. So it should not sound like a jet engine.

Any help would be appreciated.

flotus1 · October 28, 2020, 07:57

From AMDs Epyc lineup, I would recommend either the 24-core 7352 or the 32-core 7452. Depends on your budget, 5-10k is quite a large range.
As always, paired with 16x16GB DDR4-3200 reg ECC memory, or more depending on your requirements. Preferably dual-rank instead of single rank.
One possible advantage of 2nd gen Epyc for your non-CFD applications might be support for PCIe 4.0, given the right motherboard. Nvidias latest GPUs support that. You probably know best whether PCIe bandwidth matters for your applications.

On the other side of the fence, Intel is marketing its latest Xeon CPUs with a feature called DL boost: https://www.intel.com/content/www/us...ing-boost.html
It's basically another AVX-512 instruction targeted at deep learning algorithms. Whether you can make any use of it, I don't know. I have zero experience with machine learning.

Sebs · October 28, 2020, 08:39

Honestly, this will be a ML machine as a secondary priority. CPU power does not really matter much for that anyways - GPUs are orders of magnitude faster. So ideally I would find a board, that could accept at least two PCIe x16 graphics cards (and I would choose a power supply that's dimensioned big enough).

So let's say I have a budget of 6k to start, with the main priority CFD, what would be a recommended hardware combination?

Oh, and I saw you are from Germany - I am as well - just studying in the US right now - fortunately the hardware prices here are much cheaper than back in Europe

flotus1 · October 28, 2020, 09:47

Something along those lines: https://geizhals.eu/?cat=WL-1781998
Storage is intentionally left blank, since I can't know what you need. The only mass storage is a small SSD for the OS.

Sebs · November 3, 2020, 19:54

Ok,

how about this kind of configuration, as a high end system?

https://imgur.com/sNMep6m

Anything that looks out of place?

flotus1 · November 4, 2020, 04:30

Oh, look how cheap hardware is in the US compared to Europe

H11DSi, two problems:
Only revision 2 boards can take Epyc Rome CPUs. Not all shops keep up the distinction between revisions.
It's only PCIe 3.0

The memory you picked is single rank (1Rx4). If you have pay that much anyway, you should be picky about that detail.

Sebs · November 4, 2020, 10:08

Yeah, I was surprised, that the server components are equally expensive here. All pc components seem to be very expensive due to COVID atm. 'Asche auf mein Haupt'

Which RAM would you suggest, that is dual rank? I am having difficulty finding any on Newegg.
These are the only ones I can find Click

And regarding the board: there do not seem to be many options out there. The Gigabyte board does not seem to be easily available in the US?!

flotus1 · November 4, 2020, 12:06

Browsing Newegg from within the EU would require a VPN, so I can't really help you with that.
What you can do is search for the model numbers directly, and see what comes up.
https://geizhals.eu/?cat=ramddr3&sor..._RDIMM+mit+ECC
E.g. type M393A2K43DB3-CWE into Neweggs search bar.

The AMD Epyc motherboard situation for system builders has always been like that. You only have these two options - Supermicro H11DSi or Gigabyte MZ72-HB0. In fact, there was only one option up until a few weeks ago. I have read from other people that the MZ72-HB0 is still not available in the US.
That sums up AMD Epyc nicely. The CPUs are good, but the ecosystem around them is still a joke compared to Intel.

I don't know your time-frame for the purchase. If you can still wait a bit, maybe the Gigabyte board becomes available in your region. What is definitely going to happen within the next few weeks is the "launch" of Zen3-based Epyc Milan CPUs. Maybe this time there will be some new motherboards along with it. Fingers crossed...
If you can't wait, H11DSi with PCIe 3.0 is your only option right now.

Sebs · November 4, 2020, 12:50

I actually found out the Gigabyte board for sale here in the US from a different vendor. So it would be an option if it has advantages.

As for the wait. We could potentially wait a bit. Would buying a Milan instead of a Rome Zen CPU/CPUs make sense as well, or is Rome the best bang for the buck?

flotus1 · November 4, 2020, 14:12

The main advantage of that Gigabyte board is PCIe 4.0.
It also has better power delivery, but that won't make a difference with the CPUs and RAM you intend to use.

Hard to tell what price/performance of Epyc Milan might look like, there were no leaks about CPUs and prices yet. Judging by Zen3 for desktops, AMD might increase their prices quite substantially.

Sebs · November 4, 2020, 18:57

I think, that I can get my hands on the Gigabyte board. The fast PCIe will be important, if we ever decide to drop some GPUs in there for ML.

How loud will this machine be? Not louder than a regular PC, I assume? My prof asked me about it, since he only knows the fans of real servers - and we will have to keep this in our office.

flotus1 · November 5, 2020, 03:39

Without any powerful GPUs, this PC can be relatively quiet. It depends a bit on which options for fan control you have with that Gigabyte board. If it should be as bad as Supermicros fan control, there are always options to work around it.
Depending on which GPUs and how many of them you add, things will get a bit louder. But as far as air cooling is concerned, this case with 6x140mm Noctua fans is about as good as it gets.

Speaking of GPUs: the board has onboard graphics, but you should at least add some graphics card in the 100$-200$ range for anything 3D. I kind of forgot that in the configuration I posted.

Sebs · November 5, 2020, 16:39

Thanks,
yeah, we have multiple older workstation GPUs around. Those should be good enough for now. We also will probably use this machine headless most of the time.

Just a side question. Which OS is recommended for a headless machine for Fluent and/or Openfoam? Windows server or should I use a Unix based OS?

flotus1 · November 5, 2020, 16:54

I would highly recommend Linux. All software mentioned so far supports it, and OpenFOAM doesn't run natively on Windows.

Sebs · November 5, 2020, 16:55

That's what I was leaning towards to anyway. Thanks!
It should be pretty easy to setup a scheduler, so that multiple people can just send jobs to the server, right? So far we have only been using CFD software independently on our workstations.

flotus1 · November 5, 2020, 17:01

Up until this point, I successfully managed to avoid setting up a queuing system myself. So unfortunately, I can't share any insight.

Sebs · November 6, 2020, 17:21

Ok, I will figure the scheduler out.

I am having two systems together now with prices. I want to give my prof two options at different price points.
Anything that can be improved? Click me!

Sebs · November 6, 2020, 17:32

I just realized that the board is wrong in the configuration, so neglect that.

flotus1 · November 6, 2020, 18:04

Noctua NH-U12S will do the job, but the NH-U14S has more surface area and quieter fans.
Other than that, I see nothing wrong with the build.

Sebs · November 7, 2020, 11:25

I Actually reached out to some HPC suppliers and it looks like getting a pre built system would be actually cheaper than building it myself, since they get deals on the server hardware.
This was one suggestion, that I worked out with them. The only concern I have is noise level, but I was told "it is not very loud". Not sure if I believe that with a server PSU and fans.
CLick

As an alternative, I will probably try to work with them in sourcing exactly the parts I want (Non server chassis and fans and PSU) and even then come out ahead vs. building it myself. They can also source the Gigabyte board in the US, but delivery would be beginning of December.

October 27, 2020, 08:06	Desktop - CFD and Machine Learning Workstation	#1
Sebs New Member Join Date: Oct 2020 Posts: 13 Rep Power: 5	Hi, I am a doctoral student in a lab, that in the past has done work mostly in numerical simulations (Matlab/Fortran/C/Python). Recently, we have been shifting more and more in the direction of machine learning, so I build some computers with just regular hardware (Ryzen 3700x and some beefy GPUs for tensorflow), which work well. Now we also want to do more in the direction of CFD - namely StarCCM, Fluent and OpenFoam. We have the budget to build a workstation that will mainly be used for CFD, but might sometimes also be used to run some ML and or our Matlab models. I am fairly well versed in hardware, but am not so familiar with the Epyc CPUs and what the best bang for the buck would be for the following requirements: Budget:USD5-10k Usage: Mainly OpenFoam and Fluent (maybe also StarCCM and ML and Matlab simulations) Special considerations: The workstation needs to be in the office, since we don't have a server closet available. So it should not sound like a jet engine. Any help would be appreciated.

November 5, 2020, 16:54		#14
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,399 Rep Power: 46	I would highly recommend Linux. All software mentioned so far supports it, and OpenFOAM doesn't run natively on Windows. ships26 likes this.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
CFD workstation configuration calling for help 2	Freewill1	Hardware	6	July 8, 2020 21:17
Alienware Area 51 R5 as a CFD workstation	fusij	Hardware	1	June 13, 2019 10:15
Workstation for HPC CFD	tubois	Hardware	5	November 13, 2015 14:36
First CFD Machine: Some questions	chimaera	Hardware	3	June 15, 2012 16:39
PC vs. Workstation	Tim Franke	Main CFD Forum	5	September 29, 1999 15:01

October 28, 2020, 07:57		#2
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,399 Rep Power: 46	From AMDs Epyc lineup, I would recommend either the 24-core 7352 or the 32-core 7452. Depends on your budget, 5-10k is quite a large range. As always, paired with 16x16GB DDR4-3200 reg ECC memory, or more depending on your requirements. Preferably dual-rank instead of single rank. One possible advantage of 2nd gen Epyc for your non-CFD applications might be support for PCIe 4.0, given the right motherboard. Nvidias latest GPUs support that. You probably know best whether PCIe bandwidth matters for your applications. On the other side of the fence, Intel is marketing its latest Xeon CPUs with a feature called DL boost: https://www.intel.com/content/www/us...ing-boost.html It's basically another AVX-512 instruction targeted at deep learning algorithms. Whether you can make any use of it, I don't know. I have zero experience with machine learning.

October 28, 2020, 08:39		#3
Sebs New Member Join Date: Oct 2020 Posts: 13 Rep Power: 5	Honestly, this will be a ML machine as a secondary priority. CPU power does not really matter much for that anyways - GPUs are orders of magnitude faster. So ideally I would find a board, that could accept at least two PCIe x16 graphics cards (and I would choose a power supply that's dimensioned big enough). So let's say I have a budget of 6k to start, with the main priority CFD, what would be a recommended hardware combination? Oh, and I saw you are from Germany - I am as well - just studying in the US right now - fortunately the hardware prices here are much cheaper than back in Europe

October 28, 2020, 09:47		#4
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,399 Rep Power: 46	Something along those lines: https://geizhals.eu/?cat=WL-1781998 Storage is intentionally left blank, since I can't know what you need. The only mass storage is a small SSD for the OS.

November 3, 2020, 19:54		#5
Sebs New Member Join Date: Oct 2020 Posts: 13 Rep Power: 5	Ok, how about this kind of configuration, as a high end system? https://imgur.com/sNMep6m Anything that looks out of place?

November 4, 2020, 04:30		#6
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,399 Rep Power: 46	Oh, look how cheap hardware is in the US compared to Europe H11DSi, two problems: Only revision 2 boards can take Epyc Rome CPUs. Not all shops keep up the distinction between revisions. It's only PCIe 3.0 The memory you picked is single rank (1Rx4). If you have pay that much anyway, you should be picky about that detail.

November 4, 2020, 10:08		#7
Sebs New Member Join Date: Oct 2020 Posts: 13 Rep Power: 5	Yeah, I was surprised, that the server components are equally expensive here. All pc components seem to be very expensive due to COVID atm. 'Asche auf mein Haupt' Which RAM would you suggest, that is dual rank? I am having difficulty finding any on Newegg. These are the only ones I can find Click And regarding the board: there do not seem to be many options out there. The Gigabyte board does not seem to be easily available in the US?!

November 4, 2020, 12:06		#8
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,399 Rep Power: 46	Browsing Newegg from within the EU would require a VPN, so I can't really help you with that. What you can do is search for the model numbers directly, and see what comes up. https://geizhals.eu/?cat=ramddr3&sor..._RDIMM+mit+ECC E.g. type M393A2K43DB3-CWE into Neweggs search bar. The AMD Epyc motherboard situation for system builders has always been like that. You only have these two options - Supermicro H11DSi or Gigabyte MZ72-HB0. In fact, there was only one option up until a few weeks ago. I have read from other people that the MZ72-HB0 is still not available in the US. That sums up AMD Epyc nicely. The CPUs are good, but the ecosystem around them is still a joke compared to Intel. I don't know your time-frame for the purchase. If you can still wait a bit, maybe the Gigabyte board becomes available in your region. What is definitely going to happen within the next few weeks is the "launch" of Zen3-based Epyc Milan CPUs. Maybe this time there will be some new motherboards along with it. Fingers crossed... If you can't wait, H11DSi with PCIe 3.0 is your only option right now.

November 4, 2020, 12:50		#9
Sebs New Member Join Date: Oct 2020 Posts: 13 Rep Power: 5	I actually found out the Gigabyte board for sale here in the US from a different vendor. So it would be an option if it has advantages. As for the wait. We could potentially wait a bit. Would buying a Milan instead of a Rome Zen CPU/CPUs make sense as well, or is Rome the best bang for the buck?

November 4, 2020, 14:12		#10
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,399 Rep Power: 46	The main advantage of that Gigabyte board is PCIe 4.0. It also has better power delivery, but that won't make a difference with the CPUs and RAM you intend to use. Hard to tell what price/performance of Epyc Milan might look like, there were no leaks about CPUs and prices yet. Judging by Zen3 for desktops, AMD might increase their prices quite substantially.

November 4, 2020, 18:57		#11
Sebs New Member Join Date: Oct 2020 Posts: 13 Rep Power: 5	I think, that I can get my hands on the Gigabyte board. The fast PCIe will be important, if we ever decide to drop some GPUs in there for ML. How loud will this machine be? Not louder than a regular PC, I assume? My prof asked me about it, since he only knows the fans of real servers - and we will have to keep this in our office.

November 5, 2020, 03:39		#12
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,399 Rep Power: 46	Without any powerful GPUs, this PC can be relatively quiet. It depends a bit on which options for fan control you have with that Gigabyte board. If it should be as bad as Supermicros fan control, there are always options to work around it. Depending on which GPUs and how many of them you add, things will get a bit louder. But as far as air cooling is concerned, this case with 6x140mm Noctua fans is about as good as it gets. Speaking of GPUs: the board has onboard graphics, but you should at least add some graphics card in the 100$-200$ range for anything 3D. I kind of forgot that in the configuration I posted.

November 5, 2020, 16:39		#13
Sebs New Member Join Date: Oct 2020 Posts: 13 Rep Power: 5	Thanks, yeah, we have multiple older workstation GPUs around. Those should be good enough for now. We also will probably use this machine headless most of the time. Just a side question. Which OS is recommended for a headless machine for Fluent and/or Openfoam? Windows server or should I use a Unix based OS?

November 5, 2020, 16:55		#15
Sebs New Member Join Date: Oct 2020 Posts: 13 Rep Power: 5	That's what I was leaning towards to anyway. Thanks! It should be pretty easy to setup a scheduler, so that multiple people can just send jobs to the server, right? So far we have only been using CFD software independently on our workstations.

November 5, 2020, 17:01		#16
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,399 Rep Power: 46	Up until this point, I successfully managed to avoid setting up a queuing system myself. So unfortunately, I can't share any insight.

November 6, 2020, 17:21		#17
Sebs New Member Join Date: Oct 2020 Posts: 13 Rep Power: 5	Ok, I will figure the scheduler out. I am having two systems together now with prices. I want to give my prof two options at different price points. Anything that can be improved? Click me!

November 6, 2020, 17:32		#18
Sebs New Member Join Date: Oct 2020 Posts: 13 Rep Power: 5	I just realized that the board is wrong in the configuration, so neglect that.

November 6, 2020, 18:04		#19
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,399 Rep Power: 46	Noctua NH-U12S will do the job, but the NH-U14S has more surface area and quieter fans. Other than that, I see nothing wrong with the build.

November 7, 2020, 11:25		#20
Sebs New Member Join Date: Oct 2020 Posts: 13 Rep Power: 5	I Actually reached out to some HPC suppliers and it looks like getting a pre built system would be actually cheaper than building it myself, since they get deals on the server hardware. This was one suggestion, that I worked out with them. The only concern I have is noise level, but I was told "it is not very loud". Not sure if I believe that with a server PSU and fans. CLick As an alternative, I will probably try to work with them in sourcing exactly the parts I want (Non server chassis and fans and PSU) and even then come out ahead vs. building it myself. They can also source the Gigabyte board in the US, but delivery would be beginning of December.