[ByronW]

Hi,

I need to get a new (to me) workstation for a new project.

The baseline purpose is general engineering office work. Additional needs / wants are:
-pre/post processing CFD run on an HPC cluster.
-running smaller CFD cases locally (see below for details)
-usable for light office work and ideally pre/post while running a case
-upgrade path over the next few years if needed.

I'm a US based, self-employed engineer. Budget is $2-3k, although I could increase that if there was a compelling justification. I don't have the bandwidth to build from parts but used is fine or basic changes (i.e. storage / memory).

The CFD I am running is external aero, subsonic aircraft high lift configurations including prop/wing interaction.

I use NASA's FUN3D, which is a free license w/o core limits. I have access to HPC cloud computing, but it's relatively expensive $/cpu-hr due to FUN3D export control requirements, so there's incentive to run locally when possible.

So far for local simulations I've used an old 2010 12 core Intel Xeon Mac Pro with 48 GB memory, which has been fine for 2D and smaller quasi-2D (a wing section between symmetry planes) cases up to roughly 8 million nodes. Somewhere around there it runs out of memory. I've used a laptop for pre and post, which is starting to become limiting for visualization.

The cases I have run on the cluster so far are still quasi-2D, just larger (more complex geometry, finer mesh). These can finish overnight or in a working day (or faster) on a single node. The HPC node specs are 48 core, Dual Intel Xeon 8268 2.9 Ghz with 384 GB memory. I'm expecting to run a whole lot of these quasi 2D cases over the next year, along with a smaller number of larger cases i.e. a full wing or full aircraft.

I found a refurbished HP Z8 workstation for $2,200 (specs below) which seems like it fits my needs. The Intel 26xx v3 was a suggested lower cost setup on other threads, and the Z8 chassis appears to be upgradable w/ higher performance Intel CPUs in the future.

Chassis:HP Z8 G4 Workstation
Processor:2x Intel Xeon E5-2678 v3 (2.5 Ghz) (2 x 12 core)
Memory (RAM):512GB DDR4 (16x32GB)
Storage:1x 1TB SSD
Graphics Card:Nvidia QUADRO M4000
Operating System:Windows 10 Pro 64-Bit

I would fill all the 24 memory slots, add more storage, and dual boot to Linux.

Is this a reasonable setup for what I've described? Is it realistic to expect it could locally run cases overnight (say about 16 hrs max) that currently fit on one cluster node? Is there some much better value alternative (again, considering new/used buy, not build)?

Appreciate the help,

Byron

[trans(sonic)_pride]

The hardware is a bit old IMO.
The GPU is not very important for CFD usually, but still a maxwell card seems expensive to me at this price range. Are you comfortable with building your own pc using used cpu, gpu, ram and motherboard? Nowadays you can get a 3070 for 450-500$, maybe pair it up with 2x16 core epyc romes (or 1x24), enough RAM to fill the memory channels (and cover your needs), and that should be perfect for you.
Even using the same cpu's listed you could get a pc for less with a better gpu if you assembled it.

Edit: I'm sorry I didn't read the last part where you said you wouldn't awant to build it. Maybe you can find someone who will do it for you for like 50 bucks, or a friend who can help you. If not, you'll be paying a signifficant premium. The only good thing about that build is the RAM. Do you need 512 GB? Coming from a MAC with 48GB, even 128GB would be a significant improvement.

[ByronW]

Thanks.

Yeah I don't want to build myself. I don't mean this to come across as arrogant at all but the harsh reality of being self employed that my own time and money have a very explicit trade. I just can't see how sourcing used parts to assemble a workstation from parts (in this price range, for my own situation) will actually cost less. So that's why I don't want to go down that path.

However there are places around including some local shops that I've used for other things that offer customized used workstations (within the range of what they have). So that could be an intermediate path (or a freelancer as you mentioned if I could find someone). So it's really helpful to have any guidance on what's a better value for CFD hardware, and I greatly appreciate the help.

Thanks!

Byron

[wkernkamp]

Originally Posted by ByronW

Hi,

The cases I have run on the cluster so far are still quasi-2D, just larger (more complex geometry, finer mesh). These can finish overnight or in a working day (or faster) on a single node. The HPC node specs are 48 core, Dual Intel Xeon 8268 2.9 Ghz with 384 GB memory.

I found a refurbished HP Z8 workstation for $2,200 (specs below) which seems like it fits my needs. The Intel 26xx v3 was a suggested lower cost setup on other threads, and the Z8 chassis appears to be upgradable w/ higher performance Intel CPUs in the future.

Chassis:HP Z8 G4 Workstation
Processor:2x Intel Xeon E5-2678 v3 (2.5 Ghz) (2 x 12 core)
Memory (RAM):512GB DDR4 (16x32GB)
Storage:1x 1TB SSD
Graphics Card:Nvidia QUADRO M4000
Operating System:Windows 10 Pro 64-Bit

I would fill all the 24 memory slots, add more storage, and dual boot to Linux.

Byron

1. I would expect the HP Z840 (is that what this system is?) to take about twice as long as on a single cluster node. The prices on E5-2680 v4 have really come down, so that will be a cheap upgrade if you know how to replace cpu's and apply thermal paste.

2. The 512GB of memory is more than enough (and even exceeds the cluster node memory). Do not fill the additional slots, because it will reduce the bandwidth of each channel. (Check me on this, but in general three DIMMs per channel will run at a lower frequency than two.) The v4 cpu's can use DDR4-2400 memory. Performance is pretty much proportional to frequency, so if the 512 GB is already DDR4-2400 that is a plus. On some systems you can force the memory to 2400 and that normally works.

3. The cost of the system is a bit high compared with what you can find on Ebay. Possibly there is a warranty to make up the difference. With some guaranties from the refurbisher, this may be a good choice. Maybe they could deliver the system with DDR4-2400 and 2xE5-2683 v4 for 2x$75? That is a 16 core processor with 40 MB L3 cache. More cores than that won't speed up the CFD. You won't need 24 cores like the cluster.


4. Don't buy this system, if the memory is not able to run at 2400 MT/s. In that case it is better to get a Z840 on the cheap with v3 cpus and little memory. You can buy the memory and processors you want with the large savings.

[ByronW]

Thanks very much, I'll check that CPU as well. It's the HP Z8 G4 which is the next generation (gen 4) of the Z840.

Yes I'll have to see if a reseller will configure as I want, or I could probably do that myself too if the "as-is" price makes sense. I see a few of these Z8 G4s bare bones for about $800 on ebay.

Byron

Originally Posted by wkernkamp

1. I would expect the HP Z840 (is that what this system is?) to take about twice as long as on a single cluster node. The prices on E5-2680 v4 have really come down, so that will be a cheap upgrade if you know how to replace cpu's and apply thermal paste.

[wkernkamp]

The Z8 G4 has Intel scalable processors with socket FCLGA3647. That won't take the E5-26XX v3 and v4 processors. That makes me question the configuration. It probably is a Z840 after all. (Otherwise the E5-2678 V3 wont fit).



The scalable processors have six channels of memory. So you should fill all twenty four DIMM slots. Make sure that all your memory matches (same number of ranks (prefer 2R). The socket can take processors with memory speeds up to 2933 MHz. Keep that in mind when ordering memory. Note that the processors are not so cheap.

[trans(sonic)_pride]

Originally Posted by ByronW

Thanks very much, I'll check that CPU as well. It's the HP Z8 G4 which is the next generation (gen 4) of the Z840.

Yes I'll have to see if a reseller will configure as I want, or I could probably do that myself too if the "as-is" price makes sense. I see a few of these Z8 G4s bare bones for about $800 on ebay.

Byron

Hi Byron, Check this seller out maybe, (this listing would need maybe a 500$ GPU upgrade and that's it).

A first-gen Epyc build would easily get out of budget, but it could mean a 70% increase in raw performance according to this. About 3200$ for the build and 3h of your time probably, that would be the cost, I can send you a concrete quote if you want.

[ByronW]

Thanks, you're right I contacted the seller and they confirmed it was a mistake in the listing - it is actually a Z840 for $2200.

They will change it to a Z840, Intel E5-2690 v4 (2x14 core) with 2 per channel (16x16GB DDR4) for $1800. (the 2680 v4 would be slightly cheaper, but they don't have it available right now). To me that looks ballpark similar to the Z840 ebay listing Transonic posted (+video card upgrade).

Or, same seller a real Z8 G4 with 2x Intel 4108 (2x 8 core) and all memory channels filled (24x16 384 GB), otherwise equivalent would be ~$3,000. But, it gives more total memory and 1.4x memory bandwidth (107 vs 77 GB/s), and the newer Z8 is more upgradable in the future.

Any thoughts on these options? Really appreciate the help.

Byron

Originally Posted by wkernkamp

The Z8 G4 has Intel scalable processors with socket FCLGA3647. That won't take the E5-26XX v3 and v4 processors. That makes me question the configuration. It probably is a Z840 after all. (Otherwise the E5-2678 V3 wont fit).



The scalable processors have six channels of memory. So you should fill all twenty four DIMM slots. Make sure that all your memory matches (same number of ranks (prefer 2R). The socket can take processors with memory speeds up to 2933 MHz. Keep that in mind when ordering memory. Note that the processors are not so cheap.

[ByronW]

Sure, more options are better. Thanks for the help

Byron

Originally Posted by trans(sonic)_pride

Hi Byron,

A first-gen Epyc build would easily get out of budget, but it could mean a 70% increase in raw performance according to this. About 3200$ for the build and 3h of your time probably, that would be the cost, I can send you a concrete quote if you want.

[trans(sonic)_pride]

Originally Posted by ByronW

Sure, more options are better. Thanks for the help

Byron

Here you have it It didn't cost me much, like fifteen minutes so don't worry. I'm glad to help.

[wkernkamp]

Originally Posted by trans(sonic)_pride

Here you have it It didn't cost me much, like fifteen minutes so don't worry. I'm glad to help.

I like this system. (Initially, I thought you had just 4 RDIMMs, but later I saw that you have 4 kits with 4 RDIMMs each.)

[trans(sonic)_pride]

Originally Posted by wkernkamp

Transonic, your quote is not bad, but the workstation will only reach it's normal CFD performance level with 16 RDIMMs installed. Since performance us roughly proportional to memory channels filled, you would be at 25% rated performance. That would cost another $3000!

They are dimms in packs of 4. 4 packs of 4, thus 16 dimms

[wkernkamp]

Originally Posted by ByronW

Thanks, you're right I contacted the seller and they confirmed it was a mistake in the listing - it is actually a Z840 for $2200.

They will change it to a Z840, Intel E5-2690 v4 (2x14 core) with 2 per channel (16x16GB DDR4) for $1800. (the 2680 v4 would be slightly cheaper, but they don't have it available right now). To me that looks ballpark similar to the Z840 ebay listing Transonic posted (+video card upgrade).

Or, same seller a real Z8 G4 with 2x Intel 4108 (2x 8 core) and all memory channels filled (24x16 384 GB), otherwise equivalent would be ~$3,000. But, it gives more total memory and 1.4x memory bandwidth (107 vs 77 GB/s), and the newer Z8 is more upgradable in the future.

Any thoughts on these options? Really appreciate the help.

Byron

Byron,


1. The Z840 now looks better at $1800. What is the memory speed on the DDR4? You should insist on DDR4-2400.


2. The Z8 G4 with all that memory would be better if only the cpus had more cores. Basically, you will have 12 memory channels at potentially 2933 MT/s, and at least 2666 MT/s, but only 16 cores. If you want to go this route, maybe ask for at least 12 core cpus for a total of 24 cores. You could do with less memory to keep the cost the same. (As long as you keep all channels filled with two ranks)


3. To reach rated bandwith on a channel, you need at least two ranks. An RDIMM has either 1 or 2 ranks. So with 2Rx4 RDIMMs, you need only one RDIMM per channel. However with the 1Rxn RDIMMs you need at least two.



4. transonic is correct that the dual EPYC systems outperform the intel. I like his system.