[plucas]

I am working on a budget cluster for my company. We are on a super small budget. We use ANSYS Fluent mainly and I piddle in OpenFOAM, but nothing for the actual company. We have access to 32 cores currently with our HPC licensing though ANSYS. We mainly do external aerodynamics on vehicles. Average cell count is around 20-30million cells depending on meshing strategies. We currently are using an 8 core workstation which is pretty slow for results.

I plan to run this on CentOS using Rocks 7.0

Master Node:
Dell PowerEdge R710 (12core, 128GB ram)


Compute Nodes
4x Dell PowerEdge R410 (8core, 32GB ram)

I want to leave room for upgrading more compute nodes later. The R710 and R410 are very reasonable on eBay. I currently bought 2 R410's to work on trying to get this type of system up and working since servers/clusters are new to us.

Does this seem to make sense? Any recommendations on more ways to learn how to set up a cluster?

[flotus1]

Tough choice. Sure, R410 are cheap but...
Personally, I would not buy anything older than Xeon E5 v2 right now, mainly for three reasons:
1) Performance per core is much higher. This will come in handy since you are limited to 32 parallel Ansys licenses. And you need less nodes, which slightly compensates the higher price, makes management easier and reduces overhead cost for network hardware.
2) Efficiency and power consumption are much better. Depending on which part of the world you live in, electricity costs for running such a cluster can not be neglected. And you will have less issues with cooling your server room.
3) Age. Poweredge R410 are roughly 8 years old. At this point, hardware failures might become a frequent occurrence.

When you say "super small budget", can this be translated into $ or €?
If you think about it, what are your annual costs for Ansys licenses? Cheaping out on the hardware to use them seems counterintuitive.

By the way: "4x Dell PowerEdge R410 (8core, 32GB ram)"
Memory seems to be configured for normal server tasks, not for CFD. These CPUs have 3 memory channels, so RAM should come in lots of 6 identical DIMMs for optimal performance. i.e. 24GB or 48GB.

[plucas]

Originally Posted by flotus1

Tough choice. Sure, R410 are cheap but...
Personally, I would not buy anything older than Xeon E5 v2 right now, mainly for three reasons:
1) Performance per core is much higher. This will come in handy since you are limited to 32 parallel Ansys licenses. And you need less nodes, which slightly compensates the higher price, makes management easier and reduces overhead cost for network hardware.
2) Efficiency and power consumption are much better. Depending on which part of the world you live in, electricity costs for running such a cluster can not be neglected. And you will have less issues with cooling your server room.
3) Age. Poweredge R410 are roughly 8 years old. At this point, hardware failures might become a frequent occurrence.

When you say "super small budget", can this be translated into $ or €?
If you think about it, what are your annual costs for Ansys licenses? Cheaping out on the hardware to use them seems counterintuitive.

By the way: "4x Dell PowerEdge R410 (8core, 32GB ram)"
Memory seems to be configured for normal server tasks, not for CFD. These CPUs have 3 memory channels, so RAM should come in lots of 6 identical DIMMs for optimal performance. i.e. 24GB or 48GB.

1. I will look into some other xeons (e5 v2+) with more better performance. That makes sense since the limit is 32 cores.

2. USA. Electrical cost here isn't bad but good to know. Didn't even look into the electrical cost since didn't think a 32 core would be too bad.

3. Good point

I would like to stay under $5k if possible buying used off ebay. Our ANSYS cost is reasonable since we are in their start-up program. Thanks for the remarks on the RAM. Did not know that. I will keep looking. For our budget, is there other ways to go which would be more beneficial and allow room for growth?

[flotus1]

Well with 5000$ to spend, 32 cores required and since you don't seem to be averse building yourself there is one other option I can recommend: AMD Epyc.
Your budget should be just enough to buy a complete compute node with 2x Epyc 7281 and 16x8GB of RAM. Maybe two 7301 if you can get a few good deals.
Of course the same amount of money could buy you 4 nodes with 2x8 cores Xeon E5 v2 generation. Each option has its pros and cons, you will need to decide which is more important to you. Slightly better parallel performance (when running only 8 cores per node on the older Xeons) vs. new hardware in a single box without any issues concerning node interconnects or server management.

[plucas]

Originally Posted by flotus1

Well with 5000$ to spend, 32 cores required and since you don't seem to be averse building yourself there is one other option I can recommend: AMD Epyc.
Your budget should be just enough to buy a complete compute node with 2x Epyc 7281 and 16x8GB of RAM. Maybe two 7301 if you can get a few good deals.
Of course the same amount of money could buy you 4 nodes with 2x8 cores Xeon E5 v2 generation. Each option has its pros and cons, you will need to decide which is more important to you. Slightly better parallel performance (when running only 8 cores per node on the older Xeons) vs. new hardware in a single box without any issues concerning node interconnects or server management.

Forgot to thank you for your response. I have been reading other threads in here and the performance of the Epyc 7301 has me very intrigued and I will be leaning in that direction if I can stretch my budget. Also do you have any links where I can read up on hardware and how it relates to CFD? I have been reading Puget Systems blogs based on you linking to them in another thread. I just want to make sure I am reading the right info and you seem very knowledgeable on the subject.

[flotus1]

While the articles from Puget Systems are a great source for some information that is hard to find anywhere else, CFD is definitely not their focus.
Unfortunately, I have not come across any particular source that sheds light on hardware topics particularly for CFD. Everything I write here is just an accumulation of tiny bits and pieces of information I accumulated and combined over the years from various sources, Including my own testing.