32 CPUs Workstation V.S. Cluster for Fluent
 

Hi,

We are going to purchase hardware for a CFD department. Fluent and ICEM are used. We will have 32 CPUs (128 cores) to support our simulation. And 2 CFD people will use it. I'm wondering shall we buy a workstation (Large workstation shall contain 32 CPUs. And I guess there should be a way for 2 people to access the computational resource of one workstation at the same time. If not, please correct me.) or we'd better build a cluster? Any suggestions?

I don't understand what you mean by "32 CPUs (120 cores)". A 32 core workstation would be based on a board with 4 cpu sockets and 8-core cpus. Although it should be noted that only certain Xeons work on mainboards with 4 sockets.

Most straight forward setup would consist of 3 machines.
1 machine for Fluent (doesn't even need monitor, keyboard etc)
2 machines for Pre/Postprocessing
File-Transfer using SMB or SFTP

Possibly cheaper
1 big machine: run Fluent on fewer cores than available during daytime and use the computer for pre-/post-processing
1 smaller machine for engineer number 2

Are you planning on using Windows or Linux?

Edit: Apparently AMD offers a 16 core cpu (consisting on 4 quad cores), which is capable of running on an 4 socket board -> 64 cores on one mainboard.

Hi,

I'm planing on a workstation with 32 CPUs and each of them have 4 cores. So there are 128 cores in total. Sorry that 120 is a tempo. I have corrected that.

I'm planning on having at least the large workstation on Linux.

Regarding the two setup ways you mentioned, will they have any performance difference? I guess the first way you mentioned (3 machines one) could be more complex to build. I'm really wondering is this big workstation normal? And will the large size of the machine, large number of CPUs and cores cause any problem?

How are you going to fit 32 cpus into a single machine?

Ansys is restrictive when it comes to Linux distributions: http://www.ansys.com/staticassets/AN....0_11-2011.pdf
I doubt Fluent wouldn't work on a free distribution, but Ansys might not give you support if something doesn't function properly.

BTW: StarCCM+ is cheaper for that many cores and CD-Adapco allows the use of CentOS.

No that much big machine on the market? I once heard about workstation with 32 CPUs but I never checked it online. Intuitively I think it could be quite big and as heavy as me.

I don't understand what do you mean by 'free distribution' here. I'm reading the link you provide me hardly. Thanks for your information.

We don't have a problem of the fluent license. We can easily rent or borrow licenses from our supplier company at the neighbor.

Btw, what about having win XP on the large machine? I read somewhere in this forum that Fluent runs much faster on Win XP. Fluent 15 still supports win XP. If I have Windows on my big machine, how do we set up so that the CFD engineer using the small machine can run the simulation directly on the big machine while another engineer is using the big machine at the same time?

I only found mainboards/workstations with 4 cpu sockets. If you want to use more cpus, you need to connect multiple machines using something like infiniband or ethernet -> cluster.

Free as in "for free". RHEL and SLES require a fee: https://www.suse.com/products/server/how-to-buy/ (You can set the language at the top.)
They offer support and a long distribution life time.

XP support ends in April. And I doubt it is faster.

I will check the size on the market with the workstation stores. Maybe we will be running into building a cluster. Is there any tutorial on how to build a cluster? We're totally fresh on this.

On the plus side: a cluster is easier to upgrade.

Most tutorials for setting up a cluster are pretty ancient. But it should be easy if you only want run one Fluent session across multiple machines.

If you want to use a job scheduler and ressource manager it gets trickier. Possible Google Searches: Platform LSF / Torque maui / Hadoop

We are in the process of setting up a 32 core cluster ourselves. Our hardware is virtual, but the software setup should be similar.

Edit : nevermind, I was suggesting to go read the hardware forum, and I just saw you did :rolleyes:

I just checked the other threads on this forum. It seems that it will be much more expensive if you want to build a cluster with good data exchange speed between CPUs. Why don't you just choose to buy a 4 sockets, 128 cores workstation? In this way, the maintenance and management cost shall be also a little lower. Later on, if you want to upgrade it (like adding more CPUs), you can also change the workstation into a cluster.

Check out the SGI workstations I think they have the machine you need. Regards.
Gonzalo

Yes. I found hardware companies may sell it, including HP. But they are mainly for the server, not for scientific calculation, like CFD. I never saw anyone mentioned there is a this large workstation in this forum. So may I ask are they okay for CFD?

I'm also wondering what is the advantage of cluster over workstation in this case? And how much more expensive and complex would the cluster choose be for my case?

I'm quite fresh to the enterprise linux area. What I understand from the link you provide is as below.

1. Linux is open for one computer. But if it is connected to another computer (no matter it is linux or windows), we need to pay for the connection which is called SUSE Linux Enterprise Server.

2. Fluent also restrict file distribution in Linux.

Is this correct? Windows doesn't have this restriction? We already have Windows system.

Hi,
I never seen these type of worstations/servers managing more than 4 cpus.

To have an idea of the price, if you want to build a cluster based on 8 workstations connected to 1 infiniband switch, with 32gb ram 1600Mhz ecc, 2x Xeon e5-2687w (which are not the latest cpus on the market, but they're very good 8 real cores), 1000W psu gold, asus motherboard, liquid cooled, infiniband 40gb/s, based on USA costs, you are going to spend more or less about 50.000 USD (installation costs not included).

What will be the cost of the single server?

PS: from my experience I would go with windows

@ghost82: Why would you build a cluster from workstation? I think some rack mounted machines would be far easier to handle.

@Anna Tian: The problem with large amounts of cores in a single machine is probably heat. The heat production of core/CPU rises linearly with the clock rate, additionaly the voltages need to be raised to keep the system stable, which increases the heat production exponentially. So if you want a lot of cores, you will end up with slower cores.

Have you considered asking Ansys for advice?

Yes you're right, however I think the total cost will not be too much different; also 8 machines are not too many, they can be installed in asmall space, and can be easily controlled/maintained in the form of workstations.

For commercial applications the sheer computing power is not enough. You'll need a decent RAID system plus backup and an UPS (Uninterruptible Power Supply). And a rack will make handling the cables easier.

The next step should probably be getting quotes for different system.

You go with Windows only for the price reason? I read on this forum that linux is better on the memory usage.

Not for the price: I tried several linux distribution, not certified by ansys (Centos, ubuntu, kubuntu) and the Sles distribution; in all cases I had problems not with fluent itself, but with the license manager: sometimes starting 2 times and not able to connect, sometimes not starting; I found (at least for my system) that windows 7 64 bit is better since everithing was and is smooth.

Daniele

Regarding the liquid cooling, is liquid cooling important? Is it expensive? We can place it in another room so that we won't be disturbed by the noise from the machine if air cooling is used. Any other concerns? Will that give additional requirements like the quality of the water?

Just dont.
Water doesnt go together with electronics, no matter what the retailers of water cooling systems say.
They are expensive compared to air cooling, need maintenance and are just as good in terms of noise and cooling performance as a good air cooling.
Unless you are planning to overclock the CPUs to a point where they will reach lifetime cycle after 6 months, you dont need it.

I chosed liquid coolers because I have to dissipate near 150W x2 at full load.
Air coolers exist, but they are too big for my motherboard and the dissipators interfered with the ram slots.
I chosed liquid coolers from corsair, as they are small and give good performance: at 100% load I have a maximum temperature of 54 ºC.
These coolers are closed systems, composed by a block to install above the cpu, which comprises the pump, 2 flexible tubes, connected to a radiator and 2 fans attached to the radiator to have a push-pull configuration.
Liquid circulates in this system.
Each cooler is connected to the motherboard or to an external usb port to control the fans speed and to monitor temperatures.
The 2 fans on the radiator are also connected by a y cable to the pump block.
The last connection is from the pump block to the motherboard through the fan connector.
The cost is near the same a good air cooler.
Corsair warranty is 5 years for these units.

@ghost82: Are you using one machine or several?

Regarding liquid cooling: You can use certain non-conductive oils or demineralized water. But I really don't think liquid cooling is a good idea. In a well ventilated case good air coolers are more than sufficient. If a fan breaks, the CPU clocks down to keep from overheating. A fan can be replaced quickly, but if something goes wrong with your liquid cooling, you need a mop.

Get some quotes for professional systems. This starts to read like some Overclockers Forum.

Only one machine.
Unfortunately the rule is the higher the heat to be dissipated by the cpu the bigger the dissipator: and if the dissipator is "big" you need more fans attached to it to cool it down.
I looked at several professional/non professional air dissipators but all interefere with the slots of the ram (ASUS z9pe-d8 ws populated with 8 sticks of ram).
There are some air coolers which not interfere with the slots, but able to dissipate near 130 W (lower than my 150 W), and they have the fan mounted above the cpu, with air flux perpendicular to it; not a good solution at all.
You are right about the maintenance: a fan could be replaced in a few minutes, and liquid coolers have intrinsic risks..but scheduled inspections to the workstation can really low down the risks.
I'm happy with my coolers, I have very low temperature at full load, but if there were air coolers for my case I would have chosen them, to set the risks to zero.

Have anyone computed the power cost of the cluster? Is the electricity cost a significant part of the overall cost of hardware? Of course this depends on the electricity price. But how many watt? I'd like to just have a basic feeling on that.

it depends on the chosen hardware, but I think it will absorb about 4-5 kW at full load.

Is it mainly driven by the number of CPUs? Or cooling device?

A very rough estimate would proably be number_of_cpus*TDP*1,25. If you use the same cpus as ghost82, that would mean 8cpu*150w/cpu*1,25=3000W.

Most cpus use less power than their TDP, due to the way Intel sorts them.

A typical cpu fan uses less than 5 W.

Sorry, my brain was on lunch break. 8*8 is 64, not 128. So the power usage is around 6KW. Or 4 electric kettles.

I don't know where you live, but an AC unit might be in order. Which further increases the power usage.

Thanks for your information. I gain a basic feeling on that. It is quite some cost per year if we keep the machine running.

Dear Anna

Before you consider to buy your new workstation. Plese check your fluid flow problems that fluent hpc can be employed. Regarding to my knowledge, only a few case (radiation case) is confirm. I hope this information can help you.


Goodluck
Eakarach

Thanks for your information. What do you mean? You mean the license problem?

I'd guess that Eakarach is referring to GPUs in Fluent, which have limited applicability in versions earlier than 15. In 15, the AMG solver can use the GPU(s), but I haven't seen much attesting to the real speed up.

In order to speed up the computation, there is two ways including workstation with more cpus and GPU computer. For the second case, there is only a few successful case (radiation). The first one is another choice to accelerate the computation. However, if you search the topics about the acceleration of computation. The results revealed that an increasing in cpu cores can be reduced the computed time. However, the calculation time is not decreased linearly with incracing the cpu core. In order words, if you solved the problems with 2 cores workstation. The caculation time is 2 hrs. When you employ 4 cores workstation. The caculation time is not 1hr. It should be 1.5 hrs. Thus, you should to search more information about the acceleration topics.

Best regards,
Eakarach

No one was talking about GPUs. (FVM doesn't really benefit and LBM still has a long way to go.)

@Eakarach: Of course they don't scale lineary, but it is not nearly as bad as you describe. And what else are you going to do if you need to solve a complex problem? Scale back to 100.000 cells or wait half a year? Although you are right if you are suggesting, she should do a cost/benefit analysis depending on her use cases. Maybe two fast workstations with a case each would be more effective, than a small cluster. Really depends on the number of simultaneous cases.

Hi Kaskade,

If i follow ghost82's advice and build cluster by connecting 8 workstation, I guess in this way the cooling will be quite good per CPU. Then I can choose CPUs with many strong cores, right?

If the fast CPUs spend half the waiting for data from other machines, they are not a good investment. Depending on your budget and the number of parallel jobs, it might be good idea to buy cheaper CPUs and invest in an Infiniband Interconnect.

And I wouldn't buy workstations (invidual towers including graphics card) but servers/blades in a rack. Easier to maintain and cooler.

Just for reference, having a workstation doesn't mean you have to buy individual graphic cards: for example the asus z9pe d8 ws has a jumper to set onboard vga; you haven't any vga port output, the system will start without keyboard and by setting correctly that jumper, and you can control it by remote management.
I think that if you have "few" cores you can go for indivual towers: you can use each individually for "small" cases (<=16-14 cores, on newer xeon v2 cpus), but if your cases are "big" and you always need to run them on a lot of cores (>48 cores) then go for a rack mounted system.

Yes. It is usually not a good investment to fast CPUs for CFD applications. But I just realized that the CPU Intel Xeon E5-1620 v2 with 4 3.7 GHz cores and max memory bandwidth 59.7 GB/s only sells 370 US dollars in U.S. and less than 300 US dollars in China. I think its price in Europe won't be too different. Even it is for 2.5 GHz, I still didn't see a obvious better choice than it. And it is 3.7 GHz. So I go for it. This is the reason I choose fast CPU. I will connect 8 or 16 E5-1620 v2 CPUs to have 32 cores. This is the best maximum memory bandwidth (GB/s) per dollar I found.

There is another advantage of using a fast CPU. If I use a slow CPU (e.g. 2.5 GHz), I will always need to go for 2.5 GHz when I want to expand the cluster. 2.5 GHz CPUs might be two slow in the future and a main factor which limit the CFD calculation speed. In the future, the price difference between 3.7 GHz CPU and 2.5 GHz won't be as large as right now.

Btw, I will anyway need to invest in Infiniband for CFD Cluster.

I'm around the middle of the range you provide. I will have around 32 cores.

I think there is another advantage of using the '8 connecting individual tower workstations cluster'. This is just my speculation. I don't have any practical experience on that. Correct me if it is not correct.

If I build a rack mounted cluster, i can't increase the box size to contain more motherboards easily. The space in the rack mounted box is fixed. But if I go for individual towers, one day if I find i need another CPU, I can just add it by adding another individual tower workstation. This is much easier, right?