CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > ANSYS > FLUENT

32 CPUs Workstation V.S. Cluster for Fluent

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree6Likes

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   February 17, 2014, 14:40
Default
  #21
Super Moderator
 
flotus1's Avatar
 
Alex
Join Date: Jun 2012
Location: Germany
Posts: 3,395
Rep Power: 46
flotus1 has a spectacular aura aboutflotus1 has a spectacular aura about
Just dont.
Water doesnt go together with electronics, no matter what the retailers of water cooling systems say.
They are expensive compared to air cooling, need maintenance and are just as good in terms of noise and cooling performance as a good air cooling.
Unless you are planning to overclock the CPUs to a point where they will reach lifetime cycle after 6 months, you dont need it.
Anna Tian likes this.
flotus1 is offline   Reply With Quote

Old   February 17, 2014, 14:51
Default
  #22
Senior Member
 
ghost82's Avatar
 
Rick
Join Date: Oct 2010
Posts: 1,016
Rep Power: 26
ghost82 will become famous soon enough
I chosed liquid coolers because I have to dissipate near 150W x2 at full load.
Air coolers exist, but they are too big for my motherboard and the dissipators interfered with the ram slots.
I chosed liquid coolers from corsair, as they are small and give good performance: at 100% load I have a maximum temperature of 54 ºC.
These coolers are closed systems, composed by a block to install above the cpu, which comprises the pump, 2 flexible tubes, connected to a radiator and 2 fans attached to the radiator to have a push-pull configuration.
Liquid circulates in this system.
Each cooler is connected to the motherboard or to an external usb port to control the fans speed and to monitor temperatures.
The 2 fans on the radiator are also connected by a y cable to the pump block.
The last connection is from the pump block to the motherboard through the fan connector.
The cost is near the same a good air cooler.
Corsair warranty is 5 years for these units.
ghost82 is offline   Reply With Quote

Old   February 18, 2014, 03:41
Default
  #23
Senior Member
 
Onno
Join Date: Jan 2012
Location: Germany
Posts: 120
Rep Power: 15
Kaskade is on a distinguished road
@ghost82: Are you using one machine or several?

Regarding liquid cooling: You can use certain non-conductive oils or demineralized water. But I really don't think liquid cooling is a good idea. In a well ventilated case good air coolers are more than sufficient. If a fan breaks, the CPU clocks down to keep from overheating. A fan can be replaced quickly, but if something goes wrong with your liquid cooling, you need a mop.

Get some quotes for professional systems. This starts to read like some Overclockers Forum.
Kaskade is offline   Reply With Quote

Old   February 18, 2014, 04:12
Default
  #24
Senior Member
 
ghost82's Avatar
 
Rick
Join Date: Oct 2010
Posts: 1,016
Rep Power: 26
ghost82 will become famous soon enough
Only one machine.
Unfortunately the rule is the higher the heat to be dissipated by the cpu the bigger the dissipator: and if the dissipator is "big" you need more fans attached to it to cool it down.
I looked at several professional/non professional air dissipators but all interefere with the slots of the ram (ASUS z9pe-d8 ws populated with 8 sticks of ram).
There are some air coolers which not interfere with the slots, but able to dissipate near 130 W (lower than my 150 W), and they have the fan mounted above the cpu, with air flux perpendicular to it; not a good solution at all.
You are right about the maintenance: a fan could be replaced in a few minutes, and liquid coolers have intrinsic risks..but scheduled inspections to the workstation can really low down the risks.
I'm happy with my coolers, I have very low temperature at full load, but if there were air coolers for my case I would have chosen them, to set the risks to zero.
ghost82 is offline   Reply With Quote

Old   February 18, 2014, 07:20
Question
  #25
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Have anyone computed the power cost of the cluster? Is the electricity cost a significant part of the overall cost of hardware? Of course this depends on the electricity price. But how many watt? I'd like to just have a basic feeling on that.
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   February 18, 2014, 07:56
Default
  #26
Senior Member
 
ghost82's Avatar
 
Rick
Join Date: Oct 2010
Posts: 1,016
Rep Power: 26
ghost82 will become famous soon enough
it depends on the chosen hardware, but I think it will absorb about 4-5 kW at full load.
ghost82 is offline   Reply With Quote

Old   February 18, 2014, 08:37
Question
  #27
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by ghost82 View Post
it depends on the chosen hardware, but I think it will absorb about 4-5 kW at full load.
Is it mainly driven by the number of CPUs? Or cooling device?
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   February 18, 2014, 08:56
Default
  #28
Senior Member
 
Onno
Join Date: Jan 2012
Location: Germany
Posts: 120
Rep Power: 15
Kaskade is on a distinguished road
A very rough estimate would proably be number_of_cpus*TDP*1,25. If you use the same cpus as ghost82, that would mean 8cpu*150w/cpu*1,25=3000W.

Most cpus use less power than their TDP, due to the way Intel sorts them.

A typical cpu fan uses less than 5 W.
Anna Tian likes this.
Kaskade is offline   Reply With Quote

Old   February 18, 2014, 10:32
Default
  #29
Senior Member
 
Onno
Join Date: Jan 2012
Location: Germany
Posts: 120
Rep Power: 15
Kaskade is on a distinguished road
Sorry, my brain was on lunch break. 8*8 is 64, not 128. So the power usage is around 6KW. Or 4 electric kettles.

I don't know where you live, but an AC unit might be in order. Which further increases the power usage.
Kaskade is offline   Reply With Quote

Old   February 18, 2014, 10:46
Default
  #30
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by Kaskade View Post
Sorry, my brain was on lunch break. 8*8 is 64, not 128. So the power usage is around 6KW. Or 4 electric kettles.

I don't know where you live, but an AC unit might be in order. Which further increases the power usage.
Thanks for your information. I gain a basic feeling on that. It is quite some cost per year if we keep the machine running.
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   March 2, 2014, 18:08
Default
  #31
Member
 
Join Date: Aug 2013
Location: Bangkok
Posts: 32
Rep Power: 12
Eakarach is on a distinguished road
Dear Anna

Before you consider to buy your new workstation. Plese check your fluid flow problems that fluent hpc can be employed. Regarding to my knowledge, only a few case (radiation case) is confirm. I hope this information can help you.


Goodluck
Eakarach
Eakarach is offline   Reply With Quote

Old   March 3, 2014, 05:34
Question
  #32
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by Eakarach View Post
Dear Anna

Before you consider to buy your new workstation. Plese check your fluid flow problems that fluent hpc can be employed. Regarding to my knowledge, only a few case (radiation case) is confirm. I hope this information can help you.


Goodluck
Eakarach
Thanks for your information. What do you mean? You mean the license problem?
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   March 3, 2014, 11:19
Default
  #33
Senior Member
 
Andrew Kokemoor
Join Date: Aug 2013
Posts: 122
Rep Power: 13
Kokemoor is on a distinguished road
I'd guess that Eakarach is referring to GPUs in Fluent, which have limited applicability in versions earlier than 15. In 15, the AMG solver can use the GPU(s), but I haven't seen much attesting to the real speed up.
Kokemoor is offline   Reply With Quote

Old   March 3, 2014, 18:19
Default
  #34
Member
 
Join Date: Aug 2013
Location: Bangkok
Posts: 32
Rep Power: 12
Eakarach is on a distinguished road
In order to speed up the computation, there is two ways including workstation with more cpus and GPU computer. For the second case, there is only a few successful case (radiation). The first one is another choice to accelerate the computation. However, if you search the topics about the acceleration of computation. The results revealed that an increasing in cpu cores can be reduced the computed time. However, the calculation time is not decreased linearly with incracing the cpu core. In order words, if you solved the problems with 2 cores workstation. The caculation time is 2 hrs. When you employ 4 cores workstation. The caculation time is not 1hr. It should be 1.5 hrs. Thus, you should to search more information about the acceleration topics.

Best regards,
Eakarach
Eakarach is offline   Reply With Quote

Old   March 4, 2014, 02:14
Default
  #35
Senior Member
 
Onno
Join Date: Jan 2012
Location: Germany
Posts: 120
Rep Power: 15
Kaskade is on a distinguished road
No one was talking about GPUs. (FVM doesn't really benefit and LBM still has a long way to go.)

@Eakarach: Of course they don't scale lineary, but it is not nearly as bad as you describe. And what else are you going to do if you need to solve a complex problem? Scale back to 100.000 cells or wait half a year? Although you are right if you are suggesting, she should do a cost/benefit analysis depending on her use cases. Maybe two fast workstations with a case each would be more effective, than a small cluster. Really depends on the number of simultaneous cases.
ghost82 likes this.
Kaskade is offline   Reply With Quote

Old   July 16, 2014, 09:12
Question
  #36
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by Kaskade View Post
@ghost82: Why would you build a cluster from workstation? I think some rack mounted machines would be far easier to handle.

@Anna Tian: The problem with large amounts of cores in a single machine is probably heat. The heat production of core/CPU rises linearly with the clock rate, additionaly the voltages need to be raised to keep the system stable, which increases the heat production exponentially. So if you want a lot of cores, you will end up with slower cores.

Have you considered asking Ansys for advice?
Hi Kaskade,

If i follow ghost82's advice and build cluster by connecting 8 workstation, I guess in this way the cooling will be quite good per CPU. Then I can choose CPUs with many strong cores, right?
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   July 16, 2014, 11:49
Default
  #37
Senior Member
 
Onno
Join Date: Jan 2012
Location: Germany
Posts: 120
Rep Power: 15
Kaskade is on a distinguished road
If the fast CPUs spend half the waiting for data from other machines, they are not a good investment. Depending on your budget and the number of parallel jobs, it might be good idea to buy cheaper CPUs and invest in an Infiniband Interconnect.

And I wouldn't buy workstations (invidual towers including graphics card) but servers/blades in a rack. Easier to maintain and cooler.
Kaskade is offline   Reply With Quote

Old   July 16, 2014, 12:50
Default
  #38
Senior Member
 
ghost82's Avatar
 
Rick
Join Date: Oct 2010
Posts: 1,016
Rep Power: 26
ghost82 will become famous soon enough
Just for reference, having a workstation doesn't mean you have to buy individual graphic cards: for example the asus z9pe d8 ws has a jumper to set onboard vga; you haven't any vga port output, the system will start without keyboard and by setting correctly that jumper, and you can control it by remote management.
I think that if you have "few" cores you can go for indivual towers: you can use each individually for "small" cases (<=16-14 cores, on newer xeon v2 cpus), but if your cases are "big" and you always need to run them on a lot of cores (>48 cores) then go for a rack mounted system.
ghost82 is offline   Reply With Quote

Old   July 16, 2014, 21:58
Default
  #39
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by Kaskade View Post
If the fast CPUs spend half the waiting for data from other machines, they are not a good investment. Depending on your budget and the number of parallel jobs, it might be good idea to buy cheaper CPUs and invest in an Infiniband Interconnect.

And I wouldn't buy workstations (invidual towers including graphics card) but servers/blades in a rack. Easier to maintain and cooler.
Yes. It is usually not a good investment to fast CPUs for CFD applications. But I just realized that the CPU Intel Xeon E5-1620 v2 with 4 3.7 GHz cores and max memory bandwidth 59.7 GB/s only sells 370 US dollars in U.S. and less than 300 US dollars in China. I think its price in Europe won't be too different. Even it is for 2.5 GHz, I still didn't see a obvious better choice than it. And it is 3.7 GHz. So I go for it. This is the reason I choose fast CPU. I will connect 8 or 16 E5-1620 v2 CPUs to have 32 cores. This is the best maximum memory bandwidth (GB/s) per dollar I found.

There is another advantage of using a fast CPU. If I use a slow CPU (e.g. 2.5 GHz), I will always need to go for 2.5 GHz when I want to expand the cluster. 2.5 GHz CPUs might be two slow in the future and a main factor which limit the CFD calculation speed. In the future, the price difference between 3.7 GHz CPU and 2.5 GHz won't be as large as right now.

Btw, I will anyway need to invest in Infiniband for CFD Cluster.
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   July 16, 2014, 22:27
Question
  #40
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by ghost82 View Post
Just for reference, having a workstation doesn't mean you have to buy individual graphic cards: for example the asus z9pe d8 ws has a jumper to set onboard vga; you haven't any vga port output, the system will start without keyboard and by setting correctly that jumper, and you can control it by remote management.
I think that if you have "few" cores you can go for indivual towers: you can use each individually for "small" cases (<=16-14 cores, on newer xeon v2 cpus), but if your cases are "big" and you always need to run them on a lot of cores (>48 cores) then go for a rack mounted system.
I'm around the middle of the range you provide. I will have around 32 cores.

I think there is another advantage of using the '8 connecting individual tower workstations cluster'. This is just my speculation. I don't have any practical experience on that. Correct me if it is not correct.

If I build a rack mounted cluster, i can't increase the box size to contain more motherboards easily. The space in the rack mounted box is fixed. But if I go for individual towers, one day if I find i need another CPU, I can just add it by adding another individual tower workstation. This is much easier, right?
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Dual cpu workstation VS 2 node cluster single cpu workstation Verdi Hardware 18 September 2, 2013 04:09
Cluster of 8 CPUs tu FLUENT 5 November 29, 2005 02:43
How many CPU's in your cluster Charles Crosby Main CFD Forum 18 March 26, 2001 11:07
star-hpc on workstation cluster Joe Stone Siemens 1 August 9, 2000 14:06
PC vs. Workstation Tim Franke Main CFD Forum 5 September 29, 1999 16:01


All times are GMT -4. The time now is 01:27.