CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > ANSYS > FLUENT

32 CPUs Workstation V.S. Cluster for Fluent

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree6Likes

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   February 14, 2014, 05:32
Question 32 CPUs Workstation V.S. Cluster for Fluent
  #1
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Hi,

We are going to purchase hardware for a CFD department. Fluent and ICEM are used. We will have 32 CPUs (128 cores) to support our simulation. And 2 CFD people will use it. I'm wondering shall we buy a workstation (Large workstation shall contain 32 CPUs. And I guess there should be a way for 2 people to access the computational resource of one workstation at the same time. If not, please correct me.) or we'd better build a cluster? Any suggestions?
__________________
Best regards,
Meimei

Last edited by Anna Tian; February 14, 2014 at 10:01.
Anna Tian is offline   Reply With Quote

Old   February 14, 2014, 09:41
Default
  #2
Senior Member
 
Onno
Join Date: Jan 2012
Location: Germany
Posts: 120
Rep Power: 15
Kaskade is on a distinguished road
I don't understand what you mean by "32 CPUs (120 cores)". A 32 core workstation would be based on a board with 4 cpu sockets and 8-core cpus. Although it should be noted that only certain Xeons work on mainboards with 4 sockets.

Most straight forward setup would consist of 3 machines.
1 machine for Fluent (doesn't even need monitor, keyboard etc)
2 machines for Pre/Postprocessing
File-Transfer using SMB or SFTP

Possibly cheaper
1 big machine: run Fluent on fewer cores than available during daytime and use the computer for pre-/post-processing
1 smaller machine for engineer number 2

Are you planning on using Windows or Linux?

Edit: Apparently AMD offers a 16 core cpu (consisting on 4 quad cores), which is capable of running on an 4 socket board -> 64 cores on one mainboard.
Kaskade is offline   Reply With Quote

Old   February 14, 2014, 10:01
Default
  #3
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by Kaskade View Post
I don't understand what you mean by "32 CPUs (120 cores)". A 32 core workstation would be based on a board with 4 cpu sockets and 8-core cpus. Although it should be noted that only certain Xeons work on mainboards with 4 sockets.

Most straight forward setup would consist of 3 machines.
1 machine for Fluent (doesn't even need monitor, keyboard etc)
2 machines for Pre/Postprocessing
File-Transfer using SMB or SFTP

Possibly cheaper
1 big machine: run Fluent on fewer cores than available during daytime and use the computer for pre-/post-processing
1 smaller machine for engineer number 2

Are you planning on using Windows or Linux?

Edit: Apparently AMD offers a 16 core cpu (consisting on 4 quad cores), which is capable of running on an 4 socket board -> 64 cores on one mainboard.
Hi,

I'm planing on a workstation with 32 CPUs and each of them have 4 cores. So there are 128 cores in total. Sorry that 120 is a tempo. I have corrected that.

I'm planning on having at least the large workstation on Linux.

Regarding the two setup ways you mentioned, will they have any performance difference? I guess the first way you mentioned (3 machines one) could be more complex to build. I'm really wondering is this big workstation normal? And will the large size of the machine, large number of CPUs and cores cause any problem?
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   February 14, 2014, 10:10
Default
  #4
Senior Member
 
Onno
Join Date: Jan 2012
Location: Germany
Posts: 120
Rep Power: 15
Kaskade is on a distinguished road
How are you going to fit 32 cpus into a single machine?

Ansys is restrictive when it comes to Linux distributions: http://www.ansys.com/staticassets/AN....0_11-2011.pdf
I doubt Fluent wouldn't work on a free distribution, but Ansys might not give you support if something doesn't function properly.

BTW: StarCCM+ is cheaper for that many cores and CD-Adapco allows the use of CentOS.
Kaskade is offline   Reply With Quote

Old   February 14, 2014, 10:27
Default
  #5
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by Kaskade View Post
How are you going to fit 32 cpus into a single machine?

Ansys is restrictive when it comes to Linux distributions: http://www.ansys.com/staticassets/AN....0_11-2011.pdf
I doubt Fluent wouldn't work on a free distribution, but Ansys might not give you support if something doesn't function properly.

BTW: StarCCM+ is cheaper for that many cores and CD-Adapco allows the use of CentOS.

No that much big machine on the market? I once heard about workstation with 32 CPUs but I never checked it online. Intuitively I think it could be quite big and as heavy as me.

I don't understand what do you mean by 'free distribution' here. I'm reading the link you provide me hardly. Thanks for your information.

We don't have a problem of the fluent license. We can easily rent or borrow licenses from our supplier company at the neighbor.

Btw, what about having win XP on the large machine? I read somewhere in this forum that Fluent runs much faster on Win XP. Fluent 15 still supports win XP. If I have Windows on my big machine, how do we set up so that the CFD engineer using the small machine can run the simulation directly on the big machine while another engineer is using the big machine at the same time?
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   February 14, 2014, 10:38
Default
  #6
Senior Member
 
Onno
Join Date: Jan 2012
Location: Germany
Posts: 120
Rep Power: 15
Kaskade is on a distinguished road
I only found mainboards/workstations with 4 cpu sockets. If you want to use more cpus, you need to connect multiple machines using something like infiniband or ethernet -> cluster.

Free as in "for free". RHEL and SLES require a fee: https://www.suse.com/products/server/how-to-buy/ (You can set the language at the top.)
They offer support and a long distribution life time.

XP support ends in April. And I doubt it is faster.
Kaskade is offline   Reply With Quote

Old   February 14, 2014, 10:48
Question
  #7
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by Kaskade View Post
I only found mainboards/workstations with 4 cpu sockets. If you want to use more cpus, you need to connect multiple machines using something like infiniband or ethernet -> cluster.

Free as in "for free". RHEL and SLES require a fee: https://www.suse.com/products/server/how-to-buy/ (You can set the language at the top.)
They offer support and a long distribution life time.

XP support ends in April. And I doubt it is faster.
I will check the size on the market with the workstation stores. Maybe we will be running into building a cluster. Is there any tutorial on how to build a cluster? We're totally fresh on this.
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   February 14, 2014, 11:15
Default
  #8
Senior Member
 
Onno
Join Date: Jan 2012
Location: Germany
Posts: 120
Rep Power: 15
Kaskade is on a distinguished road
On the plus side: a cluster is easier to upgrade.

Most tutorials for setting up a cluster are pretty ancient. But it should be easy if you only want run one Fluent session across multiple machines.

If you want to use a job scheduler and ressource manager it gets trickier. Possible Google Searches: Platform LSF / Torque maui / Hadoop

We are in the process of setting up a 32 core cluster ourselves. Our hardware is virtual, but the software setup should be similar.
Kaskade is offline   Reply With Quote

Old   February 14, 2014, 17:30
Default
  #9
Senior Member
 
François Grégoire
Join Date: Jan 2010
Location: Canada
Posts: 392
Rep Power: 17
macfly is on a distinguished road
Edit : nevermind, I was suggesting to go read the hardware forum, and I just saw you did
macfly is offline   Reply With Quote

Old   February 15, 2014, 08:21
Question
  #10
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by Kaskade View Post
On the plus side: a cluster is easier to upgrade.

Most tutorials for setting up a cluster are pretty ancient. But it should be easy if you only want run one Fluent session across multiple machines.

If you want to use a job scheduler and ressource manager it gets trickier. Possible Google Searches: Platform LSF / Torque maui / Hadoop

We are in the process of setting up a 32 core cluster ourselves. Our hardware is virtual, but the software setup should be similar.
I just checked the other threads on this forum. It seems that it will be much more expensive if you want to build a cluster with good data exchange speed between CPUs. Why don't you just choose to buy a 4 sockets, 128 cores workstation? In this way, the maintenance and management cost shall be also a little lower. Later on, if you want to upgrade it (like adding more CPUs), you can also change the workstation into a cluster.
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   February 15, 2014, 10:43
Default
  #11
Senior Member
 
Gonzalo
Join Date: Mar 2011
Location: Argentina
Posts: 122
Rep Power: 16
gfoam is on a distinguished road
Check out the SGI workstations I think they have the machine you need. Regards.
Gonzalo
gfoam is offline   Reply With Quote

Old   February 16, 2014, 06:00
Question
  #12
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by gfoam View Post
Check out the SGI workstations I think they have the machine you need. Regards.
Gonzalo

Yes. I found hardware companies may sell it, including HP. But they are mainly for the server, not for scientific calculation, like CFD. I never saw anyone mentioned there is a this large workstation in this forum. So may I ask are they okay for CFD?

I'm also wondering what is the advantage of cluster over workstation in this case? And how much more expensive and complex would the cluster choose be for my case?
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   February 16, 2014, 07:28
Question
  #13
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by Kaskade View Post
How are you going to fit 32 cpus into a single machine?

Ansys is restrictive when it comes to Linux distributions: http://www.ansys.com/staticassets/AN....0_11-2011.pdf
I doubt Fluent wouldn't work on a free distribution, but Ansys might not give you support if something doesn't function properly.

BTW: StarCCM+ is cheaper for that many cores and CD-Adapco allows the use of CentOS.
I'm quite fresh to the enterprise linux area. What I understand from the link you provide is as below.

1. Linux is open for one computer. But if it is connected to another computer (no matter it is linux or windows), we need to pay for the connection which is called SUSE Linux Enterprise Server.

2. Fluent also restrict file distribution in Linux.

Is this correct? Windows doesn't have this restriction? We already have Windows system.
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Old   February 16, 2014, 08:04
Default
  #14
Senior Member
 
ghost82's Avatar
 
Rick
Join Date: Oct 2010
Posts: 1,016
Rep Power: 26
ghost82 will become famous soon enough
Hi,
I never seen these type of worstations/servers managing more than 4 cpus.

To have an idea of the price, if you want to build a cluster based on 8 workstations connected to 1 infiniband switch, with 32gb ram 1600Mhz ecc, 2x Xeon e5-2687w (which are not the latest cpus on the market, but they're very good 8 real cores), 1000W psu gold, asus motherboard, liquid cooled, infiniband 40gb/s, based on USA costs, you are going to spend more or less about 50.000 USD (installation costs not included).

What will be the cost of the single server?

PS: from my experience I would go with windows

Last edited by ghost82; February 17, 2014 at 01:54.
ghost82 is offline   Reply With Quote

Old   February 17, 2014, 08:28
Default
  #15
Senior Member
 
Onno
Join Date: Jan 2012
Location: Germany
Posts: 120
Rep Power: 15
Kaskade is on a distinguished road
@ghost82: Why would you build a cluster from workstation? I think some rack mounted machines would be far easier to handle.

@Anna Tian: The problem with large amounts of cores in a single machine is probably heat. The heat production of core/CPU rises linearly with the clock rate, additionaly the voltages need to be raised to keep the system stable, which increases the heat production exponentially. So if you want a lot of cores, you will end up with slower cores.

Have you considered asking Ansys for advice?
Kaskade is offline   Reply With Quote

Old   February 17, 2014, 08:36
Default
  #16
Senior Member
 
ghost82's Avatar
 
Rick
Join Date: Oct 2010
Posts: 1,016
Rep Power: 26
ghost82 will become famous soon enough
Quote:
Originally Posted by Kaskade View Post
@ghost82: Why would you build a cluster from workstation? I think some rack mounted machines would be far easier to handle.
Yes you're right, however I think the total cost will not be too much different; also 8 machines are not too many, they can be installed in asmall space, and can be easily controlled/maintained in the form of workstations.
ghost82 is offline   Reply With Quote

Old   February 17, 2014, 08:54
Default
  #17
Senior Member
 
Onno
Join Date: Jan 2012
Location: Germany
Posts: 120
Rep Power: 15
Kaskade is on a distinguished road
For commercial applications the sheer computing power is not enough. You'll need a decent RAID system plus backup and an UPS (Uninterruptible Power Supply). And a rack will make handling the cables easier.

The next step should probably be getting quotes for different system.
ghost82 and Anna Tian like this.
Kaskade is offline   Reply With Quote

Old   February 17, 2014, 11:37
Post
  #18
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by ghost82 View Post
Hi,
I never seen these type of worstations/servers managing more than 4 cpus.

To have an idea of the price, if you want to build a cluster based on 8 workstations connected to 1 infiniband switch, with 32gb ram 1600Mhz ecc, 2x Xeon e5-2687w (which are not the latest cpus on the market, but they're very good 8 real cores), 1000W psu gold, asus motherboard, liquid cooled, infiniband 40gb/s, based on USA costs, you are going to spend more or less about 50.000 USD (installation costs not included).

What will be the cost of the single server?

PS: from my experience I would go with windows
You go with Windows only for the price reason? I read on this forum that linux is better on the memory usage.
__________________
Best regards,
Meimei

Last edited by Anna Tian; February 17, 2014 at 14:25.
Anna Tian is offline   Reply With Quote

Old   February 17, 2014, 12:55
Default
  #19
Senior Member
 
ghost82's Avatar
 
Rick
Join Date: Oct 2010
Posts: 1,016
Rep Power: 26
ghost82 will become famous soon enough
Not for the price: I tried several linux distribution, not certified by ansys (Centos, ubuntu, kubuntu) and the Sles distribution; in all cases I had problems not with fluent itself, but with the license manager: sometimes starting 2 times and not able to connect, sometimes not starting; I found (at least for my system) that windows 7 64 bit is better since everithing was and is smooth.

Daniele
Anna Tian likes this.
ghost82 is offline   Reply With Quote

Old   February 17, 2014, 14:25
Question
  #20
Senior Member
 
Anna Tian's Avatar
 
Meimei Wang
Join Date: Jul 2012
Posts: 494
Rep Power: 15
Anna Tian is on a distinguished road
Quote:
Originally Posted by ghost82 View Post
Not for the price: I tried several linux distribution, not certified by ansys (Centos, ubuntu, kubuntu) and the Sles distribution; in all cases I had problems not with fluent itself, but with the license manager: sometimes starting 2 times and not able to connect, sometimes not starting; I found (at least for my system) that windows 7 64 bit is better since everithing was and is smooth.

Daniele
Regarding the liquid cooling, is liquid cooling important? Is it expensive? We can place it in another room so that we won't be disturbed by the noise from the machine if air cooling is used. Any other concerns? Will that give additional requirements like the quality of the water?
__________________
Best regards,
Meimei
Anna Tian is offline   Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Dual cpu workstation VS 2 node cluster single cpu workstation Verdi Hardware 18 September 2, 2013 04:09
Cluster of 8 CPUs tu FLUENT 5 November 29, 2005 02:43
How many CPU's in your cluster Charles Crosby Main CFD Forum 18 March 26, 2001 11:07
star-hpc on workstation cluster Joe Stone Siemens 1 August 9, 2000 14:06
PC vs. Workstation Tim Franke Main CFD Forum 5 September 29, 1999 16:01


All times are GMT -4. The time now is 06:06.