OpenFOAM requirements for a cluster
 

Hi,

my university is going to build up a new cluster and asked me for a profile of requirements for my applications! For my PhD I will work in the field of turbomachinery (10-20 Mio cells) with OpenFoam (and CFX)! Unfortunately, I don't know much about the administration of IT-Systems (Hardware/Software)...

Does anybody have some suggestions what I should tell them??


Thanks,

Michael

Cluster Specifications
 

This is a very open ended question, but I will try to point you in the right direction. As a rule of thumb, I tend to limit a single core to 1M grid points, so for your application, you would need a minimum of about 20 cores. However, you should also consider how many cases you will be running simultaneously, and then increase the desired number of cores accordingly. Also, at 1M points per core, cases will run somewhat slowly, so it would be beneficial if you could limit that to say 0.5 or 0.25M.

When it comes to memory, I tend towards 1GB of memory for 1GHz of cpu power; however, depending on your application this may be bad advice. For example, if you are going to be generating large grids on the cluster, you may need a single node with significantly more memory to generate and decompose the grids.

A final thing to consider is the required speed of the interconnect. If you plan on building nodes with dual quad core processors, or quad quad core processors, the limiting factor will quickly become the interconnect, and therefore you should strongly consider using something like infiniband.

ANSYS specifications
 

hey farbfilm,

I had the same problem the beginning of this year and I used the ANSYS guidelines. Since you need a login to find this information I will post it here:

General Hardware Recommendations

We test and certify our applications on high-end workstations or servers and partner with all of the major hardware vendors. Below is the current list of certified recommended processors. At the hardware vendor web site, look for " Workstations " or " Servers " when looking for these processors. We encourage you to configure a system and send us the specifications and we will look them over and get back to you with further recommendations, if necessary. Please email: diana.collier@ansys.com for a review. We recommend that you review the FLUENT benchmarks, a link is provided at the bottom of this document.
PROCESSORS
Intel quad-core ( Nehalem, X55xx ) processors
Intel dual-core ( Nehalem, X55xx ) processors
Intel six-core (Westmore, X56xx) Processors
AMD quad-core Shanghai processors
NOTE: Please choose this link to view ANSYS FLUENT benchmarks http://www.fluent.com/software/fluen...ench/index.htm
IMPORTANT: We DO NOT certify the Intel Core i7 processors. These are designed specifically for home use only. We only certify on the workstation class chips, Intel Nehalem or AMD Shanghai.
Intel Hyper-Threading/SMT

Hyper-Threading is now called Simultaneous multithreading or SMT. Customers are recommended to leave SMT enabled on their systems but not over-subscribe physical cores for parallel simulations. While some improvement is possible, the extra performance from the virtual threads is not cost-effective and incommensurate with the additional license costs (which are per process)."
Basically, if a section of the CPU core is not being used it tries to run a second task on these sections. For example, if one process only needs to do floating point operations while another only needs to do integer operations they can run both concurrently. For FLUENT, there is no consequence to performance if it is turned off. If SMT is on, and you run 16x (instead of 8x; assuming dual cpu quad-core nodes), you can get an additional 20% or so (compared to 8x) improvement. This is not recommended since you only get 20% more for 2x licenses (license is per process). in this scene rio, leave SMT on and run 8 way. This is the recommended approach.
GPU COMPUTING (CUDA)
CFD solvers do not currently run on GPU processors. We are doing some investigative work and will update this document when GPU computing is supported. GPU solving should be available for Structural Mechanical at release V13.x.
HARD DRIVE SPECIFICATIONS
FLUENT/CFX - No special hard drive configuration is required as FLUENT does very little Disk Read/Write.
MECHANICAL – Multiple SCSI or SAS hard drives striped with RAID 0, 15,000 RPM or better is recommended. To clarify, if you have enough memory for the sparse solver to solve in-core, which is quite possible these days, or you use the PCG solver, then the I/O is essentially the same as FLUENTs.
RAID 0 Description

Using RAID 0 mainly for tasks requiring fast access to a large capacity of temporary disk storage. RAID 0 usable capacity is 100% as all available drives are used.
MEMORY
We recommend a minimum of 4 GB RAM per core.
HIGH SPEED INTERCONNECTS
In general if you have fewer than 4 nodes, a Gigabit Ethernet switch is sufficient. For more than 4 nodes, consider a high speed switch such as Infiniband or Myrinet. The amount of speedup you see from such a switch will depend on your problems size and complexity. Currently, Infiniband and Myrinet are only available on Linux systems or Windows 2008 HPC Server (High Performance Computing).
GRAPHIC CARDS
We test and certify with the following two graphic card vendors. These cards range from the low end to the very high end. We recommend the high end
Nvidia Quadro FX
ATI FireGL
REMOTE ACCESS - BEST PRACTICES
Running any ANSYS application using remote access software is not certified or supported. If the machine you are remoting into has a PCIx16 slot with an nVidia Quadro FX or AMD ATI FirePro/FireGL high-end graphics card it could work with some considerations. Remember that most servers DO NOT have a PCIx16 slot so this will not be possible. There are servers on the market that do offer this capability. Please check your hardware vendors web site for more information.
Visit this FAQ for recommendations and best practices.
http://www.fluentusers.com/support/i...ote-access.htm
SUBMITTING JOBS TO A CLUSTER
WINDOWS > WINDOWS CLUSTER
Having a high-end 64-bit Workstation to do Pre and Post Processing locally is supported. Pre and Post Processing has heavy graphics demands and requires a high-end, certified video card. See the section above, "REMOTE ACCESS - BEST PRACTICES" and choose the FAQ link for more information.
LINUX > LINUX CLUSTER
Having a high-end 64-bit Workstation to do Pre and Post Processing locally is supported. Pre and Post Processing has heavy graphics demands and requires a high-end, certified video card. See the section above, "REMOTE ACCESS - BEST PRACTICES" and choose the FAQ link for more information.
64-BIT OPERATING SYSTEMS
We recommend that you run on 64-bit Workstations or Servers. There is a 2 GB per process memory limitation on 32-bit operating systems. Realistically, approximately 1.5 - 1.7 GB of RAM is all that will be available. Running on 64-bit processors will give you access to as much RAM as is configured on the system. To see information about Microsoft Windows Operating Systems memory limitations visit: http://msdn.microsoft.com/en-us/library/aa366778.aspx
================================================== =============

I do not known if you already have a hardware supplier, but I would like to recommend Super Micro.
Previously we had 2 generations of SGI machines to do our work, but the management software they use is really messy and their support is not so "friendly".

Hope it helps