CFD Online Discussion Forums

CFD Online Discussion Forums (
-   Main CFD Forum (
-   -   Beowulf clusters (

Sebastien Perron May 15, 2001 05:33

Beowulf clusters
This fall, I might have to implement a linux cluster to solve radiative heat transfer problems. I would like to get some good info on the subject. Especially from those who have working experience in this field.


Kike May 15, 2001 06:30

Re: Beowulf clusters
Dear Sebastien

What subject, clusters or radiative heat transfer?

If it is the first, this is a very nice site with a well suited link list:

I also like this, specially designed for Linux clusters.

You will also need this for MPI novelties http://WWW.ERC.MsState.Edu/labs/hpcl/projects/mpi/ or both are clasics sites on this subject.

We work on parallel codes for CFD, not with radiation for the moment, but in atomization.



Seb May 15, 2001 07:29

Re: Beowulf clusters
Thanks. I'm looking for info on clusters.

Jonas Larsson May 15, 2001 08:08

Re: Beowulf clusters
We have been doing CFD simulations on Linux clusters for two years with very good results. We also recently invested in a new 100-CPU cluster. This type of clusters definitely gives the most computing power per dollar. I'm not sure what kind of information you want. If you have any more specific questions please ask.

Seb May 15, 2001 08:40

Re: Beowulf clusters
I'M looking for publications and technical info on setup and performance of type of clusters. I maybe wrong, but I beleive that a beowulf cluster is better suited for computations than other type clusters.

Jonas Larsson May 15, 2001 09:24

Re: Beowulf clusters
Technical info on setup - there are hundres of web-sites and also several books about how to build clusters. Search on and and you'll find more info than you want to have. There are a few important things which you need to decide:

1. If you should run a special "cluster linux-distribution" or if you should go with something more mainstream (RedHat or so) - I would recommend to go with a mainstream distribution supported by your software vendor. Examples of specialized distributions are and We use plain RedHat 7.0 (could not use 7.1 due to a problem with Fluent on 7.1).

2. How to setup your cluster nodes - disk-less or with disk, user-data on disks, ... - I would recommend having disks on each node but only install swap and OS on the local disks and keep applications and user-data elsewhere on a mother-node.

3. Network - If you should use 100mbit fast ethernet or something faster - I would recoomend 100mbit. Faster alternatives like myrinet ( ) are too expensive. What is important for CFD is most often the latency of the network (time to initiate a communication), not the maximum bandwith. See the fluent benchmarks below for a comparison between myrinet and fast ethernet - it only makes a difference on very small cases or when you use very many CPUs.

4. Type of nodes - rack mounted or stand-alone, workstation dual-CPU type or cheap single-CPU desktop type PC - I would recommend to buy cheap signle-CPU desktop machines (small form factor) from a large vendor (Dell, Gateway, Compaq, HP, ...). These cost a lot less than the rack-mounted machines and the work-station-class machines and the performance is the same almost. However, if you don't have a good place to place these (a cooled server room or so) then you might be better of byuing rack-mounted 1U machines - these are a bit more expensive per CPU but the prices are coming down quickly due to the large volumes on these for web-hosting applications.

I'm not sure of what you mean with "beowulf". To me a beowulf cluster is a cluster running linux assembled of off-the-shelf single or double-CPU machines. This type of clusters are suitable for applications where you don't need SMP. Most CFD-codes run very well on this type of clusters using domain-decomposition.

About performance. Fluent has published a number of benchmarks with Linux and NT clusters, using both 100mbit fast ethernet and myrinet. You can find the benchmarks on:

The setup we have in our last cluster is single-CPU 1GHz PIII nodes with 512MB memory using a standard 100mbit switched network. I will soon be able to publish some benchmarks from it if you are interested. We run Fluent and a number of in-house codes compiled with Absoft Fortran and gcc.

John C. Chien May 15, 2001 13:15

Re: Beowulf clusters
(1). If you put more layers between your idea and results, then the idea will get lost sooner or later. (2). If I have to link 100 calculators together to calculate multiplication and division faster, then it will be at most a research project. (3). The principle is: if it can be done, then there is no need to actually do it.(unless you are curious about the actual process) Also, if it has been done, then it is solved already. (4). The fate of CFD is: people who are trained to do CFD (researchers) tend to push it into research-oriented project. CFD can become something like a calculator, if you push in that direction harder. (this is what I am going. I think vector-computing, parallel-computing are non-personal type of activities. It can survive only in an organized company or laboratory. The extensive programming effort invested in vector-computing is not going to be used in any PC or Workstation. (5). Recently, I got comments from an engineer using CFD saying that since the company has thousands of workstations, these workstation can be used at night to do CFD parallel computing and they have code written in this fashion already. This is pushing the CFD activities into night-shift. It is identical to the days of vector-computing, when CFD jobs must be scheduled for weekend execution. (6). Anyway, if we look at the current approach, we know that CFD is being pushed further away from the daily applications, because of the use of parallel computing, because of the modular approach in code architecture and the division into pre- and post-processing. (if you continue on this approach, you are not going to see CAD, Word processing, that means you have to do Pre-processing of CAD and word processing, solver-CAD or word crunching, and then post-processing of CAD and pre-view of word processing,etc...) (7). If you need power like vector-computer, then you know that it is still in research. But if you focus on the requirement a level lower, then you can make use of currently available computer effectively. (8). So my suggestion is: try to solve the current problem with the current technology. If one is still not sure of the solution of simple problem, then scaling up the size and speed of the equipment is not going to solve the real problem. In other words, the difficulty in CFD is not the consequence of the shortage in vector- or parallel- computing equipment or system. (the same is true for the Internet. If it is not secure and reliable, then the bigger size will give you the bigger problem.)

Sebastien Perron May 15, 2001 13:31

Re: Beowulf clusters
Thanks for the advice. But this cluster is needed to sove 3D radiative heat transfer problems. The ressources (computers) available at my university (small it is...) are not powerfull enough. This project would have three different goals

1) Research in CFD for the applied computer science

2) Research in parallel computing for the computer science departement.

3) Contracts in MHD with an industry nearby.

If I put forward a decent project (get money from researcher groups), I could be able to create a job for myself.

John C. Chien May 15, 2001 15:42

Re: Beowulf clusters
(1). Yes, I think it is a good idea, if the radiation problem does require a more powerful system. And your comment was exactly what I was expecting. (2). Your three goals justify the use of a bigger system. And the key word is "research". (3). My comment was: some companies are taking this approach as a routine approach. As a research project approach, vector-computing and parallel-computing are good approaches to promote deeper thinking. As a routine solution approach, it will inflate the resources requirement in each department, competition in getting parallel-computing resources, or push the engineers into night-shift environment.

John D. Van Workum May 15, 2001 18:35

Re: Beowulf clusters
Getting started in cluster computing can be a challenging experience. There are many options and choices to consider. If you (or anyone) would like to try out a cluster before you make any purchasing decisions; send me an email personally. I can present you with a beneficial alternative to your computing needs.

Marat Hoshim May 16, 2001 02:18

Re: Beowulf clusters

do you run 1 job on all 100 nodes ?


Jonas Larsson May 16, 2001 02:58

Re: Beowulf clusters
Only if we have very large jobs (several million cells). A one-million cell job is usually run on about 20 nodes.

Kike May 16, 2001 05:58

Re: Beowulf clusters
Dear JCChien

I agree with you with respect to the anecdote of this engineer. His proposal, as you say, could be "night-shift". As John D. Van Workum said, to program for work on a beowulf (or on any cluster type enviroment) is a challenging task, but it is necesary in many industrial and scientific cases.

Now you have the case of Sebastien, and I give two others. Could you solve the following CFD problems in your PC (say a P4 at 1.5GHz with 1GB RAM)?

- DNS for coaxial atomization (Re_gas=10^5) through a downstream distance large enought to describe secondary atomization.

- Complete (not reduced) combustion kinetics for diffusion methane flame.

I am sure your experience is extense enought to list much more problems of this kind. Are you sure you will accuratelly solve them with your PC?

I dont know exactly why, but your comments remind me some words I listened from a soviet communist leader some years ago, when the east-block still exists.

If your keep yourself stacked in your VC++ (for Windows9x or ME, I think), in the next ten years you will have to program for schools. Don't close your great developed mind to the future (in CFD calculations). The soviets did it and now you can see what had happened.



Marat Hoshim May 16, 2001 08:08

Re: Beowulf clusters
That's interesting !

Did you test the speed-up factor for running a job on 20 and on 100 nodes ?

Did you find out a maximum node number which is adequate for a job regarding the 100 MBit ethernet ?

I guess that at a job on 100 nodes (or even less) the 100 MBit ethernet might become a bottleneck ?



Jonas Larsson May 16, 2001 10:39

Re: Beowulf clusters
Yep, we've done a lot of scaling-studies. The parallel scaling is very dependent on how large your case is though. Here are a few "rules-of-thumb" for which scaling you can expect with a 100mbit switched network:

50,000 cells scales well up to only a few nodes

500,000 cells scales well up to 10 nodes (approx 8 times speedup) and reasonably well up to 20 nodes (approx 14 times speedup).

5,000,000 cells scales well up to 50 nodes (still waiting for the last nodes to come up so I haven't tried to run on all 100 yet).

John C. Chien May 16, 2001 14:19

Re: Beowulf clusters
(1). As I have said clearly that his project was a "Research Project". Research Project is important, so I said that his approach was justified. (2). I have also mentioned that "DNS" is not going to be used in industrial day-to-day CFD applications, and "Turbulent Reacting Flow modeling" is a very difficult task. So, you are talking about "CFD Research". (3). It is all right to use a large network to do research work. And you can say that it can be done or it has been done. (4). I think, ten years from now, you will still be dealing with the same issue. IBM PC is now twenty years, and we are still using the same PC. This include the workstations. Nothing has changed. (5). The problem with the Russians is, I think, that they tried to jump ahead of their time and try to change their system. You can't replace the system, unless you know how to change it. (6). Don't worry about me, I am not working on CFD, I have been working on something else for a long time. (more than fifteen years) (7). You will always run out of computer memory and computer time, even if you connect every single computer in the whole world. I think, the PC we have today is more than enough to solve our daily problems. The only thing missing is: start using it in a smart way. (8). As for the Russians, I think, it is a good idea to use their old system, instead of the western system. You can't erase the history, so both China and Russia can't use the western system successfully, even if they are trying to do so. (9). AS for the research, I think, it is individual-oriented. It has nothing to do with the nationality, or the location. You can't actually manage the research. (10). My focus is on researcher or engineer, I think, it is up to him to decide what computer to use and what approach to take. And in research, each individual is different. What I am doing can't be reproduced by someone else. This is very important. (11). Like a medical doctor, what I am doing is : to stimulate your brain activity. Don't try to follow someone's foot steps, unless that is your own approach. Russians and Chinese will always put themselves in the fighting position, because they are unable to erase the history. They are still fighting for their survival, even if currently the threat is not there. So, I think, the issue is not whether a larger computer system should be used or not. The key issue is whether you can keep individual in the active thinking status. I think, it is better to have 100 researchers than to have 100 computers. Because each individual has in his brain a very powerful computer which can also think.

Jurek May 16, 2001 15:07

Re: Beowulf clusters
Are those factors software-dependent (fluent ? with sliding mesh?)

Jonas Larsson May 16, 2001 15:13

Re: Beowulf clusters
Yes, this of course depends on the code. The numbers I gave are typical for Fluent. Our in-house code (multi-block explicit runge-kutta) scales even better. Sliding interfaces in Fluent 5.5 and below scales very badly due to a bad algorithm to update the sliding interfaces. We've worked with Fluent to improve this and their next version will have much better scaling with sliding interfaces. Today it is not practical to parallelize cases with sliding-interfaces on more than about 6 CPU - this also applies to SMP machines!

Seb May 17, 2001 18:11

Thanks for the info. EOM.

All times are GMT -4. The time now is 02:44.