CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > General Forums > Main CFD Forum

Beowulf clusters

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   May 15, 2001, 06:33
Default Beowulf clusters
  #1
Sebastien Perron
Guest
 
Posts: n/a
This fall, I might have to implement a linux cluster to solve radiative heat transfer problems. I would like to get some good info on the subject. Especially from those who have working experience in this field.

Thanks.
  Reply With Quote

Old   May 15, 2001, 07:30
Default Re: Beowulf clusters
  #2
Kike
Guest
 
Posts: n/a
Dear Sebastien

What subject, clusters or radiative heat transfer?

If it is the first, this is a very nice site with a well suited link list: www.beowulf.org

I also like this www.inifinitevoid.com/beowulf, specially designed for Linux clusters.

You will also need this for MPI novelties http://WWW.ERC.MsState.Edu/labs/hpcl/projects/mpi/ or http://www-unix.mcs.anl.gov/mpi/index.html both are clasics sites on this subject.

We work on parallel codes for CFD, not with radiation for the moment, but in atomization.

Regards

Kike
  Reply With Quote

Old   May 15, 2001, 08:29
Default Re: Beowulf clusters
  #3
Seb
Guest
 
Posts: n/a
Thanks. I'm looking for info on clusters.
  Reply With Quote

Old   May 15, 2001, 09:08
Default Re: Beowulf clusters
  #4
Jonas Larsson
Guest
 
Posts: n/a
We have been doing CFD simulations on Linux clusters for two years with very good results. We also recently invested in a new 100-CPU cluster. This type of clusters definitely gives the most computing power per dollar. I'm not sure what kind of information you want. If you have any more specific questions please ask.
  Reply With Quote

Old   May 15, 2001, 09:40
Default Re: Beowulf clusters
  #5
Seb
Guest
 
Posts: n/a
I'M looking for publications and technical info on setup and performance of type of clusters. I maybe wrong, but I beleive that a beowulf cluster is better suited for computations than other type clusters.

  Reply With Quote

Old   May 15, 2001, 10:24
Default Re: Beowulf clusters
  #6
Jonas Larsson
Guest
 
Posts: n/a
Technical info on setup - there are hundres of web-sites and also several books about how to build clusters. Search on www.google.com and www.amazon.com and you'll find more info than you want to have. There are a few important things which you need to decide:

1. If you should run a special "cluster linux-distribution" or if you should go with something more mainstream (RedHat or so) - I would recommend to go with a mainstream distribution supported by your software vendor. Examples of specialized distributions are www.scyld.com and www.mosix.com. We use plain RedHat 7.0 (could not use 7.1 due to a problem with Fluent on 7.1).

2. How to setup your cluster nodes - disk-less or with disk, user-data on disks, ... - I would recommend having disks on each node but only install swap and OS on the local disks and keep applications and user-data elsewhere on a mother-node.

3. Network - If you should use 100mbit fast ethernet or something faster - I would recoomend 100mbit. Faster alternatives like myrinet ( www.myrinet.com ) are too expensive. What is important for CFD is most often the latency of the network (time to initiate a communication), not the maximum bandwith. See the fluent benchmarks below for a comparison between myrinet and fast ethernet - it only makes a difference on very small cases or when you use very many CPUs.

4. Type of nodes - rack mounted or stand-alone, workstation dual-CPU type or cheap single-CPU desktop type PC - I would recommend to buy cheap signle-CPU desktop machines (small form factor) from a large vendor (Dell, Gateway, Compaq, HP, ...). These cost a lot less than the rack-mounted machines and the work-station-class machines and the performance is the same almost. However, if you don't have a good place to place these (a cooled server room or so) then you might be better of byuing rack-mounted 1U machines - these are a bit more expensive per CPU but the prices are coming down quickly due to the large volumes on these for web-hosting applications.

I'm not sure of what you mean with "beowulf". To me a beowulf cluster is a cluster running linux assembled of off-the-shelf single or double-CPU machines. This type of clusters are suitable for applications where you don't need SMP. Most CFD-codes run very well on this type of clusters using domain-decomposition.

About performance. Fluent has published a number of benchmarks with Linux and NT clusters, using both 100mbit fast ethernet and myrinet. You can find the benchmarks on: http://www.fluent.com/software/fluen...ench/index.htm

The setup we have in our last cluster is single-CPU 1GHz PIII nodes with 512MB memory using a standard 100mbit switched network. I will soon be able to publish some benchmarks from it if you are interested. We run Fluent and a number of in-house codes compiled with Absoft Fortran and gcc.
  Reply With Quote

Old   May 15, 2001, 14:15
Default Re: Beowulf clusters
  #7
John C. Chien
Guest
 
Posts: n/a
(1). If you put more layers between your idea and results, then the idea will get lost sooner or later. (2). If I have to link 100 calculators together to calculate multiplication and division faster, then it will be at most a research project. (3). The principle is: if it can be done, then there is no need to actually do it.(unless you are curious about the actual process) Also, if it has been done, then it is solved already. (4). The fate of CFD is: people who are trained to do CFD (researchers) tend to push it into research-oriented project. CFD can become something like a calculator, if you push in that direction harder. (this is what I am going. I think vector-computing, parallel-computing are non-personal type of activities. It can survive only in an organized company or laboratory. The extensive programming effort invested in vector-computing is not going to be used in any PC or Workstation. (5). Recently, I got comments from an engineer using CFD saying that since the company has thousands of workstations, these workstation can be used at night to do CFD parallel computing and they have code written in this fashion already. This is pushing the CFD activities into night-shift. It is identical to the days of vector-computing, when CFD jobs must be scheduled for weekend execution. (6). Anyway, if we look at the current approach, we know that CFD is being pushed further away from the daily applications, because of the use of parallel computing, because of the modular approach in code architecture and the division into pre- and post-processing. (if you continue on this approach, you are not going to see CAD, Word processing, that means you have to do Pre-processing of CAD and word processing, solver-CAD or word crunching, and then post-processing of CAD and pre-view of word processing,etc...) (7). If you need power like vector-computer, then you know that it is still in research. But if you focus on the requirement a level lower, then you can make use of currently available computer effectively. (8). So my suggestion is: try to solve the current problem with the current technology. If one is still not sure of the solution of simple problem, then scaling up the size and speed of the equipment is not going to solve the real problem. In other words, the difficulty in CFD is not the consequence of the shortage in vector- or parallel- computing equipment or system. (the same is true for the Internet. If it is not secure and reliable, then the bigger size will give you the bigger problem.)
  Reply With Quote

Old   May 15, 2001, 14:31
Default Re: Beowulf clusters
  #8
Sebastien Perron
Guest
 
Posts: n/a
Thanks for the advice. But this cluster is needed to sove 3D radiative heat transfer problems. The ressources (computers) available at my university (small it is...) are not powerfull enough. This project would have three different goals

1) Research in CFD for the applied computer science

2) Research in parallel computing for the computer science departement.

3) Contracts in MHD with an industry nearby.

If I put forward a decent project (get money from researcher groups), I could be able to create a job for myself.

  Reply With Quote

Old   May 15, 2001, 16:42
Default Re: Beowulf clusters
  #9
John C. Chien
Guest
 
Posts: n/a
(1). Yes, I think it is a good idea, if the radiation problem does require a more powerful system. And your comment was exactly what I was expecting. (2). Your three goals justify the use of a bigger system. And the key word is "research". (3). My comment was: some companies are taking this approach as a routine approach. As a research project approach, vector-computing and parallel-computing are good approaches to promote deeper thinking. As a routine solution approach, it will inflate the resources requirement in each department, competition in getting parallel-computing resources, or push the engineers into night-shift environment.
  Reply With Quote

Old   May 15, 2001, 19:35
Default Re: Beowulf clusters
  #10
John D. Van Workum
Guest
 
Posts: n/a
Getting started in cluster computing can be a challenging experience. There are many options and choices to consider. If you (or anyone) would like to try out a cluster before you make any purchasing decisions; send me an email personally. I can present you with a beneficial alternative to your computing needs.
  Reply With Quote

Old   May 16, 2001, 03:18
Default Re: Beowulf clusters
  #11
Marat Hoshim
Guest
 
Posts: n/a
Jonas,

do you run 1 job on all 100 nodes ?

Marat
  Reply With Quote

Old   May 16, 2001, 03:58
Default Re: Beowulf clusters
  #12
Jonas Larsson
Guest
 
Posts: n/a
Only if we have very large jobs (several million cells). A one-million cell job is usually run on about 20 nodes.
  Reply With Quote

Old   May 16, 2001, 06:58
Default Re: Beowulf clusters
  #13
Kike
Guest
 
Posts: n/a
Dear JCChien

I agree with you with respect to the anecdote of this engineer. His proposal, as you say, could be "night-shift". As John D. Van Workum said, to program for work on a beowulf (or on any cluster type enviroment) is a challenging task, but it is necesary in many industrial and scientific cases.

Now you have the case of Sebastien, and I give two others. Could you solve the following CFD problems in your PC (say a P4 at 1.5GHz with 1GB RAM)?

- DNS for coaxial atomization (Re_gas=10^5) through a downstream distance large enought to describe secondary atomization.

- Complete (not reduced) combustion kinetics for diffusion methane flame.

I am sure your experience is extense enought to list much more problems of this kind. Are you sure you will accuratelly solve them with your PC?

I dont know exactly why, but your comments remind me some words I listened from a soviet communist leader some years ago, when the east-block still exists.

If your keep yourself stacked in your VC++ (for Windows9x or ME, I think), in the next ten years you will have to program for schools. Don't close your great developed mind to the future (in CFD calculations). The soviets did it and now you can see what had happened.

Regards

Kike
  Reply With Quote

Old   May 16, 2001, 09:08
Default Re: Beowulf clusters
  #14
Marat Hoshim
Guest
 
Posts: n/a
That's interesting !

Did you test the speed-up factor for running a job on 20 and on 100 nodes ?

Did you find out a maximum node number which is adequate for a job regarding the 100 MBit ethernet ?

I guess that at a job on 100 nodes (or even less) the 100 MBit ethernet might become a bottleneck ?

Regards,

Marat
  Reply With Quote

Old   May 16, 2001, 11:39
Default Re: Beowulf clusters
  #15
Jonas Larsson
Guest
 
Posts: n/a
Yep, we've done a lot of scaling-studies. The parallel scaling is very dependent on how large your case is though. Here are a few "rules-of-thumb" for which scaling you can expect with a 100mbit switched network:

50,000 cells scales well up to only a few nodes

500,000 cells scales well up to 10 nodes (approx 8 times speedup) and reasonably well up to 20 nodes (approx 14 times speedup).

5,000,000 cells scales well up to 50 nodes (still waiting for the last nodes to come up so I haven't tried to run on all 100 yet).
  Reply With Quote

Old   May 16, 2001, 15:19
Default Re: Beowulf clusters
  #16
John C. Chien
Guest
 
Posts: n/a
(1). As I have said clearly that his project was a "Research Project". Research Project is important, so I said that his approach was justified. (2). I have also mentioned that "DNS" is not going to be used in industrial day-to-day CFD applications, and "Turbulent Reacting Flow modeling" is a very difficult task. So, you are talking about "CFD Research". (3). It is all right to use a large network to do research work. And you can say that it can be done or it has been done. (4). I think, ten years from now, you will still be dealing with the same issue. IBM PC is now twenty years, and we are still using the same PC. This include the workstations. Nothing has changed. (5). The problem with the Russians is, I think, that they tried to jump ahead of their time and try to change their system. You can't replace the system, unless you know how to change it. (6). Don't worry about me, I am not working on CFD, I have been working on something else for a long time. (more than fifteen years) (7). You will always run out of computer memory and computer time, even if you connect every single computer in the whole world. I think, the PC we have today is more than enough to solve our daily problems. The only thing missing is: start using it in a smart way. (8). As for the Russians, I think, it is a good idea to use their old system, instead of the western system. You can't erase the history, so both China and Russia can't use the western system successfully, even if they are trying to do so. (9). AS for the research, I think, it is individual-oriented. It has nothing to do with the nationality, or the location. You can't actually manage the research. (10). My focus is on researcher or engineer, I think, it is up to him to decide what computer to use and what approach to take. And in research, each individual is different. What I am doing can't be reproduced by someone else. This is very important. (11). Like a medical doctor, what I am doing is : to stimulate your brain activity. Don't try to follow someone's foot steps, unless that is your own approach. Russians and Chinese will always put themselves in the fighting position, because they are unable to erase the history. They are still fighting for their survival, even if currently the threat is not there. So, I think, the issue is not whether a larger computer system should be used or not. The key issue is whether you can keep individual in the active thinking status. I think, it is better to have 100 researchers than to have 100 computers. Because each individual has in his brain a very powerful computer which can also think.
  Reply With Quote

Old   May 16, 2001, 16:07
Default Re: Beowulf clusters
  #17
Jurek
Guest
 
Posts: n/a
Are those factors software-dependent (fluent ? with sliding mesh?)
  Reply With Quote

Old   May 16, 2001, 16:13
Default Re: Beowulf clusters
  #18
Jonas Larsson
Guest
 
Posts: n/a
Yes, this of course depends on the code. The numbers I gave are typical for Fluent. Our in-house code (multi-block explicit runge-kutta) scales even better. Sliding interfaces in Fluent 5.5 and below scales very badly due to a bad algorithm to update the sliding interfaces. We've worked with Fluent to improve this and their next version will have much better scaling with sliding interfaces. Today it is not practical to parallelize cases with sliding-interfaces on more than about 6 CPU - this also applies to SMP machines!
  Reply With Quote

Old   May 17, 2001, 19:11
Default Thanks for the info. EOM.
  #19
Seb
Guest
 
Posts: n/a
  Reply With Quote

Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Beowulf Cluster configuration? inginer OpenFOAM 1 December 6, 2010 10:11
parallel processing- linux clusters manoj s FLUENT 2 June 20, 2007 19:48
Parallel Fluent on Linux clusters Chengwang Lei FLUENT 3 November 24, 2004 03:05
MPI on Clusters wak Siemens 0 September 19, 2004 23:03
Beowulf Clusters, Extrem Linux Support Jonas Larsson Main CFD Forum 2 March 31, 1999 01:50


All times are GMT -4. The time now is 08:51.