[lakeat]

Dear all,

Long time no see.

I want to buy a personal supercomputer.

I am using a big mac pro now, 8 cores, 32G mem, 10000$, yes, it is really fast. BUT, I am thinking if there's better options.

Some say Cray, others say GPU. So, my ultimate questions are:
1. Will OpenFOAM parallel computing works on GPU? Or am I already out-of-date, so, is there anyone of you already using GPU for OpenFOAM, how is it? Anyinfo is appreciated!
2. Based on your experiences, what hardware and computer-company do you suggest me to go? Thanks


---------
Note.1 - There is already supercomputer for me to run large cases, I just need a "personal" supercomputer for pre/post processing, or some small CFD cases.
Note.2 - 10000$~20000$ would be good


Let's talk.

[alberto]

You should give an idea of the size of your cases too, in order to better evaluate the case.

If you use the machine mainly for post-processing, I would not consider GPU-based machines. The GPUs are good to perform the core operations (there is a library for OpenFOAM too: http://vratis.com/speedITblog/ You can contact them, they were very friendly with me.), but for pre/post-processing you still need to rely on traditional CPU's, since the post-processing tools are not designed to work on GPU's.
GPU's are very interesting to run cases however, because the power/cost is very convenient.

About the company, it depends on what kind of system you choose, and on your personal preference. I personally stay away from Apple (overpriced hardware, technological lock-in, not amazing support). At work we essentially use DELL machines: quality is good, support service is very good as well as the price/quality ratio, hardware support for Linux is excellent.

Best,

[lakeat]

Hi there, thanks

Originally Posted by alberto

You should give an idea of the size of your cases too, in order to better evaluate the case.

As I have said, I will submit large cases to school's high performance computer, so the deskside supercomputer I would use to simulate unsteady LES cases less than 2 million grids.

Originally Posted by alberto

If you use the machine mainly for post-processing, I would not consider GPU-based machines. The GPUs are good to perform the core operations (there is a library for OpenFOAM too: http://vratis.com/speedITblog/ You can contact them, they were very friendly with me.), but for pre/post-processing you still need to rely on traditional CPU's, since the post-processing tools are not designed to work on GPU's. GPU's are very interesting to run cases however, because the power/cost is very convenient.

I wonder if it is restricted to some certain matrices solvers?

Originally Posted by alberto

About the company, it depends on what kind of system you choose, and on your personal preference. I personally stay away from Apple (overpriced hardware, technological lock-in, not amazing support). At work we essentially use DELL machines: quality is good, support service is very good as well as the price/quality ratio, hardware support for Linux is excellent.

I prefer linux (open)suse.
So far, the building experience is not very pleasant on mac for me.
I will have a look at dell.

[alberto]

Originally Posted by lakeat

As I have said, I will submit large cases to school's high performance computer, so the deskside supercomputer I would use to simulate unsteady LES cases less than 2 million grids.

OK. So your current configuration should be good enough. If you're really willing to change, you might take a look at workstations from DELL, HP and others. I'm familiar with the Dell Precision line: they're not good looking (more or less they look grey/black, squared and they are quite heavy), but the seem very robust, and their "next business day" support is very efficient.

I wonder if it is restricted to some certain matrices solvers?

I think they have at least the standard solvers (different flavours of conjugate gradient methods). I'm not sure about AMG methods. You can find these details in their documentation.

I prefer linux (open)suse.

Me too! I have to use Red Hat on some machine, but my life is so much easier on openSUSE/SUSE Linux Enterprise (It has probably something to do with the SUSE mascotte: that smart chameleon !)

So far, the building experience is not very pleasant on mac for me.

Yes. I have similar problems on Red Hat, especially for the graphical tools. Their libraries are too old, and I had to rebuild everything (meaning also system libraries) from source.

I will have a look at dell.

Hint: DELL has a "N series", which comes with no operating system. If you plan to go with Linux, you might be interested, especially if your campus has a site license for some distribution.

Best,

[lakeat]

N series? Can you give me a link?

I checked the Precision workstation, here's their advice (But Win-7 is a necessary option):

Model Dell Precision™ T5500/T7500 Workstation
Processor Dual Quad Core Intel® Xeon® Processors W5677 3.46GHz
Operating System Genuine Windows® XP Professional 64bit
Memory 12 GB, 1333MHz, DDR3 SDRAM, ECC
Primary Hard Disk 500GB SATA 3.0Gb/s with NCQ and 16MB DataBurst Cache
Secondary Hard Disk
Graphics Card NVIDIA® Quadro® FX 3800
Display Dual 20", 24"Ultra Sharp or single/dual 27" or 30" UltraSharpTM

Any ideas?

[alberto]

Originally Posted by lakeat

N series? Can you give me a link?

I checked the Precision workstation, here's their advice (But Win-7 is a necessary option):

Model Dell Precision™ T5500/T7500 Workstation
Processor Dual Quad Core Intel® Xeon® Processors W5677 3.46GHz
Operating System Genuine Windows® XP Professional 64bit
Memory 12 GB, 1333MHz, DDR3 SDRAM, ECC
Primary Hard Disk 500GB SATA 3.0Gb/s with NCQ and 16MB DataBurst Cache
Secondary Hard Disk
Graphics Card NVIDIA® Quadro® FX 3800
Display Dual 20", 24"Ultra Sharp or single/dual 27" or 30" UltraSharpTM

Any ideas?

N-series seem not to have "desktop" systems anymore (third column):

http://premier.dell.com/portal/catal...47&cs=RC956904

If you do not want Windows, simply contact them, they are usually pretty friendly and you can always tell them "no windows, or I don't buy" However, usually, this is not required.

Since you are in academia, you probably have right to their academic discount program (check also if your university has deals with them). It is fairly easy to have a system without operating system from them however.

Best,

[arjun]

Originally Posted by lakeat

N series? Can you give me a link?

I checked the Precision workstation, here's their advice (But Win-7 is a necessary option):

Model Dell Precision™ T5500/T7500 Workstation
Processor Dual Quad Core Intel® Xeon® Processors W5677 3.46GHz
Operating System Genuine Windows® XP Professional 64bit
Memory 12 GB, 1333MHz, DDR3 SDRAM, ECC
Primary Hard Disk 500GB SATA 3.0Gb/s with NCQ and 16MB DataBurst Cache
Secondary Hard Disk
Graphics Card NVIDIA® Quadro® FX 3800
Display Dual 20", 24"Ultra Sharp or single/dual 27" or 30" UltraSharpTM

Any ideas?

buy the gpu with meaximim memory possible because that would be your bottle-neck.

2. About post processing, actually gpus are very much suitable for post processing but currently not many programs could exploit this.

3. I am not sure whether openfoam would be converted to gpu based or not. Somebody who knows better might comment on it.

4. About your LES simulations does your mesh move or change with iterations?

Edited to add: I just bought corei7 based desktop with gts 240 (1GB memory), so far the functions i have written i can see 10-20 times gains on speed. Not expensive but alright setup.

[alberto]

Originally Posted by arjun

buy the gpu with meaximim memory possible because that would be your bottle-neck.

2. About post processing, actually gpus are very much suitable for post processing but currently not many programs could exploit this.

3. I am not sure whether openfoam would be converted to gpu based or not. Somebody who knows better might comment on it.

4. About your LES simulations does your mesh move or change with iterations?

Edited to add: I just bought corei7 based desktop with gts 240 (1GB memory), so far the functions i have written i can see 10-20 times gains on speed. Not expensive but alright setup.

Yes, GPU would be good for post-processing, but there is not much around using them.

At the moment they seem very interesting if you have a specialized code, to obtain high performance machines at low cost. The story changes when you use commercial applications and/or more general applications, because of the limits of the hardware architecture, and of the fact there isn't many software GPU-enabled out of the box.

I find the technology very interesting, but at the moment I'm "watching what happens". I don't think it is convenient to convert codes at this point, if you do not have specific needs. Plus, you tie yourself to a platform, actually to one vendor. Never a great idea, especially if your investment is long term.

There is another factor. It seems to me you still need a relatively powerful machine (CPU-based) to work with large cases, since not all the work is done on the GPU. IBM is proposing hybrid solutions for high performance applications, but from what I read (not much), the target doesn't seem to be the workstation market.

Best,

[arjun]

Originally Posted by alberto

Yes, GPU would be good for post-processing, but there is not much around using them.

At the moment they seem very interesting if you have a specialized code, to obtain high performance machines at low cost. The story changes when you use commercial applications and/or more general applications, because of the limits of the hardware architecture, and of the fact there isn't many software GPU-enabled out of the box.

I find the technology very interesting, but at the moment I'm "watching what happens". I don't think it is convenient to convert codes at this point, if you do not have specific needs. Plus, you tie yourself to a platform, actually to one vendor. Never a great idea, especially if your investment is long term.

There is another factor. It seems to me you still need a relatively powerful machine (CPU-based) to work with large cases, since not all the work is done on the GPU. IBM is proposing hybrid solutions for high performance applications, but from what I read (not much), the target doesn't seem to be the workstation market.

Best,

At the moment the biggest hurdle is that codes have to be re-written to make them suitable for gpus.
Now this becomes big issue because all the codes - Fluent,starcmm+ (though relatively new), openfoam etc are very large codes and re-writing them to suit gpu is difficult if not impossible. Plus it would take time and people willing to work on it.

OpenFOAM as my understanding is very esoteric code (at least is beyond my brain). So chances of modifying it by an outsider are slim. So the core developers would have to do it. I do not think they made any comments about their intensions about it.

Personally i do am interested in gpu based navier stokes solver. Next few months i will find out whether for me it would be even feasible to write anything like it.

[Leandros]

Don't take a Dell. They are not expensive enough, and you will have difficulties to waste your money.
But don't worry. As Alberto wrote, Dells are just grey. Ask Gucci to design your Personal Supercomputer case, and you can get rid of another 10000$.

I thought April Fool's day was too months ago. Anybody else setting a lower limit on his budget and using Macs for CFD?

[alberto]

Hmm...as first post it looks "interesting". I guess he was stating his maximum budget.

[alberto]

Originally Posted by arjun

OpenFOAM as my understanding is very esoteric code (at least is beyond my brain). So chances of modifying it by an outsider are slim. So the core developers would have to do it. I do not think they made any comments about their intensions about it.

I'd say OF is very modular, but yes, not always simple to understand. However someone prepared a library for the core solvers as said before, and it seems using it is straightforward, at least according to their documentation (see link above).

Best,

[arjun]

Originally Posted by alberto

I'd say OF is very modular, but yes, not always simple to understand. However someone prepared a library for the core solvers as said before, and it seems using it is straightforward, at least according to their documentation (see link above).

Best,

In openFOAM's case it might be little bit easier or more difficult. I am not very sure.

There are two big hurdles

1. GPU programming code has to be c type and it does not allow you object oriented code. But openfoam is object oriented concept.

It is however possible to call gpu code from c++ type program for example openfoam so in that way it is possible to call gpu functions from openfoam.

2. Second biggest hurdle is AMG part. Solving amg cycle with gpu is easy but setup could be very difficult. openfoam uses additive corrective multigrid so for them making it parallel is easier i guess. But until someone tries we would never know.

Other than these two things i think almost every thing could be written gpu based.

So in the end i do agree with you that gpu based acceleration with openfoam is possible.

[alberto]

Originally Posted by arjun

In openFOAM's case it might be little bit easier or more difficult. I am not very sure.

There are two big hurdles

1. GPU programming code has to be c type and it does not allow you object oriented code. But openfoam is object oriented concept.

It is however possible to call gpu code from c++ type program for example openfoam so in that way it is possible to call gpu functions from openfoam.

True, it should be question of writing wrappers for the C functions.

2. Second biggest hurdle is AMG part. Solving amg cycle with gpu is easy but setup could be very difficult. openfoam uses additive corrective multigrid so for them making it parallel is easier i guess. But until someone tries we would never know.

Interesting. I still use conjugate gradient methods in my work on multiphase flow because they seem to be more reliable (stable), even if not as fast.

Best,

[lakeat]

Hi, thank you.
I would probably use AMG, and FSI research is also on my schedule.
I will ask dell.
Every time I use mac "magic" mouse, I'm going nuts