CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM

personal supercomputer

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   May 28, 2010, 23:04
Arrow personal supercomputer
  #1
Senior Member
 
lakeat's Avatar
 
Daniel WEI (老魏)
Join Date: Mar 2009
Location: South Bend, IN, USA
Posts: 688
Blog Entries: 9
Rep Power: 10
lakeat is on a distinguished road
Send a message via Skype™ to lakeat
Dear all,

Long time no see.

I want to buy a personal supercomputer.

I am using a big mac pro now, 8 cores, 32G mem, 10000$, yes, it is really fast. BUT, I am thinking if there's better options.

Some say Cray, others say GPU. So, my ultimate questions are:
1. Will OpenFOAM parallel computing works on GPU? Or am I already out-of-date, so, is there anyone of you already using GPU for OpenFOAM, how is it? Anyinfo is appreciated!
2. Based on your experiences, what hardware and computer-company do you suggest me to go? Thanks


---------
Note.1 - There is already supercomputer for me to run large cases, I just need a "personal" supercomputer for pre/post processing, or some small CFD cases.
Note.2 - 10000$~20000$ would be good


Let's talk.
__________________
~
Daniel WEI
-------------
NatHaz Modeling Laboratory
Department of Civil & Environmental Engineering & Earth Sciences
University of Notre Dame, USA
Email || My Personal CFD Blog
lakeat is offline   Reply With Quote

Old   May 29, 2010, 14:37
Default
  #2
Senior Member
 
Alberto Passalacqua
Join Date: Mar 2009
Location: Ames, Iowa, United States
Posts: 1,889
Rep Power: 25
alberto will become famous soon enoughalberto will become famous soon enough
You should give an idea of the size of your cases too, in order to better evaluate the case.

If you use the machine mainly for post-processing, I would not consider GPU-based machines. The GPUs are good to perform the core operations (there is a library for OpenFOAM too: http://vratis.com/speedITblog/ You can contact them, they were very friendly with me.), but for pre/post-processing you still need to rely on traditional CPU's, since the post-processing tools are not designed to work on GPU's.
GPU's are very interesting to run cases however, because the power/cost is very convenient.

About the company, it depends on what kind of system you choose, and on your personal preference. I personally stay away from Apple (overpriced hardware, technological lock-in, not amazing support). At work we essentially use DELL machines: quality is good, support service is very good as well as the price/quality ratio, hardware support for Linux is excellent.

Best,
__________________
Alberto

GeekoCFD - A free distribution based on openSUSE 64 bit with CFD tools, including OpenFOAM. Available as live DVD/USB, hard drive image and virtual image.
GeekoCFD 32bit - The 32bit edition of GeekoCFD.
GeekoCFD text mode - A smaller version of GeekoCFD, text-mode only, with only OpenFOAM. Available in a variety of virtual formats.
alberto is offline   Reply With Quote

Old   May 29, 2010, 15:51
Default
  #3
Senior Member
 
lakeat's Avatar
 
Daniel WEI (老魏)
Join Date: Mar 2009
Location: South Bend, IN, USA
Posts: 688
Blog Entries: 9
Rep Power: 10
lakeat is on a distinguished road
Send a message via Skype™ to lakeat
Hi there, thanks

Quote:
Originally Posted by alberto View Post
You should give an idea of the size of your cases too, in order to better evaluate the case.
As I have said, I will submit large cases to school's high performance computer, so the deskside supercomputer I would use to simulate unsteady LES cases less than 2 million grids.

Quote:
Originally Posted by alberto View Post
If you use the machine mainly for post-processing, I would not consider GPU-based machines. The GPUs are good to perform the core operations (there is a library for OpenFOAM too: http://vratis.com/speedITblog/ You can contact them, they were very friendly with me.), but for pre/post-processing you still need to rely on traditional CPU's, since the post-processing tools are not designed to work on GPU's. GPU's are very interesting to run cases however, because the power/cost is very convenient.
I wonder if it is restricted to some certain matrices solvers?

Quote:
Originally Posted by alberto View Post
About the company, it depends on what kind of system you choose, and on your personal preference. I personally stay away from Apple (overpriced hardware, technological lock-in, not amazing support). At work we essentially use DELL machines: quality is good, support service is very good as well as the price/quality ratio, hardware support for Linux is excellent.
I prefer linux (open)suse.
So far, the building experience is not very pleasant on mac for me.
I will have a look at dell.
__________________
~
Daniel WEI
-------------
NatHaz Modeling Laboratory
Department of Civil & Environmental Engineering & Earth Sciences
University of Notre Dame, USA
Email || My Personal CFD Blog
lakeat is offline   Reply With Quote

Old   May 29, 2010, 17:20
Default
  #4
Senior Member
 
Alberto Passalacqua
Join Date: Mar 2009
Location: Ames, Iowa, United States
Posts: 1,889
Rep Power: 25
alberto will become famous soon enoughalberto will become famous soon enough
Quote:
Originally Posted by lakeat View Post
As I have said, I will submit large cases to school's high performance computer, so the deskside supercomputer I would use to simulate unsteady LES cases less than 2 million grids.
OK. So your current configuration should be good enough. If you're really willing to change, you might take a look at workstations from DELL, HP and others. I'm familiar with the Dell Precision line: they're not good looking (more or less they look grey/black, squared and they are quite heavy), but the seem very robust, and their "next business day" support is very efficient.

Quote:
I wonder if it is restricted to some certain matrices solvers?
I think they have at least the standard solvers (different flavours of conjugate gradient methods). I'm not sure about AMG methods. You can find these details in their documentation.

Quote:
I prefer linux (open)suse.
Me too! I have to use Red Hat on some machine, but my life is so much easier on openSUSE/SUSE Linux Enterprise (It has probably something to do with the SUSE mascotte: that smart chameleon !)

Quote:
So far, the building experience is not very pleasant on mac for me.
Yes. I have similar problems on Red Hat, especially for the graphical tools. Their libraries are too old, and I had to rebuild everything (meaning also system libraries) from source.

Quote:
I will have a look at dell.
Hint: DELL has a "N series", which comes with no operating system. If you plan to go with Linux, you might be interested, especially if your campus has a site license for some distribution.

Best,
__________________
Alberto

GeekoCFD - A free distribution based on openSUSE 64 bit with CFD tools, including OpenFOAM. Available as live DVD/USB, hard drive image and virtual image.
GeekoCFD 32bit - The 32bit edition of GeekoCFD.
GeekoCFD text mode - A smaller version of GeekoCFD, text-mode only, with only OpenFOAM. Available in a variety of virtual formats.
alberto is offline   Reply With Quote

Old   May 29, 2010, 17:26
Default
  #5
Senior Member
 
lakeat's Avatar
 
Daniel WEI (老魏)
Join Date: Mar 2009
Location: South Bend, IN, USA
Posts: 688
Blog Entries: 9
Rep Power: 10
lakeat is on a distinguished road
Send a message via Skype™ to lakeat
N series? Can you give me a link?

I checked the Precision workstation, here's their advice (But Win-7 is a necessary option):

Model Dell Precision™ T5500/T7500 Workstation
Processor Dual Quad Core Intel® Xeon® Processors W5677 3.46GHz
Operating System Genuine Windows® XP Professional 64bit
Memory 12 GB, 1333MHz, DDR3 SDRAM, ECC
Primary Hard Disk 500GB SATA 3.0Gb/s with NCQ and 16MB DataBurst Cache
Secondary Hard Disk
Graphics Card NVIDIA® Quadro® FX 3800
Display Dual 20", 24"Ultra Sharp or single/dual 27" or 30" UltraSharpTM

Any ideas?
__________________
~
Daniel WEI
-------------
NatHaz Modeling Laboratory
Department of Civil & Environmental Engineering & Earth Sciences
University of Notre Dame, USA
Email || My Personal CFD Blog
lakeat is offline   Reply With Quote

Old   May 29, 2010, 18:44
Default
  #6
Senior Member
 
Alberto Passalacqua
Join Date: Mar 2009
Location: Ames, Iowa, United States
Posts: 1,889
Rep Power: 25
alberto will become famous soon enoughalberto will become famous soon enough
Quote:
Originally Posted by lakeat View Post
N series? Can you give me a link?

I checked the Precision workstation, here's their advice (But Win-7 is a necessary option):

Model Dell Precision™ T5500/T7500 Workstation
Processor Dual Quad Core Intel® Xeon® Processors W5677 3.46GHz
Operating System Genuine Windows® XP Professional 64bit
Memory 12 GB, 1333MHz, DDR3 SDRAM, ECC
Primary Hard Disk 500GB SATA 3.0Gb/s with NCQ and 16MB DataBurst Cache
Secondary Hard Disk
Graphics Card NVIDIA® Quadro® FX 3800
Display Dual 20", 24"Ultra Sharp or single/dual 27" or 30" UltraSharpTM

Any ideas?
N-series seem not to have "desktop" systems anymore (third column):

http://premier.dell.com/portal/catal...47&cs=RC956904

If you do not want Windows, simply contact them, they are usually pretty friendly and you can always tell them "no windows, or I don't buy" However, usually, this is not required.

Since you are in academia, you probably have right to their academic discount program (check also if your university has deals with them). It is fairly easy to have a system without operating system from them however.

Best,
__________________
Alberto

GeekoCFD - A free distribution based on openSUSE 64 bit with CFD tools, including OpenFOAM. Available as live DVD/USB, hard drive image and virtual image.
GeekoCFD 32bit - The 32bit edition of GeekoCFD.
GeekoCFD text mode - A smaller version of GeekoCFD, text-mode only, with only OpenFOAM. Available in a variety of virtual formats.
alberto is offline   Reply With Quote

Old   May 29, 2010, 18:55
Default
  #7
Senior Member
 
Arjun
Join Date: Mar 2009
Location: Nurenberg, Germany
Posts: 355
Rep Power: 9
arjun is on a distinguished road
Quote:
Originally Posted by lakeat View Post
N series? Can you give me a link?

I checked the Precision workstation, here's their advice (But Win-7 is a necessary option):

Model Dell Precision™ T5500/T7500 Workstation
Processor Dual Quad Core Intel® Xeon® Processors W5677 3.46GHz
Operating System Genuine Windows® XP Professional 64bit
Memory 12 GB, 1333MHz, DDR3 SDRAM, ECC
Primary Hard Disk 500GB SATA 3.0Gb/s with NCQ and 16MB DataBurst Cache
Secondary Hard Disk
Graphics Card NVIDIA® Quadro® FX 3800
Display Dual 20", 24"Ultra Sharp or single/dual 27" or 30" UltraSharpTM

Any ideas?

buy the gpu with meaximim memory possible because that would be your bottle-neck.

2. About post processing, actually gpus are very much suitable for post processing but currently not many programs could exploit this.

3. I am not sure whether openfoam would be converted to gpu based or not. Somebody who knows better might comment on it.

4. About your LES simulations does your mesh move or change with iterations?

Edited to add: I just bought corei7 based desktop with gts 240 (1GB memory), so far the functions i have written i can see 10-20 times gains on speed. Not expensive but alright setup.
arjun is offline   Reply With Quote

Old   May 29, 2010, 22:39
Default
  #8
Senior Member
 
Alberto Passalacqua
Join Date: Mar 2009
Location: Ames, Iowa, United States
Posts: 1,889
Rep Power: 25
alberto will become famous soon enoughalberto will become famous soon enough
Quote:
Originally Posted by arjun View Post
buy the gpu with meaximim memory possible because that would be your bottle-neck.

2. About post processing, actually gpus are very much suitable for post processing but currently not many programs could exploit this.

3. I am not sure whether openfoam would be converted to gpu based or not. Somebody who knows better might comment on it.

4. About your LES simulations does your mesh move or change with iterations?

Edited to add: I just bought corei7 based desktop with gts 240 (1GB memory), so far the functions i have written i can see 10-20 times gains on speed. Not expensive but alright setup.
Yes, GPU would be good for post-processing, but there is not much around using them.

At the moment they seem very interesting if you have a specialized code, to obtain high performance machines at low cost. The story changes when you use commercial applications and/or more general applications, because of the limits of the hardware architecture, and of the fact there isn't many software GPU-enabled out of the box.

I find the technology very interesting, but at the moment I'm "watching what happens". I don't think it is convenient to convert codes at this point, if you do not have specific needs. Plus, you tie yourself to a platform, actually to one vendor. Never a great idea, especially if your investment is long term.

There is another factor. It seems to me you still need a relatively powerful machine (CPU-based) to work with large cases, since not all the work is done on the GPU. IBM is proposing hybrid solutions for high performance applications, but from what I read (not much), the target doesn't seem to be the workstation market.

Best,
__________________
Alberto

GeekoCFD - A free distribution based on openSUSE 64 bit with CFD tools, including OpenFOAM. Available as live DVD/USB, hard drive image and virtual image.
GeekoCFD 32bit - The 32bit edition of GeekoCFD.
GeekoCFD text mode - A smaller version of GeekoCFD, text-mode only, with only OpenFOAM. Available in a variety of virtual formats.
alberto is offline   Reply With Quote

Old   May 30, 2010, 09:04
Default
  #9
Senior Member
 
Arjun
Join Date: Mar 2009
Location: Nurenberg, Germany
Posts: 355
Rep Power: 9
arjun is on a distinguished road
Quote:
Originally Posted by alberto View Post
Yes, GPU would be good for post-processing, but there is not much around using them.

At the moment they seem very interesting if you have a specialized code, to obtain high performance machines at low cost. The story changes when you use commercial applications and/or more general applications, because of the limits of the hardware architecture, and of the fact there isn't many software GPU-enabled out of the box.

I find the technology very interesting, but at the moment I'm "watching what happens". I don't think it is convenient to convert codes at this point, if you do not have specific needs. Plus, you tie yourself to a platform, actually to one vendor. Never a great idea, especially if your investment is long term.

There is another factor. It seems to me you still need a relatively powerful machine (CPU-based) to work with large cases, since not all the work is done on the GPU. IBM is proposing hybrid solutions for high performance applications, but from what I read (not much), the target doesn't seem to be the workstation market.

Best,

At the moment the biggest hurdle is that codes have to be re-written to make them suitable for gpus.
Now this becomes big issue because all the codes - Fluent,starcmm+ (though relatively new), openfoam etc are very large codes and re-writing them to suit gpu is difficult if not impossible. Plus it would take time and people willing to work on it.

OpenFOAM as my understanding is very esoteric code (at least is beyond my brain). So chances of modifying it by an outsider are slim. So the core developers would have to do it. I do not think they made any comments about their intensions about it.

Personally i do am interested in gpu based navier stokes solver. Next few months i will find out whether for me it would be even feasible to write anything like it.
arjun is offline   Reply With Quote

Old   May 30, 2010, 12:25
Default
  #10
New Member
 
Join Date: May 2010
Posts: 1
Rep Power: 0
Leandros is on a distinguished road
Don't take a Dell. They are not expensive enough, and you will have difficulties to waste your money.
But don't worry. As Alberto wrote, Dells are just grey. Ask Gucci to design your Personal Supercomputer case, and you can get rid of another 10000$.

I thought April Fool's day was too months ago. Anybody else setting a lower limit on his budget and using Macs for CFD?
Leandros is offline   Reply With Quote

Old   May 30, 2010, 13:56
Default
  #11
Senior Member
 
Alberto Passalacqua
Join Date: Mar 2009
Location: Ames, Iowa, United States
Posts: 1,889
Rep Power: 25
alberto will become famous soon enoughalberto will become famous soon enough
Hmm...as first post it looks "interesting". I guess he was stating his maximum budget.
__________________
Alberto

GeekoCFD - A free distribution based on openSUSE 64 bit with CFD tools, including OpenFOAM. Available as live DVD/USB, hard drive image and virtual image.
GeekoCFD 32bit - The 32bit edition of GeekoCFD.
GeekoCFD text mode - A smaller version of GeekoCFD, text-mode only, with only OpenFOAM. Available in a variety of virtual formats.
alberto is offline   Reply With Quote

Old   May 30, 2010, 14:09
Default
  #12
Senior Member
 
Alberto Passalacqua
Join Date: Mar 2009
Location: Ames, Iowa, United States
Posts: 1,889
Rep Power: 25
alberto will become famous soon enoughalberto will become famous soon enough
Quote:
Originally Posted by arjun View Post
OpenFOAM as my understanding is very esoteric code (at least is beyond my brain). So chances of modifying it by an outsider are slim. So the core developers would have to do it. I do not think they made any comments about their intensions about it.
I'd say OF is very modular, but yes, not always simple to understand. However someone prepared a library for the core solvers as said before, and it seems using it is straightforward, at least according to their documentation (see link above).

Best,
__________________
Alberto

GeekoCFD - A free distribution based on openSUSE 64 bit with CFD tools, including OpenFOAM. Available as live DVD/USB, hard drive image and virtual image.
GeekoCFD 32bit - The 32bit edition of GeekoCFD.
GeekoCFD text mode - A smaller version of GeekoCFD, text-mode only, with only OpenFOAM. Available in a variety of virtual formats.
alberto is offline   Reply With Quote

Old   May 30, 2010, 16:27
Default
  #13
Senior Member
 
Arjun
Join Date: Mar 2009
Location: Nurenberg, Germany
Posts: 355
Rep Power: 9
arjun is on a distinguished road
Quote:
Originally Posted by alberto View Post
I'd say OF is very modular, but yes, not always simple to understand. However someone prepared a library for the core solvers as said before, and it seems using it is straightforward, at least according to their documentation (see link above).

Best,
In openFOAM's case it might be little bit easier or more difficult. I am not very sure.

There are two big hurdles

1. GPU programming code has to be c type and it does not allow you object oriented code. But openfoam is object oriented concept.

It is however possible to call gpu code from c++ type program for example openfoam so in that way it is possible to call gpu functions from openfoam.

2. Second biggest hurdle is AMG part. Solving amg cycle with gpu is easy but setup could be very difficult. openfoam uses additive corrective multigrid so for them making it parallel is easier i guess. But until someone tries we would never know.

Other than these two things i think almost every thing could be written gpu based.

So in the end i do agree with you that gpu based acceleration with openfoam is possible.
arjun is offline   Reply With Quote

Old   May 30, 2010, 19:16
Default
  #14
Senior Member
 
Alberto Passalacqua
Join Date: Mar 2009
Location: Ames, Iowa, United States
Posts: 1,889
Rep Power: 25
alberto will become famous soon enoughalberto will become famous soon enough
Quote:
Originally Posted by arjun View Post
In openFOAM's case it might be little bit easier or more difficult. I am not very sure.

There are two big hurdles

1. GPU programming code has to be c type and it does not allow you object oriented code. But openfoam is object oriented concept.

It is however possible to call gpu code from c++ type program for example openfoam so in that way it is possible to call gpu functions from openfoam.
True, it should be question of writing wrappers for the C functions.

Quote:
2. Second biggest hurdle is AMG part. Solving amg cycle with gpu is easy but setup could be very difficult. openfoam uses additive corrective multigrid so for them making it parallel is easier i guess. But until someone tries we would never know.
Interesting. I still use conjugate gradient methods in my work on multiphase flow because they seem to be more reliable (stable), even if not as fast.

Best,
__________________
Alberto

GeekoCFD - A free distribution based on openSUSE 64 bit with CFD tools, including OpenFOAM. Available as live DVD/USB, hard drive image and virtual image.
GeekoCFD 32bit - The 32bit edition of GeekoCFD.
GeekoCFD text mode - A smaller version of GeekoCFD, text-mode only, with only OpenFOAM. Available in a variety of virtual formats.
alberto is offline   Reply With Quote

Old   May 30, 2010, 19:30
Default
  #15
Senior Member
 
lakeat's Avatar
 
Daniel WEI (老魏)
Join Date: Mar 2009
Location: South Bend, IN, USA
Posts: 688
Blog Entries: 9
Rep Power: 10
lakeat is on a distinguished road
Send a message via Skype™ to lakeat
Hi, thank you.
I would probably use AMG, and FSI research is also on my schedule.
I will ask dell.
Every time I use mac "magic" mouse, I'm going nuts
__________________
~
Daniel WEI
-------------
NatHaz Modeling Laboratory
Department of Civil & Environmental Engineering & Earth Sciences
University of Notre Dame, USA
Email || My Personal CFD Blog
lakeat is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Compiling personal BC fsalvucci OpenFOAM 0 April 27, 2010 18:10
Pressure inlet BC for supercomputer gailms FLUENT 0 November 12, 2009 10:56
Problems in personal library compilation shrina OpenFOAM 3 February 23, 2009 12:12
Advertising and Personal Promotion in the CFD-Wiki Jonas Larsson CFD-Wiki 0 November 20, 2005 20:20
Parallel Computing Classes at San Diego Supercomputer Center Jan. 20-22 Amitava Majumdar Main CFD Forum 0 January 5, 1999 13:00


All times are GMT -4. The time now is 08:14.