CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM

OpenFOAM on Amazon EC2

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree1Likes

Reply
 
LinkBack Thread Tools Display Modes
Old   August 3, 2010, 13:16
Default OpenFOAM on Amazon EC2
  #1
bjr
Member
 
Ben Racine
Join Date: Mar 2009
Location: Seattle, WA, USA
Posts: 62
Rep Power: 8
bjr is on a distinguished road
Send a message via AIM to bjr Send a message via Skype™ to bjr
Cluster compute instances announced...
http://aws.amazon.com/ec2/instance-types/

I hope to be able to share an openfoam-extend machine image and instructions of how to sign up and start using it on the wiki before too long.

Questions, comments, recommendations much appreciated.

Regards,
Ben Racine
bjr is offline   Reply With Quote

Old   November 11, 2010, 22:37
Default
  #2
New Member
 
Charles McCreary
Join Date: Jun 2010
Posts: 9
Rep Power: 7
crmccreary is on a distinguished road
And the AMI is?
crmccreary is offline   Reply With Quote

Old   November 27, 2010, 16:12
Smile
  #3
New Member
 
Joel Cugnoni
Join Date: Nov 2010
Posts: 4
Rep Power: 6
jcugnoni is on a distinguished road
Quote:
Originally Posted by crmccreary View Post
And the AMI is?
You can try the CAELinux image in EC2, it contains openfoam 1.7 and many other software packages like Discretizer, enGrid, netgen, Salome, GMSH
see http://www.caelinux.com for more details...
jcugnoni is offline   Reply With Quote

Old   November 28, 2010, 05:43
Default
  #4
Member
 
Lukasz Miroslaw
Join Date: Dec 2009
Location: Poland
Posts: 65
Rep Power: 7
Lukasz is on a distinguished road
Send a message via Skype™ to Lukasz
If there is a gpu card we could accelerate OF simulations. We have already good results with simplefoam. See http://sourceforge.net/projects/openfoamspeedit/ for details or drop me a line.

Last edited by Lukasz; November 28, 2010 at 11:54.
Lukasz is offline   Reply With Quote

Old   November 28, 2010, 11:36
Default
  #5
New Member
 
Zhijun
Join Date: Nov 2010
Posts: 3
Rep Power: 6
xizhijun is on a distinguished road
lukazs, you are annoying me and coworkers more and more .you spam in many threads to advertise a commercial offer even if nobody has asked anything remotely related .you even ask people to leave the discussion on this forum and use your site instead .

i remind you of the forum rules that you have agreed :

"Link spammers and other forms of spam are hunted very actively, both automatically and manually."

  • "State your affiliation explicitly in your message if you:
    • Recommend a product sold by your employer"
  • "Be very restrictive in posting links to other web sites, especially commercial sites"
"Forbidden content:
  • Advertising, both commercial and personal"
i ask you to stop your practice .
without regards ,zhijun !
xizhijun is offline   Reply With Quote

Old   November 29, 2010, 06:11
Default
  #6
Member
 
Lukasz Miroslaw
Join Date: Dec 2009
Location: Poland
Posts: 65
Rep Power: 7
Lukasz is on a distinguished road
Send a message via Skype™ to Lukasz
I agree with zhijun. We should not provide links to our web pages. Apologies for that. Our intention was to share our GPL-based technology to OF users, and not to advertise it on the forum.

Therefore, from now on we will stop providing a link to our web page but only to sourceforge where the OF plugin and the opensource speedit library can be downloaded: http://sourceforge.net/projects/openfoamspeedit/

Apologies again to those who felt offended.

Best regards,
Lukasz
Lukasz is offline   Reply With Quote

Old   January 17, 2011, 07:28
Default
  #7
Member
 
alexey2petrov's Avatar
 
Alexey
Join Date: Feb 2010
Posts: 33
Blog Entries: 1
Rep Power: 8
alexey2petrov is on a distinguished road
Quote:
Originally Posted by crmccreary View Post
And the AMI is?
Here they are :

'us-east-1' :
  • 'openfoam171_0-1ubuntu2' - 'ami-ecf50385',
  • 'openfoam171_0-1' - 'ami-62fa0c0b',
  • 'openfoam-dev-1.5' - 'ami-98f701f1',
  • 'openfoam-1.6-ext' : 'ami-b2fd0cdb',

'eu-west-1' :
  • 'openfoam171_0-1ubuntu2' - 'ami-519ca925',
  • 'openfoam171_0-1' - 'ami-8d9da8f9',
  • 'openfoam-dev-1.5' - 'ami-4d9ca939',
  • 'openfoam-1.6-ext' - 'ami-f998ad8d',

'ap-southeast-1' :
  • 'openfoam171_0-1ubuntu2' - 'ami-d2423c80',
  • 'openfoam171_0-1' - 'ami-28423c7a',
  • 'openfoam-dev-1.5' - 'ami-2c423c7e',
  • 'openfoam-1.6-ext' - 'ami-3a433d68',

'us-west-1' :
  • 'openfoam171_0-1ubuntu2' : 'ami-bb7e2efe',
  • 'openfoam171_0-1' - 'ami-8d7e2ec8',
  • 'openfoam-dev-1.5' - 'ami-a97e2eec',
  • 'openfoam-1.6-ext' - 'ami-5f60301a'

There is a SourceForge project dedicated to the "Cloud Computing for OpenFOAM (R) users" - CloudFlu

And corresponding documentation page
alexey2petrov is offline   Reply With Quote

Old   January 17, 2011, 08:26
Default
  #8
Senior Member
 
Elvis
Join Date: Mar 2009
Location: Sindelfingen, Germany
Posts: 579
Blog Entries: 5
Rep Power: 13
elvis is on a distinguished road
Dear Alexey,

could you tell me something more about "vulashaka" or how to use openfoam with python.
It would be great to get a starting point.

Another question what is the difference between CloudFlu and balloonfoam

thank you in advance.

elvis
elvis is offline   Reply With Quote

Old   January 17, 2011, 08:35
Default
  #9
Member
 
alexey2petrov's Avatar
 
Alexey
Join Date: Feb 2010
Posts: 33
Blog Entries: 1
Rep Power: 8
alexey2petrov is on a distinguished road
Quote:
Originally Posted by elvis View Post
what is the difference between CloudFlu and balloon-foam
We were forced to change the initial "balloon-foam" project name to the new one - "CloudFlu" according to OpenCFD Trademark Guidelines
alexey2petrov is offline   Reply With Quote

Old   May 23, 2011, 17:29
Default
  #10
New Member
 
Paul Bemis
Join Date: Nov 2010
Posts: 2
Rep Power: 0
PaulBemis is on a distinguished road
Hi Ben,
How'd this experiment go? How does OpenFoam run on the Amazon cloud in terms of performance? Can you also run it in parallel?

Thx

Quote:
Originally Posted by bjr View Post
Cluster compute instances announced...
http://aws.amazon.com/ec2/instance-types/

I hope to be able to share an openfoam-extend machine image and instructions of how to sign up and start using it on the wiki before too long.

Questions, comments, recommendations much appreciated.

Regards,
Ben Racine
PaulBemis is offline   Reply With Quote

Old   July 11, 2011, 10:29
Default
  #11
Member
 
Kevin
Join Date: May 2011
Posts: 32
Rep Power: 6
murrdpirate is on a distinguished road
Has anyone tried the Cluster Compute instance? I saw some results for the High-CPU Extra Large Compute instance and they weren't that impressive, but the Cluster Compute instance is supposedly tailor made for CFD type computing.

For some reason I can't seem to connect to the amazon machine image posted on openfoam.com. If someone can help me connect to an AMI with OpenFoam, I'd be glad to run a few tests and share the results.
murrdpirate is offline   Reply With Quote

Old   July 12, 2011, 09:19
Default
  #12
Member
 
alexey2petrov's Avatar
 
Alexey
Join Date: Feb 2010
Posts: 33
Blog Entries: 1
Rep Power: 8
alexey2petrov is on a distinguished road
Hi Kevin,
Quote:
Originally Posted by murrdpirate View Post
Has anyone tried the Cluster Compute instance? I saw some results for the High-CPU Extra Large Compute instance and they weren't that impressive, but the Cluster Compute instance is supposedly tailor made for CFD type computing.
Could you share the sources of your information, please?

From my turn, I read "Benchmarking Amazon EC2 for high-performance scientific computing" written by Edward Walker.
It is not OpenFOAM specific, but what he found out is "approximately 40%1000% performance degradation in the EC2 High-CPU Medium Instance runs compared to the NCSA (National Center for Supercomputing Applications, USA) runs"
And I have experienced the same situation when I tested my Amazon EC2 front-end for OpenFOAM - cloudFlu

As a consequence, our conclusion for using Amazon EC2 for HPC could be, "use Amazon EC2 if you have no access to the real cluster, only".
At the same time, this investigation by Edward Walker (Oct 2008) were made before Amazon had introduced specific Cluster Computing Instances (July 2010).
So, there is a real hope that things are changed from that time. Unfortunately, I still could see no data or articles which would indicate this shift.

Quote:
Originally Posted by murrdpirate View Post
If someone can help me connect to an AMI with OpenFoam, I'd be glad to run a few tests and share the results.
Ye, I will try to prepare specific Amazon EC2 Cluster Computing Instance with OpenFOAM pre-installed.

Best regards,
Alexey
alexey2petrov is offline   Reply With Quote

Old   July 12, 2011, 09:36
Default
  #13
Member
 
Kevin
Join Date: May 2011
Posts: 32
Rep Power: 6
murrdpirate is on a distinguished road
Hi Alexey,

I'm just going by the description by Amazon:

Quote:
The Amazon EC2 Cluster Compute and Cluster GPU instance types are specifically designed to combine high compute performance with high performance network capability to meet the needs of HPC applications. Unique to Cluster Compute and Cluster GPU instances is the ability to group them into clusters of instances for use with HPC applications. This is particularly valuable for those applications that rely on protocols like Message Passing Interface (MPI) for tightly coupled inter-node communication.
I don't know how Walker was able to test a cluster of High-CPU Medium Instances, as this description says that clusters can only be done with Cluster Compute Instances. But however he did it, I doubt it compares well to the new Cluster Compute instance. Although I am definitely not an expert on any of this stuff.

I did finally did connect to the OpenFoam AMI last night and got it running on a High-CPU instance with solid results. I'm looking to test the Cluster Compute Instance asap.
murrdpirate is offline   Reply With Quote

Old   July 12, 2011, 10:34
Default
  #14
Member
 
alexey2petrov's Avatar
 
Alexey
Join Date: Feb 2010
Posts: 33
Blog Entries: 1
Rep Power: 8
alexey2petrov is on a distinguished road
Hi Kevin,
Quote:
Originally Posted by murrdpirate View Post
I don't know how Walker was able to test a cluster of High-CPU Medium Instances, as this description says that clusters can only be done with Cluster Compute Instances.
You see, cluster notion is not defined in hardware terms. According to wikipedia - computer cluster is a group of linked computers, working together closely thus in many respects forming a single computer could be build. You can make a cluster even from a few interconnected PC (know as Beowulf cluster). What enables group of computers become a cluster is low latency interconnection.

I have used this understanding in my cloudFlu implementation and saw the same idea implemented in the widely used common purpose StartCluster - it is really working.

If you look at the Amazon description on account of Cluster Computing Instance, it will be possible to see specification of "I/O Performance: 10 Gigabit Ethernet", which provides comparable performance with the "Infiniband switch" available at 2008 year, claimed by Walker's in his article. So, once we will prepare a Cluster Computing Instance AMI (with OpenFOAM pre-installed), we will be somehow close to the referenced Walker NCSA cluster configuration (except CPU performance; Amazon's should be faster).
alexey2petrov is offline   Reply With Quote

Old   July 12, 2011, 10:55
Default
  #15
Member
 
Kevin
Join Date: May 2011
Posts: 32
Rep Power: 6
murrdpirate is on a distinguished road
I just wasn't sure if you could link instances other than the Cluster Compute Instance (as Amazon's descriptions seems to say that is the only clusterable instance). I suppose there's a way to make it happen, but the fact that the Cluster Compute Instance was specifically built for this purpose gives me confidence.

And like you said, the 10 Gigabit I/O performance is comparable to non-cloud clusters used for CFD type calculations. I did see that that is comparable to some Infiniband Switch specs, but I didn't know which was typical at the time of Walker's report. To hear that he probably had something around 10 Gigabits is very encouraging. I suppose the latency could be problematic, but again, the fact that this instance is build for HPC applications gives me hope.
murrdpirate is offline   Reply With Quote

Old   July 13, 2011, 23:27
Default
  #16
Member
 
Kevin
Join Date: May 2011
Posts: 32
Rep Power: 6
murrdpirate is on a distinguished road
I don't suppose anyone has an openfoam AMI on US East that I could copy. I'm not familiar enough with openfoam, linux, and amazon AWS to figure out how to do this...it was painful enough just to start running on the EU region.
murrdpirate is offline   Reply With Quote

Old   July 14, 2011, 01:57
Default
  #17
Senior Member
 
Gijsbert Wierink
Join Date: Mar 2009
Posts: 383
Rep Power: 9
gwierink is on a distinguished road
Hi Kevin,

Is this useful perhaps? http://www.openfoam.com/resources/ec2.php
__________________
Regards, Gijs
gwierink is offline   Reply With Quote

Old   July 14, 2011, 10:14
Default
  #18
Member
 
Kevin
Join Date: May 2011
Posts: 32
Rep Power: 6
murrdpirate is on a distinguished road
Hi Gijsbert,

That's actually the tutorial I used to run openfoam on the EU region server, but the best I can do on that server is a single 8-core processor. In order to use the cluster compute instance, I have to be on the US-east region server, so I have to copy the instance from EU to US-east somehow.
murrdpirate is offline   Reply With Quote

Old   July 26, 2011, 09:05
Default
  #19
Member
 
alexey2petrov's Avatar
 
Alexey
Join Date: Feb 2010
Posts: 33
Blog Entries: 1
Rep Power: 8
alexey2petrov is on a distinguished road
Hi Kevin,
Quote:
Originally Posted by murrdpirate View Post
I'm looking to test the Cluster Compute Instance asap.
Here it is - ami-4fad6a26, Amazon EC2 AMI with the pre-installed OpenFOAM-2.0.0 especially dedicated to Cluster Computing Instances ('cc1.4xlarge' type, in Amazon terms). As it was expected, this image could be run in 'us-east-1' region, only.

Important : It is not enough just to run a number of AMI to get a cluster. Configuration of the cluster itself from the given set of machine instances is the next important step. And this requires a lot of Linux experience and accuracy. I could suggest to use cloudFlu library (especially developed for cloud based HPC OpenFOAM calculations), to automate all these steps and make your way to cloud computing really simple.

Enjoy,
Alexey
alexey2petrov is offline   Reply With Quote

Old   January 8, 2013, 06:44
Default Cluster Computing Instance for OF-1.6-ext
  #20
New Member
 
Join Date: Mar 2012
Posts: 23
Rep Power: 5
matthiass is on a distinguished road
Hi Alexey and all the others,

my goal is to do cluster computing on amazon using OpenFOAM-1.6-ext. I found this thread and cloudFlu and am now trying to get it working.

Using cloudflu I only get ami-4fad6a26 working, that was mentioned in the last post.

Am I right that the ami's mentioned in an earlier post by Alaxey for earlier Openfoam versions and the extended project version do not work anymore for the new cluster computing instances (like e.g. cc2.8xlarge).

Did anyone already prepare such an AMI or can give me a bit support in building it up by myself.
Thanks in advance!

Best regards,
Matthias
matthiass is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Superlinear speedup in OpenFOAM 13 msrinath80 OpenFOAM Running, Solving & CFD 18 March 3, 2015 06:36
OpenFOAM on Amazon Elastic Compute Cloud (Amazon EC2) piina OpenFOAM Running, Solving & CFD 13 February 20, 2013 13:01
Modified OpenFOAM Forum Structure and New Mailing-List pete Site News & Announcements 0 June 29, 2009 05:56
64bitrhel5 OF installation instructions mirko OpenFOAM Installation 2 August 12, 2008 18:07
Adventure of fisrst openfoam installation on Ubuntu 710 jussi OpenFOAM Installation 0 April 24, 2008 14:25


All times are GMT -4. The time now is 16:53.