Increase speed of parallel computation

Purushothama · November 29, 2010, 20:16

Dear StarCD colleagues,

I am using starcd 3.26 for fuel cell simulation. We have cluster facility with 48GB server and 8 nodes of 8GB memory each. When I do simulation of large-scale geometry having cells about 4 million with 32 processors, it takes about 4 minutes/iteration. This is too much. I issued the command,

star -dp -mpi=hp -decompmeth=s node0,4 node1,4 etc..

I have also tried with 8 processors, but the time taken per iteration is almost same. I use ufiles and CG solving method.

Is there anyone know how I can reduce simulation time? I have tried with AMG method, but found it is not working well with my ufiles.

TMG · November 30, 2010, 14:03

First of all, you might try running on a version of STAR that's not 10 years out of date. Second, its often the case (not always) that your own user subroutines are at fault. If they don't scale, then the solution certainly won't. Finally, you are forcing the decomposition definition yourself by using sets. If that is not an optimum decomposition, then you will not be able to scale to higher numbers of domains.

Pauli · November 30, 2010, 14:51

"48GB server and 8 nodes of 8GB" ?? My math says 8nodes x 8GB = 64[IMG]file:///home/home-titan/ptroxler/tmp/moz-screenshot.png[/IMG]GB

Are you running on all cores of all nodes? If so, that could also be contributing to lack of scalability. Whether it does or not is problem and cluster configuration dependant.

Have you tried leaving one core "free on each node"? That along with using Metis for decomposition might help performance. That is unless you have a good reason to decompose by sets.

November 29, 2010, 20:16	Increase speed of parallel computation	#1
Purushothama New Member Purushothama Chippar Join Date: May 2010 Posts: 15 Rep Power: 15	Dear StarCD colleagues, I am using starcd 3.26 for fuel cell simulation. We have cluster facility with 48GB server and 8 nodes of 8GB memory each. When I do simulation of large-scale geometry having cells about 4 million with 32 processors, it takes about 4 minutes/iteration. This is too much. I issued the command, star -dp -mpi=hp -decompmeth=s node0,4 node1,4 etc.. I have also tried with 8 processors, but the time taken per iteration is almost same. I use ufiles and CG solving method. Is there anyone know how I can reduce simulation time? I have tried with AMG method, but found it is not working well with my ufiles.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
load balancing in parallel computation	shyamdsundar	Main CFD Forum	0	September 2, 2009 23:53
Errors in parallel computation	ILia Skryabin	FLUENT	1	January 19, 2007 08:20
how to parallel computation	Jane	Siemens	2	April 28, 2004 06:11
Parallel computation problem in Tascflow	dandy	CFX	3	April 21, 2002 00:32
speed up ratio at parallel processing	Kim hak-gyu	Main CFD Forum	1	October 25, 2000 09:57

November 30, 2010, 14:03		#2
TMG Member Join Date: Mar 2009 Posts: 44 Rep Power: 17	First of all, you might try running on a version of STAR that's not 10 years out of date. Second, its often the case (not always) that your own user subroutines are at fault. If they don't scale, then the solution certainly won't. Finally, you are forcing the decomposition definition yourself by using sets. If that is not an optimum decomposition, then you will not be able to scale to higher numbers of domains.

November 30, 2010, 14:51		#3
Pauli Senior Member Pauli Join Date: Mar 2009 Posts: 189 Rep Power: 17	"48GB server and 8 nodes of 8GB" ?? My math says 8nodes x 8GB = 64[IMG]file:///home/home-titan/ptroxler/tmp/moz-screenshot.png[/IMG]GB Are you running on all cores of all nodes? If so, that could also be contributing to lack of scalability. Whether it does or not is problem and cluster configuration dependant. Have you tried leaving one core "free on each node"? That along with using Metis for decomposition might help performance. That is unless you have a good reason to decompose by sets.