CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > CFX

Cluster setup for FSI

Register Blogs Members List Search Today's Posts Mark Forums Read

Reply
 
LinkBack Thread Tools Display Modes
Old   October 27, 2009, 12:06
Default Cluster setup for FSI
  #1
Senior Member
 
Lance
Join Date: Mar 2009
Posts: 522
Rep Power: 11
Lance is on a distinguished road
Hi,
Im trying to get our linux cluster to understand that it needs to run the Ansys mechanical part of my 2-way FSI on eight cores, and not just one core as it does now.

My start script looks like:

#SBATCH -N5 -n40

deffile=deffile.def
inpfile=inpfile.dat
inifile=restartfile.res
name=my_name

cfx5solve -def $deffile -ansys-input $inpfile -ansys-arguments "-np 8" -name $name -initial-file $inifile -start-method "HP MPI Distributed Parallel" -par-dist $(hostlist -e -s, -a'*8' $SLURM_NODELIST)

which makes the fluid solver run on 5 nodes with 40 cores, and the "-np 8" makes the solid solver to run on 1 node with 8 cores with shared memory (I assume??). But the solid solver's performance is really slow and Im suspecting that it only runs on one core.

Im running the simulation sequentially, starting with solid solver. Fluid model is pure hex mesh with ~3M cells, solid mesh is pure hex with ~0.4M cells and the memory of each node is 32 GB so I guess that that shouldnt be a problem. There are no large displacements in the beginning of the simulation and so far I've only managed to do 2 time steps, which took 15 WC-hours...

1) Is there a way to see if the solid solver only runs on one core? There's nothing in the ANSYS.stdout file that indicates that, except the poor/slow performance.

2) Could someone give an example of their own 2-way fsi start script?

Any suggestion/input would be very helpful!

regards
Lance
Lance is offline   Reply With Quote

Old   October 27, 2009, 14:48
Default
  #2
Senior Member
 
Join Date: Apr 2009
Posts: 513
Rep Power: 12
stumpy is on a distinguished road
Near the top of the ANSYS.stdout file it would indicate it was using 8 processors. If you don't see anything, then it's likely running seriel (you can run "top" to see how many processors are in use when ANSYS is solving). Are you using v11 or v12? In v12 "-np 8" should work OK. In v11 this doesn't work (bug). You need to use
' -np 8'
exactly as shown. Note the space after the first single quote.
stumpy is offline   Reply With Quote

Old   October 28, 2009, 03:33
Default
  #3
Senior Member
 
Lance
Join Date: Mar 2009
Posts: 522
Rep Power: 11
Lance is on a distinguished road
Im using V12 and in the top of ANSYS.stdout is says:
8 PARALLEL CPUS REQUESTED.
Thats all the info I get about wether it runs serial or parallel. I guess it could be that the solver is in parallel mode but there's something else thats wrong...? I have HPC license btw.

There's an example in xmgr.pdf that shows how to start a multi-field run and only launching CFX and telling it to communicate with the already running mechanical solver. Is there anyone that uses that start method? How do I get the mechanical solver to start and wait for input from CFX?

/L
Lance is offline   Reply With Quote

Old   October 29, 2009, 10:30
Default
  #4
Senior Member
 
Lance
Join Date: Mar 2009
Posts: 522
Rep Power: 11
Lance is on a distinguished road
Ok, a small update:
I started Ansys alone on one node with

cfx5solve -def $deffile -ansys-input $inpfile -mfx-run-mode "Start ANSYS only" -ansys-arguments "-np 8"

and then started CFX on another node and connected it to Ansys with

cfx5solve -def $deffile -initial-file $inifile -mfx-run-mode "Start CFX only" -start-method "HP MPI Distributed Parallel" -par-dist $(hostlist -e -s, -a'*8' $SLURM_NODELIST) -cplg-host #####@n### -cplg-slave CFX

which almost worked out fine. The simulation starts, and CFX runs on 8 cores with 100 % CPU load each, but Ansys still runs on one node and with 800% CPU load!

So I guess that Ansys makes 8 partitions on one core instead of 8 partitions on 8 cores... and I have no idea on how to fix this.

Any suggestions?

Lance
Lance is offline   Reply With Quote

Old   October 29, 2009, 14:07
Default
  #5
Senior Member
 
Join Date: Apr 2009
Posts: 513
Rep Power: 12
stumpy is on a distinguished road
Ansys can only run in local parallel for FSI. So your requesting, and getting, 8 partitions. Therefore you'll need 8 cores on that machine. I think I've seen the 800% thing before - this might just be a quirk of how the parallel process is reported. If it's running faster than in serial then everything might be working OK.
stumpy is offline   Reply With Quote

Old   October 30, 2009, 03:13
Default
  #6
Senior Member
 
Lance
Join Date: Mar 2009
Posts: 522
Rep Power: 11
Lance is on a distinguished road
I do have 8 cores on that machine, it's two quad-cores connected on one compute node. That should work for shared memory I guess?

The thing is that the parallel simulation is running as fast as the serial (or as slow I should say), so Im still thinking that the 800% CPU load is the error here...
Lance is offline   Reply With Quote

Old   October 30, 2009, 10:15
Default
  #7
Senior Member
 
Join Date: Apr 2009
Posts: 513
Rep Power: 12
stumpy is on a distinguished road
Yes, that sounds fine for shared memory. From what's you've said I suspect that ANSYS is indeed running on 8 cores. The message in the stdout file and the 800% CPU load is consistent with what I've seen, except that I get some speed up. Double-check that no other processes are running on those nodes too. So I think the question is why aren't you getting any speed up. I would run in serial, then on 2 partitions and compare. Perhaps performance is dropping off with 8 partitions due to overhead? Make sure you are only comparing the time ANSYS is taking. You could also try a structual-only run to see if this gets speed up. If not, then I'm no structural expect, so I would contact ANSYS support.
stumpy is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
setup windows cluster radiatn STAR-CD 2 March 30, 2009 13:52
setup vof test problem Fang Jin FLUENT 1 June 14, 2005 08:27
Linux Cluster Performance with a bi-processor PC M. FLUENT 1 April 22, 2005 09:25
Linux Cluster Setup Problems Bob CFX 1 October 3, 2002 18:08
To Jonas: Linux Cluster Mark Render Main CFD Forum 1 June 20, 2002 04:19


All times are GMT -4. The time now is 06:10.