Massive speed penalty when using HPC Pack 2012 Cluster Manager
 

Hi,

I've been using Converge at work for about six months now and starting runs manually from the command line after placing the relevant files on our cluster.

I'm trying to do a factorial study of a device with different BC's and key geometry dimensions and running HPC Pack 2012 R2. I queued up about 27 sims to run over this last weekend and came in on Tuesday to a nasty shock - only two had run, and the remainder were split roughly 50/50 between not started and crashed.

I spent yesterday cleaning things up to the point that I can queue sims that will run, but they do so at a small fraction of our cluster's capacity at ~5%. If I return to running manually then its at ~95%.

Does anyone have any idea how I can avoid this massive performance penalty so we can use the queue system in the future?

Thanks.

Hi Chris,

Thank you for your question.
Can you tell me which MPI you are using? and what is the version?
When your simulation crashes, what is the error message?
Are you trying to run multiple jobs on the same node?
Your IT person may help you out.

Best,

Yunliang

Hi, sorry for the delay, I've been fighting other fires at work.

We're using HP-MPI but I dont know the version.

I fixed the sims manually, just a garden variety CFL problem with a lowest time step being too high.

We have one node dedicated for our use with 60+ cores we use on a regular basis. As far as I can tell, if I run a job through the HPC Pack 2012 scheduler then it only runs on a single core despite the number of cores we specify in the HPC 2012 "Edit Task" command line dialog.

We are trying to run at least two in parallel at any one time, but really what we need is the ability to simply queue up a list of simulations so that the hardware is actually busy.

Hi Chris,

Nice to hear that you figured out the issue with dt_min. Frankly speaking, we don't have many clients who run CONVERGE on Windows. You mentioned that you were using HPMPI. I was wondering if you ever tried MSMPI instead.

Thanks,

Yunliang

Hi Chris,

I just talked to our GUI team and we ever helped a client for a similar issue with HPC. It was the setup issue. Please email me so that we may take a look at your setting remotely and fix the problem.

Thanks,

Yunliang
ywang@convergecfd.com