[jzhen]

I'm trying to run a case in parallel on a cluster using my qsub script:

#PBS -N SU2_inlet_test
#PBS -q default
#PBS -S /bin/bash
#PBS -l nodes=compute-0-0: ppn=16

# Location of Case
CASEDIR=$PBS_O_WORKDIR

# Scratch directory
SCRATCH=/tmp/jzhen/SU2_inlet/completed

#make directory in scratch
mkdir -p ${SCRATCH}

# Change directories to Location of Case
cd ${CASEDIR}

# Copy files to the scratch directory
cp * ${SCRATCH}

# Change directories to scratch directory (must exist)
cd ${SCRATCH}

# Clear out any files from previous runs
rm -fr *.dat *.csv *.png

pwd

# MPIRun of SU2_CFD
mpirun -n 8 SU2_CFD inlet.cfg &>z.out


# Copy files back to original directory

cp -r ${SCRATCH} ${CASEDIR}

When Ie execute this, I get the error message:

mpirun noticed that process rank 0 with PID 16581 on node compute-0-0.local exited on signal 9 (Killed).


Does anyone experience this?

[RcktMan77]

Have you partitioned the mesh already? Typically I would invoke SU2 using their parallel_computation.py wrapper python script. In order to do so using a cluster environment with PBS/Torque schedulers you may need to modify $SU2_HOME/SU2_PY/SU2/run/interface.py. On line 58 of this file you can modify the unique mpirun command that gets executed when the parallel_computation.py wrapper is invoked. For PBS/Torque with OpenMPI, you could modify this line to read:

mpi_Command = 'mpirun --hostfile ${PBS_NODEFILE} -np %i %s'

Although, with OpenMPI the library should automatically get a lists of hosts from the scheduler, so this example isn't very instructive. However, you can adapt this to suit the specific MPI library used to compile SU2 on your cluster environment. For MPICH as another example, you would modify this line to read:

mpi_Command = 'mpirun -f ${PBS_NODEFILE} -n %i %s'

[CapSizer]

Is there a simple series of command line actions that one can reliably take instead of the parallel python script? The problem with the python script is that it does a bunch of stuff before actually getting around to launching the mpirun command. In a cluster environment, you don't want to do any of that on the login node, so one would typically make sure that the decomposition and merging steps are done on a compute node. Basically, what I'm after is the commands for:

decompose

mpirun solver

merge

sort out the output stuff

Most of this is easy, but it looks like the odd one is the decomposition command. The guide on the website says that SU2_PRT is used for this ... but I don't find a binary for that in the current version. Any ideas?

[jzhen]

RcktMan77: I tried your suggestion of using the python script with the modification of: mpi_Command = mpirun -f ${PBS_NODEFILE} -n %i %s. I launched an interactive PBS job using qsub -I -l nodes=1pn=8 and did a cat $PBS_NODEFILE to make sure I had the nodes, and got a similar error.

I wasn't sure if this problem was specific to my *.cfg and mesh files or my PBS setup, so I used the same PBS process on the Onera tutorial case, and that worked! I'm led to believe there is some issue with my mesh (maybe it's a memory problem?) or my *.cfg file, and not with my PBS setup itself. I remade a mesh in Pointwise that was axisymmetric, and was able to get that to work.

CapSizer: I didn’t run the decomposition command (SU2_PRT) because I’ve come to understand that SU2_CFD has built-in the partitioning step. That's why it is not included in the version of SU2 I have installed (v 3.2.8). The command-line option you can use is just: mpirun -n NP SU2_CFD your_config_file.cfg where NP is the number of processors.

[RcktMan77]

The command(s) that are included in your scheduler's submission script should only run on the compute nodes which are reserved for the job. Thus, anything that is run via the the Python parallel_computation.py script will only be run on those nodes reserved for the job. You shouldn't need to be concerned that some of the components that are run via the Python script will be run on the node from where the job was submitted.

That being said, I'm not sure what they did with the SU2_PRT C++ module as I don't have it in my installation either.

[RcktMan77]

Originally Posted by jzhen

CapSizer: I didn’t run the decomposition command (SU2_PRT) because I’ve come to understand that SU2_CFD has built-in the partitioning step. That's why it is not included in the version of SU2 I have installed (v 3.2.8). The command-line option you can use is just: mpirun -n NP SU2_CFD your_config_file.cfg where NP is the number of processors.

Thanks for clarifying that, jzhen. CapSizer left me wondering how partitioning was being handled in 3.2.8. I hadn't even noticed it had gone missing.

[CapSizer]

Got it figured out OK, example PBS script is up at http://wiki.chpc.ac.za/howto:su2 . Thanks for the advice!

[jzhen]

I'm running into problems again with running my case in parallel using PBS. I'm using the parallel_computation.py script, and get the following error message:

---------------------- Read Grid File Information -----------------------
Three dimensional problem.
8341318 interior elements.
8124838 tetrahedra.
179400 hexahedra.
37080 pyramids.
1645805 points.
8124838 tetrahedra.
179400 hexahedra.
37080 pyramids.
1645805 points.
8124838 tetrahedra.
179400 hexahedra.
37080 pyramids.
1645805 points.
8124838 tetrahedra.
179400 hexahedra.
37080 pyramids.
1645805 points.
5 surface markers.
16424 boundary elements in index 0 (Marker = box).
5 surface markers.
16424 boundary elements in index 0 (Marker = box).
5 surface markers.
16424 boundary elements in index 0 (Marker = box).
3092 boundary elements in index 1 (Marker = inflow).
3092 boundary elements in index 1 (Marker = inflow).
3092 boundary elements in index 1 (Marker = inflow).
3994 boundary elements in index 2 (Marker = outflow).
3994 boundary elements in index 2 (Marker = outflow).
3994 boundary elements in index 2 (Marker = outflow).
254470 boundary elements in index 3 (Marker = symmetry).
254470 boundary elements in index 3 (Marker = symmetry).
254470 boundary elements in index 3 (Marker = symmetry).
5 surface markers.
16424 boundary elements in index 0 (Marker = box).
3092 boundary elements in index 1 (Marker = inflow).
3994 boundary elements in index 2 (Marker = outflow).
254470 boundary elements in index 3 (Marker = symmetry).
74060 boundary elements in index 4 (Marker = walls).
74060 boundary elements in index 4 (Marker = walls).
74060 boundary elements in index 4 (Marker = walls).
74060 boundary elements in index 4 (Marker = walls).
Traceback (most recent call last):
File "/home/jzhen/SU2/SU2_Install/bin/parallel_computation.py", line 110, in <module>
main()
File "/home/jzhen/SU2/SU2_Install/bin/parallel_computation.py", line 61, in main
options.compute )
File "/home/jzhen/SU2/SU2_Install/bin/parallel_computation.py", line 88, in parallel_computation
info = SU2.run.CFD(config)
File "/home/jzhen/SU2/SU2_Install/bin/SU2/run/interface.py", line 88, in CFD
run_command( the_Command )
File "/home/jzhen/SU2/SU2_Install/bin/SU2/run/interface.py", line 276, in run_command
raise exception , message
RuntimeError: Path = /tmp/jzhen/scratch/WD001/,
Command = mpirun -n 4 /home/jzhen/SU2/SU2_Install/bin/SU2_CFD config_CFD.cfg
SU2 process returned error '137'


I don't know what the Runtime Error means. Any thoughts?

[RcktMan77]

@jzhen Which MPI library did you compile SU2 with?

[jzhen]

I'm using OpenMPI v 1.6.2

[RcktMan77]

The command which you edited in the interface.py file isn't exactly consistent with OpenMPI. You will want to ensure that your OpenMPI library was installed with Torque/PBS scheduler support--it's an option you can pass in during the configure step. Otherwise you need to pass in a machinefile to mpirun. Additionally, with OpenMPI you specify the number of MPI ranks using the -np flag, not -n as you have done. The -n flag would be appropriate for MPICH--another MPI library. Thus, to be safe you could edit your interface.py file as follows:


mpi_Command = 'mpirun --hostfile ${PBS_NODEFILE} -np %i%s'

If you're certain that your OpenMPI library was built with Torque/PBS scheduler support, then the --hostfile flag and it's ${PBS_NODEFILE} argument shouldn't be required. I hope this helps!

[NKever]

Edit: Asked in a different post Parallelizing SU2 in PBS


Hi,

I am trying to run the shape_optimization.py script and use all my available nodes in PBS and I don't think I am doing this right.

Here is my script:

#PBS -q devel

##PBS -q long

##PBS -q normal

#PBS -W group_list=a1489

#PBS -lselect=50:ncpus=28:model=bro,walltime=02:00:00

##PBS -lselect=80:ncpus=28:model=bro,walltime=08:00:00

##PBS -lselect=80:ncpus=28:model=bro,walltime=50:00:00

#PBS -N test_su2

#

# Request that regular output and terminal output go to the same file

#PBS -j oe

#PBS -m n

# go to where the qsub was performed.

cd $PBS_O_WORKDIR

# Running Program

export SU2_RUN="/home6/nkever/su2/install/SU2/bin"

export SU2_HOME="/home6/nkever/su2/src/SU2"

export PATH=$PATH:$SU2_RUN

export PYTHONPATH=$PYTHONPATH:$SU2_RUN

module load mpi-sgi/mpt

module load python/2.7.15

module load comp-intel/2018.0.128

./shape_optimization.py -g CONTINUOUS_ADJOINT -o SLSQP -f inv_NACA0012_basic.cfg -n 400

echo "finished mpirun"

So I think the issue may be the -n, my interpretation is that the number of partitions tells su2 the amount of nodes. How do I use all my nodes?