[CapSizer]

I've been using cfmesh to build a mesh for the Drag Prediction Workshop model. It's a reasonably large model, around 27 million cells, and I thought that it would make a handy SMP benchmarking test case, given how nicely it runs in parallel with OpenMP. The workstation I use for this is a dual socket Xeon, with 6 cores per socket, and hyper-theading turned on, therefore the default number of threads is 24. And it creates the mesh very satisfactorily indeed.

However, to get a feel for the parallel scaling, I tried it with a smaller number of threads, by setting the environment variable OMP_NUM_THREADS. Now this is where it starts to get a bit strange. 24 threads, OK. 23 threads OK. 22 threads OK. 21 threads OK. Anything smaller than 21, it get stuck on the "Refining boundary boxes to the given size" phase, and then segfaults.

Has anybody else seen anything like this?

[franjo_j]

What operating system are you running on? How much memory have you got, and what percentage of memory is used for meshing?
I have noticed that changing the value of OMP_NUM_THREADS often can cause strange behaviour and crashes from time to time. So far, I could not figure the pattern when that happens.
Can you please try to compile cfMesh in debug mode and let me know if you notice any problems that occur when you start changing the value of OMP_NUM_THREADS?

Regards,

Franjo

[CapSizer]

Hi Franjo, the O/S is Fedora 19, the system has 128 GB of DDR 1333/ECC. Cfmesh is compiled with an OpenFOAM-2.4.0 environment. When it meshes successfully (with 24 threads), the memory usage seems to stay quite low, around 12 GB or so. When I set the number of threads lower (say 12), it dies quite quickly, while on the " Refining boundary boxes to the given size" step. But it spikes to a higher memory usage, top reports VIRT around 13 GB and RES around 8 GB. I will try to compile it in debug mode and report what I find.

On a different topic, I would like to take the mesh into SU2, which means converting to either CGNS or SU2 format. Any advice on the best way to do that?

Charles

[franjo_j]

Thanks, please let me know if your notice anything strange.
Regarding the SU2, the code uses only a few standard cell types. You can meet those requirements with tetMesh. cartesianMesh is also an option when you use decomposePolyhedraIntoTetsAndPyrs setting in meshDict. Once you are sure that your meshes can be converted, writing a converter itself shall not be a problem. Feel free to contact me if you need assistance with it.

Regards,

Franjo

[cfdFreshman]

Hello to all,

allthough this thread seems not to be active I hope someone answers.

I would like to know how to set the OMP_NUM_THREADS environment variable. It should be a fix value of processors so it can not cause any problems to other running calculations.
I read a article of IBM (https://www-01.ibm.com/support/knowl...1/ruomprun.htm) which explains lot of options to OMP, but don't know if it is possible with cfMesh.
Can you help?

ThanX!

[franjo_j]

Hi,

Originally Posted by cfdFreshman

Hello to all,

allthough this thread seems not to be active I hope someone answers.

I would like to know how to set the OMP_NUM_THREADS environment variable. It should be a fix value of processors so it can not cause any problems to other running calculations.
I read a article of IBM (https://www-01.ibm.com/support/knowl...1/ruomprun.htm) which explains lot of options to OMP, but don't know if it is possible with cfMesh.
Can you help?

ThanX!

These options control the behaviour of openmp used by cfMesh, and all of them apply to cfMesh, too.
You can set OMP_NUM_THREADS as follows:
bash shell: export OMP_NUM_THREADS=<num threads>, <num threads in nested loops>
csh shell: setenv OMP_NUM_THREADS <num threads>, <num threads in nested loops>

However, some algorithms in cfMesh use more threads than specified above, and it last only for a short period of time. It does not do harm to running jobs unless the system runs out of memory.

Regards,

Franjo

[cfdFreshman]

Originally Posted by franjo_j

Hi,



These options control the behaviour of openmp used by cfMesh, and all of them apply to cfMesh, too.
You can set OMP_NUM_THREADS as follows:
bash shell: export OMP_NUM_THREADS=<num threads>, <num threads in nested loops>
csh shell: setenv OMP_NUM_THREADS <num threads>, <num threads in nested loops>

However, some algorithms in cfMesh use more threads than specified above, and it last only for a short period of time. It does not do harm to running jobs unless the system runs out of memory.

Regards,

Franjo

Hello,

somehow it doesn't work anyway. The computer I use has 20 cores and I want it to use f.e. 8 of them for the whole calculation. So I type in my bash script:
export OMP_NUM_THREADS=8,8
runApplication ...Mesh
... and execute it.
If I look at the system monitor it seems to still use all processors. Viewed with "top" command in terminal it is executed just with one.

Regards,
cfdFreshman

[franjo_j]

Hi,

Originally Posted by cfdFreshman

Hello,

somehow it doesn't work anyway. The computer I use has 20 cores and I want it to use f.e. 8 of them for the whole calculation. So I type in my bash script:
export OMP_NUM_THREADS=8,8
runApplication ...Mesh
... and execute it.
If I look at the system monitor it seems to still use all processors. Viewed with "top" command in terminal it is executed just with one.

Regards,
cfdFreshman

Please try setting the variable in a shell, before you start your Allrun script. It is expected to see a single process only with process usage above 100%. For example, 400% means that you case is using 4 cores.

Regards,

Franjo