CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums

Notes about running OpenFOAM in parallel

Register Blogs Community New Posts Updated Threads Search

Rating: 5 votes, 4.80 average.

Notes about running OpenFOAM in parallel

Posted February 27, 2011 at 09:22 by wyldckat
Updated October 31, 2015 at 08:25 by wyldckat (added more links)
This is a rough list of informations to be compiled and posted on openfoamwiki.net on a page about running OpenFOAM in parallel. If someone picks up on this and wants to start the article in the wiki, be my guest. Leave a comment pointing to the wiki page, so we can cooperate

List of already catalogued applications on the wiki for pre/post processing for parallel mode: http://openfoamwiki.net/index.php/Ca...sing_utilities
Yet to be catalogued:
  • redistributeMesh
  • renumberMesh - useful for optimizing the mesh in each processor for improving cooperative performance.
  • reconstructParMesh

OpenFOAM's script for running applications in parallel: foamJob
OpenFOAM's script with helpful functions, including for running in parallel: "bin/tools/RunFunctions", namely function runParallel

Threads with very useful information:
Also a must read: http://www.openfoam.com/docs/user/ru...s-parallel.php

About NFS and OpenMPI tweaks:
Items to talk about:
  • Intro on how things work
  • Passwordless ssh and possibly rsh
  • NFS and in case the network can't handle NFS
  • .bashrc and foamExec
  • using mpirun app, foamJob script, and RunParallel function
  • Testing operationality
  • Tweaking mpirun, including telling it how to use rsh instead of ssh
  • Running cases and benchmarks
  • Tips on improving performance with OpenFOAM tools.
Views 27912 Comments 15 Edit Tags Email Blog Entry
« Prev     Main     Next »
Total Comments 15

Comments

  1. Old Comment
    Do you have any notes on how to use simple decomposition for parallel processing. I am running on a dual hexacore (12 core) machine, and no matter what I try, my decomposeParDict in sloshingtank2D just wont work, and I get errors something like:
    Decomposing mesh region0

    Create mesh

    Calculating distribution of cells
    Selecting decompositionMethod hierarchical


    --> FOAM FATAL ERROR:
    Wrong number of processor divisions in geomDecomp:
    Number of domains : 12
    Wanted decomposition : (4 2 2)

    From function geomDecomp::geomDecomp(const dictionary& decompositionDict)
    in file geomDecomp/geomDecomp.C at line 50.

    Any suggestions? why wont (6 6 1) work? Isnt that 12 sub domains? I look forward to your reply.
    :
    permalink
    Posted July 17, 2013 at 23:43 by musahossein musahossein is offline
  2. Old Comment
    Quote:
    Originally Posted by musahossein View Comment
    Do you have any notes on how to use simple decomposition for parallel processing. I am running on a dual hexacore (12 core) machine, and no matter what I try, my decomposeParDict in sloshingtank2D just wont work, and I get errors something like:
    Decomposing mesh region0

    Create mesh

    Calculating distribution of cells
    Selecting decompositionMethod hierarchical


    --> FOAM FATAL ERROR:
    Wrong number of processor divisions in geomDecomp:
    Number of domains : 12
    Wanted decomposition : (4 2 2)

    From function geomDecomp::geomDecomp(const dictionary& decompositionDict)
    in file geomDecomp/geomDecomp.C at line 50.

    Any suggestions? why wont (6 6 1) work? Isnt that 12 sub domains? I look forward to your reply.
    :

    Well, 4*2*2 would make 16 Regions. 6*6*1 separations would be 36.
    What would work for 12 domains/processors is (3 2 2).
    The Numbers represent the slides through the Mesh parallel to the related coordinate-axe.
    permalink
    Posted August 16, 2013 at 08:26 by Kenna Kenna is offline
  3. Old Comment
    Tobi's Avatar
    Dear Bruno,

    do you know any Benchmark of using the x,y,z direction and the Splitting of a geometry into different parts like:

    what is faster in a Special case:

    a) decompose in (10 1 1)
    b) decompose in (1 10 1)
    c) decompose in (1 1 10)
    d) decompose in (2 5 1)
    and so on...

    Regards
    Tobi
    permalink
    Posted September 16, 2013 at 06:55 by Tobi Tobi is offline
  4. Old Comment
    Quote:
    what is faster in a Special case:

    a) decompose in (10 1 1)
    b) decompose in (1 10 1)
    c) decompose in (1 1 10)
    d) decompose in (2 5 1)
    and so on...
    You want to use the maximum capacity of your cores (10 are a lot cores ). You need to anticipate where in the mesh the biggest work will appear. Those are mostly the regions with high gradients of pressure, velocity or what ever it calculates(cause of more iterations). For example if you have a straight jet stream going parallel to the x-Axis, you should choose something like this (10 1 1) where the x gets sliced 10 times and every part has a piece of a workload. The program will automatically slice in a manner, that every piece has about the same amount of cells. Consider it when you use refined regions.

    Also if you use ubuntu you can use the command 'top´ in a further console to look up the actual use of processor capacity.
    permalink
    Posted September 16, 2013 at 10:27 by Kenna Kenna is offline
  5. Old Comment
    Tobi's Avatar
    Hi Kenna,

    10 cores are a lot? Not really or - I think 100 cores are okay

    Well I know the fact of splitting the mesh in a optimal way so that not only 2 cores get the whole work.

    But I just wanted to know if there are any benchmarks where you can see the effect of a "wrong" splitted mesh.

    Maybe in time increase of 10% or sth. like that
    permalink
    Posted September 16, 2013 at 12:04 by Tobi Tobi is offline
  6. Old Comment
    The decomposition in geomDecomp.C looks something like this:
    nProcessors_!=n_.x()*n_.y()*n_.z().

    So the product cannot exceed the number of cores that you specify. If you are running 2D problems and the mesh is more finer in one direction and coarse in the other direction, then cores in the x direction is 1 and you vary the two others as necessary. In 3d, however, you can utilise the cores in all directions. What I have found useful is to dedicate more cores in the direction where the mesh is fine or varies greatly. Also since the product must equal to or be less than the number of crores available, the user is limited in how they can vary the cores in all directions, since the product of the number or cores in each direction cannot exceed the number of cores available.

    IN other words trial and error would be the norm.
    permalink
    Posted September 16, 2013 at 19:58 by musahossein musahossein is offline
  7. Old Comment
    OK... I've hit the limit of 10000 characters on this blog post... I'm going to have to move this to a wiki page sometime soon... Anyone would like to help?

    By the way, the link for the document by Shannon Keough was sent me by our fellow forum member KateEisenhower: http://www.cfd-online.com/Forums/ope...mulations.html
    permalink
    Posted October 31, 2015 at 08:28 by wyldckat wyldckat is offline
  8. Old Comment
    And another link: http://www.cfd-online.com/Forums/ope...m-2-3-1-a.html - neat solutions for SSH issues with Open-MPI newer than 1.6.5
    permalink
    Posted January 6, 2016 at 17:30 by wyldckat wyldckat is offline
  9. Old Comment
    And another one, this time on PBS jobs: http://www.cfd-online.com/Forums/ope...ble-basis.html
    permalink
    Posted May 1, 2016 at 19:11 by wyldckat wyldckat is offline
  10. Old Comment
    Another one, as a reminder for selecting any other MPI versions based on module load: https://www.cfd-online.com/Forums/op...fic-linux.html
    permalink
    Posted December 22, 2016 at 10:15 by wyldckat wyldckat is offline
  11. Old Comment
    alia's Avatar
    Hi Bruno! I am working with OpenFOAM for three years somehow a newcomer . I would like to participate in documentation of parallelization in OpenFOAM. I'm particularly interested in application of parallel computation in FsiFoam.
    Regards,
    Ali
    permalink
    Posted April 7, 2017 at 12:13 by alia alia is offline
  12. Old Comment
    Hi Ali,

    Quote:
    Originally Posted by alia View Comment
    I'm particularly interested in application of parallel computation in FsiFoam.
    For fsiFoam, you need to contact the current author, Željko Tuković. Because as far as I know, running in parallel is not yet fully implemented, not even in the latest source package for foam-extend 4.0, that is provided here: https://openfoamwiki.net/index.php/E...re_interaction

    If you have figured out how to run those solvers in parallel, feel free to edit that wiki page and add a section for it.

    As for running in parallel in general, feel free to start the wiki page https://openfoamwiki.net/index.php/H...un_In_Parallel

    Best regards,
    Bruno
    permalink
    Posted April 8, 2017 at 07:39 by wyldckat wyldckat is offline
  13. Old Comment
    alia's Avatar
    Quote:
    Originally Posted by wyldckat View Comment
    Hi Ali,


    For fsiFoam, you need to contact the current author, Željko Tuković. Because as far as I know, running in parallel is not yet fully implemented, not even in the latest source package for foam-extend 4.0, that is provided here: https://openfoamwiki.net/index.php/E...re_interaction

    If you have figured out how to run those solvers in parallel, feel free to edit that wiki page and add a section for it.

    As for running in parallel in general, feel free to start the wiki page https://openfoamwiki.net/index.php/H...un_In_Parallel

    Best regards,
    Bruno
    Thank you Bruno!
    permalink
    Posted April 10, 2017 at 06:55 by alia alia is offline
  14. Old Comment
    Yet another solution for the error message:
    Code:
      opal_shmem_base_select failed
  --> Returned value -1 instead of OPAL_SUCCESS
    Is given in the following posts:
    permalink
    Posted June 4, 2017 at 20:14 by wyldckat wyldckat is offline
  15. Old Comment
    Since I've already hit the 10000 characters limit on the main blog post:
    permalink
    Posted April 8, 2018 at 14:45 by wyldckat wyldckat is offline
 

All times are GMT -4. The time now is 14:33.

Contact Us - CFD Online - Privacy Statement - Top