CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums

Notes about running OpenFOAM in parallel

Register Blogs Members List Search Today's Posts Mark Forums Read

Rating: 4 votes, 4.75 average.

Notes about running OpenFOAM in parallel

Posted February 27, 2011 at 10:22 by wyldckat
Updated June 15, 2015 at 15:57 by wyldckat (added more links)

This is a rough list of informations to be compiled and posted on openfoamwiki.net on a page about running OpenFOAM in parallel. If someone picks up on this and wants to start the article in the wiki, be my guest. Leave a comment pointing to the wiki page, so we can cooperate

List of already catalogued applications on the wiki for pre/post processing for parallel mode: http://openfoamwiki.net/index.php/Ca...sing_utilities
Yet to be catalogued:
  • redistributeMesh
  • renumberMesh - useful for optimizing the mesh in each processor for improving cooperative performance.
  • reconstructParMesh

OpenFOAM's script for running applications in parallel: foamJob
OpenFOAM's script with helpful functions, including for running in parallel: "bin/tools/RunFunctions", namely function runParallel

Threads with very useful information:
Also a must read: http://www.openfoam.com/docs/user/ru...s-parallel.php

About NFS and OpenMPI tweaks:
Items to talk about:
  • Intro on how things work
  • Passwordless ssh and possibly rsh
  • NFS and in case the network can't handle NFS
  • .bashrc and foamExec
  • using mpirun app, foamJob script, and RunParallel function
  • Testing operationality
  • Tweaking mpirun, including telling it how to use rsh instead of ssh
  • Running cases and benchmarks
  • Tips on improving performance with OpenFOAM tools.
« Prev     Main     Next »
Total Comments 6

Comments

  1. Old Comment
    Do you have any notes on how to use simple decomposition for parallel processing. I am running on a dual hexacore (12 core) machine, and no matter what I try, my decomposeParDict in sloshingtank2D just wont work, and I get errors something like:
    Decomposing mesh region0

    Create mesh

    Calculating distribution of cells
    Selecting decompositionMethod hierarchical


    --> FOAM FATAL ERROR:
    Wrong number of processor divisions in geomDecomp:
    Number of domains : 12
    Wanted decomposition : (4 2 2)

    From function geomDecomp::geomDecomp(const dictionary& decompositionDict)
    in file geomDecomp/geomDecomp.C at line 50.

    Any suggestions? why wont (6 6 1) work? Isnt that 12 sub domains? I look forward to your reply.
    :
    permalink
    Posted July 17, 2013 at 23:43 by musahossein musahossein is offline
  2. Old Comment
    Quote:
    Originally Posted by musahossein View Comment
    Do you have any notes on how to use simple decomposition for parallel processing. I am running on a dual hexacore (12 core) machine, and no matter what I try, my decomposeParDict in sloshingtank2D just wont work, and I get errors something like:
    Decomposing mesh region0

    Create mesh

    Calculating distribution of cells
    Selecting decompositionMethod hierarchical


    --> FOAM FATAL ERROR:
    Wrong number of processor divisions in geomDecomp:
    Number of domains : 12
    Wanted decomposition : (4 2 2)

    From function geomDecomp::geomDecomp(const dictionary& decompositionDict)
    in file geomDecomp/geomDecomp.C at line 50.

    Any suggestions? why wont (6 6 1) work? Isnt that 12 sub domains? I look forward to your reply.
    :

    Well, 4*2*2 would make 16 Regions. 6*6*1 separations would be 36.
    What would work for 12 domains/processors is (3 2 2).
    The Numbers represent the slides through the Mesh parallel to the related coordinate-axe.
    permalink
    Posted August 16, 2013 at 08:26 by Kenna Kenna is offline
  3. Old Comment
    Tobi's Avatar
    Dear Bruno,

    do you know any Benchmark of using the x,y,z direction and the Splitting of a geometry into different parts like:

    what is faster in a Special case:

    a) decompose in (10 1 1)
    b) decompose in (1 10 1)
    c) decompose in (1 1 10)
    d) decompose in (2 5 1)
    and so on...

    Regards
    Tobi
    permalink
    Posted September 16, 2013 at 06:55 by Tobi Tobi is offline
  4. Old Comment
    Quote:
    what is faster in a Special case:

    a) decompose in (10 1 1)
    b) decompose in (1 10 1)
    c) decompose in (1 1 10)
    d) decompose in (2 5 1)
    and so on...
    You want to use the maximum capacity of your cores (10 are a lot cores ). You need to anticipate where in the mesh the biggest work will appear. Those are mostly the regions with high gradients of pressure, velocity or what ever it calculates(cause of more iterations). For example if you have a straight jet stream going parallel to the x-Axis, you should choose something like this (10 1 1) where the x gets sliced 10 times and every part has a piece of a workload. The program will automatically slice in a manner, that every piece has about the same amount of cells. Consider it when you use refined regions.

    Also if you use ubuntu you can use the command 'topī in a further console to look up the actual use of processor capacity.
    permalink
    Posted September 16, 2013 at 10:27 by Kenna Kenna is offline
  5. Old Comment
    Tobi's Avatar
    Hi Kenna,

    10 cores are a lot? Not really or - I think 100 cores are okay

    Well I know the fact of splitting the mesh in a optimal way so that not only 2 cores get the whole work.

    But I just wanted to know if there are any benchmarks where you can see the effect of a "wrong" splitted mesh.

    Maybe in time increase of 10% or sth. like that
    permalink
    Posted September 16, 2013 at 12:04 by Tobi Tobi is offline
  6. Old Comment
    The decomposition in geomDecomp.C looks something like this:
    nProcessors_!=n_.x()*n_.y()*n_.z().

    So the product cannot exceed the number of cores that you specify. If you are running 2D problems and the mesh is more finer in one direction and coarse in the other direction, then cores in the x direction is 1 and you vary the two others as necessary. In 3d, however, you can utilise the cores in all directions. What I have found useful is to dedicate more cores in the direction where the mesh is fine or varies greatly. Also since the product must equal to or be less than the number of crores available, the user is limited in how they can vary the cores in all directions, since the product of the number or cores in each direction cannot exceed the number of cores available.

    IN other words trial and error would be the norm.
    permalink
    Posted September 16, 2013 at 19:58 by musahossein musahossein is offline
 

All times are GMT -4. The time now is 22:54.