[RL-S]

Hey there,

I found a couple of old threads about the manual option in decomposeParDict, and how to conveniently prepare the file for it with the setFields utility:

DecomposePar utility
Manually divide sub-domain for parallel computing

It was possible to piece the answer together from these threads, but it took me a couple of hours, so I wanted to share my solution here. There might be more elegant ways and I'm open to suggestions, but it works and it's fast enough.
First, I'll explain the problem, then I'll describe my solution.

The problem:

The manual option in the decomposeParDict file offers a lot of versatility, but is difficult to use, because you have to provide a file with a labelList the size of your mesh cell count. Each entry signifies the processor that cell is on. The exact file formatting is not completely trivial to find in the docs, but that's doable. However, actually creating that file manually doesn't really make sense, especially for larger meshes.
While the setFields utility provides a very convenient way of writing that list, its input- and output formatting is slightly different to the one for the needed file.

My solution:

Make sure your mesh is created and your system/decomposeParDict looks something like this:

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  4.x                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    note        "mesh decomposition control dictionary";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains  2;

method          scotch;

scotchCoeffs
{
}

manualCoeffs
{
    dataFile    "cellDist";
}

// ************************************************************************* //

Then open a terminal in your case directory and get your current decomposition file by executing

decomposePar -cellDist

That will write a file called cellDist in 0/.

Now we need to write our desired decomposition into the system/setFieldsDict. Just as an example, it can be sth like this:

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  1.7.1                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      setFieldsDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

defaultFieldValues ( volScalarFieldValue cellDist 0 );

regions 
(
      boxToCell 
      {
           box ( 0.5 0 0 ) ( 1 0.5 1 ) ;
           fieldValues ( volScalarFieldValue cellDist 1); 
      }
      boxToCell 
      {
           box ( 0 0.5 0 ) ( 0.5 1 1 ) ;
           fieldValues ( volScalarFieldValue cellDist 2); 
      }
      boxToCell 
      {
           box ( 0.5 0.5 0 ) ( 1 1 1 ) ;
           fieldValues ( volScalarFieldValue cellDist 3); 
      }
);


// ************************************************************************* //

For a cube with an edge length of 1, this would cut it into four regions of equal size. That's really just an example, and you'll know better what suits your case. In my case, I needed to put boundary patches on opposite sides of the mesh on the same processor, and that was easier with this method than with the others.

Now go back to your terminal and execute

setFields ; touch prepDecomp.py

This will rewrite your file cellDist in 0/ according to your desired decomposition. It also creates an empty file called prepDecomp.py (you may change the name if you want), which we open and paste the following into:

#read file
with open("0/cellDist", "r") as cdFile:
    lines = cdFile.readlines()

#replace and delete unwanted lines
lines[11] = lines[11].replace("volScalarField", "labelList")
lines[12] = lines[12].replace("0", "constant")
del lines[17:21]
bfline = 0
for i, line in enumerate(lines):
    if (line.find("boundaryField") != -1):
        bfline = i
        break
del lines[i:]

#write file
with open("constant/cellDist", "w") as cdFile:
    for line in lines:
        cdFile.write(line)

Now we execute that python script through the terminal with

python prepDecomp.py

This will modify the file according to the needed input format for the decomposePar utility.

Next, we set the keyword method in system/decomposeParDict to manual, and we're good to go.

Your workflow for designing your preferred decomposition may be this:

1. Edit setFieldsDict
2. Execute

   rm -r processor* ; setFields ; python prepDecomp.py ; decomposePar

3. Check the cell and boundary face counts and see whether it looks good. If yes, you may run your solver, if not, start over from 1.

I needed this, because I used a codedFixedValue BC that accessed a patch on another processor core. This way, I could just place them on the same core, at least as a stopgap solution.


Hope this might help someone.


Lennart

Edit: I didn't try this method with binary format, so if you run into problems, make sure the writeFormat keyword in system/controlDict is set to ascii for the whole process. That way you can at least see what's happening.