[askjak]

Hi,

I am interested in doing a lot of parallel postprocessing. I have noticed problems when using patchIntegrate and patchAverage on a cyclic patch.

I have compared results from a parallel case and a seriel case. On a non-cyclic patch I get identical results. But the results are not correct for cyclic patches. The areas of patches are not even identical:

Serial:

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  1.6                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 1.6-f802ff2d6c5a
Exec   : patchIntegrate p port17 -latestTime
Date   : Dec 17 2009
Time   : 13:47:12
Host   : c1-045
PID    : 10180
Case   : /home/asfj/OpenFOAM/asfj-1.5/run/linerStudy/noScaleAlt-refineCurvatureInport-poly
nProcs : 1
SigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 4000

Time = 4000
    Cyclic patch vector area: 
    - half 1 = (0.0015018 4.07798e-08 0.00702065), 0.00717948
    - half 2 = (-0.0015018 -4.07798e-08 -0.00702065), 0.00717948
    - total  = (0 7.27919e-22 0), 7.27919e-22
    Cyclic patch area magnitude = 0.00718769
    Reading volScalarField p
    Integral of p over vector area of patch port17[26] = (-2.01271e-15 1.14604e-18 7.84746e-16)
    Integral of p over area magnitude of patch port17[26] = 10.1629

End

Parallel:

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  1.6                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 1.6-f802ff2d6c5a
Exec   : patchIntegrate p port17 -latestTime -parallel
Date   : Dec 17 2009
Time   : 13:46:20
Host   : c1-045
PID    : 10110
Case   : /home/asfj/OpenFOAM/asfj-1.5/run/linerStudy/noScaleAlt-refineCurvatureInport-poly
nProcs : 8
Slaves : 
7
(
c1-046.10017
c1-049.9888
c1-050.10134
c1-008.23707
c1-006.22837
c1-014.17367
c1-018.13667
)

Pstream initialized with:
    floatTransfer     : 0
    nProcsSimpleSum   : 0
    commsType         : nonBlocking
SigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 4000

Time = 4000
    Cyclic patch vector area: 
    - half 1 = (0.00139876 5.13032e-08 0.00665429), 0.00679971
    - half 2 = (-0.00139876 -5.13032e-08 -0.00665429), 0.00679971
    - total  = (4.33681e-19 3.63959e-22 0), 4.33681e-19
    Cyclic patch area magnitude = 0.00680708
    Reading volScalarField p
    Integral of p over vector area of patch port17[26] = (-1.10697e-16 -8.47033e-21 -3.79818e-15)
    Integral of p over area magnitude of patch port17[26] = 9.48489

End

Finalising parallel run

I have looked at the code for patchAverage and patchIntegrate and it looks correct. It uses "reduce" and everything should work. Help would be greatly appreciated.

Best regards, Ask

[askjak]

I have now also downloaded and compiled the newest version of 1.6.x and the problems persists.

It appears to be a real bug and also quite troublesome since it does not fail in an obvious way but reports numbers that (in this case) is 5-10% wrong.

BTW: When an application has the option "-parallel" can one then safely assume that the application works in parallel?

[henry]

The problem you are seeing is a consequence of cyclic patches transforming either partially or completely into processor patches during decomposition. In fact for efficiency reasons the sets of cyclic faces which require parallel communications are spliced into the sets of "normal" processor faces so that the transfers are combined into a single block. This means that integrating over cyclic patches when running in parallel is VERY complicated and requires looping over the corresponding processor patches and finding which faces correspond to which cyclic patches.

We are considering re-organising the way cyclic patches are handled, splitting each into two, one for either side, and maintaining separate processor patches for each as required. This will make integrating over them in parallel slightly easier as it would simply be a matter of handling the residual cyclic patch and the corresponding processor patch as required.

In the meantime it is more reliable to create a faceZone for the set of faces you want to integrate over rather that integrating over the cyclic patch directly.

H

[askjak]

Thanks. That makes sense computationally.

Since the patchAverage/patchIntegrate takes steps to support cyclic patches and runs in parallel mode should there not be thrown an error in the case cyclic && parallel?

Any suggestions on how to define faceZones internally in a (complex) cylindrical geometry? I need to analyze on many small patches that are flush with the cylinder walls.

[henry]

> Since the patchAverage/patchIntegrate takes steps to support cyclic patches and runs in parallel mode should there not be thrown an error in the case cyclic && parallel?

Possibly, but under some conditions it will work correctly but unfortunately those conditions depend on the decomposition. We will discuss if we should add such an error/warning message.

> Any suggestions on how to define faceZones internally in a (complex) cylindrical geometry? I need to analyze on many small patches that are flush with the cylinder walls.

You can use faceSet or setSet and then setsToZones for this purpose.

H

[askjak]

I have now succeed creating a faceZone that I want to do my analysis on. I have tried to find an example as a template. So far the best guess is the functionObject faceZoneIntegration, but it appears to have been removed in 1.6.x.

Can anyone point me to a tool that does postprocessing on a faceZone so I can get started?

Ask

[henry]

We have enhanced the functionality in the OpenFOAM-1.6.x/src/postProcessing/functionObjects/field functionObjects to include support for cellZones and faceZones which superceds faceZoneIntegration and renders it redundant so we removed it.

H

[askjak]

Great. I will have a look at it.

I guess this cyclic patch issue could be mitigated in the decomposing step by adding a "preservePatches" in decomposeParDict

//- Keep owner and neighbour on same processor for faces in patches:
// (makes sense only for cyclic patches)
preservePatches (port01);

Ask

[henry]

Yes using "preservePatches" should help.

H

[tstovall]

Hi,

I am trying to model a box with two cyclic boundaries. The front and back of the box are coupled with one cyclic patch, and the left and right are coupled with another cyclic patch. The other two faces (top and bottom) are generic patches. The simulation will run on a single processor, but when I decompose the domain and try to run with multiple processors, the solution explodes. I suspect the issue is with the two cyclic patches. I've tried cutting the domain in different orientations, but this does not alleviate the issue. Does anyone have any help/suggestions for running this case in parallel?

Thanks,

Tim

[mattijs]

You could use the preservePatches option in the decomposeParDict (see the sample library in $FOAM_UTILITIES)

What solver are you running? There should be no problem running standard solvers with decomposed cyclics.

[tstovall]

I'm using the preservePatches option in decomposePar and verified all the faces of the cyclic patches are present in the processor directories. I'm using pisoFoam with LES and the oneEqEddy SGS model with standard coefficients.

After some further review, I've isolated the problem. The domain consists of two cyclic conditions for the four vertical boundaries, each paired with the opposite surface. The top boundary is a symmetryPlane. The bottom surface is a wall. The simulation will run with these conditions in parallel with the domain only cut in the X (flow direction). When I switch the nuSgs bottom wall condition from zeroGradient to nuSgsWallFunction, the solution fails to converge. Each momentum solver maxes out at 1001 itterations and the courant number blows up by the second time step.

I've run another simulation with the sides are cyclic, inlet is directMapped, outlet is inletOutlet, top is symmetryPlane, and bottom is wall with nuSgsWallFunction. I'm curious why changing the inlet/outlet from directMapped/inletOutlet to a coupled cyclic patch would cause an error.

Any ideas why the wall function would cause this?

[mattijs]

Can you post a simple testcase which exhibits the problem (run non-parallel, not run in parallel)?

[tstovall]

Mattijs,

I tried scaling down the case so reproduce the error for a simple test case, but the error does not occur for the smaller domain. I found that having a smaller domain prevents the CFL from maxing at the early time steps. With the resized domain, the solutions converge. Thanks for your help. Attached is a log file from one of the bad runs (16 processors) and the same run one a single processor.

-Tim

log-single.txt

log-16proc.txt

[mattijs]

Can you recreate the case with a blockMesh so we can recreate the case over here?

[tstovall]

Mattijs,

Here's the case file. Included are all the log files from the steps I took to build and decompose the case. The case was decomposed on a 1.6.x build, but the cluster for parallel and single processor runs only has a 1.6 build. The log-pisoFoamParallel shows the iterations maxing for the momentum equations that does not occur for the same case run on a single processor shown in log-pisoFoamSingle.

DualCyclics.tar.gz

Thanks,

Tim

[mattijs]

Hi Tim,

I just tried on 16 processors, 1.6.x and first time step seems to run ok - slightly more iterations on the solvers as expected but no maxing out at 1001. Attached my log. Can you try 16x?

[mchurchf]

Hello,

I am using OpenFOAM 1.6. I am running 2D case where the flow is from left to right and the inflow and outflow are cyclic boundaries. A schematic is below where the vertical lines are the cyclic boundary a.

PHP Code:

    -->
|         |
|         |
|_________|
a         a 


In order to keep the corresponding cyclic faces on single processors when running in parallel, I've been resorting to splitting the domain into, say 3, vertically stacked zones like this

PHP Code:

           -->
|_________| 0
|_________| 1
|_________| 2
a         a 


The larger the grid gets in the horizontal (especially when going to 3D) and the more processors I use this becomes unreasonable because I will have a stack of very thin pieces of grid alotted to each processor.

As an experiment, I have tried using preservePatches in the decomposeParDict like:

numberOfSubdomains 3;
method simple;
preservePatches (a);

simpleCoeffs
{
n (3 1 1);
delta 0.001;
}

Hoping that the domain would be divided like

PHP Code:

-->
 0 1  2  0
|_|__|__|_|
|_|__|__|_|
|_|__|__|_|
a         a 


So that the cyclic patch a is processed by processor0 only. Instead the result is

PHP Code:

-->
 0  1  2  
|__|__|__|
|__|__|__|
|__|__|__|
a        a 


so that corresponding faces of the cyclic patch a are divided between two processors. Is there a way to make the situation desired occur?

Thank you,

Matt

[maddalena]

Originally Posted by henry

You can use faceSet or setSet and then setsToZones for this purpose

Hi everyone,
I have faced the same problem when decomposing a domain with cyclics.
In order to use the preserveFace option, I must define faceSet and then setsToZones, as explained above. In the faceSetDict I defined:

name Cy01;
action new;
topoSetSources
(
    patchToFace
    {
        name "Cy01"; 
    }
);

Then setsToZones and decomposePar. That worked.
Now my question: I have 24 cyclics and changing and running 24 times faceSet is annoying... Is there any fast method to define 24 faceSet basing on the 24 "Cy.*" patches?
Thanks

mad