CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Bugs

Cyclic patches and parallel postprocessing problems

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree2Likes
  • 2 Post By askjak

Reply
 
LinkBack Thread Tools Display Modes
Old   December 17, 2009, 09:02
Default Cyclic patches and parallel postprocessing problems
  #1
New Member
 
Join Date: Apr 2009
Posts: 26
Rep Power: 8
askjak is on a distinguished road
Hi,

I am interested in doing a lot of parallel postprocessing. I have noticed problems when using patchIntegrate and patchAverage on a cyclic patch.

I have compared results from a parallel case and a seriel case. On a non-cyclic patch I get identical results. But the results are not correct for cyclic patches. The areas of patches are not even identical:

Serial:

Code:
/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  1.6                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 1.6-f802ff2d6c5a
Exec   : patchIntegrate p port17 -latestTime
Date   : Dec 17 2009
Time   : 13:47:12
Host   : c1-045
PID    : 10180
Case   : /home/asfj/OpenFOAM/asfj-1.5/run/linerStudy/noScaleAlt-refineCurvatureInport-poly
nProcs : 1
SigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 4000

Time = 4000
    Cyclic patch vector area: 
    - half 1 = (0.0015018 4.07798e-08 0.00702065), 0.00717948
    - half 2 = (-0.0015018 -4.07798e-08 -0.00702065), 0.00717948
    - total  = (0 7.27919e-22 0), 7.27919e-22
    Cyclic patch area magnitude = 0.00718769
    Reading volScalarField p
    Integral of p over vector area of patch port17[26] = (-2.01271e-15 1.14604e-18 7.84746e-16)
    Integral of p over area magnitude of patch port17[26] = 10.1629

End
Parallel:

Code:
/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  1.6                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 1.6-f802ff2d6c5a
Exec   : patchIntegrate p port17 -latestTime -parallel
Date   : Dec 17 2009
Time   : 13:46:20
Host   : c1-045
PID    : 10110
Case   : /home/asfj/OpenFOAM/asfj-1.5/run/linerStudy/noScaleAlt-refineCurvatureInport-poly
nProcs : 8
Slaves : 
7
(
c1-046.10017
c1-049.9888
c1-050.10134
c1-008.23707
c1-006.22837
c1-014.17367
c1-018.13667
)

Pstream initialized with:
    floatTransfer     : 0
    nProcsSimpleSum   : 0
    commsType         : nonBlocking
SigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 4000

Time = 4000
    Cyclic patch vector area: 
    - half 1 = (0.00139876 5.13032e-08 0.00665429), 0.00679971
    - half 2 = (-0.00139876 -5.13032e-08 -0.00665429), 0.00679971
    - total  = (4.33681e-19 3.63959e-22 0), 4.33681e-19
    Cyclic patch area magnitude = 0.00680708
    Reading volScalarField p
    Integral of p over vector area of patch port17[26] = (-1.10697e-16 -8.47033e-21 -3.79818e-15)
    Integral of p over area magnitude of patch port17[26] = 9.48489

End

Finalising parallel run
I have looked at the code for patchAverage and patchIntegrate and it looks correct. It uses "reduce" and everything should work. Help would be greatly appreciated.

Best regards, Ask
askjak is offline   Reply With Quote

Old   December 18, 2009, 12:15
Default
  #2
New Member
 
Join Date: Apr 2009
Posts: 26
Rep Power: 8
askjak is on a distinguished road
I have now also downloaded and compiled the newest version of 1.6.x and the problems persists.

It appears to be a real bug and also quite troublesome since it does not fail in an obvious way but reports numbers that (in this case) is 5-10% wrong.

BTW: When an application has the option "-parallel" can one then safely assume that the application works in parallel?
askjak is offline   Reply With Quote

Old   December 19, 2009, 05:50
Default
  #3
Senior Member
 
Join Date: Mar 2009
Posts: 854
Rep Power: 13
henry is on a distinguished road
The problem you are seeing is a consequence of cyclic patches transforming either partially or completely into processor patches during decomposition. In fact for efficiency reasons the sets of cyclic faces which require parallel communications are spliced into the sets of "normal" processor faces so that the transfers are combined into a single block. This means that integrating over cyclic patches when running in parallel is VERY complicated and requires looping over the corresponding processor patches and finding which faces correspond to which cyclic patches.

We are considering re-organising the way cyclic patches are handled, splitting each into two, one for either side, and maintaining separate processor patches for each as required. This will make integrating over them in parallel slightly easier as it would simply be a matter of handling the residual cyclic patch and the corresponding processor patch as required.

In the meantime it is more reliable to create a faceZone for the set of faces you want to integrate over rather that integrating over the cyclic patch directly.

H
henry is offline   Reply With Quote

Old   December 22, 2009, 04:49
Default
  #4
New Member
 
Join Date: Apr 2009
Posts: 26
Rep Power: 8
askjak is on a distinguished road
Thanks. That makes sense computationally.

Since the patchAverage/patchIntegrate takes steps to support cyclic patches and runs in parallel mode should there not be thrown an error in the case cyclic && parallel?

Any suggestions on how to define faceZones internally in a (complex) cylindrical geometry? I need to analyze on many small patches that are flush with the cylinder walls.
askjak is offline   Reply With Quote

Old   December 22, 2009, 05:00
Default
  #5
Senior Member
 
Join Date: Mar 2009
Posts: 854
Rep Power: 13
henry is on a distinguished road
> Since the patchAverage/patchIntegrate takes steps to support cyclic patches and runs in parallel mode should there not be thrown an error in the case cyclic && parallel?

Possibly, but under some conditions it will work correctly but unfortunately those conditions depend on the decomposition. We will discuss if we should add such an error/warning message.

> Any suggestions on how to define faceZones internally in a (complex) cylindrical geometry? I need to analyze on many small patches that are flush with the cylinder walls.

You can use faceSet or setSet and then setsToZones for this purpose.

H
henry is offline   Reply With Quote

Old   December 26, 2009, 14:37
Default
  #6
New Member
 
Join Date: Apr 2009
Posts: 26
Rep Power: 8
askjak is on a distinguished road
I have now succeed creating a faceZone that I want to do my analysis on. I have tried to find an example as a template. So far the best guess is the functionObject faceZoneIntegration, but it appears to have been removed in 1.6.x.

Can anyone point me to a tool that does postprocessing on a faceZone so I can get started?

Ask

Last edited by askjak; December 28, 2009 at 19:21.
askjak is offline   Reply With Quote

Old   December 26, 2009, 17:53
Default
  #7
Senior Member
 
Join Date: Mar 2009
Posts: 854
Rep Power: 13
henry is on a distinguished road
We have enhanced the functionality in the OpenFOAM-1.6.x/src/postProcessing/functionObjects/field functionObjects to include support for cellZones and faceZones which superceds faceZoneIntegration and renders it redundant so we removed it.

H
henry is offline   Reply With Quote

Old   January 4, 2010, 05:28
Default
  #8
New Member
 
Join Date: Apr 2009
Posts: 26
Rep Power: 8
askjak is on a distinguished road
Great. I will have a look at it.

I guess this cyclic patch issue could be mitigated in the decomposing step by adding a "preservePatches" in decomposeParDict

Quote:
//- Keep owner and neighbour on same processor for faces in patches:
// (makes sense only for cyclic patches)
preservePatches (port01);
Ask
Pagoda and iurnus like this.
askjak is offline   Reply With Quote

Old   January 4, 2010, 05:54
Default
  #9
Senior Member
 
Join Date: Mar 2009
Posts: 854
Rep Power: 13
henry is on a distinguished road
Yes using "preservePatches" should help.

H
henry is offline   Reply With Quote

Old   January 11, 2010, 19:48
Default Cyclics and decomposePar
  #10
New Member
 
Tim Stovall
Join Date: Mar 2009
Posts: 12
Rep Power: 8
tstovall is on a distinguished road
Hi,

I am trying to model a box with two cyclic boundaries. The front and back of the box are coupled with one cyclic patch, and the left and right are coupled with another cyclic patch. The other two faces (top and bottom) are generic patches. The simulation will run on a single processor, but when I decompose the domain and try to run with multiple processors, the solution explodes. I suspect the issue is with the two cyclic patches. I've tried cutting the domain in different orientations, but this does not alleviate the issue. Does anyone have any help/suggestions for running this case in parallel?

Thanks,

Tim
tstovall is offline   Reply With Quote

Old   January 13, 2010, 08:22
Default
  #11
Super Moderator
 
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,416
Rep Power: 16
mattijs is on a distinguished road
You could use the preservePatches option in the decomposeParDict (see the sample library in $FOAM_UTILITIES)

What solver are you running? There should be no problem running standard solvers with decomposed cyclics.
mattijs is offline   Reply With Quote

Old   January 13, 2010, 14:04
Default
  #12
New Member
 
Tim Stovall
Join Date: Mar 2009
Posts: 12
Rep Power: 8
tstovall is on a distinguished road
I'm using the preservePatches option in decomposePar and verified all the faces of the cyclic patches are present in the processor directories. I'm using pisoFoam with LES and the oneEqEddy SGS model with standard coefficients.

After some further review, I've isolated the problem. The domain consists of two cyclic conditions for the four vertical boundaries, each paired with the opposite surface. The top boundary is a symmetryPlane. The bottom surface is a wall. The simulation will run with these conditions in parallel with the domain only cut in the X (flow direction). When I switch the nuSgs bottom wall condition from zeroGradient to nuSgsWallFunction, the solution fails to converge. Each momentum solver maxes out at 1001 itterations and the courant number blows up by the second time step.

I've run another simulation with the sides are cyclic, inlet is directMapped, outlet is inletOutlet, top is symmetryPlane, and bottom is wall with nuSgsWallFunction. I'm curious why changing the inlet/outlet from directMapped/inletOutlet to a coupled cyclic patch would cause an error.

Any ideas why the wall function would cause this?
tstovall is offline   Reply With Quote

Old   January 14, 2010, 01:10
Default
  #13
Super Moderator
 
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,416
Rep Power: 16
mattijs is on a distinguished road
Can you post a simple testcase which exhibits the problem (run non-parallel, not run in parallel)?
mattijs is offline   Reply With Quote

Old   January 18, 2010, 15:43
Default
  #14
New Member
 
Tim Stovall
Join Date: Mar 2009
Posts: 12
Rep Power: 8
tstovall is on a distinguished road
Mattijs,

I tried scaling down the case so reproduce the error for a simple test case, but the error does not occur for the smaller domain. I found that having a smaller domain prevents the CFL from maxing at the early time steps. With the resized domain, the solutions converge. Thanks for your help. Attached is a log file from one of the bad runs (16 processors) and the same run one a single processor.

-Tim

log-single.txt

log-16proc.txt
tstovall is offline   Reply With Quote

Old   January 26, 2010, 11:41
Default
  #15
Super Moderator
 
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,416
Rep Power: 16
mattijs is on a distinguished road
Can you recreate the case with a blockMesh so we can recreate the case over here?
mattijs is offline   Reply With Quote

Old   January 26, 2010, 12:43
Default
  #16
New Member
 
Tim Stovall
Join Date: Mar 2009
Posts: 12
Rep Power: 8
tstovall is on a distinguished road
Mattijs,

Here's the case file. Included are all the log files from the steps I took to build and decompose the case. The case was decomposed on a 1.6.x build, but the cluster for parallel and single processor runs only has a 1.6 build. The log-pisoFoamParallel shows the iterations maxing for the momentum equations that does not occur for the same case run on a single processor shown in log-pisoFoamSingle.

DualCyclics.tar.gz

Thanks,

Tim
tstovall is offline   Reply With Quote

Old   January 26, 2010, 13:37
Default
  #17
Super Moderator
 
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,416
Rep Power: 16
mattijs is on a distinguished road
Hi Tim,

I just tried on 16 processors, 1.6.x and first time step seems to run ok - slightly more iterations on the solvers as expected but no maxing out at 1001. Attached my log. Can you try 16x?
mattijs is offline   Reply With Quote

Old   April 30, 2010, 11:19
Default PreservePatches with Cyclic Boundaries
  #18
Member
 
Matthew J. Churchfield
Join Date: Nov 2009
Location: Boulder, Colorado, USA
Posts: 49
Rep Power: 8
mchurchf is on a distinguished road
Hello,

I am using OpenFOAM 1.6. I am running 2D case where the flow is from left to right and the inflow and outflow are cyclic boundaries. A schematic is below where the vertical lines are the cyclic boundary a.
PHP Code:
    -->
|         |
|         |
|
_________|
a         a 

In order to keep the corresponding cyclic faces on single processors when running in parallel, I've been resorting to splitting the domain into, say 3, vertically stacked zones like this
PHP Code:
           -->
|
_________0
|_________1
|_________2
a         a 

The larger the grid gets in the horizontal (especially when going to 3D) and the more processors I use this becomes unreasonable because I will have a stack of very thin pieces of grid alotted to each processor.

As an experiment, I have tried using preservePatches in the decomposeParDict like:

numberOfSubdomains 3;
method simple;
preservePatches (a);

simpleCoeffs
{
n (3 1 1);
delta 0.001;
}

Hoping that the domain would be divided like
PHP Code:
-->
 
0 1  2  0
|_|__|__|_|
|
_|__|__|_|
|
_|__|__|_|
a         a 

So that the cyclic patch a is processed by processor0 only. Instead the result is
PHP Code:
-->
 
0  1  2  
|__|__|__|
|
__|__|__|
|
__|__|__|
a        a 

so that corresponding faces of the cyclic patch a are divided between two processors. Is there a way to make the situation desired occur?

Thank you,

Matt
mchurchf is offline   Reply With Quote

Old   October 27, 2010, 03:35
Default
  #19
Senior Member
 
maddalena's Avatar
 
maddalena
Join Date: Mar 2009
Posts: 436
Rep Power: 12
maddalena is on a distinguished road
Quote:
Originally Posted by henry View Post
You can use faceSet or setSet and then setsToZones for this purpose
Hi everyone,
I have faced the same problem when decomposing a domain with cyclics.
In order to use the preserveFace option, I must define faceSet and then setsToZones, as explained above. In the faceSetDict I defined:
Code:
name Cy01;
action new;
topoSetSources
(
    patchToFace
    {
        name "Cy01"; 
    }
);
Then setsToZones and decomposePar. That worked.
Now my question: I have 24 cyclics and changing and running 24 times faceSet is annoying... Is there any fast method to define 24 faceSet basing on the 24 "Cy.*" patches?
Thanks

mad
maddalena is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Parallel Moving Mesh Bug for Multi-patch Case albcem OpenFOAM Bugs 17 April 28, 2013 23:44
Postprocessing large data sets in parallel evrikon OpenFOAM Post-Processing 27 June 22, 2011 03:34
MRFSimpleFoam with cyclicGgi in parallel hannes OpenFOAM Bugs 7 April 29, 2010 12:50
Working with patches in parallel mode mkraposhin OpenFOAM 0 September 7, 2009 11:44
Cyclic patch in parallel calculations didomenico OpenFOAM Running, Solving & CFD 4 March 7, 2007 06:46


All times are GMT -4. The time now is 20:16.