|
[Sponsors] |
Cyclic patches and parallel postprocessing problems |
|
LinkBack | Thread Tools | Search this Thread | Display Modes |
December 17, 2009, 09:02 |
Cyclic patches and parallel postprocessing problems
|
#1 |
New Member
Join Date: Apr 2009
Posts: 26
Rep Power: 17 |
Hi,
I am interested in doing a lot of parallel postprocessing. I have noticed problems when using patchIntegrate and patchAverage on a cyclic patch. I have compared results from a parallel case and a seriel case. On a non-cyclic patch I get identical results. But the results are not correct for cyclic patches. The areas of patches are not even identical: Serial: Code:
/*---------------------------------------------------------------------------*\ | ========= | | | \\ / F ield | OpenFOAM: The Open Source CFD Toolbox | | \\ / O peration | Version: 1.6 | | \\ / A nd | Web: www.OpenFOAM.org | | \\/ M anipulation | | \*---------------------------------------------------------------------------*/ Build : 1.6-f802ff2d6c5a Exec : patchIntegrate p port17 -latestTime Date : Dec 17 2009 Time : 13:47:12 Host : c1-045 PID : 10180 Case : /home/asfj/OpenFOAM/asfj-1.5/run/linerStudy/noScaleAlt-refineCurvatureInport-poly nProcs : 1 SigFpe : Enabling floating point exception trapping (FOAM_SIGFPE). // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * // Create time Create mesh for time = 4000 Time = 4000 Cyclic patch vector area: - half 1 = (0.0015018 4.07798e-08 0.00702065), 0.00717948 - half 2 = (-0.0015018 -4.07798e-08 -0.00702065), 0.00717948 - total = (0 7.27919e-22 0), 7.27919e-22 Cyclic patch area magnitude = 0.00718769 Reading volScalarField p Integral of p over vector area of patch port17[26] = (-2.01271e-15 1.14604e-18 7.84746e-16) Integral of p over area magnitude of patch port17[26] = 10.1629 End Code:
/*---------------------------------------------------------------------------*\ | ========= | | | \\ / F ield | OpenFOAM: The Open Source CFD Toolbox | | \\ / O peration | Version: 1.6 | | \\ / A nd | Web: www.OpenFOAM.org | | \\/ M anipulation | | \*---------------------------------------------------------------------------*/ Build : 1.6-f802ff2d6c5a Exec : patchIntegrate p port17 -latestTime -parallel Date : Dec 17 2009 Time : 13:46:20 Host : c1-045 PID : 10110 Case : /home/asfj/OpenFOAM/asfj-1.5/run/linerStudy/noScaleAlt-refineCurvatureInport-poly nProcs : 8 Slaves : 7 ( c1-046.10017 c1-049.9888 c1-050.10134 c1-008.23707 c1-006.22837 c1-014.17367 c1-018.13667 ) Pstream initialized with: floatTransfer : 0 nProcsSimpleSum : 0 commsType : nonBlocking SigFpe : Enabling floating point exception trapping (FOAM_SIGFPE). // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * // Create time Create mesh for time = 4000 Time = 4000 Cyclic patch vector area: - half 1 = (0.00139876 5.13032e-08 0.00665429), 0.00679971 - half 2 = (-0.00139876 -5.13032e-08 -0.00665429), 0.00679971 - total = (4.33681e-19 3.63959e-22 0), 4.33681e-19 Cyclic patch area magnitude = 0.00680708 Reading volScalarField p Integral of p over vector area of patch port17[26] = (-1.10697e-16 -8.47033e-21 -3.79818e-15) Integral of p over area magnitude of patch port17[26] = 9.48489 End Finalising parallel run Best regards, Ask |
|
December 18, 2009, 12:15 |
|
#2 |
New Member
Join Date: Apr 2009
Posts: 26
Rep Power: 17 |
I have now also downloaded and compiled the newest version of 1.6.x and the problems persists.
It appears to be a real bug and also quite troublesome since it does not fail in an obvious way but reports numbers that (in this case) is 5-10% wrong. BTW: When an application has the option "-parallel" can one then safely assume that the application works in parallel? |
|
December 19, 2009, 05:50 |
|
#3 |
Senior Member
Join Date: Mar 2009
Posts: 854
Rep Power: 22 |
The problem you are seeing is a consequence of cyclic patches transforming either partially or completely into processor patches during decomposition. In fact for efficiency reasons the sets of cyclic faces which require parallel communications are spliced into the sets of "normal" processor faces so that the transfers are combined into a single block. This means that integrating over cyclic patches when running in parallel is VERY complicated and requires looping over the corresponding processor patches and finding which faces correspond to which cyclic patches.
We are considering re-organising the way cyclic patches are handled, splitting each into two, one for either side, and maintaining separate processor patches for each as required. This will make integrating over them in parallel slightly easier as it would simply be a matter of handling the residual cyclic patch and the corresponding processor patch as required. In the meantime it is more reliable to create a faceZone for the set of faces you want to integrate over rather that integrating over the cyclic patch directly. H |
|
December 22, 2009, 04:49 |
|
#4 |
New Member
Join Date: Apr 2009
Posts: 26
Rep Power: 17 |
Thanks. That makes sense computationally.
Since the patchAverage/patchIntegrate takes steps to support cyclic patches and runs in parallel mode should there not be thrown an error in the case cyclic && parallel? Any suggestions on how to define faceZones internally in a (complex) cylindrical geometry? I need to analyze on many small patches that are flush with the cylinder walls. |
|
December 22, 2009, 05:00 |
|
#5 |
Senior Member
Join Date: Mar 2009
Posts: 854
Rep Power: 22 |
> Since the patchAverage/patchIntegrate takes steps to support cyclic patches and runs in parallel mode should there not be thrown an error in the case cyclic && parallel?
Possibly, but under some conditions it will work correctly but unfortunately those conditions depend on the decomposition. We will discuss if we should add such an error/warning message. > Any suggestions on how to define faceZones internally in a (complex) cylindrical geometry? I need to analyze on many small patches that are flush with the cylinder walls. You can use faceSet or setSet and then setsToZones for this purpose. H |
|
December 26, 2009, 14:37 |
|
#6 |
New Member
Join Date: Apr 2009
Posts: 26
Rep Power: 17 |
I have now succeed creating a faceZone that I want to do my analysis on. I have tried to find an example as a template. So far the best guess is the functionObject faceZoneIntegration, but it appears to have been removed in 1.6.x.
Can anyone point me to a tool that does postprocessing on a faceZone so I can get started? Ask Last edited by askjak; December 28, 2009 at 19:21. |
|
December 26, 2009, 17:53 |
|
#7 |
Senior Member
Join Date: Mar 2009
Posts: 854
Rep Power: 22 |
We have enhanced the functionality in the OpenFOAM-1.6.x/src/postProcessing/functionObjects/field functionObjects to include support for cellZones and faceZones which superceds faceZoneIntegration and renders it redundant so we removed it.
H |
|
January 4, 2010, 05:28 |
|
#8 | |
New Member
Join Date: Apr 2009
Posts: 26
Rep Power: 17 |
Great. I will have a look at it.
I guess this cyclic patch issue could be mitigated in the decomposing step by adding a "preservePatches" in decomposeParDict Quote:
|
||
January 4, 2010, 05:54 |
|
#9 |
Senior Member
Join Date: Mar 2009
Posts: 854
Rep Power: 22 |
Yes using "preservePatches" should help.
H |
|
January 11, 2010, 19:48 |
Cyclics and decomposePar
|
#10 |
New Member
Tim Stovall
Join Date: Mar 2009
Posts: 12
Rep Power: 17 |
Hi,
I am trying to model a box with two cyclic boundaries. The front and back of the box are coupled with one cyclic patch, and the left and right are coupled with another cyclic patch. The other two faces (top and bottom) are generic patches. The simulation will run on a single processor, but when I decompose the domain and try to run with multiple processors, the solution explodes. I suspect the issue is with the two cyclic patches. I've tried cutting the domain in different orientations, but this does not alleviate the issue. Does anyone have any help/suggestions for running this case in parallel? Thanks, Tim |
|
January 13, 2010, 08:22 |
|
#11 |
Senior Member
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,419
Rep Power: 26 |
You could use the preservePatches option in the decomposeParDict (see the sample library in $FOAM_UTILITIES)
What solver are you running? There should be no problem running standard solvers with decomposed cyclics. |
|
January 13, 2010, 14:04 |
|
#12 |
New Member
Tim Stovall
Join Date: Mar 2009
Posts: 12
Rep Power: 17 |
I'm using the preservePatches option in decomposePar and verified all the faces of the cyclic patches are present in the processor directories. I'm using pisoFoam with LES and the oneEqEddy SGS model with standard coefficients.
After some further review, I've isolated the problem. The domain consists of two cyclic conditions for the four vertical boundaries, each paired with the opposite surface. The top boundary is a symmetryPlane. The bottom surface is a wall. The simulation will run with these conditions in parallel with the domain only cut in the X (flow direction). When I switch the nuSgs bottom wall condition from zeroGradient to nuSgsWallFunction, the solution fails to converge. Each momentum solver maxes out at 1001 itterations and the courant number blows up by the second time step. I've run another simulation with the sides are cyclic, inlet is directMapped, outlet is inletOutlet, top is symmetryPlane, and bottom is wall with nuSgsWallFunction. I'm curious why changing the inlet/outlet from directMapped/inletOutlet to a coupled cyclic patch would cause an error. Any ideas why the wall function would cause this? |
|
January 14, 2010, 01:10 |
|
#13 |
Senior Member
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,419
Rep Power: 26 |
Can you post a simple testcase which exhibits the problem (run non-parallel, not run in parallel)?
|
|
January 18, 2010, 15:43 |
|
#14 |
New Member
Tim Stovall
Join Date: Mar 2009
Posts: 12
Rep Power: 17 |
Mattijs,
I tried scaling down the case so reproduce the error for a simple test case, but the error does not occur for the smaller domain. I found that having a smaller domain prevents the CFL from maxing at the early time steps. With the resized domain, the solutions converge. Thanks for your help. Attached is a log file from one of the bad runs (16 processors) and the same run one a single processor. -Tim log-single.txt log-16proc.txt |
|
January 26, 2010, 11:41 |
|
#15 |
Senior Member
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,419
Rep Power: 26 |
Can you recreate the case with a blockMesh so we can recreate the case over here?
|
|
January 26, 2010, 12:43 |
|
#16 |
New Member
Tim Stovall
Join Date: Mar 2009
Posts: 12
Rep Power: 17 |
Mattijs,
Here's the case file. Included are all the log files from the steps I took to build and decompose the case. The case was decomposed on a 1.6.x build, but the cluster for parallel and single processor runs only has a 1.6 build. The log-pisoFoamParallel shows the iterations maxing for the momentum equations that does not occur for the same case run on a single processor shown in log-pisoFoamSingle. DualCyclics.tar.gz Thanks, Tim |
|
January 26, 2010, 13:37 |
|
#17 |
Senior Member
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,419
Rep Power: 26 |
Hi Tim,
I just tried on 16 processors, 1.6.x and first time step seems to run ok - slightly more iterations on the solvers as expected but no maxing out at 1001. Attached my log. Can you try 16x? |
|
April 30, 2010, 12:19 |
PreservePatches with Cyclic Boundaries
|
#18 |
Member
Matthew J. Churchfield
Join Date: Nov 2009
Location: Boulder, Colorado, USA
Posts: 49
Rep Power: 19 |
Hello,
I am using OpenFOAM 1.6. I am running 2D case where the flow is from left to right and the inflow and outflow are cyclic boundaries. A schematic is below where the vertical lines are the cyclic boundary a. PHP Code:
In order to keep the corresponding cyclic faces on single processors when running in parallel, I've been resorting to splitting the domain into, say 3, vertically stacked zones like this PHP Code:
The larger the grid gets in the horizontal (especially when going to 3D) and the more processors I use this becomes unreasonable because I will have a stack of very thin pieces of grid alotted to each processor. As an experiment, I have tried using preservePatches in the decomposeParDict like: numberOfSubdomains 3; method simple; preservePatches (a); simpleCoeffs { n (3 1 1); delta 0.001; } Hoping that the domain would be divided like PHP Code:
So that the cyclic patch a is processed by processor0 only. Instead the result is PHP Code:
so that corresponding faces of the cyclic patch a are divided between two processors. Is there a way to make the situation desired occur? Thank you, Matt |
|
October 27, 2010, 04:35 |
|
#19 | |
Senior Member
maddalena
Join Date: Mar 2009
Posts: 436
Rep Power: 23 |
Quote:
I have faced the same problem when decomposing a domain with cyclics. In order to use the preserveFace option, I must define faceSet and then setsToZones, as explained above. In the faceSetDict I defined: Code:
name Cy01; action new; topoSetSources ( patchToFace { name "Cy01"; } ); Now my question: I have 24 cyclics and changing and running 24 times faceSet is annoying... Is there any fast method to define 24 faceSet basing on the 24 "Cy.*" patches? Thanks mad |
||
Thread Tools | Search this Thread |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Postprocessing large data sets in parallel | evrikon | OpenFOAM Post-Processing | 28 | June 28, 2016 04:43 |
Parallel Moving Mesh Bug for Multi-patch Case | albcem | OpenFOAM Bugs | 17 | April 29, 2013 00:44 |
MRFSimpleFoam with cyclicGgi in parallel | hannes | OpenFOAM Bugs | 7 | April 29, 2010 13:50 |
Working with patches in parallel mode | mkraposhin | OpenFOAM | 0 | September 7, 2009 12:44 |
Cyclic patch in parallel calculations | didomenico | OpenFOAM Running, Solving & CFD | 4 | March 7, 2007 06:46 |