CFD Online Discussion Forums

CFD Online Discussion Forums (
-   OpenFOAM Bugs (
-   -   parallel error with cyclic BCs for pimpleDyMFoam and trouble in resuming running (

sunliming November 28, 2012 07:51

parallel error with cyclic BCs for pimpleDyMFoam and trouble in resuming running
Two problems found when using pimpleDyMFoam and waveDisplacement:

1. When I use pimpleDyMFoam together with cyclic boundary conditions,(actually I use waveDisplacement, you can try the movingCylinder case, with a pair of cyclic boundaries and oscillatingVelocity for the cylinder, the same thing happens) parallel running cracks: after about 2 or 3 timeSteps, running stopped with the message " face o area does not match neighbour by 0.0195% --possible face ordering problem", no matter what kind of decomposition method I use, the same problem always appears. But when I use a single core to run it, no such problem happens. So I assume there might be some bugs in parallel running of dynamic mesh cases with cyclic BCs. Has anyone encountered this before? If anyone is interested, just let me know, I can post the tutorial.

2. When I use waveDisplacement to simulate a imposed wave motion of the bottom surface, something even more strange happens: if I stop the running, and resume it, it just explodes with the error "Floating point exception (core dumped)"! This is serious cause I cannot assure the running keeps going until I get satisfactory results. Once it stops, it cannot be resumed! Again, if anyone is interested, I can post the tutorial.

calim_cfd November 29, 2012 14:21

do ur cases run fine in serial mode?

bag rows December 2, 2012 02:09

I got same problem (1) when using pimpleDyMfoam with cyclic bc in parallel run (2.1.0).
In my case (LES), serial run with above and parallel run with pimpleform with cyclic was ok. I tried to loose match torlence (0.9) and decompose with scotch and simple method, but always encountered same error within 2-3 time steps.

Could you post how to solve it?

calim_cfd December 2, 2012 22:20

check tutorials.. u might have to use this patch type instead "cyclicAMI"

sunliming December 4, 2012 03:12


Originally Posted by calim_cfd (Post 394909)
do ur cases run fine in serial mode?

What is serial mode? I'v never heard of that before, could you explain a little bit? And is it ok to replace cyclic bc with cyclicAMI since the latter stands for cyclic arbitrary moving interface? Sorry to be ignorant.

By the way:
I tried mpirun -np 2 gradPchannelDyMFoam, which I think is the so called "serial run", and it seems ok until now.
How could cyclicAMI be used for the inlet, outlet, front and back wall since there is no arbitrary moving interface?

bag rows December 4, 2012 11:09

Hi, sunliming

Serial run is just to use only one CPU for far as I know...
In my case, I used 4 CPUs (-np 4 pimpleDyMFoam with cyclic BCs) in my desktop but failed around 2000 time step. In case of 200 CPUs, failed within 2-3 time step.

Maybe, as posted by mauricio, it seems cyclicAMI is the only option for dynamic mesh problem. however, it looks quite tricky for me to use cyclicAMI ...:(
If you managed to solve the problem, please let me know..


sunliming December 4, 2012 11:16

I think serial run is running with several cores one after another rather than parallel, so I run "mpirun -np 2 channelFoam (here -parallel is removed so that it becomes serial mode)". But I think this mode is much slower than the parallel mode, and I doubt wether it is any faster than the single core mode. Anyway, I still think there are some bugs for parallel running with cyclic BCs, but I've no idea how to handle at the moment.

calim_cfd December 4, 2012 16:16

hi everyone!

yes serial is when u launch only one process using 100% of a thread.

when u set -np x it means your case should have x domains and you'll have to launch x processes. If you have x threads available then you will have x process running at 100% cpu each. Else if x is less than the number of threads available the process you launched you will have to share cpu resources.

cyclicAMI is for dynamic meshes because it handles non-conforming meshes ( you'll have to be careful how u set ur domain patches, bcs and the settings in the system folder since its a transient solution method.


sunliming December 16, 2012 05:26

parallel running error with cyclic BCs for pimpleDyMFoam
1 Attachment(s)
Problem remains unsolved, here is the case file, hope anyone can check it and give some help!
Just run the following applications:
mpirun -np 2 pimpleDyMFoam -parallel

sunliming December 16, 2012 23:00

I guess there must be something wrong with the updatecoeffs() in the source code of e.g. oscillatingDisplacement. So when running in parallel one processor updated the coordinates and set the switch to "updated already", then the other processors get the message and stop updating the coordinates which should be updated. As a result, two cyclic patches will which contradict due to the inconsistence of coordinates. Does anybody know whether I am right or not? Thank you in advance for comments.

sunliming December 19, 2012 10:02

I'm now more sure about the existence of bugs, since it runs well in parallel mode with the same BCs(2 pair of cyclics,i.e. channel395 case) for static mesh and runs well in serial mode but crashes in parallel for dynamic mesh. I tried ro use "createPatch" as someone mentioned in the forum, but it still runs well in serial and craches in parallel. So I think there must be someting wrong for cyclic BCs to go well with dynamic mesh. Yet, I've no idea how to fix it. Has anyone encountered this before?

wyldckat December 22, 2012 17:30

2 Attachment(s)
Greetings to all!

I've used an example case I really like from this bug report: - and I adapted it to run with 4 cores and using only "cyclic" patches.

I've tested with 2.1.1 and the same error occurs, where "obj" files are dumped and it then crashes.
I've also tested with 2.1.x and it interestingly locks up instead of crashing...

Then I did some searching and adapted to "cyclicAMI" patches. Basically you need to:
  1. Change all references from "type cyclic;" to "type cyclicAMI;".
  2. And in "blockMeshDict" you need this:

    transform      translational;
    separationVector (0.001 0 0);

    Well, you should adjust the last vector value accordingly to your case/mesh. And keep in mind that the other patch needs to use the respective negative position, namely:

    transform      translational;
    separationVector (-0.001 0 0);

Attached you will find the two examples.

Note: the cyclicAMI version ends up crashing at around 0.0038s :rolleyes:, so it is not a perfected example ;).

To run either one of the cases, simply run:

Best regards,

sunliming December 23, 2012 00:28

cyclicAMI works for parallel running where cyclic cannot
OH MY GOD, it works!!! You've solved the problem that has bothered me for months! Great great thanks to you, Bruno! Come to China the othet day and let me treat you to dinner!:)
A few days ago, I tried cyclicAMI with translational transform, but I was not clear about the vector and got the wrong direction and size and ended up in another failure.
Now I'm quite clear! The separationVector conveys 2 of important factors: dierction and the distance between the two cyclic patches (These two "brothers" are great trouble-makers in my opinion!) in that direction.
By the way, cyclicAMI with tanslational transform can ensure the two patches have equal corresponding field variables as cyclic can do, right? If so, it's perfect for me!
Best wishes!
Nick (sunliming)

wyldckat December 24, 2012 08:29

Hi Nick,

You're welcome! To complete the answer, there are some more hints in the tutorials, more specifically in the file "incompressible/pimpleDyMFoam/mixerVesselAMI2D/constant/polyMesh/blockMeshDict.m4", namely the commented code:


        type            cyclicAMI;
        neighbourPatch  AMI1;
        transform      noOrdering;
/* optional
            type            searchableCylinder; // none
            point1          (0 0 -1);
            point2          (0 0 1);
            radius          0.5;

The translational detail came from the file "applications/utilities/mesh/manipulation/createPatch/createPatchDict".

Best regards,

sunliming December 27, 2012 09:21

Mesh goes into a mess with cyclicAMI
2 Attachment(s)
Hi Bruno,
Thank you for the supplement!

I tried cyclicAMI for my case, and found the mesh intersects using cyclicAMI which will not happen with cyclic.

Attacted are my cyclic and cyclicAMI cases, cyclic case can run in serial mode without disturbing the mesh, but with cyclicAMI meshes all go wrong, could have a look at the files, run with the 'Allrun' script for each case and see what is wrong with cyclicAMI.

Thank you! Merry Christmas!


wyldckat December 29, 2012 19:44

2 Attachment(s)
Hi Nick,

:eek: This is one crazy bug :D Attached are two iterations detailing the issue!

So, on both images you'll see on the left the mesh with "cyclic" patches and on the right with "cyclicAMI".
In the iteration 0.25, we start to see a slight difference. At 0.5, they are veeery different! This is why it crashes with "cyclicAMI", since there are cells overlapping each other!
You can see this effect on either one of the 4 bottom corners.

I suggest that you simplify a bit more the examples you've attached. More specifically:
  • Make both cases only run up to 0.75. This way you'll get 3 more iterations, where the 3rd iteration wont be possible with AMI.
  • The AMI case should run in serial as well. It's easier that way to diagnose.
Then create a new report at with these simplified examples and you can also include these 2 images I attached as well. This way it will make life a lot easier for the developers to diagnose the problem ;)

Best regards,

sunliming December 31, 2012 02:52

Bug lies in waveDisplacement due to update the internalFields alone without other BCs
5 Attachment(s)
Hi Bruno,

Thank you for the quick reply and good advice!

I'll make a report to the bug centre later. I guess it is not a problem with the cyclicAMI BC, for when I replace 'cyclic' and/or 'cyclicAMI' with 'patch',(see attached case file) the mesh still goes wrong, so it is concluded that only 'cyclic' can be used for the motion type 'waveDisplacement' where the bug is supposed to be.

When I use 'patch' for all lateral walls, the mesh is good for the internal field but wrong at the inlet and outlet.(see attached pictures) So I checked the source code "opemfoam211/src/fvMotionSolver/pointPatchFields/derived/waveDisplacement/waveDisplacementPointPatchVectorField.C",the member function updateCoeffs() only updates the internalField and the moving boundary, excluding the other boundarys, see "const polyMesh& mesh = this->dimensionedInternalField().mesh()();".

I don't know how to include other boundary pathes for point motion, for I'm not quite familiar with OpenFOAM syntax.

Best wishes!

sunliming December 31, 2012 02:57

2 Attachment(s)
Two more pictures attached for the lower part of inlet at 4s and outlet at 4s.

wyldckat January 1, 2013 09:51

Hi Nick,

The reason why I asked you to report on the bug tracker is also because I'm not experienced enough to quickly figure out what's the solution on this one.
But I can tell you that:
  1. "dimensionedInternalField().mesh()()" is only used in this case for getting information about the current time.
  2. It's the "waveNumber_ & patch().localPoints()" that does the actual point retrieval.
  3. My guess is that when using "cyclicAMI", there's a step somewhere that is missing for updating the positions of the points for the previous vs next iteration.
But the "waveNumber" reminded me of something... have you looked into this :confused::
I'm not certain of the problem/solution you're after is related to the information on that page, but it sort of looks like it might be.

Best regards,

sunliming January 1, 2013 10:32

Hi Bruno,

Actually, I'm doing research on wind-wave interaction. I'm a graduate student of wind engineering, my final goal is to study mean wind profiles over the ocean under strong wind and give some instructions of proper wind load on offshore structures.

So the most suitable model for my case should be two-phase flow, but I wanna start with a relatively easy model--imposed solid wave undulation, this is just what waveDisplacement can do. As for 'waves2Foam', I have studied it for a while, but later I found its focus is on the wave rather than wind field which is my concern. And I did some simple simulation, finding only near wave layer of the upper wind field has evident change. The wave cannot take into accout of wind energy transfered to it, thus modification must be made to satisfy my need. So this may be my next task providing current step can be accomplished.

Problems keep coming one after another, so there is still a lot to do.

Anyway, thank you giving so much help!

Best wishes!

All times are GMT -4. The time now is 23:23.