CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Bugs

parallel error with cyclic BCs for pimpleDyMFoam and trouble in resuming running

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree5Likes

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   November 28, 2012, 07:51
Default parallel error with cyclic BCs for pimpleDyMFoam and trouble in resuming running
  #1
Member
 
Nick
Join Date: Nov 2011
Location: Tongji University,Shanghai,China
Posts: 33
Blog Entries: 6
Rep Power: 14
sunliming is on a distinguished road
Two problems found when using pimpleDyMFoam and waveDisplacement:

1. When I use pimpleDyMFoam together with cyclic boundary conditions,(actually I use waveDisplacement, you can try the movingCylinder case, with a pair of cyclic boundaries and oscillatingVelocity for the cylinder, the same thing happens) parallel running cracks: after about 2 or 3 timeSteps, running stopped with the message " face o area does not match neighbour by 0.0195% --possible face ordering problem", no matter what kind of decomposition method I use, the same problem always appears. But when I use a single core to run it, no such problem happens. So I assume there might be some bugs in parallel running of dynamic mesh cases with cyclic BCs. Has anyone encountered this before? If anyone is interested, just let me know, I can post the tutorial.

2. When I use waveDisplacement to simulate a imposed wave motion of the bottom surface, something even more strange happens: if I stop the running, and resume it, it just explodes with the error "Floating point exception (core dumped)"! This is serious cause I cannot assure the running keeps going until I get satisfactory results. Once it stops, it cannot be resumed! Again, if anyone is interested, I can post the tutorial.
sunliming is offline   Reply With Quote

Old   November 29, 2012, 14:21
Default
  #2
Senior Member
 
calim_cfd's Avatar
 
mauricio
Join Date: Jun 2011
Posts: 172
Rep Power: 17
calim_cfd is on a distinguished road
do ur cases run fine in serial mode?
__________________
Best Regards
/calim

"Elune will grant us the strength"
calim_cfd is offline   Reply With Quote

Old   December 2, 2012, 02:09
Default
  #3
New Member
 
sung han kim
Join Date: Apr 2012
Posts: 2
Rep Power: 0
bag rows is on a distinguished road
I got same problem (1) when using pimpleDyMfoam with cyclic bc in parallel run (2.1.0).
In my case (LES), serial run with above and parallel run with pimpleform with cyclic was ok. I tried to loose match torlence (0.9) and decompose with scotch and simple method, but always encountered same error within 2-3 time steps.

Could you post how to solve it?

Last edited by bag rows; December 2, 2012 at 03:02.
bag rows is offline   Reply With Quote

Old   December 2, 2012, 22:20
Default
  #4
Senior Member
 
calim_cfd's Avatar
 
mauricio
Join Date: Jun 2011
Posts: 172
Rep Power: 17
calim_cfd is on a distinguished road
check tutorials.. u might have to use this patch type instead "cyclicAMI"
__________________
Best Regards
/calim

"Elune will grant us the strength"
calim_cfd is offline   Reply With Quote

Old   December 4, 2012, 03:12
Default
  #5
Member
 
Nick
Join Date: Nov 2011
Location: Tongji University,Shanghai,China
Posts: 33
Blog Entries: 6
Rep Power: 14
sunliming is on a distinguished road
Quote:
Originally Posted by calim_cfd View Post
do ur cases run fine in serial mode?
What is serial mode? I'v never heard of that before, could you explain a little bit? And is it ok to replace cyclic bc with cyclicAMI since the latter stands for cyclic arbitrary moving interface? Sorry to be ignorant.
Best!
Sunliming

By the way:
I tried mpirun -np 2 gradPchannelDyMFoam, which I think is the so called "serial run", and it seems ok until now.
How could cyclicAMI be used for the inlet, outlet, front and back wall since there is no arbitrary moving interface?

Last edited by sunliming; December 4, 2012 at 11:03.
sunliming is offline   Reply With Quote

Old   December 4, 2012, 11:09
Default
  #6
New Member
 
sung han kim
Join Date: Apr 2012
Posts: 2
Rep Power: 0
bag rows is on a distinguished road
Hi, sunliming

Serial run is just to use only one CPU for computation...as far as I know...
In my case, I used 4 CPUs (-np 4 pimpleDyMFoam with cyclic BCs) in my desktop but failed around 2000 time step. In case of 200 CPUs, failed within 2-3 time step.

Maybe, as posted by mauricio, it seems cyclicAMI is the only option for dynamic mesh problem. however, it looks quite tricky for me to use cyclicAMI ...
If you managed to solve the problem, please let me know..

Thanks
bag rows is offline   Reply With Quote

Old   December 4, 2012, 11:16
Default
  #7
Member
 
Nick
Join Date: Nov 2011
Location: Tongji University,Shanghai,China
Posts: 33
Blog Entries: 6
Rep Power: 14
sunliming is on a distinguished road
I think serial run is running with several cores one after another rather than parallel, so I run "mpirun -np 2 channelFoam (here -parallel is removed so that it becomes serial mode)". But I think this mode is much slower than the parallel mode, and I doubt wether it is any faster than the single core mode. Anyway, I still think there are some bugs for parallel running with cyclic BCs, but I've no idea how to handle at the moment.
sunliming is offline   Reply With Quote

Old   December 4, 2012, 16:16
Default
  #8
Senior Member
 
calim_cfd's Avatar
 
mauricio
Join Date: Jun 2011
Posts: 172
Rep Power: 17
calim_cfd is on a distinguished road
hi everyone!


yes serial is when u launch only one process using 100% of a thread.

when u set -np x it means your case should have x domains and you'll have to launch x processes. If you have x threads available then you will have x process running at 100% cpu each. Else if x is less than the number of threads available the process you launched you will have to share cpu resources.

cyclicAMI is for dynamic meshes because it handles non-conforming meshes (http://www.openfoam.org/version2.1.0/ami.php). you'll have to be careful how u set ur domain patches, bcs and the settings in the system folder since its a transient solution method.

gl
__________________
Best Regards
/calim

"Elune will grant us the strength"
calim_cfd is offline   Reply With Quote

Old   December 16, 2012, 05:26
Default parallel running error with cyclic BCs for pimpleDyMFoam
  #9
Member
 
Nick
Join Date: Nov 2011
Location: Tongji University,Shanghai,China
Posts: 33
Blog Entries: 6
Rep Power: 14
sunliming is on a distinguished road
Problem remains unsolved, here is the case file, hope anyone can check it and give some help!
Just run the following applications:
blockMesh
decomposePar
mpirun -np 2 pimpleDyMFoam -parallel
Attached Files
File Type: gz bugReport.tar.gz (10.0 KB, 46 views)
sunliming is offline   Reply With Quote

Old   December 16, 2012, 23:00
Default
  #10
Member
 
Nick
Join Date: Nov 2011
Location: Tongji University,Shanghai,China
Posts: 33
Blog Entries: 6
Rep Power: 14
sunliming is on a distinguished road
I guess there must be something wrong with the updatecoeffs() in the source code of e.g. oscillatingDisplacement. So when running in parallel one processor updated the coordinates and set the switch to "updated already", then the other processors get the message and stop updating the coordinates which should be updated. As a result, two cyclic patches will which contradict due to the inconsistence of coordinates. Does anybody know whether I am right or not? Thank you in advance for comments.
sunliming is offline   Reply With Quote

Old   December 19, 2012, 10:02
Default
  #11
Member
 
Nick
Join Date: Nov 2011
Location: Tongji University,Shanghai,China
Posts: 33
Blog Entries: 6
Rep Power: 14
sunliming is on a distinguished road
I'm now more sure about the existence of bugs, since it runs well in parallel mode with the same BCs(2 pair of cyclics,i.e. channel395 case) for static mesh and runs well in serial mode but crashes in parallel for dynamic mesh. I tried ro use "createPatch" as someone mentioned in the forum, but it still runs well in serial and craches in parallel. So I think there must be someting wrong for cyclic BCs to go well with dynamic mesh. Yet, I've no idea how to fix it. Has anyone encountered this before?
sunliming is offline   Reply With Quote

Old   December 22, 2012, 17:30
Default
  #12
Retired Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,974
Blog Entries: 45
Rep Power: 128
wyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to all
Greetings to all!

I've used an example case I really like from this bug report: http://www.openfoam.org/mantisbt/view.php?id=566 - and I adapted it to run with 4 cores and using only "cyclic" patches.

I've tested with 2.1.1 and the same error occurs, where "obj" files are dumped and it then crashes.
I've also tested with 2.1.x and it interestingly locks up instead of crashing...

Then I did some searching and adapted to "cyclicAMI" patches. Basically you need to:
  1. Change all references from "type cyclic;" to "type cyclicAMI;".
  2. And in "blockMeshDict" you need this:
    Code:
    transform       translational;
    separationVector (0.001 0 0);
    Well, you should adjust the last vector value accordingly to your case/mesh. And keep in mind that the other patch needs to use the respective negative position, namely:
    Code:
    transform       translational;
    separationVector (-0.001 0 0);
Attached you will find the two examples.


Note: the cyclicAMI version ends up crashing at around 0.0038s , so it is not a perfected example .


To run either one of the cases, simply run:
Code:
./Allrun
Best regards,
Bruno
Attached Files
File Type: gz massConservation_Cyclic4x.tar.gz (3.9 KB, 112 views)
File Type: gz massConservation_CyclicAMI4x.tar.gz (3.9 KB, 142 views)
__________________

Last edited by wyldckat; December 24, 2012 at 08:23. Reason: it's "blockMeshDict", not "blockMeshMeshDict" :)
wyldckat is offline   Reply With Quote

Old   December 23, 2012, 00:28
Default cyclicAMI works for parallel running where cyclic cannot
  #13
Member
 
Nick
Join Date: Nov 2011
Location: Tongji University,Shanghai,China
Posts: 33
Blog Entries: 6
Rep Power: 14
sunliming is on a distinguished road
OH MY GOD, it works!!! You've solved the problem that has bothered me for months! Great great thanks to you, Bruno! Come to China the othet day and let me treat you to dinner!
A few days ago, I tried cyclicAMI with translational transform, but I was not clear about the vector and got the wrong direction and size and ended up in another failure.
Now I'm quite clear! The separationVector conveys 2 of important factors: dierction and the distance between the two cyclic patches (These two "brothers" are great trouble-makers in my opinion!) in that direction.
By the way, cyclicAMI with tanslational transform can ensure the two patches have equal corresponding field variables as cyclic can do, right? If so, it's perfect for me!
Best wishes!
Nick (sunliming)
sunliming is offline   Reply With Quote

Old   December 24, 2012, 08:29
Default
  #14
Retired Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,974
Blog Entries: 45
Rep Power: 128
wyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to all
Hi Nick,

You're welcome! To complete the answer, there are some more hints in the tutorials, more specifically in the file "incompressible/pimpleDyMFoam/mixerVesselAMI2D/constant/polyMesh/blockMeshDict.m4", namely the commented code:
Quote:
Code:
    AMI2
    {
        type            cyclicAMI;
        neighbourPatch  AMI1;
        transform       noOrdering;
/* optional
        surface
        {
            type            searchableCylinder; // none
            point1          (0 0 -1);
            point2          (0 0 1);
            radius          0.5;
        }
*/
The translational detail came from the file "applications/utilities/mesh/manipulation/createPatch/createPatchDict".

Best regards,
Bruno
__________________
wyldckat is offline   Reply With Quote

Old   December 27, 2012, 09:21
Default Mesh goes into a mess with cyclicAMI
  #15
Member
 
Nick
Join Date: Nov 2011
Location: Tongji University,Shanghai,China
Posts: 33
Blog Entries: 6
Rep Power: 14
sunliming is on a distinguished road
Hi Bruno,
Thank you for the supplement!

I tried cyclicAMI for my case, and found the mesh intersects using cyclicAMI which will not happen with cyclic.

Attacted are my cyclic and cyclicAMI cases, cyclic case can run in serial mode without disturbing the mesh, but with cyclicAMI meshes all go wrong, could have a look at the files, run with the 'Allrun' script for each case and see what is wrong with cyclicAMI.

Thank you! Merry Christmas!

Nick
Attached Files
File Type: gz propagation_cyclic_bug.tar.gz (10.0 KB, 20 views)
File Type: gz propagation_cyclicAMI_bug.tar.gz (10.0 KB, 18 views)

Last edited by sunliming; December 27, 2012 at 10:08.
sunliming is offline   Reply With Quote

Old   December 29, 2012, 19:44
Default
  #16
Retired Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,974
Blog Entries: 45
Rep Power: 128
wyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to all
Hi Nick,

This is one crazy bug Attached are two iterations detailing the issue!

So, on both images you'll see on the left the mesh with "cyclic" patches and on the right with "cyclicAMI".
In the iteration 0.25, we start to see a slight difference. At 0.5, they are veeery different! This is why it crashes with "cyclicAMI", since there are cells overlapping each other!
You can see this effect on either one of the 4 bottom corners.

I suggest that you simplify a bit more the examples you've attached. More specifically:
  • Make both cases only run up to 0.75. This way you'll get 3 more iterations, where the 3rd iteration wont be possible with AMI.
  • The AMI case should run in serial as well. It's easier that way to diagnose.
Then create a new report at http://www.openfoam.org/mantisbt/ with these simplified examples and you can also include these 2 images I attached as well. This way it will make life a lot easier for the developers to diagnose the problem

Best regards,
Bruno
Attached Images
File Type: jpg 0.25.jpg (78.7 KB, 167 views)
File Type: jpg 0.5.jpg (78.3 KB, 144 views)
__________________

Last edited by wyldckat; December 29, 2012 at 19:45. Reason: 4 iterations --> 3 more iterations
wyldckat is offline   Reply With Quote

Old   December 31, 2012, 02:52
Default Bug lies in waveDisplacement due to update the internalFields alone without other BCs
  #17
Member
 
Nick
Join Date: Nov 2011
Location: Tongji University,Shanghai,China
Posts: 33
Blog Entries: 6
Rep Power: 14
sunliming is on a distinguished road
Hi Bruno,

Thank you for the quick reply and good advice!

I'll make a report to the bug centre later. I guess it is not a problem with the cyclicAMI BC, for when I replace 'cyclic' and/or 'cyclicAMI' with 'patch',(see attached case file) the mesh still goes wrong, so it is concluded that only 'cyclic' can be used for the motion type 'waveDisplacement' where the bug is supposed to be.

When I use 'patch' for all lateral walls, the mesh is good for the internal field but wrong at the inlet and outlet.(see attached pictures) So I checked the source code "opemfoam211/src/fvMotionSolver/pointPatchFields/derived/waveDisplacement/waveDisplacementPointPatchVectorField.C",the member function updateCoeffs() only updates the internalField and the moving boundary, excluding the other boundarys, see "const polyMesh& mesh = this->dimensionedInternalField().mesh()();".

I don't know how to include other boundary pathes for point motion, for I'm not quite familiar with OpenFOAM syntax.

Best wishes!
Nick
Attached Images
File Type: jpg initial_grids.jpg (35.8 KB, 96 views)
File Type: jpg initial_inlet.jpg (34.2 KB, 94 views)
File Type: jpg grids_time_6.jpg (39.0 KB, 123 views)
File Type: jpg time_6_outlet.jpg (33.2 KB, 87 views)
Attached Files
File Type: gz propagation_fixed.tar.gz (40.0 KB, 17 views)
sunliming is offline   Reply With Quote

Old   December 31, 2012, 02:57
Default
  #18
Member
 
Nick
Join Date: Nov 2011
Location: Tongji University,Shanghai,China
Posts: 33
Blog Entries: 6
Rep Power: 14
sunliming is on a distinguished road
Two more pictures attached for the lower part of inlet at 4s and outlet at 4s.
Attached Images
File Type: jpg time_4_inlet.jpg (42.7 KB, 40 views)
File Type: jpg time_4_outlet.jpg (32.1 KB, 38 views)
sunliming is offline   Reply With Quote

Old   January 1, 2013, 09:51
Default
  #19
Retired Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,974
Blog Entries: 45
Rep Power: 128
wyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to allwyldckat is a name known to all
Hi Nick,

The reason why I asked you to report on the bug tracker is also because I'm not experienced enough to quickly figure out what's the solution on this one.
But I can tell you that:
  1. "dimensionedInternalField().mesh()()" is only used in this case for getting information about the current time.
  2. It's the "waveNumber_ & patch().localPoints()" that does the actual point retrieval.
  3. My guess is that when using "cyclicAMI", there's a step somewhere that is missing for updating the positions of the points for the previous vs next iteration.
But the "waveNumber" reminded me of something... have you looked into this : http://openfoamwiki.net/index.php/Contrib/waves2Foam
I'm not certain of the problem/solution you're after is related to the information on that page, but it sort of looks like it might be.

Best regards,
Bruno
__________________
wyldckat is offline   Reply With Quote

Old   January 1, 2013, 10:32
Default
  #20
Member
 
Nick
Join Date: Nov 2011
Location: Tongji University,Shanghai,China
Posts: 33
Blog Entries: 6
Rep Power: 14
sunliming is on a distinguished road
Hi Bruno,

Actually, I'm doing research on wind-wave interaction. I'm a graduate student of wind engineering, my final goal is to study mean wind profiles over the ocean under strong wind and give some instructions of proper wind load on offshore structures.

So the most suitable model for my case should be two-phase flow, but I wanna start with a relatively easy model--imposed solid wave undulation, this is just what waveDisplacement can do. As for 'waves2Foam', I have studied it for a while, but later I found its focus is on the wave rather than wind field which is my concern. And I did some simple simulation, finding only near wave layer of the upper wind field has evident change. The wave cannot take into accout of wind energy transfered to it, thus modification must be made to satisfy my need. So this may be my next task providing current step can be accomplished.

Problems keep coming one after another, so there is still a lot to do.

Anyway, thank you giving so much help!

Best wishes!
Nick
wyldckat likes this.
sunliming is offline   Reply With Quote

Reply

Tags
cyclic boundaries, parallel error, pimpledymfoam, trouble with resuming, wavedisplacement

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
License trouble: ANSLIC_ADMIN, FLEXlm not running, but two lmgrd processes running! dancoz87 ANSYS 17 November 14, 2016 02:10
Trouble with parallel runs cfdmarkus OpenFOAM Running, Solving & CFD 9 February 27, 2009 04:59


All times are GMT -4. The time now is 22:41.