|
[Sponsors] |
May 29, 2013, 01:05 |
How to gather mesh to one processor?
|
#1 |
Member
yijin Mao
Join Date: May 2010
Location: Columbia, MO
Posts: 62
Rep Power: 15 |
Hi All,
I would like to know how to gather all distributed meshes to one processor, such that all the processor can see the entire mesh. I was trying to make it by using Pstream::gatherList as, List<fvMesh*> allListMesh(Pstream::nProcs()); allListMesh[Pstream::myProcNo()] = & mesh; // everything goes well until now Pstream::gatherList(allListMesh); it throw out errors like error: no match for ‘operator>>’ in ‘fromBelow >> It looks that Pstream::gatherList is written only for certain type. If I try in this way, // this line can not be compiled because of lack of fvMesh constructor. List<fvMesh> allListMesh(Pstream::nProcs()); allListMesh[Pstream::myProcNo()] = mesh; Pstream::gatherList(allListMesh); a similar question: http://www.cfd-online.com/Forums/ope...time-step.html I would like to know if there are other alternatives to make it. Thank you! |
|
May 29, 2013, 16:08 |
|
#2 |
Member
yijin Mao
Join Date: May 2010
Location: Columbia, MO
Posts: 62
Rep Power: 15 |
does anyone have suggestions?
|
|
May 30, 2013, 17:50 |
|
#3 |
Member
Thomas Boucheres
Join Date: May 2013
Posts: 41
Rep Power: 12 |
Hi,
gather meshes on one processor is clearly a very (very) difficult task, mainly because gather or all related MPI functionalities assume a kind of copy construct and a kind of adjacency in datas. It is by no means the case for the fvMesh class. If I can give you my opinion, I never seen the need of such an operation in a CFD software. Working in // is usually done to never have to deal with such a job. Why do you want to do that? Is it really necessary? If you need a kind of knowledge of mesh datas from others processors, you have all the necessary machinery in OF. Try to look at OpenFOAM/mesh/polyMesh/globalMeshData from example. Lastly, if you are REALLY sure you need a full transmission of meshes, please explain why. |
|
May 30, 2013, 18:47 |
|
#4 | |
Member
yijin Mao
Join Date: May 2010
Location: Columbia, MO
Posts: 62
Rep Power: 15 |
Quote:
I think I really need a full transmission of meshes, because I am trying to couple OpenFOAM with Lammps(http://lammps.sandia.gov/) where I need to find the particles based on the meshes info by findCell() function. If I don't have the whole mesh on each processor, then I cannot locate particles correctly, some of them may missing. So I was thinking to gather all the meshes on each processor before do transformation. |
||
May 31, 2013, 08:38 |
|
#5 |
Senior Member
Kyle Mooney
Join Date: Jul 2009
Location: San Francisco, CA USA
Posts: 323
Rep Power: 17 |
If you feel that the problem can only be computed using such huge data synchronizations then you might want to consider simply doing the problem in serial. A mesh transfer like this would inhibit speed scaling as you try to do large problems.
I wouldn't assume that this is the only way to approach the problem. The parallel programming paradigm can be difficult to grasp especially at first so there may be ways to properly parallelize the problem. I've seen lammps coupled to OF in the past so it must be possible. |
|
May 31, 2013, 12:11 |
|
#6 | |
Member
yijin Mao
Join Date: May 2010
Location: Columbia, MO
Posts: 62
Rep Power: 15 |
Quote:
I feel the best way of solving my problem is synchronizing the mesh. If I can have all the mesh info combined on one processor, it will be good, because the mesh domain is not that huge, but I have many particles to be solved with lammps which paralleled solving is much preferred. Solve it in serial is my last choice... One example of coupling LIGGGHTs(lammps) to OF is CFDEM, is this the one you have seen? In my opinion, it is not necessary to transfer mesh into LIGGGHTS(lammps) from OF in CFDEM package. |
||
June 5, 2013, 19:02 |
|
#7 |
Member
Thomas Boucheres
Join Date: May 2013
Posts: 41
Rep Power: 12 |
Hi,
just 2 remarks : 1- have you think about simply reading the primitive global mesh (i.e before running decomposePar) from file with the master processor? It seems to be the most direct / efficient answer to your problem. 2- my knowledge about the parallel data structures associated with the domain decomposition of the mesh being too poor, I find it is a good exercise not to use answer 1 and to try to build a utilitary for gathering meshes without refering with initial mesh. So i will try to do it. If I success, I will send you the source. Probably not very difficult when all parallel data structures are understood. The core class seems to be src/OpenFOAM/mesh/polyMesh/globalPolyMesh. All stuff needed is here I think. The src/parallel/reconstruct/reconstruct utilities can also be usefull. More on this later maybe. |
|
June 6, 2013, 23:49 |
|
#8 | |
Member
yijin Mao
Join Date: May 2010
Location: Columbia, MO
Posts: 62
Rep Power: 15 |
Quote:
|
||
February 9, 2016, 13:38 |
|
#9 |
New Member
Alex Landsberg
Join Date: Sep 2009
Location: Australia
Posts: 15
Rep Power: 0 |
I tried to read the global mesh on the master processor without success. Did you try the second option ?
|
|
February 10, 2016, 05:14 |
|
#10 |
Senior Member
Matvey Kraposhin
Join Date: Mar 2009
Location: Moscow, Russian Federation
Posts: 355
Rep Power: 21 |
Hi, I've done reading serial mesh on master process,
Today later i can upload pieces of code with explanation. Main problem was to turn off OpenFOAM MPI interprocess communications
__________________
MDPI Fluids (Q2) special issue for OSS software: https://www.mdpi.com/journal/fluids/..._modelling_OSS GitHub: https://github.com/unicfdlab Linkedin: https://linkedin.com/in/matvey-kraposhin-413869163 RG: https://www.researchgate.net/profile/Matvey_Kraposhin |
|
February 10, 2016, 12:08 |
reading the global mesh in the master processor
|
#11 |
New Member
Alex Landsberg
Join Date: Sep 2009
Location: Australia
Posts: 15
Rep Power: 0 |
Hello Matvey
Have you got some news Thanks |
|
February 10, 2016, 12:12 |
|
#12 |
Senior Member
Matvey Kraposhin
Join Date: Mar 2009
Location: Moscow, Russian Federation
Posts: 355
Rep Power: 21 |
The main problem, is to turn off OpenFOAM MPI communications
First of all, you need to create static functions, that will do all needed dirty work, let's say suspendMPI() and resumeMPI. Also, suggest that this functions are located in class myClass: Code:
class myClass { public: //- static List<label> oldProcIDs_; //- static List<label> newProcIDs_; //- static void suspendMPI(); //- static void resumeMPI(); }; To make this functions working, you must implement them somewhere in .C file: Code:
List<label> myClass::oldProcIDs_(0); List<label> myClass::newProcIDs_(0); void myClass::suspendMPI() { Pstream::parRun() = false; label comm = Pstream::worldComm; oldProcIDs_ = Pstream::procID(comm); newProcIDs_ = List<label> (1, 0); Pstream::procID(comm) = newProcIDs_; } void myClass::resumeMPI() { label comm = Pstream::worldComm; Pstream::procID(comm) = oldProcIDs_; Pstream::parRun() = true; } Code:
autoPtr<Time> globalTimePtr_; autroPtr<fvMesh> globalMeshPtr_; Time& localTime = runTime; fvMesh& localMesh = mesh; if (Pstream::master()) { const word globalConstant = localTime.rootPath() + "/" + localTime.globalCaseName() + "/constant"; const fileName gRootPath = localTime.rootPath(); const fileName gCaseName = localTime.globalCaseName(); myClass::suspendMPI(); globalTimePtr_.reset ( new Time ( gRootPath, gCaseName ) ); Time& globalTime = globalTimePtr_(); Info << "Creating global mesh " << endl; globalMeshPtr_.reset ( new fvMesh ( IOobject ( fvMesh::defaultRegion, globalTime.timeName(), globalTime, IOobject::MUST_READ ) ) ); myClass::resumeMPI(); }
__________________
MDPI Fluids (Q2) special issue for OSS software: https://www.mdpi.com/journal/fluids/..._modelling_OSS GitHub: https://github.com/unicfdlab Linkedin: https://linkedin.com/in/matvey-kraposhin-413869163 RG: https://www.researchgate.net/profile/Matvey_Kraposhin |
|
February 11, 2016, 07:27 |
gather the global mesh on the master
|
#13 |
New Member
Alex Landsberg
Join Date: Sep 2009
Location: Australia
Posts: 15
Rep Power: 0 |
Thank you for your prompt answer.
I got 2 compilation errors with OF2.1, so i only keep the line with the parallel flag on/off in the functions suspendMPI and resumeMPI and it works Last edited by landsber; February 11, 2016 at 09:05. |
|
February 11, 2016, 23:58 |
|
#14 |
Senior Member
Matvey Kraposhin
Join Date: Mar 2009
Location: Moscow, Russian Federation
Posts: 355
Rep Power: 21 |
Ok, i forgot to note, that this piece if code worked for OF2.3.0
__________________
MDPI Fluids (Q2) special issue for OSS software: https://www.mdpi.com/journal/fluids/..._modelling_OSS GitHub: https://github.com/unicfdlab Linkedin: https://linkedin.com/in/matvey-kraposhin-413869163 RG: https://www.researchgate.net/profile/Matvey_Kraposhin |
|
February 12, 2016, 03:56 |
|
#15 |
Senior Member
Join Date: Oct 2013
Posts: 397
Rep Power: 18 |
Great news! Just for reference, it is also possible to combine fields from multiple processors. This way, you can get information about the mesh such as cell positions, volumes, patches, field values etc...
Code:
template<class Type> void combineFields(Field<Type>& field) { List<Field<Type> > allValues(Pstream::nProcs()); allValues[Pstream::myProcNo()] = field; Pstream::gatherList(allValues); Pstream::scatterList(allValues); field = ListListOps::combine<Field<Type> > ( allValues, accessOp<Field<Type> >() ); } |
|
February 21, 2016, 08:40 |
releasing the memory for the global mesh
|
#16 |
New Member
Alex Landsberg
Join Date: Sep 2009
Location: Australia
Posts: 15
Rep Power: 0 |
I tried to release the memory as globalMeshPtr_.reset(0) but got the following errors :
[0] #0 Foam::errorprintStack(Foam::Ostream&) in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so" [0] #1 Foam::sigSegv::sigHandler(int) in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so" [0] #2 [0] at sigaction.c:0 [0] #3 Foam::List<int>::setSize(int) in "myexecutable" [0] #4 Foam::fileMonitor::removeWatch(int) in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so" [0] #5 Foam::regIOobject::checkOut() in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so" [0] #6 Foam::regIOobject::~regIOobject() in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so" [0] #7 Foam::fvMesh::~fvMesh() Any idea ? |
|
February 22, 2016, 11:34 |
|
#17 |
Senior Member
David Gaden
Join Date: Apr 2009
Location: Winnipeg, Canada
Posts: 437
Rep Power: 21 |
What happens with globalMeshPtr_.clear()?
__________________
~~~ Follow me on twitter @DavidGaden |
|
February 23, 2016, 05:16 |
releasing the memory for the global mesh
|
#18 |
New Member
Alex Landsberg
Join Date: Sep 2009
Location: Australia
Posts: 15
Rep Power: 0 |
same error message as above. This error occurs when I used an autoPtr pointer to read the mesh but without it it works.
|
|
March 10, 2016, 07:15 |
|
#19 |
New Member
jaming
Join Date: Dec 2010
Location: Berlin
Posts: 9
Rep Power: 15 |
Hi Chriss85
for fields your class works fine, but how can I map the mesh to get the information about global neighbours? many thanks jaming |
|
March 11, 2016, 07:13 |
|
#20 |
Senior Member
Join Date: Oct 2013
Posts: 397
Rep Power: 18 |
I think you would have to run this in a loop for every cell of the mesh after you distributed the cell indices. Then you can synchronize each list of neighbours for every cell.
|
|
Thread Tools | Search this Thread |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
[snappyHexMesh] Add Mesh Layers doesnt work on the whole surface | Kryo | OpenFOAM Meshing & Mesh Conversion | 13 | February 17, 2022 07:34 |
AMI speed performance | danny123 | OpenFOAM | 21 | October 24, 2020 04:13 |
Superlinear speedup in OpenFOAM 13 | msrinath80 | OpenFOAM Running, Solving & CFD | 18 | March 3, 2015 05:36 |
Problem with decomposePar tool | vinz | OpenFOAM Pre-Processing | 18 | January 26, 2011 02:17 |
basic of mesh refinement | arya | CFX | 4 | June 19, 2007 12:21 |