CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM Programming & Development

How to gather mesh to one processor?

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree1Likes
  • 1 Post By mkraposhin

Reply
 
LinkBack Thread Tools Display Modes
Old   May 29, 2013, 01:05
Default How to gather mesh to one processor?
  #1
Member
 
yijin Mao
Join Date: May 2010
Location: Columbia, MO
Posts: 48
Rep Power: 8
alundilong is on a distinguished road
Hi All,
I would like to know how to gather all distributed meshes to one processor, such that all the processor can see the entire mesh.

I was trying to make it by using Pstream::gatherList as,

List<fvMesh*> allListMesh(Pstream::nProcs());
allListMesh[Pstream::myProcNo()] = & mesh;

// everything goes well until now
Pstream::gatherList(allListMesh);

it throw out errors like
error: no match for ‘operator>>’ in ‘fromBelow >>

It looks that Pstream::gatherList is written only for certain type.

If I try in this way,
// this line can not be compiled because of lack of fvMesh constructor.
List<fvMesh> allListMesh(Pstream::nProcs());
allListMesh[Pstream::myProcNo()] = mesh;
Pstream::gatherList(allListMesh);

a similar question:
Map fields every time step

I would like to know if there are other alternatives to make it. Thank you!
alundilong is offline   Reply With Quote

Old   May 29, 2013, 16:08
Default
  #2
Member
 
yijin Mao
Join Date: May 2010
Location: Columbia, MO
Posts: 48
Rep Power: 8
alundilong is on a distinguished road
does anyone have suggestions?
alundilong is offline   Reply With Quote

Old   May 30, 2013, 17:50
Default
  #3
Member
 
Thomas Boucheres
Join Date: May 2013
Posts: 37
Rep Power: 5
thomasArk47 is on a distinguished road
Hi,
gather meshes on one processor is clearly a very (very) difficult task, mainly because gather or all related MPI functionalities assume a kind of copy construct and a kind of adjacency in datas. It is by no means the case for the fvMesh class.
If I can give you my opinion, I never seen the need of such an operation in a CFD software. Working in // is usually done to never have to deal with such a job. Why do you want to do that? Is it really necessary?

If you need a kind of knowledge of mesh datas from others processors, you have all the necessary machinery in OF. Try to look at OpenFOAM/mesh/polyMesh/globalMeshData from example.

Lastly, if you are REALLY sure you need a full transmission of meshes, please explain why.
thomasArk47 is offline   Reply With Quote

Old   May 30, 2013, 18:47
Default
  #4
Member
 
yijin Mao
Join Date: May 2010
Location: Columbia, MO
Posts: 48
Rep Power: 8
alundilong is on a distinguished road
Quote:
Originally Posted by thomasArk47 View Post
Hi,
gather meshes on one processor is clearly a very (very) difficult task, mainly because gather or all related MPI functionalities assume a kind of copy construct and a kind of adjacency in datas. It is by no means the case for the fvMesh class.
If I can give you my opinion, I never seen the need of such an operation in a CFD software. Working in // is usually done to never have to deal with such a job. Why do you want to do that? Is it really necessary?

If you need a kind of knowledge of mesh datas from others processors, you have all the necessary machinery in OF. Try to look at OpenFOAM/mesh/polyMesh/globalMeshData from example.

Lastly, if you are REALLY sure you need a full transmission of meshes, please explain why.
thank you for the suggestion.
I think I really need a full transmission of meshes, because I am trying to couple OpenFOAM with Lammps(http://lammps.sandia.gov/) where I need to find the particles based on the meshes info by findCell() function. If I don't have the whole mesh on each processor, then I cannot locate particles correctly, some of them may missing. So I was thinking to gather all the meshes on each processor before do transformation.
alundilong is offline   Reply With Quote

Old   May 31, 2013, 08:38
Default
  #5
Senior Member
 
kmooney's Avatar
 
Kyle Mooney
Join Date: Jul 2009
Location: Amherst, MA USA - San Diego, CA USA
Posts: 319
Rep Power: 9
kmooney is on a distinguished road
If you feel that the problem can only be computed using such huge data synchronizations then you might want to consider simply doing the problem in serial. A mesh transfer like this would inhibit speed scaling as you try to do large problems.

I wouldn't assume that this is the only way to approach the problem. The parallel programming paradigm can be difficult to grasp especially at first so there may be ways to properly parallelize the problem. I've seen lammps coupled to OF in the past so it must be possible.
kmooney is offline   Reply With Quote

Old   May 31, 2013, 12:11
Default
  #6
Member
 
yijin Mao
Join Date: May 2010
Location: Columbia, MO
Posts: 48
Rep Power: 8
alundilong is on a distinguished road
Quote:
Originally Posted by kmooney View Post
If you feel that the problem can only be computed using such huge data synchronizations then you might want to consider simply doing the problem in serial. A mesh transfer like this would inhibit speed scaling as you try to do large problems.

I wouldn't assume that this is the only way to approach the problem. The parallel programming paradigm can be difficult to grasp especially at first so there may be ways to properly parallelize the problem. I've seen lammps coupled to OF in the past so it must be possible.
Thanks!
I feel the best way of solving my problem is synchronizing the mesh. If I can have all the mesh info combined on one processor, it will be good, because the mesh domain is not that huge, but I have many particles to be solved with lammps which paralleled solving is much preferred. Solve it in serial is my last choice...

One example of coupling LIGGGHTs(lammps) to OF is CFDEM, is this the one you have seen? In my opinion, it is not necessary to transfer mesh into LIGGGHTS(lammps) from OF in CFDEM package.
alundilong is offline   Reply With Quote

Old   June 5, 2013, 19:02
Default
  #7
Member
 
Thomas Boucheres
Join Date: May 2013
Posts: 37
Rep Power: 5
thomasArk47 is on a distinguished road
Hi,

just 2 remarks :

1- have you think about simply reading the primitive global mesh (i.e before running decomposePar) from file with the master processor? It seems to be the most direct / efficient answer to your problem.

2- my knowledge about the parallel data structures associated with the domain decomposition of the mesh being too poor, I find it is a good exercise not to use answer 1 and to try to build a utilitary for gathering meshes without refering with initial mesh. So i will try to do it. If I success, I will send you the source. Probably not very difficult when all parallel data structures are understood. The core class seems to be src/OpenFOAM/mesh/polyMesh/globalPolyMesh. All stuff needed is here I think. The src/parallel/reconstruct/reconstruct utilities can also be usefull.

More on this later maybe.
thomasArk47 is offline   Reply With Quote

Old   June 6, 2013, 23:49
Default
  #8
Member
 
yijin Mao
Join Date: May 2010
Location: Columbia, MO
Posts: 48
Rep Power: 8
alundilong is on a distinguished road
Quote:
Originally Posted by thomasArk47 View Post
Hi,

just 2 remarks :

1- have you think about simply reading the primitive global mesh (i.e before running decomposePar) from file with the master processor? It seems to be the most direct / efficient answer to your problem.

2- my knowledge about the parallel data structures associated with the domain decomposition of the mesh being too poor, I find it is a good exercise not to use answer 1 and to try to build a utilitary for gathering meshes without refering with initial mesh. So i will try to do it. If I success, I will send you the source. Probably not very difficult when all parallel data structures are understood. The core class seems to be src/OpenFOAM/mesh/polyMesh/globalPolyMesh. All stuff needed is here I think. The src/parallel/reconstruct/reconstruct utilities can also be usefull.

More on this later maybe.
Thank you for your update. I'm trying to construct mesh within lammps with fvMesh class, but it is difficult. I still think gathering all the mesh data within one processor is the best way. So I will also read through the source code you suggested. Looking forward your success!
alundilong is offline   Reply With Quote

Old   February 9, 2016, 14:38
Default
  #9
New Member
 
Alex Landsberg
Join Date: Sep 2009
Location: Australia
Posts: 5
Rep Power: 8
landsber is on a distinguished road
I tried to read the global mesh on the master processor without success. Did you try the second option ?
landsber is offline   Reply With Quote

Old   February 10, 2016, 06:14
Default
  #10
Senior Member
 
mkraposhin's Avatar
 
Matvey Kraposhin
Join Date: Mar 2009
Location: Moscow, Russian Federation
Posts: 330
Rep Power: 11
mkraposhin is on a distinguished road
Hi, I've done reading serial mesh on master process,
Today later i can upload pieces of code with explanation.
Main problem was to turn off OpenFOAM MPI interprocess communications
landsber likes this.
__________________
Winter OpenFOAM days in Moscow - http://www.opencloudconf.ru/en/oscome.html
mkraposhin is offline   Reply With Quote

Old   February 10, 2016, 13:08
Default reading the global mesh in the master processor
  #11
New Member
 
Alex Landsberg
Join Date: Sep 2009
Location: Australia
Posts: 5
Rep Power: 8
landsber is on a distinguished road
Hello Matvey

Have you got some news
Thanks
landsber is offline   Reply With Quote

Old   February 10, 2016, 13:12
Default
  #12
Senior Member
 
mkraposhin's Avatar
 
Matvey Kraposhin
Join Date: Mar 2009
Location: Moscow, Russian Federation
Posts: 330
Rep Power: 11
mkraposhin is on a distinguished road
The main problem, is to turn off OpenFOAM MPI communications

First of all, you need to create static functions, that will do all needed dirty work,
let's say suspendMPI() and resumeMPI. Also, suggest that this functions are located in class myClass:
Code:
class myClass
{
public:
    
    //-
    static List<label> oldProcIDs_;
    
    //-
    static List<label> newProcIDs_;
    
    //-
    static void suspendMPI();
    
    //-
    static void resumeMPI();
};
This functions uses class members newProcIDs_ and oldProcIDs_;
To make this functions working, you must implement them somewhere in .C file:

Code:
List<label> myClass::oldProcIDs_(0);

List<label> myClass::newProcIDs_(0);

void myClass::suspendMPI()
{
    Pstream::parRun() = false;
    label comm        = Pstream::worldComm;
    oldProcIDs_       = Pstream::procID(comm);
    newProcIDs_       = List<label> (1, 0);

    Pstream::procID(comm) = newProcIDs_;
}

void myClass::resumeMPI()
{
    label comm        = Pstream::worldComm;
    Pstream::procID(comm) = oldProcIDs_;
    Pstream::parRun() = true;
}
Now you can use this functions to manage MPI communications - whether they are enabled or not in OpenFOAM:
Code:
    autoPtr<Time> globalTimePtr_;
    autroPtr<fvMesh> globalMeshPtr_;
    Time& localTime = runTime;
    fvMesh& localMesh = mesh;

	if (Pstream::master())
	{
	    const word globalConstant = localTime.rootPath() + "/" + localTime.globalCaseName() + "/constant";
    
	    const fileName gRootPath = localTime.rootPath();
	    const fileName gCaseName = localTime.globalCaseName();
	    
	    myClass::suspendMPI();
	    
	    globalTimePtr_.reset
	    (
		new Time
		(
		    gRootPath,
		    gCaseName
		)
	    );
	
	    Time& globalTime = globalTimePtr_();
	    
	    Info << "Creating global mesh " << endl;
	    
	    globalMeshPtr_.reset
	    (
		new fvMesh
		(
		    IOobject
		    (
			fvMesh::defaultRegion,
			globalTime.timeName(),
			globalTime,
			IOobject::MUST_READ
		    )
		)
	    );
            
           myClass::resumeMPI();
	}
__________________
Winter OpenFOAM days in Moscow - http://www.opencloudconf.ru/en/oscome.html
mkraposhin is offline   Reply With Quote

Old   February 11, 2016, 08:27
Default gather the global mesh on the master
  #13
New Member
 
Alex Landsberg
Join Date: Sep 2009
Location: Australia
Posts: 5
Rep Power: 8
landsber is on a distinguished road
Thank you for your prompt answer.
I got 2 compilation errors with OF2.1, so i only keep the line with the parallel flag on/off in the functions suspendMPI and resumeMPI and it works

Last edited by landsber; February 11, 2016 at 10:05.
landsber is offline   Reply With Quote

Old   February 12, 2016, 00:58
Default
  #14
Senior Member
 
mkraposhin's Avatar
 
Matvey Kraposhin
Join Date: Mar 2009
Location: Moscow, Russian Federation
Posts: 330
Rep Power: 11
mkraposhin is on a distinguished road
Quote:
Originally Posted by landsber View Post
Thank you for your prompt answer.
I got 2 compilation errors with OF2.1, so i only keep the line with the parallel flag on/off in the functions suspendMPI and resumeMPI and it works
Ok, i forgot to note, that this piece if code worked for OF2.3.0
__________________
Winter OpenFOAM days in Moscow - http://www.opencloudconf.ru/en/oscome.html
mkraposhin is offline   Reply With Quote

Old   February 12, 2016, 04:56
Default
  #15
Senior Member
 
Join Date: Oct 2013
Posts: 361
Rep Power: 6
chriss85 is on a distinguished road
Great news! Just for reference, it is also possible to combine fields from multiple processors. This way, you can get information about the mesh such as cell positions, volumes, patches, field values etc...
Code:
template<class Type>
    void combineFields(Field<Type>& field)
    {
        List<Field<Type> > allValues(Pstream::nProcs());

        allValues[Pstream::myProcNo()] = field;

        Pstream::gatherList(allValues);
        Pstream::scatterList(allValues);

        field =
        ListListOps::combine<Field<Type> >
        (
            allValues,
            accessOp<Field<Type> >()
        );
    }
chriss85 is offline   Reply With Quote

Old   February 21, 2016, 09:40
Default releasing the memory for the global mesh
  #16
New Member
 
Alex Landsberg
Join Date: Sep 2009
Location: Australia
Posts: 5
Rep Power: 8
landsber is on a distinguished road
I tried to release the memory as globalMeshPtr_.reset(0) but got the following errors :
[0] #0 Foam::errorprintStack(Foam::Ostream&) in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
[0] #1 Foam::sigSegv::sigHandler(int) in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
[0] #2
[0] at sigaction.c:0
[0] #3 Foam::List<int>::setSize(int) in "myexecutable"
[0] #4 Foam::fileMonitor::removeWatch(int) in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
[0] #5 Foam::regIOobject::checkOut() in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
[0] #6 Foam::regIOobject::~regIOobject() in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
[0] #7 Foam::fvMesh::~fvMesh()

Any idea ?
landsber is offline   Reply With Quote

Old   February 22, 2016, 12:34
Default
  #17
Senior Member
 
David Gaden
Join Date: Apr 2009
Location: Winnipeg, Canada
Posts: 430
Rep Power: 14
marupio is on a distinguished road
What happens with globalMeshPtr_.clear()?
__________________
~~~
Follow me on twitter @DavidGaden
marupio is offline   Reply With Quote

Old   February 23, 2016, 06:16
Default releasing the memory for the global mesh
  #18
New Member
 
Alex Landsberg
Join Date: Sep 2009
Location: Australia
Posts: 5
Rep Power: 8
landsber is on a distinguished road
same error message as above. This error occurs when I used an autoPtr pointer to read the mesh but without it it works.
landsber is offline   Reply With Quote

Old   March 10, 2016, 08:15
Default
  #19
New Member
 
jaming
Join Date: Dec 2010
Location: Berlin
Posts: 9
Rep Power: 7
jaming is on a distinguished road
Hi Chriss85
for fields your class works fine, but how can I map the mesh to get the information about global neighbours?

many thanks
jaming
jaming is offline   Reply With Quote

Old   March 11, 2016, 08:13
Default
  #20
Senior Member
 
Join Date: Oct 2013
Posts: 361
Rep Power: 6
chriss85 is on a distinguished road
I think you would have to run this in a loop for every cell of the mesh after you distributed the cell indices. Then you can synchronize each list of neighbours for every cell.
chriss85 is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Superlinear speedup in OpenFOAM 13 msrinath80 OpenFOAM Running, Solving & CFD 18 March 3, 2015 06:36
AMI speed performance danny123 OpenFOAM 19 October 24, 2012 07:44
Add Mesh Layers doesnt work on the whole surface Kryo OpenFOAM Native Meshers: snappyHexMesh and Others 8 September 13, 2012 09:28
Problem with decomposePar tool vinz OpenFOAM Pre-Processing 18 January 26, 2011 03:17
basic of mesh refinement arya CFX 4 June 19, 2007 12:21


All times are GMT -4. The time now is 15:23.