[alundilong]

Hi All,
I would like to know how to gather all distributed meshes to one processor, such that all the processor can see the entire mesh.

I was trying to make it by using Pstream::gatherList as,

List<fvMesh*> allListMesh(Pstream::nProcs());
allListMesh[Pstream::myProcNo()] = & mesh;

// everything goes well until now
Pstream::gatherList(allListMesh);

it throw out errors like
error: no match for ‘operator>>’ in ‘fromBelow >>

It looks that Pstream::gatherList is written only for certain type.

If I try in this way,
// this line can not be compiled because of lack of fvMesh constructor.
List<fvMesh> allListMesh(Pstream::nProcs());
allListMesh[Pstream::myProcNo()] = mesh;
Pstream::gatherList(allListMesh);

a similar question:
http://www.cfd-online.com/Forums/ope...time-step.html

I would like to know if there are other alternatives to make it. Thank you!

[alundilong]

does anyone have suggestions?

[thomasArk47]

Hi,
gather meshes on one processor is clearly a very (very) difficult task, mainly because gather or all related MPI functionalities assume a kind of copy construct and a kind of adjacency in datas. It is by no means the case for the fvMesh class.
If I can give you my opinion, I never seen the need of such an operation in a CFD software. Working in // is usually done to never have to deal with such a job. Why do you want to do that? Is it really necessary?

If you need a kind of knowledge of mesh datas from others processors, you have all the necessary machinery in OF. Try to look at OpenFOAM/mesh/polyMesh/globalMeshData from example.

Lastly, if you are REALLY sure you need a full transmission of meshes, please explain why.

[alundilong]

Originally Posted by thomasArk47

Hi,
gather meshes on one processor is clearly a very (very) difficult task, mainly because gather or all related MPI functionalities assume a kind of copy construct and a kind of adjacency in datas. It is by no means the case for the fvMesh class.
If I can give you my opinion, I never seen the need of such an operation in a CFD software. Working in // is usually done to never have to deal with such a job. Why do you want to do that? Is it really necessary?

If you need a kind of knowledge of mesh datas from others processors, you have all the necessary machinery in OF. Try to look at OpenFOAM/mesh/polyMesh/globalMeshData from example.

Lastly, if you are REALLY sure you need a full transmission of meshes, please explain why.

thank you for the suggestion.
I think I really need a full transmission of meshes, because I am trying to couple OpenFOAM with Lammps(http://lammps.sandia.gov/) where I need to find the particles based on the meshes info by findCell() function. If I don't have the whole mesh on each processor, then I cannot locate particles correctly, some of them may missing. So I was thinking to gather all the meshes on each processor before do transformation.

[kmooney]

If you feel that the problem can only be computed using such huge data synchronizations then you might want to consider simply doing the problem in serial. A mesh transfer like this would inhibit speed scaling as you try to do large problems.

I wouldn't assume that this is the only way to approach the problem. The parallel programming paradigm can be difficult to grasp especially at first so there may be ways to properly parallelize the problem. I've seen lammps coupled to OF in the past so it must be possible.

[alundilong]

Originally Posted by kmooney

If you feel that the problem can only be computed using such huge data synchronizations then you might want to consider simply doing the problem in serial. A mesh transfer like this would inhibit speed scaling as you try to do large problems.

I wouldn't assume that this is the only way to approach the problem. The parallel programming paradigm can be difficult to grasp especially at first so there may be ways to properly parallelize the problem. I've seen lammps coupled to OF in the past so it must be possible.

Thanks!
I feel the best way of solving my problem is synchronizing the mesh. If I can have all the mesh info combined on one processor, it will be good, because the mesh domain is not that huge, but I have many particles to be solved with lammps which paralleled solving is much preferred. Solve it in serial is my last choice...

One example of coupling LIGGGHTs(lammps) to OF is CFDEM, is this the one you have seen? In my opinion, it is not necessary to transfer mesh into LIGGGHTS(lammps) from OF in CFDEM package.

[thomasArk47]

Hi,

just 2 remarks :

1- have you think about simply reading the primitive global mesh (i.e before running decomposePar) from file with the master processor? It seems to be the most direct / efficient answer to your problem.

2- my knowledge about the parallel data structures associated with the domain decomposition of the mesh being too poor, I find it is a good exercise not to use answer 1 and to try to build a utilitary for gathering meshes without refering with initial mesh. So i will try to do it. If I success, I will send you the source. Probably not very difficult when all parallel data structures are understood. The core class seems to be src/OpenFOAM/mesh/polyMesh/globalPolyMesh. All stuff needed is here I think. The src/parallel/reconstruct/reconstruct utilities can also be usefull.

More on this later maybe.

[alundilong]

Originally Posted by thomasArk47

Hi,

just 2 remarks :

1- have you think about simply reading the primitive global mesh (i.e before running decomposePar) from file with the master processor? It seems to be the most direct / efficient answer to your problem.

2- my knowledge about the parallel data structures associated with the domain decomposition of the mesh being too poor, I find it is a good exercise not to use answer 1 and to try to build a utilitary for gathering meshes without refering with initial mesh. So i will try to do it. If I success, I will send you the source. Probably not very difficult when all parallel data structures are understood. The core class seems to be src/OpenFOAM/mesh/polyMesh/globalPolyMesh. All stuff needed is here I think. The src/parallel/reconstruct/reconstruct utilities can also be usefull.

More on this later maybe.

Thank you for your update. I'm trying to construct mesh within lammps with fvMesh class, but it is difficult. I still think gathering all the mesh data within one processor is the best way. So I will also read through the source code you suggested. Looking forward your success!

[landsber]

I tried to read the global mesh on the master processor without success. Did you try the second option ?

[mkraposhin]

Hi, I've done reading serial mesh on master process,
Today later i can upload pieces of code with explanation.
Main problem was to turn off OpenFOAM MPI interprocess communications

[landsber]

Hello Matvey

Have you got some news
Thanks

[mkraposhin]

The main problem, is to turn off OpenFOAM MPI communications

First of all, you need to create static functions, that will do all needed dirty work,
let's say suspendMPI() and resumeMPI. Also, suggest that this functions are located in class myClass:

class myClass
{
public:
    
    //-
    static List<label> oldProcIDs_;
    
    //-
    static List<label> newProcIDs_;
    
    //-
    static void suspendMPI();
    
    //-
    static void resumeMPI();
};

This functions uses class members newProcIDs_ and oldProcIDs_;
To make this functions working, you must implement them somewhere in .C file:

List<label> myClass::oldProcIDs_(0);

List<label> myClass::newProcIDs_(0);

void myClass::suspendMPI()
{
    Pstream::parRun() = false;
    label comm        = Pstream::worldComm;
    oldProcIDs_       = Pstream::procID(comm);
    newProcIDs_       = List<label> (1, 0);

    Pstream::procID(comm) = newProcIDs_;
}

void myClass::resumeMPI()
{
    label comm        = Pstream::worldComm;
    Pstream::procID(comm) = oldProcIDs_;
    Pstream::parRun() = true;
}

Now you can use this functions to manage MPI communications - whether they are enabled or not in OpenFOAM:

    autoPtr<Time> globalTimePtr_;
    autroPtr<fvMesh> globalMeshPtr_;
    Time& localTime = runTime;
    fvMesh& localMesh = mesh;

	if (Pstream::master())
	{
	    const word globalConstant = localTime.rootPath() + "/" + localTime.globalCaseName() + "/constant";
    
	    const fileName gRootPath = localTime.rootPath();
	    const fileName gCaseName = localTime.globalCaseName();
	    
	    myClass::suspendMPI();
	    
	    globalTimePtr_.reset
	    (
		new Time
		(
		    gRootPath,
		    gCaseName
		)
	    );
	
	    Time& globalTime = globalTimePtr_();
	    
	    Info << "Creating global mesh " << endl;
	    
	    globalMeshPtr_.reset
	    (
		new fvMesh
		(
		    IOobject
		    (
			fvMesh::defaultRegion,
			globalTime.timeName(),
			globalTime,
			IOobject::MUST_READ
		    )
		)
	    );
            
           myClass::resumeMPI();
	}

[landsber]

Thank you for your prompt answer.
I got 2 compilation errors with OF2.1, so i only keep the line with the parallel flag on/off in the functions suspendMPI and resumeMPI and it works

[mkraposhin]

Originally Posted by landsber

Thank you for your prompt answer.
I got 2 compilation errors with OF2.1, so i only keep the line with the parallel flag on/off in the functions suspendMPI and resumeMPI and it works

Ok, i forgot to note, that this piece if code worked for OF2.3.0

[chriss85]

Great news! Just for reference, it is also possible to combine fields from multiple processors. This way, you can get information about the mesh such as cell positions, volumes, patches, field values etc...

template<class Type>
    void combineFields(Field<Type>& field)
    {
        List<Field<Type> > allValues(Pstream::nProcs());

        allValues[Pstream::myProcNo()] = field;

        Pstream::gatherList(allValues);
        Pstream::scatterList(allValues);

        field =
        ListListOps::combine<Field<Type> >
        (
            allValues,
            accessOp<Field<Type> >()
        );
    }

[landsber]

I tried to release the memory as globalMeshPtr_.reset(0) but got the following errors :
[0] #0 Foam::errorprintStack(Foam::Ostream&) in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
[0] #1 Foam::sigSegv::sigHandler(int) in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
[0] #2
[0] at sigaction.c:0
[0] #3 Foam::List<int>::setSize(int) in "myexecutable"
[0] #4 Foam::fileMonitor::removeWatch(int) in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
[0] #5 Foam::regIOobject::checkOut() in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
[0] #6 Foam::regIOobject::~regIOobject() in "/applhome/openfoam/openfoam21/OpenFOAM-2.1.0/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
[0] #7 Foam::fvMesh::~fvMesh()

Any idea ?

[marupio]

What happens with globalMeshPtr_.clear()?

[landsber]

same error message as above. This error occurs when I used an autoPtr pointer to read the mesh but without it it works.

[jaming]

Hi Chriss85
for fields your class works fine, but how can I map the mesh to get the information about global neighbours?

many thanks
jaming

[chriss85]

I think you would have to run this in a loop for every cell of the mesh after you distributed the cell indices. Then you can synchronize each list of neighbours for every cell.