[vkoppejan]

Hi Everyone,

I'm running simulations using CFDEM, which couples OpenFOAM to a DEM package called liggghts. I have a system in which the particles (Npart 5K-1M) require much more cpu's than the cfd mesh (Ncells < 50K).

As you can imagine running 50K cells over several nodes with 24 cpu's each wastes a lot of time on communication and forms a bottleneck to other optimization. To solve this my idea was to change the Pstreams files and create a smaller MPI group from MPI_COMM_WORLD for OpenFOAM to work with.

I'm going through the code, the mpi implementation in the sourcecode appears to be confined to the files Pstream/mpi folder.

Any thoughts on this or any stories on how this didn't work? Both are welcome.

Thanks in advance for your feedback.

Cheers,

Victor

[blais.bruno]

Hey Victor,

I don't have an exact solution to your issue. However, you have to note that all of the processors will help you calculate the interphase coupling (which is actually sometimes more expensive than calculating the CFD), therefore I don't think there is that much to gain here. I suggest you use a smarter decomposition on the CFD side (METIS or SCOTCH), which will reduce the cost associated with the parallel aspect of the CFD.

Obviously you are using too many processors for 50k cells, but it should not be THAT detrimental to your simulation speed.
Also, be careful, since the information of ALL particles is spread onto all processors via some sort of broadcast to all. Therefore, the coupling does not scale that well past 12 processors (from what I have observed)

Cheers!
BB

Originally Posted by vkoppejan

Hi Everyone,

I'm running simulations using CFDEM, which couples OpenFOAM to a DEM package called liggghts. I have a system in which the particles (Npart 5K-1M) require much more cpu's than the cfd mesh (Ncells < 50K).

As you can imagine running 50K cells over several nodes with 24 cpu's each wastes a lot of time on communication and forms a bottleneck to other optimization. To solve this my idea was to change the Pstreams files and create a smaller MPI group from MPI_COMM_WORLD for OpenFOAM to work with.

I'm going through the code, the mpi implementation in the sourcecode appears to be confined to the files Pstream/mpi folder.

Any thoughts on this or any stories on how this didn't work? Both are welcome.

Thanks in advance for your feedback.

Cheers,

Victor

[vkoppejan]

Hi Bruno,

Thanks for the reply. I didn't think of the coupling on the CFD side. I'm using a very simple cylindrical domain so I don't think scotch will do a lot, but I'll give it a try.

With the scaling past twelve cores, do you mean the coupling? As far as I can see LIGGGHTS itself scales quite nicely up to 48 cores (I haven't tried more due to the small CFD mesh).

Cheers,

Victor

[blais.bruno]

LIGGGHTS scale very well up to hundreds of core from my experience, but the coupling scales a bit more poorly. The reason for this is that LIGGGHTS does not know which particles belong to which CFD processor, so it sends all the particle information to all the processors, therefore if you have 1e6 particles, all processors know the position and the velocity of the 1e6 particles. Obviously this is not efficient. There exist a turbo model, but right now it is not public I believe.

I think this is more what is killing your efficiency than the resolution of the CFD équations in this case.

Originally Posted by vkoppejan

Hi Bruno,

Thanks for the reply. I didn't think of the coupling on the CFD side. I'm using a very simple cylindrical domain so I don't think scotch will do a lot, but I'll give it a try.

With the scaling past twelve cores, do you mean the coupling? As far as I can see LIGGGHTS itself scales quite nicely up to 48 cores (I haven't tried more due to the small CFD mesh).

Cheers,

Victor