[nskelly]

Hi,

I'm looking to use directMappedFixedValue for LES simulations in foam-extend 3.2. However, I have come across a problem when working in parallel (which has been discussed before).

The mapped boundary condition doesn't decompose properly. I have tried decomposing with different methods (scotch, metis, hierarchial) and with different number of processors.

It seems to decompose up to 6, but anything beyond this it gives me the following error.

--> FOAM FATAL ERROR:
Did not find sample (-0.0205 0.000156695 0) on any processor of region region0

    From function directMappedPatchBase::findSamples(const pointField&, labelList&, labelList&, pointField&)
    in file directMapped/directMappedPolyPatch/directMappedPatchBase.C at line 381.

FOAM aborting

I am looking to use at least 60 or 120 processors, so at the moment it is not looking promising.

Any help would be great!

N.B. I have also tried on foam-extend 4.1 and it does the same.

[Santiago]

Originally Posted by nskelly

Hi,

I'm looking to use directMappedFixedValue for LES simulations in foam-extend 3.2. However, I have come across a problem when working in parallel (which has been discussed before).

The mapped boundary condition doesn't decompose properly. I have tried decomposing with different methods (scotch, metis, hierarchial) and with different number of processors.

It seems to decompose up to 6, but anything beyond this it gives me the following error.

--> FOAM FATAL ERROR:
Did not find sample (-0.0205 0.000156695 0) on any processor of region region0

    From function directMappedPatchBase::findSamples(const pointField&, labelList&, labelList&, pointField&)
    in file directMapped/directMappedPolyPatch/directMappedPatchBase.C at line 381.

FOAM aborting

I am looking to use at least 60 or 120 processors, so at the moment it is not looking promising.

Any help would be great!

N.B. I have also tried on foam-extend 4.1 and it does the same.

put the patch in question in one single processor. You can do that in decomposeParDict.

[Ardali]

Originally Posted by Santiago

put the patch in question in one single processor. You can do that in decomposeParDict.

OpenFoam extended is an issue. Those versions are older than 5 years.
Try OFplus versions.

[Santiago]

Originally Posted by Ardali

OpenFoam extended is an issue. Those versions are older than 5 years.
Try OFplus versions.

Why foam extend is an issue? latest version of extend is one year old, and its constantly updated.

[Ardali]

Originally Posted by Santiago

Why foam extend is an issue? latest version of extend is one year old, and its constantly updated.

When I was at Chalmers I used to publish with OF-extended3.2. It goes back to 2015-16.
Entended uses many codes and many funtionalities which are not fully debugged, like immersed boundary, and many dynamic meshes.

The BC that you are using may work in extended version but for sure it comes from foundation version.

[nskelly]

I have used other OpenFOAM versions (OF5) and used the mapped condition on that version without any issues.

However, I am using specific solvers only in foam-extend 3.2, hence why I am using this over the 'standard' versions.

[nskelly]

Originally Posted by Santiago

put the patch in question in one single processor. You can do that in decomposeParDict.

I'm too sure what you mean.

[Santiago]

Originally Posted by nskelly

I'm too sure what you mean.

i assume you wanted to say "not sure"... what I mean is that foam-extend uses non-blocking mpi communications that, when reading data from a file may not find the chunck of data "on time"''. This is easily fixed by just putting the "directMapped" patch in a single processor. you can do it by adding your patch in the preservePatches entry in decomposeParDict dictionary...

Remember, IO during runtime is better done by one single process. In OpenFOAM (foundation, etc) they have created buffered commnications, which avoid race conditions, but make the code slower. In extend there us no such thing, so it puts a bit more responsibility on the user...

[Santiago]

Originally Posted by Ardali

Entended uses many codes and many funtionalities which are not fully debugged, like immersed boundary, and many dynamic meshes.

I have not found those "many bugs" that you mention, but i do agree that extend is not as foolproof as OF.

[nskelly]

Hi Santiago,

Originally Posted by Santiago

i assume you wanted to say "not sure"...

Yes (sorry it was a long day yesterday!)

Originally Posted by Santiago

what I mean is that foam-extend uses non-blocking mpi communications that, when reading data from a file may not find the chunck of data "on time"''. This is easily fixed by just putting the "directMapped" patch in a single processor. you can do it by adding your patch in the preservePatches entry in decomposeParDict dictionary...

Remember, IO during runtime is better done by one single process. In OpenFOAM (foundation, etc) they have created buffered commnications, which avoid race conditions, but make the code slower. In extend there us no such thing, so it puts a bit more responsibility on the user...

Thanks for this information, I am actually doing some benchmarking on a new HPC system and this is useful information to know!

I will definitely give it a try, and get back to you.

[Kombinator]

Originally Posted by nskelly

Hi,

I'm looking to use directMappedFixedValue for LES simulations in foam-extend 3.2. However, I have come across a problem when working in parallel (which has been discussed before).

The mapped boundary condition doesn't decompose properly. I have tried decomposing with different methods (scotch, metis, hierarchial) and with different number of processors.

It seems to decompose up to 6, but anything beyond this it gives me the following error.

--> FOAM FATAL ERROR:
Did not find sample (-0.0205 0.000156695 0) on any processor of region region0

    From function directMappedPatchBase::findSamples(const pointField&, labelList&, labelList&, pointField&)
    in file directMapped/directMappedPolyPatch/directMappedPatchBase.C at line 381.

FOAM aborting

I am looking to use at least 60 or 120 processors, so at the moment it is not looking promising.

Any help would be great!

N.B. I have also tried on foam-extend 4.1 and it does the same.

Hi Nat,

I had recently exactly the same problem. There can be three reasons for that:

1. The mapped direction should be always decomposed in one core. So, if you mapped in x-direction then decompose parameters should be like that:

simpleCoeffs
{
n ( 1 y z);
delta 1e-16;
}

where y and z are varied parameters.

2. Reducing the delta parameter to e-16 can also help.

3. Sometimes if the division result of the number of cores in a specific direction by the amount of the mesh nodes in that direction is not an integer you will get the error. To cut a long story short - if you split z-direction into 6 cores and the amount of mesh nodes in the z-direction is 10 then it is possible to get errors (the amount of cores should be 2 or 5 in that case).

I hope it helps.

Regards,
Artem

[nskelly]

Hi Artem,

Thank you for your suggestions. I gave them all a go, but still no luck with get it to decompose properly.

None of your suggestions seem to work for both the pitzDailyDirectMapped tutorial case and my own cases.

Thanks anyway

Nat

[Kombinator]

Originally Posted by nskelly

Hi Artem,

Thank you for your suggestions. I gave them all a go, but still no luck with get it to decompose properly.

None of your suggestions seem to work for both the pitzDailyDirectMapped tutorial case and my own cases.

Thanks anyway

Nat

Hi Nat,

I just checked pitzDailyDirectMappedtutorial and decomposed it for 4 domains without any trouble. I am using foam-extend 4.0
Tutorial decomposition seems to be okay for me..
Did you change something in the tutorial?

regards,
Artem

[nskelly]

Hi Artem,

Yes that tutorial works fine decomposing up to 6 processors. However, I will be using over 60 CPUs so it looks like this will not work for my cases.

Thanks