| hanness |
August 25, 2017 05:06 |
Sharing large scalarList among multiple processors
Hi All,
I'm currently looking for a solution to efficiently share data in a parallel simulation.
The problem is as follows:
In a custom (temperature) boundary condition I want to interpolate the temperature based on several parameters. The data for the interpolation (list of scalars) is read from a file during initialisation of the bc. This list is rather large (36Mio values). My current implementation works fine, but it reads and stores this list for every thread which makes it very slow in the beginning and causes trouble with memory when running on 20 or more processors.
Instead, we thought of using a pointer to the list and reading the list only on the master processor but somehow I can't make the code run properly. And here comes the problem. The relevant constructor of the BC currently looks like this:
Code:
Foam::myTemperatureFvPatchScalarField::
myTemperatureFvPatchScalarField
(
const fvPatch& p,
const DimensionedField<scalar, volMesh>& iF,
const dictionary& dict
)
:
fixedValueFvPatchScalarField(p, iF),
phiName_(dict.lookupOrDefault<word>("phi", "phi")),
airTempIntTblName_(dict.lookup("airTempIntTblName")),
airTempIntTbl_
(
airTempIntTblName_,
this->patch().boundaryMesh().mesh(),
IOdictionary
(
IOobject
(
airTempIntTblName_,
this->patch().boundaryMesh().mesh().time().constant(),
this->patch().boundaryMesh().mesh(),
IOobject::MUST_READ, // must exist, otherwise failure
IOobject::NO_WRITE, // dict is only read by the solver
false // register
)
),
true //initialise only
),
dataPointer_(),
...
{
Pout << " in constructor " << endl;
...
if (Pstream::master())
{
Pout << "in master loop ... " << endl;
uniformMatrixInterpolationTable<scalar> foo
(
uniformMatrixInterpolationTable<scalar>
(
IOobject
(
airTempIntTblName_,
this->patch().boundaryMesh().mesh().time().constant(),
this->patch().boundaryMesh().mesh(),
IOobject::MUST_READ, // must exist, otherwise failure
IOobject::NO_WRITE, // dict is only read by the solver
true
)
)
);
scalarList tblData=scalarList(foo.size(),Zero);
scalarList *dataPtr;
forAll(foo,i)
{
tblData[i]=foo[i];
}
dataPtr = &tblData;
dataPointer_.set(dataPtr);
Pout << "dataPtr " << *dataPtr << endl;
}
// Pstream::scatter(dataPointer_,Pstream::blocking);
}
dataPointer_ is defined in the respective .H file as
Code:
autoPtr<scalarList> dataPointer_;
Now, although it compiles fine, there are several problems with this code. First of all, when running on a single thread it ends with the following error (after finishing the iterations):
Code:
End
*** Error in `buoyantSimpleFoam': double free or corruption (out): 0x00007ffd10596a30 ***
Aborted (core dumped)
When running on multiple threads (here two) it crashes with the following error message during initialisation ( - for debugging purposes the table was shortened to 32 entries):
Code:
[1] in constructor
[0] in constructor
[0] in master loop ...
[0] dataPtr 32{800}
[1]
[1]
[1] --> FOAM FATAL IO ERROR:
[1] incorrect first token, expected <int> or '(', found on line 0 the word 'dictionary'
[1]
[1] file: IOstream at line 0.
[1]
[1] From function Foam::Istream& Foam::operator>>(Foam::Istream&, Foam::List<T>&) [with T = int]
[1] in file /home/ubuntu/OpenFOAM/OpenFOAM-4.1/src/OpenFOAM/lnInclude/ListIO.C at line 148.
[1]
FOAM parallel run exiting
[1]
I thought this happens because the pointer is only available on the master and that I'd have to scatter the pointer to the other processors as well. But when uncommenting the Pstream line in the above code it won't compile:
Code:
g++ -std=c++0x -m64 -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -O3 -DNoRepository -ftemplate-depth-100 -I/opt/openfoam4/src/finiteVolume/lnInclude -I/opt/openfoam4/src/TurbulenceModels/incompressible/lnInclude -I/opt/openfoam4/src/TurbulenceModels/compressible/lnInclude -I/opt/openfoam4/src/turbulenceModels/compressible/lnInclude -I/opt/openfoam4/src/TurbulenceModels/turbulenceModels/lnInclude -I/opt/openfoam4/src/transportModels/compressible/lnInclude -I/opt/openfoam4/src/thermophysicalModels/basic/lnInclude -IlnInclude -I. -I/opt/openfoam4/src/OpenFOAM/lnInclude -I/opt/openfoam4/src/OSspecific/POSIX/lnInclude -fPIC -c myTemperatureFvPatchScalarField.C -o Make/linux64GccDPInt32Opt/myTemperatureFvPatchScalarField.o
In file included from /opt/openfoam4/src/OpenFOAM/lnInclude/Pstream.H:352:0,
from /opt/openfoam4/src/OpenFOAM/lnInclude/mapDistributeBase.H:75,
from /opt/openfoam4/src/OpenFOAM/lnInclude/Field.C:30,
from /opt/openfoam4/src/OpenFOAM/lnInclude/Field.H:405,
from /opt/openfoam4/src/OpenFOAM/lnInclude/labelField.H:39,
from /opt/openfoam4/src/OpenFOAM/lnInclude/primitiveFields.H:37,
from /opt/openfoam4/src/OpenFOAM/lnInclude/pointField.H:36,
from /opt/openfoam4/src/OpenFOAM/lnInclude/edge.H:40,
from /opt/openfoam4/src/OpenFOAM/lnInclude/edgeList.H:32,
from /opt/openfoam4/src/OpenFOAM/lnInclude/PrimitivePatch.H:56,
from /opt/openfoam4/src/OpenFOAM/lnInclude/primitivePatch.H:35,
from /opt/openfoam4/src/OpenFOAM/lnInclude/polyPatch.H:43,
from /opt/openfoam4/src/finiteVolume/lnInclude/fvPatch.H:39,
from /opt/openfoam4/src/finiteVolume/lnInclude/fvPatchField.H:47,
from /opt/openfoam4/src/finiteVolume/lnInclude/fvPatchFields.H:29,
from myTemperatureFvPatchScalarField.H:55,
from myTemperatureFvPatchScalarField.C:26:
/opt/openfoam4/src/OpenFOAM/lnInclude/gatherScatter.C: In instantiation of ‘static void Foam::Pstream::scatter(const Foam::List<Foam::UPstream::commsStruct>&, T&, int, Foam::label) [with T = Foam::autoPtr<Foam::List<double> >; Foam::label = int]’:
/opt/openfoam4/src/OpenFOAM/lnInclude/gatherScatter.C:227:70: required from ‘static void Foam::Pstream::scatter(T&, int, Foam::label) [with T = Foam::autoPtr<Foam::List<double> >; Foam::label = int]’
myTemperatureFvPatchScalarField.C:303:52: required from here
/opt/openfoam4/src/OpenFOAM/lnInclude/gatherScatter.C:184:27: error: no match for ‘operator>>’ (operand types are ‘Foam::IPstream’ and ‘Foam::autoPtr<Foam::List<double> >’)
fromAbove >> Value;
^
/opt/openfoam4/src/OpenFOAM/lnInclude/gatherScatter.C:184:27: note: candidates are:
In file included from /opt/openfoam4/src/OpenFOAM/lnInclude/string.H:48:0,
from /opt/openfoam4/src/OpenFOAM/lnInclude/word.H:42,
from /opt/openfoam4/src/OpenFOAM/lnInclude/wordList.H:41,
from /opt/openfoam4/src/OpenFOAM/lnInclude/patchIdentifier.H:38,
from /opt/openfoam4/src/OpenFOAM/lnInclude/polyPatch.H:42,
from /opt/openfoam4/src/finiteVolume/lnInclude/fvPatch.H:39,
from /opt/openfoam4/src/finiteVolume/lnInclude/fvPatchField.H:47,
from /opt/openfoam4/src/finiteVolume/lnInclude/fvPatchFields.H:29,
from myTemperatureFvPatchScalarField.H:55,
from myTemperatureFvPatchScalarField.C:26:
/opt/openfoam4/src/OpenFOAM/lnInclude/char.H:49:10: note: Foam::Istream& Foam::operator>>(Foam::Istream&, char&)
Istream& operator>>(Istream&, char&);
...
So, I've got a few questions.
1. Is the approach using pointers in this case reasonable or is there a better alternative?
2. Is there anything wrong in the way the pointer is used. (see the error message for a single thread run above)
3. Why doesn't it run in parallel?
I really appreciate any comments and thoughts regarding the above and thanks a lot!
Hannes |
