[psilkeit]

Dear FOAMers,
I'm working on an apllication which utilizes the values of a scalarField interpolated to the faces and the area of the faces of a faceZone to calculate a scalar value at an observer point at an artificial time step which is stored in a scalarListList. The application runs fine on a single core and produces the correct results, when the calculation is performed in parallel the application stalls without aborting or an error message when the calculation is performed for the first time.

After some research into the matter i think the parallelization does not work automatically since i am not working with the usual fields. The search lead me to try the following approach to write out the data ( i think the calculation runs fine in parallel aswell but its a processor comunication problem but thats just a guess...)

// The important variable
scalarListList  stuff  // the stuff i want to calculate
stuff = scalarListList( number of observers);
forAll(observers,obsI)
{
   //artificially calculated timesteps
   stuff[obsI]=scalarList( number of artificial time steps ) 
}
// The calculation of the variable
forAll(observerPoints,pointI)
{
  forAll(zoneFaces,faceI)
  {
      stuff[pointI][artificialTimeI]= calculation dependent on pointI and faceI; 
      // the correct artificialTimeI is found in side the above calculation
}
}

// attempt to gather and write the variable
    List<List< List<scalar> >> gatheredStuff(Pstream::nProcs());
    gatheredStuff[Pstream::myProcNo()] = stuff;
    Pstream::gatherList(gatheredStuff);

// Only the master proc should write out the accumulated data
    if (Pstream::master())
    {
        if(runTime.write())
        {
                mkDir(runTime.path()/"Results");
                fileName outputFile("Results.TXT");
                OFstream os(runTime.path()/"Results"/outputFile);
                os << "Observer Points Test " << endl;
                forAll(artificialTimeSteps, stepI)
                {
                  os << timeSteps[stepI];
                  os << "\t";
                     forAll(observerPoints,pointI)
                      {
                         os << gatheredStuff[pointI][stepI];
                         os << "\t";
                      }
                  os << endl;
                }
        }
    }

As I said the calculation and the writing of the data works like a charm on a single processor but the application freezes in parallel when it is performed for the first time.

Any input, suggestion or help giving me a hint why this freezes and how to prevent this is highly appreciated.

Best regards,
Patrick

[andersking]

I think you should use runTime.outputTime() in your test instead of runTime.write(). write() is linked to fields, which have parallel aware IO. starting this only on the master ends up with a communications mismatch.

The other option is to switch the Pstream::master() check with runTime.write() (but this may also rewrite the fields).

Cheers,
Andrew

[psilkeit]

Dear Andrew,

thanks for your reply that makes sense and I changed it accordingly, it doesn't solve my problem though...

To further locate the problem I added the following Pout statements:

Pout << "Processor Numer 1 " << Pstream::myProcNo() << endl;    
List<List< List<scalar> >> gatheredStuff(Pstream::nProcs());

Pout << "Processor Numer 2 " << Pstream::myProcNo() << endl;    
gatheredStuff[Pstream::myProcNo()] = stuff;

Pout << "Processor Numer 3 " << Pstream::myProcNo() << endl;    
Pstream::gatherList(gatheredStuff);

Pout << "Processor Numer 4 " << Pstream::myProcNo() << endl;    

if (Pstream::master())
{ 
   if (runTime.outputTime())
   {
        etc...

The output gives:

Performing the New Application 

[1] Processor Number 1 1
[1] Processor Number 2 1
[1] Processor Number 3 1

So my guess is that the Pstream::gatherList() function does not work properly in my case... Could it be that it cannot operate on a ListListList?

Secondly I parallelize on two processors shouldn't the Pout give two outputs for each call then ? Or did is misunderstand something there?

Any further help is highly appreciated.

Cheers

[psilkeit]

Dear All,
I managed to solve the problem. The Pstream::gatherList comand does work. What i didn't realize was that I had to do further operations on the list to add together the corresponding parts that are contained in the list after gathering and then write it out.

I thus reduce the gatheredStuff to a list equal in size to stuff by adding the corresponding parts in a for loop

Therefor, the skeleton code above is actually working

I changed the write procedure according to this post http://www.cfd-online.com/Forums/ope...tml#post459498
so the output file is written to the case directory and not the master processor directory.

Thanks to everyone who looked at this.

Greetz