CFD Online Discussion Forums - Cannot run the code in parallel. Help please (urgent).

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)

- OpenFOAM (https://www.cfd-online.com/Forums/openfoam/)

- - Cannot run the code in parallel. Help please (urgent). (https://www.cfd-online.com/Forums/openfoam/129608-cannot-run-code-parallel-help-please-urgent.html)

Cannot run the code in parallel. Help please (urgent).

Hi all,

I can run my simulation in series, but I have a problem every time I run my simulation in parallel. I am using OpenFOAM 2.1.1 to simulate a data center cooling performance with k-epsilon model and buoyantBoussinesqSimpleFoam as solver. I always get this following error message:

--> FOAM FATAL IO ERROR:
wrong token type - expected Scalar, found on line 3 the punctuation token '-'

file: /............/processor9/system/data::solverPerformance::epsilon at line 3.

From function operator>>(Istream&, Scalar&)
in file lnInclude/Scalar.C at line 91.

FOAM parallel run exiting

I have tried to search for system/data::solverPerformance::epsilon and also lnInclude/Scalar.C but I couldn't find those files.

Everytime I run the code, I type the command as suggested in tutorial, i.e.

mpirun -np <number of processors> <solver> -parallel > log

Do you have any idea about what's going on? I will appreciate your help so much.

Thanks

You are giving too less information to say something about your case. But this line gives a hint:

Quote:

wrong token type - expected Scalar, found on line 3 the punctuation token '-'

I guess there must be something wrong in your Boundary Conditions. Is the simulation running in serial?

I found no problem when the case was run in serial. The problem occurred when the case was run in parallel.

Previously, I had run exactly the same code in my own laptop (OpenFOAM 2.2.2) and got no problem (so I think my boundary conditions are ok). Now, I have to run using a supercomputer with OpenFOAM 2.1.1 and the problem occurs.

Yesterday, I tried to play with the number of processors used. For some cases, I got no error message only if using 12 processors, and for other cases I got no error message only if using 48 processors. This is so strange. I totally have no idea about this phenomenon. Any clue will be appreciated..:)

Cheers

Hi,

What type of decomposition do you use? Or just post your decomposeParDict.

Hi,

I use exactly similar decomposition as suggested in the tutorial. Here is my decomposeParDict.

Cheers:)

Tutorial is just an example not a suggestion ;) I'd suggest you using scotch decomposition method ;)

I once had similar problem when due to decomposition rather small fragment of mesh become separate domain.

are you using cyclic BCs?

Hi all,

Thanks for your reply. I have tried scotch decomposition method and still doesn't work. I got the following error message:

--> FOAM FATAL IO ERROR:
wrong token type - expected Scalar, found on line 3 the word 'nan'

file: /.............../processor46/system/data::solverPerformance::epsilon at line 3.

From function operator>>(Istream&, Scalar&)
in file lnInclude/Scalar.C at line 91.

I put in bold the difference with the previous error message, i.e. the punctuation token '-' (previously) and the word 'nan' (currently). Referring to the user guide, the two entries inside "scotchCoeffs" are optional. Hence, I made the processorWeights equally distributed and the strategy as default (attached file). I don't know if this induces the error.

@hxaxtma: No, I don't use any cyclic. I just use wall and patch.

Regards,

Steven

Well,

We can continue to play in "guess what's wrong".

Basically error means that during decomposition something goes wrong with epsilon calculation so instead of a number in decomposed variant of epsilon file 'nan' or '-' is written.

0. Does this error appear on the first iteration of solver?
1. Do you use standard solver?
2. Do you use standard BCs? (once I had similar error when BC was implemented incorrectly)
3. What BC do you use for epsilon?
4. What IC do you use?

The easiest way to answer these question is surely to post case files.

Hi,

Sorry for the late reply. To answer for alexeym's question:

0. Does this error appear on the first iteration of solver?
Ans: No, it doesn't appear on the first iteration. The simulation can run up to several time steps but with an exponentially growing time step continuity errors (until 10^100 or even more for sum local, global, and cumulative errors) before terminates.

1. Do you use standard solver?
Ans: I am using buoyantBoussinesqSimpleFoam. Not sure whether it's a standard solver or not. But it is also used in heatTransfer tutorials.

2. Do you use standard BCs? (once I had similar error when BC was implemented incorrectly)
Ans: I found no problem when running the case in series. The only problem I get is when running the case in parallel. Thus, I don't think the BC is incorrect. Even for parallel run, there is 1 "magic" number of processors which can give me no error, while the other numbers always give me error. So strange, right?
Before I run the case in supercomputer, I tried to run it in my laptop (OpenFOAM 2.2.2) using 4 processors and everything was ok. Then, I tried to run it in supercomputer (OpenFOAM 2.1.1) using also 4 processors and I got the error message. I don't know whether it's due to version difference or due to "improper" OpenFOAM installation at the supercomputer or another possible factor.

3. What BC do you use for epsilon?
Ans: I use epsilonWallFunction for all walls, turbulentMixingLengthDissipationRateInlet for all flows going to simulation domain, and zeroGradient for all flows leaving simulation domain.

4. What IC do you use?
Ans: What do you mean IC? Sorry, I'm still new to OpenFOAM :p.

I have also posted the case files :).

Cheers

Well, though I needed to correct boundary conditions to run it on my desktop with OF 2.2.2, it was running ok (at least for 300 time steps).

If you'd like to check if OF installation on the supercomputer is correct - you can install 2.1.1 on your laptop and test if it will blow up. Or you can try to install 2.2.2 in your home folder on supercomputer ;)

But I guess the problem can be in (this is a code from epsilonWallFunctionFvPatchScalarField.C)

Code:

        epsilon[faceCellI] = Cmu75*pow(k[faceCellI], 1.5)/(kappa_*y[faceI]);



        G[faceCellI] =

            (nutw[faceI] + nuw[faceI])

           *magGradUw[faceI]

           *Cmu25*sqrt(k[faceCellI])

           /(kappa_*y[faceI]);

I've checked yPlus values and the grows rather rapidly

Code:

...

Patch 5 named frontWall y+ : min: 5477.84 max: 140055 average: 53592.3



Patch 6 named rack1Wall y+ : min: 11612.2 max: 440140 average: 109040



Patch 7 named rack2Wall y+ : min: 23519.2 max: 297508 average: 103464

...

The difference between 2.1.x and 2.2.x is that when you divide something by very large value 2.1.x will return you nan while 2.2.x will return you 0 (IIRC).

And just to clarify - IC stands for initial conditions ;)

Hi Steven! (and everyone else in this discussion :p)

I'm trying to learn more about performing a CFD analysis for data centers specifically using OF. If any one could suggest some reading material which would help me get on my way I'd be really grateful!

Cheers! :)

P.S: I'm also a relative newbie :p