CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > OpenFOAM

Parallel Computing decomposePar

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree10Likes

Reply
 
LinkBack Thread Tools Display Modes
Old   April 11, 2012, 21:57
Default
  #21
New Member
 
xiaoweii
Join Date: Mar 2012
Posts: 7
Rep Power: 5
xiaow_g is on a distinguished road
Hi Bruno,
Thanks for your reply. I run snappyHexMesh in OpenFOAM-2.1.0 on Tianhe-1A, a supercomputer. So the memory shouldn't be a problem.
OMG! I will re-download a OpenFOAM-2.1.0 and try again, hope that it will work. Thank you again!
Best wishes!
Xiaow-g










Quote:
Originally Posted by wyldckat View Post
Greetings to all!

@xiaow_g: I know I've had a similar problem, where snappyHexMesh kept running at 100% in parallel and wouldn't continue, but I can't remember what the problem was. It might be a memory issue like Chris said, but it could also be a bug you might be triggering somehow.

It would be useful to know the Linux distribution and architecture (uname -m) you are using, as well how much RAM is available to your Linux installation (in case you are running it inside a virtual machine).

The other possibility is that you need to upgrade to OpenFOAM 2.1.x. Several bugs have been fixed since 2.1.0 was released and I vaguely remember having problems with snappyHexMesh in parallel with one versions that wasn't from git...

Best regards,
Bruno
xiaow_g is offline   Reply With Quote

Old   May 21, 2012, 14:18
Default
  #22
Senior Member
 
lore
Join Date: Mar 2010
Location: Italy
Posts: 463
Rep Power: 9
lovecraft22 is on a distinguished road
Send a message via Skype™ to lovecraft22
Hi, as I'm experiencing the same error (Parallel run: boundary condition missing on processorX/0) is there any way to set up a case from scratch in such a way that this error here is avoided without the need to run changeDict?

Thank you!
lovecraft22 is offline   Reply With Quote

Old   June 13, 2012, 10:44
Default strange behaviour DecomposePar
  #23
Member
 
Lev
Join Date: Dec 2010
Posts: 31
Rep Power: 6
levka is on a distinguished road
Hello OF experts and users,
please assist me in struggling the following problem:

during run of icoFoam i included my utility:
Code:
....
        runTime.write();
    #include "utility.H";

        Info<< "ExecutionTime = " << runTime.elapsedCpuTime() << " s"
            << "  ClockTime = " << runTime.elapsedClockTime() << " s"
            << nl << endl;
......
"utility. H" consists of one loop, that prints values of U in cell centres along Y direction in the center of the channel (dimensions: x=0..6,Y=-1..1,Z=0..3; Nx=120,Ny=64,Nz=120):

Code:
scalar x,y,z;

forAll(mesh.cellCentres(),ii)
{
    x=y=z=0;
    x=mesh.cellCentres()[ii].x();
    y=mesh.cellCentres()[ii].y();
    z=mesh.cellCentres()[ii].z();

      if(x<3.10&&x>3.03&&z<1.58&&z>1.55)
    {
     Info<<"ii="<<ii<<" y="<<y<<" u="<<U[ii]<<endl;
    }

}
in single run output is perfect: 64 values of U

Code:
ii=315457 y=-0.996054 u=(-0.00620626 6.79366e-05 0.0108981)
ii=315585 y=-0.987859 u=(-0.00614572 -0.000243301 0.0243829)
ii=315713 y=-0.979032 u=(-0.00606576 -0.000949787 0.0380821)
ii=315841 y=-0.969524 u=(-0.00595958 -0.00210503 0.0521407)
ii=315969 y=-0.959283 u=(-0.00582333 -0.00378294 0.0664832)
ii=316097 y=-0.948252 u=(-0.00565297 -0.00606614 0.0810126)
ii=316225 y=-0.936371 u=(-0.00544439 -0.00904414 0.0956093)
ii=316353 y=-0.923574 u=(-0.00519362 -0.012812 0.110129)
ii=316481 y=-0.90979 u=(-0.00489682 -0.0174689 0.124403)
ii=316609 y=-0.894943 u=(-0.00455088 -0.0231161 0.13823)
ii=316737 y=-0.878951 u=(-0.00415365 -0.0298542 0.151383)
ii=316865 y=-0.861726 u=(-0.00370377 -0.0377802 0.163607)
ii=316993 y=-0.843174 u=(-0.00320252 -0.0469805 0.174613)
ii=317121 y=-0.82319 u=(-0.002653 -0.0575266 0.184092)
ii=317249 y=-0.801666 u=(-0.00206075 -0.0694652 0.191718)
ii=317377 y=-0.778482 u=(-0.00143554 -0.0828093 0.197153)
ii=317505 y=-0.753511 u=(-0.000789686 -0.0975293 0.200063)
ii=317633 y=-0.726614 u=(-0.000139452 -0.113541 0.200139)
ii=317761 y=-0.697644 u=(0.000496615 -0.130699 0.197118)
ii=317889 y=-0.66644 u=(0.00109615 -0.148783 0.190809)
ii=318017 y=-0.632829 u=(0.00163618 -0.167488 0.181132)
ii=318145 y=-0.596627 u=(0.00209476 -0.186432 0.168159)
ii=318273 y=-0.557634 u=(0.00245017 -0.205152 0.152146)
ii=318401 y=-0.515634 u=(0.00268635 -0.223128 0.133575)
ii=318529 y=-0.470396 u=(0.00279119 -0.239814 0.113173)
ii=318657 y=-0.42167 u=(0.00276001 -0.254686 0.0919138)
ii=318785 y=-0.369186 u=(0.002593 -0.267305 0.0709739)
ii=318913 y=-0.312656 u=(0.00229841 -0.277386 0.0516354)
ii=319041 y=-0.251767 u=(0.00188806 -0.284874 0.0351015)
ii=319169 y=-0.186184 u=(0.00137428 -0.289967 0.0222259)
ii=319297 y=-0.115543 u=(0.000771825 -0.293112 0.0131436)
ii=319425 y=-0.0394559 u=(0.000105613 -0.294848 0.0068923)
ii=839745 y=0.0394559 u=(-0.000582071 -0.295449 0.00134095)
ii=839873 y=0.115543 u=(-0.00122611 -0.294747 -0.00626435)
ii=840001 y=0.186184 u=(-0.00177453 -0.292263 -0.0177436)
ii=840129 y=0.251767 u=(-0.00220134 -0.287433 -0.0335546)
ii=840257 y=0.312656 u=(-0.00249681 -0.279854 -0.053115)
ii=840385 y=0.369186 u=(-0.00265327 -0.269405 -0.075247)
ii=840513 y=0.42167 u=(-0.00266883 -0.256237 -0.0985297)
ii=840641 y=0.470396 u=(-0.00254961 -0.240723 -0.121556)
ii=840769 y=0.515634 u=(-0.00230948 -0.223372 -0.143094)
ii=840897 y=0.557634 u=(-0.00196405 -0.204763 -0.162169)
ii=841025 y=0.596627 u=(-0.00153323 -0.185474 -0.178093)
ii=841153 y=0.632829 u=(-0.00103735 -0.166047 -0.190452)
ii=841281 y=0.66644 u=(-0.000497004 -0.146948 -0.199071)
ii=841409 y=0.697644 u=(7.11539e-05 -0.128564 -0.203973)
ii=841537 y=0.726614 u=(0.000650192 -0.111193 -0.205332)
ii=841665 y=0.753511 u=(0.00122649 -0.0950506 -0.203423)
ii=841793 y=0.778482 u=(0.00178946 -0.0802759 -0.198595)
ii=841921 y=0.801666 u=(0.00233001 -0.0669428 -0.191226)
ii=842049 y=0.82319 u=(0.0028412 -0.0550705 -0.181708)
ii=842177 y=0.843174 u=(0.00331821 -0.0446353 -0.170421)
ii=842305 y=0.861726 u=(0.00375808 -0.0355805 -0.15773)
ii=842433 y=0.878951 u=(0.00415926 -0.027827 -0.143969)
ii=842561 y=0.894943 u=(0.00452063 -0.0212799 -0.129439)
ii=842689 y=0.90979 u=(0.00484285 -0.0158363 -0.114408)
ii=842817 y=0.923574 u=(0.00512709 -0.0113894 -0.099104)
ii=842945 y=0.936371 u=(0.00537485 -0.00783297 -0.0837236)
ii=843073 y=0.948252 u=(0.00558836 -0.00506389 -0.0684298)
ii=843201 y=0.959283 u=(0.00576992 -0.00298402 -0.0533565)
ii=843329 y=0.969524 u=(0.00592204 -0.00150175 -0.0386111)
ii=843457 y=0.979032 u=(0.00604735 -0.00053254 -0.0242779)
ii=843585 y=0.987859 u=(0.00614837 -5.53222e-07 -0.0104258)
ii=843713 y=0.996054 u=(0.00622505 0.000147164 0.00250257)
But parrallel run for 6 proc outputs only 11 values!

Code:
ii=103398 y=-0.996054 u=(-0.00619963 6.84788e-05 0.0108932)
ii=103526 y=-0.987859 u=(-0.00613874 -0.000242565 0.0243827)
ii=103654 y=-0.979032 u=(-0.00605887 -0.000950786 0.0380804)
ii=103782 y=-0.969524 u=(-0.00595275 -0.00210967 0.0521367)
ii=103910 y=-0.959283 u=(-0.00581652 -0.00379168 0.0664762)
ii=104038 y=-0.948252 u=(-0.00564615 -0.00607851 0.0810015)
ii=104166 y=-0.936371 u=(-0.00543755 -0.00905979 0.0955933)
ii=104294 y=-0.923574 u=(-0.00518676 -0.0128303 0.110108)
ii=104422 y=-0.90979 u=(-0.00488997 -0.0174897 0.124375)
ii=104550 y=-0.894943 u=(-0.00454407 -0.0231361 0.138196)
ii=104678 y=-0.878951 u=(-0.00414708 -0.0298703 0.151337)
But i expect the same result as single run...Why it is not the same?

DecomposePar Dict the following:

Code:
numberOfSubdomains 6;

method          simple;

simpleCoeffs
{
    n               ( 1 6 1 );
    delta           0.00001;
}

hierarchicalCoeffs
{
    n               ( 1 2 2 );
    delta           0.001;
    order           xyz;
}

manualCoeffs
{
    dataFile        "cellDecomposition";
}

scotchCoeffs
{
}
Also the result is sensitive for the number of proc. i am using...
levka is offline   Reply With Quote

Old   June 13, 2012, 16:59
Default
  #24
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,312
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings Lev,

It's quite simple:
  • Info is for single process output on the master thread; or for each slave, when each slave is launched with it's own shell, which can be achieved by using foamJobDebug if I'm not mistaken.
    Run:
    Code:
    foamJobDebug -help
    for more information.
  • PInfo will gather all Info's and output on the master thread! (edit: apparently PInfo doesn't exist... I guess my memory isn't all that well...)
Best regards,
Bruno

Last edited by wyldckat; June 14, 2012 at 16:35. Reason: see "edit:"
wyldckat is online now   Reply With Quote

Old   June 14, 2012, 01:29
Default
  #25
Member
 
Lev
Join Date: Dec 2010
Posts: 31
Rep Power: 6
levka is on a distinguished road
Hello, Bruno
unfortunately command "PInfo" give a compilation error as "PInfo was not declared in this scope ".
But the problem is not only at the screen output, but also i do not know how to order the code to analyze whole mesh that is spread to many processors (as is working with single run). And i need the results to be collected and to be written in the log file.
Like this command in single run
Code:
OFstream k(runTime.path()/"k.dat");
....
k<<"velocity "<<U[ii]<<endl;
Where can i read about this topics?

Regards
Lev
levka is offline   Reply With Quote

Old   June 14, 2012, 16:41
Default
  #26
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,312
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Hi Lev,

Sorry about that. Apparently PInfo doesn't exist. But I vaguely remember reading about it... weird

OK, by the description of what you're trying to do, it looks like to me that the probe or sampling utilities and/or function objects would be more suitable for what you want to do.
And then there is also "swak4Foam": http://openfoamwiki.net/index.php/Contrib/swak4Foam

Other than this... try searching this forum for more answers... I know they're in here somewhere...

Best regards,
Bruno
wyldckat is online now   Reply With Quote

Old   August 31, 2012, 18:25
Default
  #27
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,312
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings!

I saw just now the solution and remembered about this thread.
Quote:
Originally Posted by wyldckat View Post
Sorry about that. Apparently PInfo doesn't exist. But I vaguely remember reading about it... weird
Apparently my memory mangled the mnemonic incorrectly:
  • It's not: Info + Pout - cout => PInfo
  • It's: Info-Info + Parallel + cout => Pout
Kudos to mfmohdyasin:
Quote:
Originally Posted by mfmohdyasin View Post
Just figured out that I supposed to use "Pout" instead of "Info" to get output from each processor.

Best regards,
Bruno
wyldckat is online now   Reply With Quote

Old   February 22, 2013, 11:47
Default
  #28
New Member
 
Marco Müller
Join Date: Feb 2013
Posts: 5
Rep Power: 4
marco.müller is on a distinguished road
Hi all,

I'm following up with the issue that decomposePar does not preserve some boundaries and later on the solver complains about missing boundary definitions.

Furthermore potentialFoam complains about the processor-boundaries.

My question is: Apart from the workarounds that have been constructed in the meanwhile to solve this: is there a out-of-box-solution for OpenFOAM 2.1.1 that I am using out of box on ubuntu???

my script looks like this (without reconstructing)

Code:
#!/bin/sh
# Source tutorial run functions
. $WM_PROJECT_DIR/bin/tools/RunFunctions

cp -r 0.org 0 > /dev/null 2>&1

runApplication blockMesh
cp system/decomposeParDict.simple system/decomposeParDict
runApplication decomposePar
runParallel snappyHexMesh 2 -overwrite -parallel

find . -type f -iname "*level*" -exec rm {} \;

ls -d processor* | xargs -i cp -r 0.org/* ./{}/0/ $1

runParallel potentialFoam 2 -initialiseUBCs -noFunctionObjects -writep

runParallel `getApplication` 2

Thanks very much!!
Marco
marco.müller is offline   Reply With Quote

Old   February 23, 2013, 08:58
Default
  #29
Super Moderator
 
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 8,312
Blog Entries: 34
Rep Power: 84
wyldckat is just really nicewyldckat is just really nicewyldckat is just really nicewyldckat is just really nice
Greetings Marco,

Well... there are 2... 2.5 tricks that I know of:
  1. The one used in the tutorial "incompressible/pisoFoam/les/motorBike/motorBike", which you seem to be basing your case on.
  2. The one used in the case examples available here: http://code.google.com/p/bluecfd-sin...untimes202_211 - those rely on reconstructing the mesh and then decomposing all of it once again.
  3. And the half-trick still relies on mesh reconstruction, but decomposes only the fields based on the existing decomposition - the following can be used to replace the tutorial "incompressible/pisoFoam/les/motorBike/motorBike/Allrun":
    Code:
    #!/bin/sh
    cd ${0%/*} || exit 1    # run from this directory
    
    # Source tutorial run functions
    . $WM_PROJECT_DIR/bin/tools/RunFunctions
    
    # copy motorbike surface from resources folder
    cp $FOAM_TUTORIALS/resources/geometry/motorBike.obj.gz constant/triSurface/
    
    #make sure the write tolerance is upped for mesh reconstruction
    sed -i -e 's=writePrecision  6;=writePrecision  7;=' system/controlDict
    
    # Copy 0.org to 0
    cp -r 0.org 0
    
    runApplication blockMesh
    
    cp system/decomposeParDict.hierarchical system/decomposeParDict
    
    runApplication decomposePar
    mv log.decomposePar log.decomposePar.1
    
    cp system/decomposeParDict.ptscotch system/decomposeParDict
    
    runParallel snappyHexMesh 8 -overwrite -parallel
    
    find . -type f -iname "*level*" -exec rm {} \;
    
    #ls -d processor* | xargs -i cp -r 0.org/* ./{}/0/ $1
    
    runParallel renumberMesh 8 -overwrite
    
    #Need to reconstruct the mesh, in order to be able to re-decompose the fields
    runApplication reconstructParMesh -constant
    runApplication decomposePar -fields
    
    runParallel potentialFoam 8 -initialiseUBCs -noFunctionObjects
    
    runParallel `getApplication` 8
    
    # ----------------------------------------------------------------- end-of-file
Anything beyond this depends on how you defined your boundary condition patch names in the first place!

Best regards,
Bruno
wyldckat is online now   Reply With Quote

Old   February 25, 2013, 07:09
Default
  #30
New Member
 
Marco Müller
Join Date: Feb 2013
Posts: 5
Rep Power: 4
marco.müller is on a distinguished road
Hey Bruno,

great help, thanks! I took version 2, which works fine.

Thanks
marco
marco.müller is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
unchangeable continuity residuals in parallel computing wlt_1985 FLUENT 0 August 1, 2011 12:15
Diffusion equation solved using Parallel Computing Sachin Paramane Main CFD Forum 0 June 11, 2007 23:48
Parallel Computing on Multi-Core Processors Upgrading Hardware CFX 6 June 7, 2007 15:54
Parallel Computing peter Main CFD Forum 7 May 15, 2006 09:53
Parallel Computing Classes at San Diego Supercomputer Center Jan. 20-22 Amitava Majumdar Main CFD Forum 0 January 5, 1999 13:00


All times are GMT -4. The time now is 19:11.