CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Running, Solving & CFD

How to use parallel running to the most?

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree5Likes
  • 2 Post By GerhardHolzinger
  • 2 Post By GerhardHolzinger
  • 1 Post By 6863523

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   January 13, 2017, 04:26
Smile How to use parallel running to the most?
  #1
Senior Member
 
Bill Wang
Join Date: Aug 2014
Posts: 109
Rep Power: 11
6863523 is on a distinguished road
Dear All,
I am trying to run cases in OpenFOAM, here is the configuration of the server I am using.
Code:
 ywang@srv-calmipo01 ~/OpenFOAM/ywang-3.0.1/run/UCUCUC/Re17500 $ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                40
On-line CPU(s) list:   0-39
Thread(s) per core:    2
Core(s) per socket:    10
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Stepping:              1
CPU MHz:               2399.976
BogoMIPS:              4396.31
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              25600K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
Q1: Here is the decomposeParDict I want to use, and I want to make the most use of the server. What should I modify about file?
Code:
 /*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  3.0.1                                 |
|   \\  /    A nd           | Web:      [IMG]file:///C:\Users\wyb\AppData\Local\Temp\%W@GJ$ACOF(TYDYECOKVDYB.png[/IMG]www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    note        "mesh decomposition control dictionary";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains  12;

method          scotch;


multiLevelCoeffs
{
    
    level0
    {
        numberOfSubdomains  64;
         method scotch;
    }
    level1
    {
        numberOfSubdomains  4;
        method scotch;
    }
}

simpleCoeffs
{
    n           (2 1 1);
    delta       0.001;
}

hierarchicalCoeffs
{
    n           (1 2 1);
    delta       0.001;
    order       xyz;
}

metisCoeffs
{
 /*
    processorWeights
    (
        1
        1
        1
        1
    );
  */
}

scotchCoeffs
{
   
}
manualCoeffs
{
    dataFile    "decompositionData";
}

structuredCoeffs
{
    patches     (movingWall);
    method      scotch;
}

// ************************************************************************* //
After the decomposePar command, I entered
mpirun -np 12 icoFoam -parallel > log
Q2: This running process will be saved in the log file, but I also want to see them running on the screen. What should I add to the command?

I want to store the solution according to the controlDict file below
Code:
 // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

application     myIcoFoam;

startFrom       startTime;

startTime       0;

stopAt          endTime;

endTime         4000;

deltaT          0.0125;

writeControl    timeStep;

writeInterval   20000;

purgeWrite      0;

writeFormat     ascii;

writePrecision  6;

writeCompression off;

timeFormat      general;

timePrecision   6;

runTimeModifiable true;

Q3:
But if I run in parallel, the files will be saved automatically in separate folders, named Processor0, processor1, ...What is the difference between these separate files and those files obtained without parallel?

Q4: How can I continue the simulations based on these files, if I do not use parallel running?

Thank you in advance.
Best Regards,
Bill
6863523 is offline   Reply With Quote

Old   January 13, 2017, 05:22
Default
  #2
Senior Member
 
Gerhard Holzinger
Join Date: Feb 2012
Location: Austria
Posts: 335
Rep Power: 28
GerhardHolzinger will become famous soon enoughGerhardHolzinger will become famous soon enough
Quote:
Originally Posted by 6863523 View Post
Dear All,
I am trying to run cases in OpenFOAM, here is the configuration of the server I am using.
Q1: Here is the decomposeParDict I want to use, and I want to make the most use of the server. What should I modify about file?

Q2: This running process will be saved in the log file, but I also want to see them running on the screen. What should I add to the command?

Q3:
But if I run in parallel, the files will be saved automatically in separate folders, named Processor0, processor1, ...What is the difference between these separate files and those files obtained without parallel?

Q4: How can I continue the simulations based on these files, if I do not use parallel running?

Thank you in advance.
Best Regards,
Bill
Q1: The setting numberOfSubdomains controls how many parallel processes will be used in the simulation. You can tune parallel performance by playing around with the decomposition. Dividing the load (the number of processes) is one thing to consider, the extra amount of work (communication between the parallel processes) is another topic. Both have influence, but the latter can be very tricky. It is hard to give a general answer, other than do your own testing and do more research.

Q2: Having the solver output permanently on the screen seems to slow the simulation down. Redirect the solver output to a file, and display the solver output only on demand, e.g. by the tail command or by pyFoamPlotWatcher.py

Q3: A parallel simulation is a large task done is smaller chunks. This is why you do domain decomposition. The sum of the content of the processor* directories is your original case. After you finished, you can run reconstructPar to recombine the small chunks into your large simulation again.

Q4: What is the point of continuing a simulation you did in parallel before, in serial (one process)? How you do it? Run reconstructPar, make sure startTime in controlDict is set to latestTime, check the settings for ending the simulation (stopAt and stopTime), and run the solver.
What makes more sense is to change the number of parallel processes, e.g. because you initially used to little or too much. Recombine, decompose with the new number of subdomains, and continue the parallel run with the new number of parallel processes.
jeytsav and Ramzy1990 like this.
GerhardHolzinger is offline   Reply With Quote

Old   January 13, 2017, 06:19
Smile
  #3
Senior Member
 
Bill Wang
Join Date: Aug 2014
Posts: 109
Rep Power: 11
6863523 is on a distinguished road
Dear Dr. Holzinger,
Thanks for the quick reply.
Q1: I will test to see what is the best for my cases.
Q2: Thank you for the suggestions.
Q3: That's correct!!! I looked through the forum and found reconstructPar function. I know what to do after parallel running.
Q4: Based on the answer for Q3, I understand my problem now and thanks for the instructions.

Thanks for the detailed instructions. If I understand your point and the computer configuration correctly, I may start from
Code:
 /*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  3.0.1                                 |
|   \\  /    A nd           | Web:       [IMG]file:///C:\Users\wyb\AppData\Local\Temp\%W@GJ$ACOF(TYDYECOKVDYB.png[/IMG]www.OpenFOAM.org                       |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    note        "mesh decomposition control dictionary";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 20;
method          scotch;
multiLevelCoeffs
{    
    level0
    {
        numberOfSubdomains  10; //changed from 64 to 10
         method scotch;
    }
    level1
    {
        numberOfSubdomains  8; //changed from 4 to 8
        method scotch;
    }
}

simpleCoeffs
{
    n           (2 1 1);
    delta       0.001;
}

hierarchicalCoeffs
{
    n           (1 2 1);
    delta       0.001;
    order       xyz;
}

metisCoeffs
{
 /*
    processorWeights
    (
        1
        1
        1
        1
    );
  */
}

scotchCoeffs
{
   
}
manualCoeffs
{
    dataFile    "decompositionData";
}

structuredCoeffs
{
    patches     (movingWall);
    method      scotch;
}

// ************************************************************************* //
Actually, I am confused on how to link the numberOfSubdomains to the computer configurations, as shown in the red parts.
Thank you.
Best Regards,
Bill
6863523 is offline   Reply With Quote

Old   January 13, 2017, 06:35
Default
  #4
Senior Member
 
Gerhard Holzinger
Join Date: Feb 2012
Location: Austria
Posts: 335
Rep Power: 28
GerhardHolzinger will become famous soon enoughGerhardHolzinger will become famous soon enough
Quote:
Originally Posted by 6863523 View Post
Dear Dr. Holzinger,
Thanks for the quick reply.
Q1: I will test to see what is the best for my cases.
Q2: Thank you for the suggestions.
Q3: That's correct!!! I looked through the forum and found reconstructPar function. I know what to do after parallel running.
Q4: Based on the answer for Q3, I understand my problem now and thanks for the instructions.

Thanks for the detailed instructions. If I understand your point and the computer configuration correctly, I may start from
Code:
 /*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  3.0.1                                 |
|   \\  /    A nd           | Web:       [IMG]file:///C:\Users\wyb\AppData\Local\Temp\%W@GJ$ACOF(TYDYECOKVDYB.png[/IMG]www.OpenFOAM.org                       |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    note        "mesh decomposition control dictionary";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 20;
method          scotch;

scotchCoeffs
{
}


// ************************************************************************* //
Actually, I am confused on how to link the numberOfSubdomains to the computer configurations, as shown in the red parts.
Thank you.
Best Regards,
Bill
The numberOfSubdomains should be less or equal to the number of CPU cores.
Since you use the scotch method for decomposition, you can delete all other entries in the decomposeParDict.

When you use a method X, then the XCoeffs are relevant. Using scotch and changing the multiLevelCoeffs makes no sense and will change nothing.

First, you may want to stick to the scotch method, as it handles decomposition for you. Try different numbers of processes to see where the sweetspot lies. If the case is large enough, conducting parallel simulation will speed-up your calculations. As you increase the number of parallel processes the speed-up will increase up to the point, when the additional work (communication between the processes) will become dominant.
6863523 and Ramzy1990 like this.
GerhardHolzinger is offline   Reply With Quote

Old   January 13, 2017, 10:25
Smile
  #5
Senior Member
 
Bill Wang
Join Date: Aug 2014
Posts: 109
Rep Power: 11
6863523 is on a distinguished road
Dear Dr. Holzinger,,
Thank you for your clear explanations, they are really very helpful to me!!!
I start from
Code:
numberofSubdomains 40;
and set all scotchCoeffs to 1 to make a test.

Since I used cyclic boundary, the decompose method is recommended here: https://github.com/OpenFOAM/OpenFOAM...composeParDict
Thank you in advance.
Best Regards,
Bill

Last edited by 6863523; January 19, 2017 at 04:39.
6863523 is offline   Reply With Quote

Old   January 19, 2017, 03:22
Default still rather long time needed in parallel
  #6
Senior Member
 
Bill Wang
Join Date: Aug 2014
Posts: 109
Rep Power: 11
6863523 is on a distinguished road
Dear,
I have tested the numberofSubdomains(n) with several numbers. Here are some results I have got.
Code:
n=8, for t=3.7s ClockTime=863s;
ClockTime/t ≈ 233;

n=12, for t=5s, ClockTime=916s;
         for t=7.538s, ClockTime=1363s;
         for t=10s, ClockTime=1803s;
         for t=10.425s, ClockTime=1898s;
ClockTime/t ≈ 180;

n=15, for t=5s, ClockTime=909s;
         for t=7.5s, ClockTime=1375s;
         for t=10s, ClockTime=1802s;
ClockTime/t ≈ 182

n=20, for t=2.325s, ClockTime=349s;         
         ClockTime/t ≈ 150
        for t=321.09s, ClockTime=21574s;
        ClockTime/t ≈ 67.2
It seems that when the numberofSubdomains=20, it works the fastest.

In addition, I would like to recommend the parallel tests mentioned here: Optimising the Parallelisation of OpenFOAM Simulations
And also the basic knowledge about parallel strategy here: http://quantumwise.com/documents/tut...rategy.systems
Thank you in advance.
Best Regards,
Bill
Ramzy1990 likes this.

Last edited by 6863523; February 1, 2017 at 14:28. Reason: wrong info
6863523 is offline   Reply With Quote

Reply

Tags
parallel calculation, parallel code

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
error while running in parallel using openmpi on local mc 6 processors suryawanshi_nitin OpenFOAM 10 February 22, 2017 22:33
Running mapFields with Parallel Source and Parallel Target RDanks OpenFOAM Pre-Processing 4 August 2, 2016 06:24
running OpenFoam in parallel vishwa OpenFOAM Running, Solving & CFD 22 August 2, 2015 09:53
Fluent 14.0 file not running in parallel mode in cluster tejakalva FLUENT 0 February 4, 2015 08:02
Problems running in parallel - missing controlDict Argen OpenFOAM Running, Solving & CFD 4 June 7, 2012 04:50


All times are GMT -4. The time now is 09:49.