One more question concerning nCellsInCoarsestLevel, how to set it?
sqrt(mesh_in_total), or 10~30, or what? Is SAMG much more better than GAMG? Any experience on that? When will DICGaussSeidel be superior over GaussSeidel? Or there is even better options? (Suppose this is an external incompressible flow with pisoFoam, have 10m grids pts, hex mesh). Thanks |
hello,
You can have a look at this interesting thread. The answer is to keep a small value here (10-20). Olivier |
Since I have compared different nCellsInCoarsestLevel values both for a 5m grids case and also for a 10m grids case, I could not get any conclusion. For my 5m grids case, it seems 100 is better than setting it to 10~20, for 10m grids, case, I even find its good to set it around 3000.
That's why I feel kind of confused.. |
Quote:
Anyway, good thing from that thread is that I designed a new multigrid scheme that is very efficient. That exploits the fact that direct solver size affects the solution rate. Previously Bi-conjugate gradient preconditioned by AMG was the best combination I have tried. Now the timings with new method are at least two times faster than previous best. Will be testing it against W and F cycles in coming days. :-D |
Quote:
Quote:
Quote:
Quote:
|
And also, It seems, scotch sometimes behaves better than metis... And I don;t know how to tune up the hierarchical method to find a optimized one, so I give up.
|
Quote:
Quote:
Quote:
Anyway the crux was that the solver where AMG used W cycle was the fastest. Now if you look at W cycle, what it does is that it tries to solve problem at coarser levels to better convergence compared to V cycle. In case of direct solver, you do not iterate and solve the coarse problem to machine precision. In both cases you are trying to essentially doing the same thing, that is to solve coarse problem to as high convergence level as possible without taking much time. What I improved is to replace W cycle by some scheme more robust and better converging (on coarser levels), that takes similar or less time (but much higher convergence). So far I am able to see speed up, but I have yet to make serious tests and comparisons. Specially with large sizes and W cycles. PS: I am using my c++ library but the same scheme could be applied to any AMG scheme. |
Quote:
Not related to openFOAM but i wrote a small parallel data exchange code, that i am using. Good thing about it is that no matter what you use, its parallel efficiency do not go down. So far, at least my code there is no issue of partitioning. It is roughly 5 times faster than fluent :-D . (needless to say i am happy and relaxing :-D). |
Quote:
Quote:
Interesting, when would you share your work to public:). And I also hope that the improvement is not just channel-flow friendly or box-turbulence friendly. Thanks |
Quote:
For my case, I am not fully sure, but having tried different cpus, I found 4s/timeStep is the best I can get... ANY "formula"? |
Quote:
|
Quote:
|
Quote:
(I added few things to fluent mesh format, if others also follow that we can have a universal mesh format for parallel calculations. Other things are information related to parallel interfaces.). Anyway the trick to efficiency is that when data is exchanged, there shall be no collision. Means that if a process is sending data to another process, no other process shall be sending data to it. The program takes care of it. This is why, data exchange remains highly efficient no matter how i partition. |
Quote:
We have run some additional tests and found metis/scotch is in general better than hierarchical. Which is different than what we thought previously. Hierarchical can be better under very specific circumstances where the cell distribution is favourable, but for cases with highly non-homogeneous cell density metis/scotch has shown up to 20% better performance in some instances. The only problem for us at the moment is that our version of snappyHexMesh does not generally work with parMetis under 1.6 - we have not tried to debug this yet. I hope the parallel scotch in 1.8 will rectify this. |
Haha, I tried for many cases using hierarchical, and then I got an impression that it is too tricky to use hierarchical, so I gave up.
Btw, just curious, are you all able to make every time-step run within one physical second, even when you are doing some time-averaging operations? |
Quote:
|
A (more basic) question:
Does anyone have a recommended method to determine if your system is performing optimally? I've run a simple case (3D cavity tut with 1M cells) repeatedly on a various number of processors (N=8,16,..,64) and I get very erratic results. The run times (even when using the same number of processors) can vary dramatically (>x4). The jobs are submitted through an LSF scheduler and I've tried a few things to test this issue, like submitting the job sequentially (e.g. 3 times in a row) on the same set of processors, or running three instances of the job simultaneously on three different sets of N-processors (if that makes sense). We added some code to Pstream to output the total amount of time each processor spent in MPI, and we find that MPI time can vary substantially across the cores, which we've been interpreting as some of the cores are running slower. However, this slow down isn't repeatable, and running the job on the same nodes doesn't reproduce the issue. I apologize because my knowledge of MPI is essentially zero. Is this a hardware issue or a problem with how the cluster is constructed? Or is it some setting in MPI? I can provide more information about the details (cluster architecture, etc.) but wanted to describe the problem in general first. Running OF-1.6 using supplied openmpi-1.3.3. Any thoughts or avenues of investigation would be great. |
Quote:
1. What did you mean by saying "I get very erratic results", are you saying that the results each time are completely different? 2. Are there any other persons who is using your computing nodes or your memory, if so that would cause differences.:) |
Thanks for the reply Daniel.
1. The results from the simulation (i.e. flow field) are the same every time. The run time (amount of time to do the same number of iterations) varies greatly. 2. Aware of this, and agreed that I can't really control for it. However, that's why I ran 3 jobs at the same time (i.e. same exact simulation, just on different nodes) because I figured these would be subject to the same network traffic at that time. Even these cases can be very different (~x2). Furthermore, I've gotten in the habit of checking the cluster load when I submit. I've submitted jobs when there's little to no load and they've taken much longer when network traffic was medium/high. I'll add to that I've also tried looking at how the job is distributed (i.e. 8 cores on 1 node, 1 core on 8 different nodes, etc. Note: each node has 4 dual-core processors) and I don't see a definite pattern - that is, it's not like running a job with N=64 on 8 x 8 is better than some other distribution... |
Hi jdiorio,
If you cannot guarantee that the nodes work only for you, there is no surprise that your computation time varies greatly. One time the machine is working for your job only, and another time it is splitting the resources between X jobs. If you are using the LSF scheduler, it is possible to reserve the nodes for your job only. Then your results will be consistent. |
All times are GMT -4. The time now is 11:01. |