|
I fully agree that communication matters. On high-end HPC clusters we were able to get about only 1.5x acceleration (see report) for a motorbike test with 32M cells (nGPUs vs. nCPUs, where n was a number of threads/GPU cards). This is probably due to communication. Please note, that on CPU GAMG was run.
This is also the reason why we implemented PISO and SIMPLE fully on a GPU card. Now, we can run the whole case on a single GPU card up to 10M cells and the inter-node communication problem is now an intra-node communication (GPU<->CPU on a single node bounded by memory bandwith). |
Quote:
This was just a quick timing comparison. I wasn't trying to nor am I planning on delivering a full benchmark on this stuff. The point of all of this is that after about 20 minutes of library installations I was able to get a foam solver to run at 1/8th the run time on a relatively old gpu, for free. |
anyone figure out how to compile the testCufflinkFoam application?
Openfoam, CUDA and the nvcc test scripts are all working on Ubuntu 10.04; I'm sure I'm missing something obvious but I can't see an obvious way to compile the testCufflinkFoam application to run the test examples. What am I missing? Thanks.
PS this is all from the GettingStarted page HERE |
Quote:
|
OpenFoam 1.6-ext
Yes it is looking pretty much like the result in the getting started document: (OF:1.6-ext Opt) caelinux@karl-OF:cufflink-library-read-only$ nvcc -o testcusp testcg.cu /usr/local/cuda/bin/../include/thrust/detail/tuple_transform.h(130): Warning: Cannot tell what pointer points to, assuming global memory space /usr/local/cuda/bin/../include/thrust/detail/tuple_transform.h(130): Warning: Cannot tell what pointer points to, assuming global memory space (OF:1.6-ext Opt) caelinux@karl-OF:cufflink-library-read-only$ ./testcusp Solver will continue until residual norm 0.01 or reaching 100 iterations Iteration Number | Residual Norm 0 1.000000e+01 1 1.414214e+01 2 1.093707e+01 3 8.949320e+00 4 6.190057e+00 5 3.835190e+00 6 1.745482e+00 7 5.963549e-01 8 2.371135e-01 9 1.152524e-01 10 3.134469e-02 11 1.144416e-02 12 1.824177e-03 Successfully converged after 12 iterations. Quote:
|
For some reason the test cannot find blockMesh, or at least I think that is what is going on:
(OF:1.6-ext Opt) caelinux@karl-OF:testCases$ sudo ./runSerialTests ./runSerialTests: line 3: /bin/tools/RunFunctions: No such file or directory /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_CG/N10 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_CG/N50 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_CG/N100 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_CG/N500 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_CG/N1000 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_CG/N2000 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_DiagPCG/N10 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_DiagPCG/N50 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_DiagPCG/N100 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_DiagPCG/N500 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_DiagPCG/N1000 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_DiagPCG/N2000 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_SmAPCG/N10 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_SmAPCG/N50 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_SmAPCG/N100 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_SmAPCG/N500 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_SmAPCG/N1000 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/cufflink/cufflink_SmAPCG/N2000 ./runSerialTests: line 16: runApplication: command not found ./runSerialTests: line 17: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/CG/N10 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/CG/N50 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/CG/N100 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/CG/N500 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/CG/N1000 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/CG/N2000 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/DPCG/N10 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/DPCG/N50 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/DPCG/N100 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/DPCG/N500 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/DPCG/N1000 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/DPCG/N2000 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/GAMG/N10 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/GAMG/N50 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/GAMG/N100 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/GAMG/N500 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/GAMG/N1000 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found /home/caelinux/OpenFOAM/caelinux-1.6-ext/run/cufflinkTest/testCases/OpenFOAM/GAMG/N2000 ./runSerialTests: line 34: runApplication: command not found ./runSerialTests: line 35: runApplication: command not found |
|
Quote:
Looking at your error log....I can tell you forgot your lduInterface.H that is discussed on the getting started page. Go there and follow those steps and you should be good. |
Sorry I thought that was only required for multi GPU. My mistake.
Thanks Very Much! Karl |
M2090
Quadro600 Six cores of E1280 are better than one CPU core and a Quadro 600 (96 gpu cores). In fact, four CPU cores on the E1280 are faster than six, and either is faster than the Quadro 600. But with the M2090, 512 CUDA cores (and a Quadro? It said 2 processes and the Quadro was heating up) crush the six core CPU by about five times apparently, at least for these test tasks. 75 seconds vs ~360 I hope this is of some benefit for an incompressible simpleFoam case. Time will tell I suppose. Thanks Dan and Kyle for your help. Karl |
Karl,
I'm glad to help. Just to add a little to your last statement Quote:
|
Quote:
|
Quote:
Quote:
steady-state solution strategies that use relative residual convergence criteria with dominant outer-iterations (e.g. one could use simpleFoam) to drive down convergence will have more instances that data will have to be transferred to the GPU. therefore it will spend less time solving the linear system and spend more time transferring things back and forth if it is a large system. Its also highly dependent on your hardware setup. Quote:
|
Dan, as an example have a look at this document that compares the performance of simpleFoam and pisoFoam using a GPU accelerated linear solver: http://am.ippt.gov.pl/index.php/am/article/view/516/196 (or also http://www.slideshare.net/LukaszMiro...el-performance).
|
1) Areal offloads the pressure velocity coupling completely on the GPU, so there is not as much back and forth data transfer (this is how they are getting around the problem I suggested)
2) The first link makes no mention of relative residual and they are using and convergence criteria. So there is no way to know if they were using a strategy of driving down the residual with many outer-iterations with a relative residual convergence criteria like I am suggesting. |
Thanks Dan I had read your earlier comment but am just coming to terms with the outer vs inner iteration part. It is just a reflection of my poor grasp of the fundamentals at play here.
Along the CPU/GPU communication line however I should add that the Quadro600 in the above test was only working at 8x, whereas the Tesla was in the 16x slot (I only have one 16x slot on the board). So the Quadro might do better in the faster slot vs. the CPU. It will be interesting to note how much better PCI 3.0 slots perform, though the overall hardware seems likely to take a quantum leap at about the same time with the Kepler stuff. I will post some comparative run times when I get a case going in simpleFoam. Quote:
|
In keeping with Dan's advice to compare the best cpu solver with the best gpu solver, I ran a few tests. On the cavity tutorial re-blockmeshed to 1000x1000 cells, I get a speedup of 1.5 (310 vs 463s) for one hundred timesteps of this:
p solver cufflink SmAPCG preconditioner FDIC U solver cufflink DiagPBiCGStab U preconditioner DILU over a cpu run on single processor of xeon e1280 with: p: FDIC/GAMG U: DILU/PBiCG I am still at the stage where I don't really know the solvers very well, so take it with a grain of salt, but it was all set at the same relative tolerances with 1e-08 for p and 1e-05 for U. I tried the parallel version of cpu, but it was slower; 491 vs 463s for 4 vs 1 core on the cpu. The gpu is an M2090 Tesla with like 512 cores or something and 6Gb memory I think. I didn't try anything with multiple gpus. So it would appear that modest speedup is obtained for this example in icoFoam. I do not know why my parallel cpu runs are slower on 4 cores vs 1; I expected it to be at least somewhat faster. |
Quote:
Edit: the preconditioner FDIC for the SmAPCG will do not actually use the FDIC preconditioner, it will use the Smoothed Aggregate AMG preconditioner. I guess I need to change this in the examples to be more straight forward. The preconditioners aren't runtime selectable with the preconditoner name, but rather with the name of the solver. |
Quote:
|
2 Attachment(s)
Hello everyone,
I have been using cufflink for some applications for incompressible flow with good results using one Tesla C2050. However I don't obtain speedup using more than one GPU with the parallel versions of the cufflink solvers and unexpectedly using 2 GPUs results slower than the serial one. I discussed some issues in the cufflink-users linked in the code page. Since I tested the same behavior in different machines and GPUs I wanna share some test case with you if anyone would check it or have experienced also that. The test case is the same provided by Dan with testCufflinkFoam application. I have fixed some wrong values in some blockMeshDict and tolerances and maxIter for the Parallel solvers since they were different. The test case can be reproduced running the same scripts as the original folder: ./runParallelGPU ./runSerialGPU ./runGetTimes I attach the case here and some figures for CG and diagPCG tested in a GeForce GTX 690. smAPCG for some reason the number of iterations don't fix for 1GPU and 2GPU and I must to check it first. Thank you very much. I find this library useful, for my experience there are situation where one can't use GAMG solver for p, especially in applications with bad quality meshes so I guess that the speed up can't be always comparable with CG since it should be robust. There the speed up is considerable. Regards |
| All times are GMT -4. The time now is 14:41. |