problems with turbDyMEngineFoam using dynamicTopoFvMesh and load balancing
Hi,
I am running an internal combustion engine with turbDyMEngineFoam using a dynamicTopoFvMesh approach. The case works good in serial but in parallel it cannot load the decomposition libraries. Every 200 time-steps there is a dynamic parallel load-balancing process using parMetis and this is what the terminal says: Code:
--> FOAM Warning : Finally the case crashes almost at the end with 4 subdomains (with 2 subdomains it gets to the end without the following error) showing this error: Code:
[0] --> FOAM FATAL ERROR: Thank you very much. |
Compiling Scotch method
Hello, I am trying to run some case in parallel with scotch, I compiled and installed it but I did not do it as I should because I always have the following error:
Code:
--> FOAM Warning : Thank you for your help. |
Greetings Mibte and welcome to the forum!
I believe you're referring to foam-extend 3.0 and not to OpenFOAM 3.0, since OpenFOAM is currently only in version 2.3. Please indicate which Linux distribution you're using, what architecture (32 or 64 bit) and which installation instructions did you follow. Best regards, Bruno PS: I've moved your two posts to the same thread, as one complements information for the other. |
Hi Bruno.
Yes, I am referring to foam-extend 3.0. It is 64 bits and in order to install it I followed the instructions from this webpage for Ubuntu 13.10: http://openfoamwiki.net/index.php/In...u#Ubuntu_13.10 I could use either parMetis or scotch but I have the same problem with both. When compiling and installing parMetis from the ThirdPARTY Packages from the BUILD directory it says that it is made by default in usr/local. The problem is that I should point the parMetisDecomp sources to the correct compiled location, but I do not know how to do it. Has it to be either to the ThirdParty directories or to usr/local? Should it be pointed in the files or options files within the make directory from the parMetisDecomp folder within the decompositionMethods directory ? or both? This is what I currently have in the files and options files respectively: Quote:
Quote:
|
Hi Mibte,
Mmm... something is missing. OK, the first detail is the instructions at http://openfoamwiki.net/index.php/In...u#Ubuntu_13.10 might be a bit outdated, because foam-extend 3.0 is gradually evolving. Therefore, as suggested at the top of that page, the up-to-date installation instructions are these: http://openfoamwiki.net/index.php/In...oam-extend-3.0 The other detail is that even though you've indicated you're following the instructions meant for Ubuntu 13.10, you didn't specify if you are indeed using Ubuntu 13.10?! As for the problems with building parMetis, I get the feeling that there were some steps you've done differently for installing foam-extend 3.0!? Because the "/usr/local" installation location is only meant for system-wide installation and the installation instructions usually focus on keeping things installed in the user's home folder. The other feeling is that you might be having problems with a conflict between having more than one OpenFOAM version/variant. If you do have more than one version installed, please describe what procedure are you using to change between each version. So, beyond the requests/ideas above, my question is this: if you follow the step #11 from this section http://openfoamwiki.net/index.php/In...u#Ubuntu_13.10 - what is the content of the file "log.AllMake"? Best regards, Bruno |
1 Attachment(s)
Hello Bruno,
Should I delete it and reinstall it again following the new instructions you send to me? Actually I am running Ubuntu 13.10. With respect to building parMetis I just install foam and the Thirparty as usual, but like it did not work I tried to install it from the following BUILD.txt file within /ThirdParty/rpmBuild/BUILD/parMetis-4.0.3: Code:
------------------------------------------------------------------------------ With respect to other versions, I just have the foam extended-3.0 version, I never installed any other different version. I attach in this message the log.AllMake file from step number 11. I would like also to show you what happens when running the foamTestInstallation. Before I did not get any error (although parMetis was not working anyway), but yesterday I reinstalled everything again and I get the following: Code:
Executing /home/alvaro/foam/foam-extend-3.0/bin/foamInstallationTest: Best regards, Alvaro. |
Hi Alvaro,
The log file indicates that there is nothing wrong with the build for the ThirdParty software. The output from foamInstallationTest can be ignored, as the code is a bit outdated. The other important log file is the "make.log" file from step #13. Can you provide a simple test case where I can test turbDyMEngineFoam to check this error myself? I ask this because I haven't been able to find any example online. Best regards, Bruno |
1 Attachment(s)
Hi Bruno,
I attach with this message the make.log from step number 13. With respect to the turbDyMEngineFoam case I tried to attach it but the size is too big, it does not allow me to attach it. Could you provide me with an email address where I could send it to you please? Best regards, Alvaro |
Hi Alvaro,
The log file doesn't indicate any build errors. The problem must be something more specific, such as turbDyMEngineFoam not being prepared to properly load up the decomposition libraries... As for sending me the case, please use Dropbox or a similar service. If the data is sensitive, you can send me the link over private message. The reason why I ask this, is because I have to keep OpenFOAM related questions on the forum, otherwise I'll loose track of the questions that have been asked to me :( Best regards, Bruno |
Hello Bruno,
I understand, I just sent it to you with a dropbox link in a private message. Thank you very much, Alvaro |
Hi Alvaro,
OK, I've run the case with "parMetis" and the first issue is now clear to me: it's just a warning and doesn't affect operations. The reason for this happening is as follows:
Now, for it to be crashing, the problem seems to be that you have disregarded the importance of the mesh and of the Courant Number. Here is the output of the iteration where for me it started to go wrong and lead to crashing a few steps later: Code:
Volume: new = 0.00021076 old = 0.00021076 change = 0 If you analyze the "time step continuity errors" lines right after each load balance, you'll see that the continuity has some issues, even though it seems to solve them later on. My suspicion is that since a fixed step of 0.1s leads to a Courant Number greater than 0.5 and 1.0, this causes the simulation to be rather unstable and leads to the eventual crash. Regarding the mesh: you only have a 1 or 2 cells between certain walls. This leads to a considerable high amount of mass to be transfered between cells, which can lead to simulation problems. In this blog post of mine you'll see some examples of what this can lead to: OpenFOAM: Interesting cases of bad meshes and bad initial conditions As for the issue with Scotch... it's not parallel-aware, therefore it cannot be used during the run. Which means that either you have to use "ptscotch" or stay with "parMetis".... er, "ptscotch" doesn't exist in foam-extend, so it can't be used. But as far as I can see, although I have not confirmed this, the best approach seems to be for you to use "engineScotch" for the initial decomposition and then to not use load balancing during the run. Best regards, Bruno |
Hi Bruno, thank you for all the information.
When you say that it does not affect the process that means that it distributes equitably the number of cells among the processors for you? Could I see your log file when the load balancing process takes part? With respect to why it is crashing, after all the changes I made to the case I sent to you, it has been working differently than before. I have sent you another case along with the log file and a picture of the error, it is almost the same but it works perfectly for me until 358.4º out of 360º. The error is not apparently related with the Courant Number, it says the following: Code:
--> FOAM FATAL ERROR: Best regards, Alvaro. |
1 Attachment(s)
Hi Alvaro,
Attached is the log file from the run I made yesterday. By the way, in order to preserve the whole output in a single log file, for example use this: Code:
mpirun -np 4 moveDynamicMesh -parallel > log.move 2>&1 As for the moveDynamicMesh, it's because I'm testing only the mesh motion, instead of running the whole simulation. ----------- edit: The result from moveDynamicMesh was not what I expected. It simply didn't move :( ----------- And I did some digging and testing but have not been successful in using the "engineScotch" decomposition method. All I know is this:
But in your case, you might instead want to rely on adding the entry "preservePatches", e.g.: Code:
numberOfSubdomains 4; My first guess is that this is a bug in either in foam-extend or in dynamicTopoFvMesh. But only after the mesh motion run is complete, will I be able to try and figure out what happened. In the mean time, I'll be moving this thread to the "solving" forum, as this isn't an issue with installation. Best regards, Bruno |
Hi Bruno,
I have looked at your output file and it seems to be happening the same issue as I had, not making correctly the load balancing process. I tried to run it without the load balancing method, but it crashes even earlier. I am currectly running again the case as you proposed by using "preservePatches" in order to see if it gets to the end without any error, apart from the warnings of the decomposition libraries: Code:
numberOfSubdomains 4; Code:
include $(RULES)/mplib$(WM_MPLIB) Code:
g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3 -DNoRepository -ftemplate-depth-200 -DOMPI_SKIP_MPICXX -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread -I/home/alvaro/foam/foam-extend-3.0/ThirdParty/packages/parmetis-4.0.3/platforms/linux64Gcc48DPOpt/include -I../decompositionMethods/lnInclude -I../metisDecomp/lnInclude -I/home/alvaro/foam/foam-extend-3.0/src/decompositionMethods/decompositionMethods/lnInclude -I/home/alvaro/foam/foam-extend-3.0/ThirdParty/packages/parmetis-4.0.3/platforms/linux64GccDPOpt/include/ -I../scotchDecomp/lnInclude -IlnInclude -I. -I/home/alvaro/foam/foam-extend-3.0/src/foam/lnInclude -I/home/alvaro/foam/foam-extend-3.0/src/OSspecific/POSIX/lnInclude -fPIC -fpermissive -Xlinker --add-needed -Xlinker --no-as-needed Make/linux64Gcc48DPOpt/parMetisDecomp.o -L/home/alvaro/foam/foam-extend-3.0/lib/linux64Gcc48DPOpt \ Best regards, Alvaro. |
Hi Alvaro,
It's a bit strange that the build scripts at the "ThirdParty" folder did not properly handle the build process of "parMetis". And if it ran without problems during so many iterations, it's even more confusing... Anyway, the correct way for you to point the configuration to the new installation of "parMetis" is as follows:
The case finally reached the crash point... it took 18750s to reach the same angle 358.4º and then crash. I'm going to try and diagnose why this happened... but I'm not 100% certain I'll be able to. I'll give more feedback as soon as I can. edit: This error has also occurred for another person a few days ago: http://www.cfd-online.com/Forums/ope...tml#post488775 - post #91 Best regards, Bruno |
Hi Alvaro,
OK, when "dynamicTopoFvMesh" is in play, debugging dynamic meshes is waaaaay over my head :( Since fortunately this issue is identical to another one reported recently here: http://www.cfd-online.com/Forums/ope...tml#post488775 post #91 - I ask you to follow these steps below for preparing the case for sending it to Sandeep Menon (see post #92), the author of "dynamicTopoFvMesh":
If you wish, you should also post at that other thread indicating that this same issue has been reproduced with another case and informing that you'll send or have sent the case over private message. Best regards, Bruno |
Hi Alvaro,
I'm going to answer half of your question here on the public forum, if you don't mind ;) Quote:
Code:
Sourcing: /home/ofuser/foam/foam-extend-3.0/ThirdParty/packages/parmetis-4.0.3/platforms/linux64Gcc46DPOpt/etc/parmetis-4.0.3.sh I do believe that parMetis is built properly by the build scripts that foam-extend provides, namely those "AllMake*" scripts that are inside the folder "ThirdParty". When you're finished building foam-extend 3.0, if you run the following command, you should get the full path to where parMetis is installed: Code:
echo $PARMETIS_DIR
I'll answer to the rest via private message. Best regards, Bruno |
Hello, Mibte:
I have read posts above mentioned. You said that you used dynamicTopoMesh (mesquitesolver) to simulate engine. So I wonder if you have successfully run your engine simulation case in parallel. I really am interested in whether combination of Mesquite solver and enginescorch decomposition method is applicable. Another question I want to ask is that: because I want to perform full cycle simulation, so how to implement attach/detach in mesquite solver? Looking forward to you reply, thanks very much. |
All times are GMT -4. The time now is 12:24. |