|
bash script for pseudo-parallel usage of reconstructPar
1 Attachment(s)
All,
I thought I would post this as maybe someone else will find it useful. I wrote a short script (not necessarily pretty, but it works...) that runs reconstructPar in pseudo-parallel mode by breaking the time directories into a number of ranges and running multiple instances of reconstructPar. For lack of a better name I have called it parReconstructPar. Enjoy. -Kent (The forum won't take a file without an extension so it is uploaded as a .txt -- just save it somewhere in your path as parReconstructPar and make it executable.) |
parallel reconstruction (slightly improved)
All,
Here is an updated version of my bash script for parallel reconstruction with a few minor modifications--it lists the total number of directories remaining to be reconstructed. I suppose it would be relatively simple to also put in an estimate for the time remaining but I think I'll save that for later. parReconstructPar Note that this is geared toward reconstruction of a large number of time directories from a long transient run--if you just want to reconstruct a single time directory this is NOT what you need. Also, it basically assumes you currently have no other reconstructed time directories in your base case directory besides 0. I would love to hear if anyone has found this to be useful. -Kent |
Nice and useful
Works well for me. Thank you for sharing such nice and useful tool. Helps to save some time on multiprocessor machines.
Regards. |
Nice!
Always good to have utilities that make using OpenFOAM more efficient (and that you don't have to study for many days to figure out how to use).
...user interface, when oh when? |
Bugs?
Hi,
I tried to use it with a case I started from time=10 s (to continue it). First reconstructPar job exists with an error. In the end, I have the files that were reconstructed by the 2. and following instances of reconstructPar (so everything from 11.3 s). But the last timeStep (15 s) is missing as well. My fault or does it just work from 0 on? Would like to use it though, so if you would fix that in case it's a bug... Thanks! Bastian |
Well, I guess strictly speaking that is not a bug, just a deficiency of the simplistic method I am using and I mentioned that it assumes you are reconstructing from the smallest time dir in your processor0 directory since it figures out the total number of time directories there and breaks it up into chunks. There is probably an easy fix to generalize this so that you could start the reconstruction from a different value (like t=10s in your case)--I don't know that I have time in the near term (next few weeks) to look at this again, so if someone else wants to run with it and post back that would be awesome.
|
Hhhmm I'm pretty sure that that (10 s) was the smallest dir in my processor0, since I usually delete all processor dirs after reconstructing. And only if i reconstruct I can continue from another time. But I'll have a look again.
Bastian |
1 Attachment(s)
Kent,
thank you very much to your contribution, I found this script very useful for me! Here I attach my bash script, which reconstructs only those case files, which are not reconstructed yet. It is not parallel, but perhaps someone will find it useful for common usage))) |
Thanks to Kent, until now it's working perfectly for me.
|
Thanks a lot, Kent!
It worked nicely for me, using a dual-core with hyperthreading. A thing a little bit surprising though: It counted down nicely from around 1000 timesteps. But the last one before it ended was around 200. This was causing not more than a bit of curiosity, for sure it's not a problem! Thanks for the tool again! |
Hi,
I wonder, if anyone tried to rewrite reconstructPar/Mesh to run in real parallel... foamToEnsight is able to write data and mesh in parallel to a 'serial' file quite quickly, so it should be possible to write direct openfoam data as well!? Did anyone tried this yet? Best REgards Fabian |
Hi Kent,
This works great, only one question, in my case it always run 2 or 3 processors for reconstructing the case, is there some way to change the code and push it to use more processors, or I am doing something wrong there? Best Regards Nikola |
Maybe you have already sorted this out, but I just noticed I never replied to this post. Not sure what is happening. If you run the script name without args you will see that you run it using:
parReconstructPar <number of processors for reconstruction> <name of output file, optional> Is it not using 8 processes if you run the script as: parReconstructPar 8 ? |
Yes I figured it out, in the same way as you have explained in your answer.
Sorry for a late reply. Thanks. |
Hello kent !
I 'm also counter this problem, and Ithink this script is useful to me ! I have download it but I don't know how to run it ? would you mind telling me how to get it work? Thanks ! |
Dear Kent,
Thank you very much for your sharing the file. In my case, I only need to reconstruct one of the variables, so the command is: Code:
reconstructPar -fields '(U)'Code:
appflag="-noZero -fields '(U)'"Thank you. OFFO |
1 Attachment(s)
Dear all,
thank you Kent for this great tool. I picked it up and added some extra features. With this script you can specify the fields to reconstruct and it is possible to specify the timeframe (start,end). The syntax for using slightly differs from the normal reconstructPar but it is explained in the file. Hope somebody finds it useful Hannes |
Thank you, it works very well.
|
Hannes,
Beautiful work! Thanks for your contribution in adding several features that make the script more flexible and that I am sure will be useful to many people. -Kent |
Hello Hannes,
When I have four parallel runs to preform the reconstruct, there only two are running, the other are not. Is this caused by that fact that the my computer memory is not enough to support the reconstruct? |
Hi,
could you post some more information about your case? From what you are writing I would assume that you only have two timesteps to reconstruct because the way the script is written it will never start more jobs than there are timesteps left. Hannes |
Error while running parReconstructPar
Hi,
I get the following error while trying to run parReconstructPar. Starting Job 1 - reconstructing time = 0.1 through 166.6 Job started with PID 10452 Starting Job 2 - reconstructing time = 166.7 through 234.1 Job started with PID 10462 Starting Job 3 - reconstructing time = 234.2 through 31.7 Job started with PID 10472 Starting Job 4 - reconstructing time = 31.8 through 300 Job started with PID 10482 --> FOAM FATAL ERROR: No times selected From function reconstructPar in file reconstructPar.C at line 210. FOAM exiting If you notice the time ranges in job 3 and 4 there a bit strange too. Can someone please tell me how to correct this? Thanks kcn |
Hi kcn!
I can only GUESS, but is it maybe related to the timestep given in your controlDict? Otherwise: Did you already try if it would work with two processors only? Maybe the division of timesteps is giving values which do not work? Or maybe you could try to decompose timestep 0 as well, so it would work beginning from timestep 0 instead of 0.1? As I said: It is all only guessing, but I think these would be the approaches I would take for further testing... Cheers, Bernhard |
Minor Fix to get all numeric directories
1 Attachment(s)
The errors reported above are due to a bug in the way the script lists list the time files in processor0. They needed to be ordered by value using the "1v" flag, otherwise you can get the times in the wrong order, e.g. ... 0.7, 0.8, 1, 10, 1.1, 1.2 ....
If the number of processors is such that one processor picks up the range from 10 to 1.9 (say), when it should be from 1.1 to 1.9, then you have a problem. Simply replace all occurrences of "ls processor0 | ...." with "ls processor0 -1v | ...." Modified script attached. Attachment 33756 Otherwise a great script. Thankyou. |
Dear Will,
Thank you very much for the corrected script. kcn |
Quote:
I tried to put your file in my case directory and execute ./parReconstructPar but it doesn't work. Again I put it in my opt/bin directory for in case but also failed. Can you tell me where Im making mistake?? |
Have you set the executable flag?
Code:
chmod a+x parReconstructParCode:
sh parReconstructPar |
Hallo,
thanks for sharing. I'm doing a parallel simulation in OpenFoam and because of the limitation of files numbers in my computer, I have to prevent, that data be increased exponentially. I have somehow no experience in bash scripting and I want to write a script, that after a certains time lets say 1 min search if there is a new time in processor0. It should than reconstract this new time and delete it in all the processors. Its clear that it shoud be able to run during the simulation. I would appreciate if sombody can help me thx |
Dear Will,
I'm very happy with this script, but I would like to point out that there still is a small overlap in the detected time ranges when decimals are involved: Starting Job 1 - reconstructing time = 0.25 through 30.5 Starting Job 2 - reconstructing time = 30.25 through 60.5 (I can only run 2 reconstructs in parallel on a single machine (16 GB ram limit) but the script parameters make it very easy to manually divide the reconstructing over 2 or 3 machines.) edit: this example is a bit more problematic, I may have to run 30.5 manually: Starting Job 1 - reconstructing time = 0.25 through 30.25 Starting Job 2 - reconstructing time = 30.75 through 61 |
Quote:
hello, jwstolk I am using the tool on HPC, but can tell me how to tun this application, I have put the script in the directory where pro* files are stored. and when I run "parReconstructPar 12",it simply give me a error of "Command not found". Quote:
very much appreciated if you can do any help. /Allen |
I don't know exactly what HPC is in this case, but I will assume this is some form of Linux.
You are on the right track. ./parReconstructPar should normally work, but since you downloaded the script, it is not marked as executable, and results in an error. if you run: ls -l parReconstructPar the "x" flag should be missing. in that case, run something like: chmod a+x parReconstructPar (oh, and possibly read scripts you download from the internet before giving them executing privileges :-) Normally you only need to use the -n option, like: ./parReconstructPar -n 12 (see the USAGE line in your quote) Note that reconstructing takes quite a bit of RAM. I recommend just running first with 1 or 2 instead of 12, and then checking how much ram it ends up using (for example with top or htop), and then decide how many cases you can run in parallel without running out of ram. The OS can swap other programs to disk, but the ram used by parReconstructPar is used continuously, and when even a small part of that needs to be swapped to disk, everything slows down to a crawl. With my current cases, I can run upto "-n 3" with 16 GB ram. Since my files are on an NFS drive, I can use the "-t start,end" option to process only half the time directories, and run the other half from another computer, with another 16 GB of ram. (The standard reconstructPar tool can only rebuild all time directories, or a list of timestamps, and does not have the neat "start,end" option like this script.) If you are using decimals in your saved timestamps, check that the script does not skip a timestep between the split time ranges, because bash has trouble with sorting numbers with decimals. |
reconstructPar in parallel using GNU Parallel with a bash one-liner
Hi All,
I didn't know there was a script for this - really nice. I usually just do this with a bash one-liner: Code:
$ foamListTimes -processor > log.foamTimes; awk 'NR%4==1' log.foamTimes | parallel --halt=0 -j8 reconstructPar -newTimes -time {}:Code:
foamListTimes -processorCode:
awk 'NR%4==1' log.foamTimesCode:
parallel --halt=0 -j8 reconstructPar -newTimes -time {}:Anyway, that's the solution I've been using - hope this helps. Peter |
Kwardly,
Thank you very much for sharing the parReconstructPar script. It is perfectly working. Best, Meth. |
Thanks Peter. Your one-line code works like a charm, concise and efficient.
|
Thanks, it real helps.
It looks good, but I've notice that something possible a bug? This is my input $ sh reconPar 24 test running reconstructPar -noZero in pseudo-parallel mode on 24 processors reconstructing 134 time directories making temp dir Starting Job 1 - reconstructing time = 0 through 10.5 Starting Job 2 - reconstructing time = 10.8 through 1.2 Starting Job 3 - reconstructing time = 12.3 through 13.8 Starting Job 4 - reconstructing time = 14.1 through 15.3 Starting Job 5 - reconstructing time = 15.6 through 17.1 Starting Job 6 - reconstructing time = 17.4 through 18.6 Starting Job 7 - reconstructing time = 18.9 through 20.4 Starting Job 8 - reconstructing time = 20.7 through 21.9 Starting Job 9 - reconstructing time = 22.2 through 23.7 Starting Job 10 - reconstructing time = 24 through 25.2 Starting Job 11 - reconstructing time = 25.5 through 27 Starting Job 12 - reconstructing time = 2.7 through 28.5 Starting Job 13 - reconstructing time = 28.8 through 30 Starting Job 14 - reconstructing time = 30.3 through 31.8 Starting Job 15 - reconstructing time = 32.1 through 33.3 Starting Job 16 - reconstructing time = 33.6 through 35.1 Starting Job 17 - reconstructing time = 35.4 through 36.6 Starting Job 18 - reconstructing time = 36.9 through 38.4 Starting Job 19 - reconstructing time = 38.7 through 39.9 Starting Job 20 - reconstructing time = 4.2 through 5.7 Starting Job 21 - reconstructing time = 6 through 7.5 Starting Job 22 - reconstructing time = 7.8 through 9.3 Starting Job 23 - reconstructing time = 9.6 through Starting Job 24 - reconstructing time = through 39.9 =============================================== what does it mean for Job2 : 10.8 though 1.2? Is this a bug? Starting Job 2 - reconstructing time = 10.8 through 1.2 Also notice that Starting Job 23 - reconstructing time = 9.6 through Starting Job 24 - reconstructing time = through 39.9 I can not understand what those two lines mean? It only deals with the timeStamp of 9.6 and 39.9? After a chunk of time, the program stuck at some timeStamp. Here's the output from this script after running a long time. sh reconPar 24 test running reconstructPar -noZero in pseudo-parallel mode on 24 processors reconstructing 134 time directories making temp dir Starting Job 1 - reconstructing time = 0 through 10.5 Starting Job 2 - reconstructing time = 10.8 through 1.2 Starting Job 3 - reconstructing time = 12.3 through 13.8 Starting Job 4 - reconstructing time = 14.1 through 15.3 Starting Job 5 - reconstructing time = 15.6 through 17.1 Starting Job 6 - reconstructing time = 17.4 through 18.6 Starting Job 7 - reconstructing time = 18.9 through 20.4 Starting Job 8 - reconstructing time = 20.7 through 21.9 Starting Job 9 - reconstructing time = 22.2 through 23.7 Starting Job 10 - reconstructing time = 24 through 25.2 Starting Job 11 - reconstructing time = 25.5 through 27 Starting Job 12 - reconstructing time = 2.7 through 28.5 Starting Job 13 - reconstructing time = 28.8 through 30 Starting Job 14 - reconstructing time = 30.3 through 31.8 Starting Job 15 - reconstructing time = 32.1 through 33.3 Starting Job 16 - reconstructing time = 33.6 through 35.1 Starting Job 17 - reconstructing time = 35.4 through 36.6 Starting Job 18 - reconstructing time = 36.9 through 38.4 Starting Job 19 - reconstructing time = 38.7 through 39.9 Starting Job 20 - reconstructing time = 4.2 through 5.7 Starting Job 21 - reconstructing time = 6 through 7.5 Starting Job 22 - reconstructing time = 7.8 through 9.3 Starting Job 23 - reconstructing time = 9.6 through Starting Job 24 - reconstructing time = through 39.9 --> FOAM FATAL ERROR: No times selected From function int main(int, char**) in file reconstructPar.C at line 225. FOAM exiting 134 directories remaining... 112 directories remaining... 110 directories remaining... 108 directories remaining... 105 directories remaining... 100 directories remaining... 89 directories remaining... 88 directories remaining... 87 directories remaining... 85 directories remaining... 83 directories remaining... 79 directories remaining... 77 directories remaining... 73 directories remaining... 63 directories remaining... 61 directories remaining... 58 directories remaining... 56 directories remaining... 54 directories remaining... 52 directories remaining... 49 directories remaining... 39 directories remaining... 37 directories remaining... 35 directories remaining... 33 directories remaining... 32 directories remaining... 28 directories remaining... 24 directories remaining... 20 directories remaining... 19 directories remaining... 17 directories remaining... 15 directories remaining... 11 directories remaining... 10 directories remaining... 9 directories remaining... 8 directories remaining... 7 directories remaining... 6 directories remaining... E.O.F I am using O.F. v4.1 with the flowing hardware: 24 cores/node, Memory per node 32 G, infiniband, AMD @2.1GHz CPU, Centos 6.8. Third update: After 01:43:43 running, it finish without error. Thanks man！ But according to my observation, the final several directories were much slower than others. Anybody has ideas about this? This is full record of the output. $ sh reconPar 24 test running reconstructPar -noZero in pseudo-parallel mode on 24 processors reconstructing 134 time directories making temp dir Starting Job 1 - reconstructing time = 0 through 10.5 Starting Job 2 - reconstructing time = 10.8 through 1.2 Starting Job 3 - reconstructing time = 12.3 through 13.8 Starting Job 4 - reconstructing time = 14.1 through 15.3 Starting Job 5 - reconstructing time = 15.6 through 17.1 Starting Job 6 - reconstructing time = 17.4 through 18.6 Starting Job 7 - reconstructing time = 18.9 through 20.4 Starting Job 8 - reconstructing time = 20.7 through 21.9 Starting Job 9 - reconstructing time = 22.2 through 23.7 Starting Job 10 - reconstructing time = 24 through 25.2 Starting Job 11 - reconstructing time = 25.5 through 27 Starting Job 12 - reconstructing time = 2.7 through 28.5 Starting Job 13 - reconstructing time = 28.8 through 30 Starting Job 14 - reconstructing time = 30.3 through 31.8 Starting Job 15 - reconstructing time = 32.1 through 33.3 Starting Job 16 - reconstructing time = 33.6 through 35.1 Starting Job 17 - reconstructing time = 35.4 through 36.6 Starting Job 18 - reconstructing time = 36.9 through 38.4 Starting Job 19 - reconstructing time = 38.7 through 39.9 Starting Job 20 - reconstructing time = 4.2 through 5.7 Starting Job 21 - reconstructing time = 6 through 7.5 Starting Job 22 - reconstructing time = 7.8 through 9.3 Starting Job 23 - reconstructing time = 9.6 through Starting Job 24 - reconstructing time = through 39.9 --> FOAM FATAL ERROR: No times selected From function int main(int, char**) in file reconstructPar.C at line 225. FOAM exiting 134 directories remaining... 112 directories remaining... 110 directories remaining... 108 directories remaining... 105 directories remaining... 100 directories remaining... 89 directories remaining... 88 directories remaining... 87 directories remaining... 85 directories remaining... 83 directories remaining... 79 directories remaining... 77 directories remaining... 73 directories remaining... 63 directories remaining... 61 directories remaining... 58 directories remaining... 56 directories remaining... 54 directories remaining... 52 directories remaining... 49 directories remaining... 39 directories remaining... 37 directories remaining... 35 directories remaining... 33 directories remaining... 32 directories remaining... 28 directories remaining... 24 directories remaining... 20 directories remaining... 19 directories remaining... 17 directories remaining... 15 directories remaining... 11 directories remaining... 10 directories remaining... 9 directories remaining... 8 directories remaining... 7 directories remaining... 6 directories remaining... 5 directories remaining... 4 directories remaining... 3 directories remaining... 2 directories remaining... 1 directories remaining... cleaning up temp files finished E.O.F |
I found the creator [O. Tange (2011)] of GNU parallel was funny.
To silence the citation notice: run 'parallel --bibtex'. That's a good way to remind users. Anyway, thanks. Quote:
|
Another minor fix to get all time steps in order
1 Attachment(s)
Dear all,
I just wanted to upload a small fix that handles situations in which Will's last version still struggles (e.g. if you have outputs at times 0.125, 0.25, 0.375 and 0.5, when the proposed "ls -v1" by itself doesn't capture well the order). The idea is simply to replace the occurrences of "ls processor0 -1v | ..." with "ls processor0 -1v | sort -g | ...", which guarantees the ordering is correct regardless of how many decimal digits are used in different folders. Regardless, this is a great script, very useful to get a faster reconstruction. Thanks to all those who contributed. |
Is it possible to reconstruct specific timestamps using -t option?
|
Quote:
This script assigns different reconstruct jobs to different threads. I have not used it for a while but if you set the time range to include only a single timestemp, the script will only use a single thread, and should be identical to just running "reconstructPar -time x.xx" I now mostly use the ParaFoam option to visualize a decomposed case, without the need for reconstructing. |
Very usefull scripts !
|
| All times are GMT -4. The time now is 18:12. |