CFD Online Discussion Forums

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)
-   OpenFOAM Running, Solving & CFD (https://www.cfd-online.com/Forums/openfoam-solving/)
-   -   Log File too Large (https://www.cfd-online.com/Forums/openfoam-solving/66749-log-file-too-large.html)

Julian K. July 23, 2009 06:21
Log File too Large
 
Hallo,

I do have the following, rather banal problem: I'd like to direct the residuals of my simulation to a log file, in order to be able to plot them at some time. However, since my time step is very small, the log file expands quickly. I estimated that, if I run my simulation for 24 hours, the log file will have a size of approx. 3 gigabyte.

So, how do you handle the size of your log files?
Is there a possibility to restrict the size of the file to a certain limit, so that, for instance, only the last 1000 residuals are written?


Cheers,

Julian

gschaider July 23, 2009 07:14
Quote:
Originally Posted by Julian K. (Post 223797)
Hallo,

I do have the following, rather banal problem: I'd like to direct the residuals of my simulation to a log file, in order to be able to plot them at some time. However, since my time step is very small, the log file expands quickly. I estimated that, if I run my simulation for 24 hours, the log file will have a size of approx. 3 gigabyte.

So, how do you handle the size of your log files?
Is there a possibility to restrict the size of the file to a certain limit, so that, for instance, only the last 1000 residuals are written?


Cheers,

Julian
Possible solutions:

julianFoam | gzip >theLog.gz

or

julianFoam | tail -10000 >theLog

henrik July 23, 2009 07:21
Bernhard,

not sure that the second option will help. I think this will simply collect the first 10000 lines, write them into theLog and then stop.

However, Julian want to have the last 10000 lines dynamically. So 10-10010 which implies rewriting the file from the start. I think the system tools for log-rotation will be helpful here, but don't know more than that.

Henrik

gschaider July 23, 2009 08:01
Quote:
Originally Posted by henrik (Post 223809)
Bernhard,

not sure that the second option will help. I think this will simply collect the first 10000 lines, write them into theLog and then stop.
I think you're confusing heads with tails. What you mean would have been

julianFoam | head -10000 >theLog

But you're right. tail won't know the last 10k lines before the simulation ended

Quote:
Originally Posted by henrik (Post 223809)
However, Julian want to have the last 10000 lines dynamically. So 10-10010 which implies rewriting the file from the start. I think the system tools for log-rotation will be helpful here, but don't know more than that.
I think even those tools would have a hard time truncating a file that is still being written.

A workaround (beware: advertisement) that could reduce the amount of data written (but would still write all residuals so it would still be a lot of data) would be to run the solver with "pyFoamRunner.py --no-log". That would not write a log-file but extract the residuals from the standard-output and write them to the .analyzed-directory. Similar things might also work with the foamLog-etc-scripts that come with OF

Bernhard

Julian K. July 23, 2009 08:11
Thank you Bernhard and Henrik for your quick reply.

I tried both commands that you posted

Quote:
gschaider:
julianFoam | gzip >theLog.gz
Quote:
gschaider:
julianFoam | tail -10000 >theLog
However, both did not work out. They solely created a file named 'theLog' and 'theLog.gz', respectively. The files are empty and there is no computation process started.

Where's the Problem?

Quote:
henrik:
I think the system tools for log-rotation will be helpful here, but don't know more than that.
After a quick research with google, I found this page http://www.linux-praxis.de/lpic1/man...logrotate.html (in German)/http://linuxcommand.org/man_pages/logrotate8.html (in English). I will go through it and see if this is an option for me.

henrik July 23, 2009 08:23
Bernhard,

you simply write myLog and once it contains more than x lines you flush the handle, move the file to myLog.last and start a fresh myLog.

We must not use use a pipe at the exit! So something like this
Code:
julianFoam | betterLogger 10000 myLog
When finished do

Code:
cat myLog.last myLog > myLog
and you are guaranteed to have at least x+1 lines.

Actually, it should not be hard to write such a shell-filter or pyFoam?

Henrik

gschaider July 23, 2009 09:05
Quote:
Originally Posted by Julian K. (Post 223820)
Thank you Bernhard and Henrik for your quick reply.

I tried both commands that you posted

However, both did not work out. They solely created a file named 'theLog' and 'theLog.gz', respectively. The files are empty and there is no computation process started.

Where's the Problem?
You DID use an existing solver instead of julianFoam?

Quote:
Originally Posted by Julian K. (Post 223820)
After a quick research with google, I found this page http://www.linux-praxis.de/lpic1/man...logrotate.html (in German)/http://linuxcommand.org/man_pages/logrotate8.html (in English). I will go through it and see if this is an option for me.
In my experience logrotate usually temporarily stops the daemon whose logs it rotates for "busy" logfiles. Which of course is not an option for a solver

Julian K. July 23, 2009 09:09
Quote:
You DID use an existing solver instead of julianFoam?
:DYes, of cause.


I tried the command

Code:
rhoCentralFoam | betterLogger 10000 myLog
and got the following output:

Code:
bash: betterLogger: command not found
Thus, I looked for a package named 'betterLogger' with YaST, but couldn't find anything. Sorry for my ignorance.

gschaider July 23, 2009 09:11
Quote:
Originally Posted by henrik (Post 223825)
Bernhard,

you simply write myLog and once it contains more than x lines you flush the handle, move the file to myLog.last and start a fresh myLog.

We must not use use a pipe at the exit! So something like this
Code:
julianFoam | betterLogger 10000 myLog
When finished do

Code:
cat myLog.last myLog > myLog
and you are guaranteed to have at least x+1 lines.

Actually, it should not be hard to write such a shell-filter or pyFoam?
Not really. I'll think about it.

What it already has is --compress-option for PyFoamRunner.py that writes the logfile as a gz (the compression ratio is not optimal). I'm just not sure whether it is already in the last released version

gschaider July 23, 2009 09:17
Quote:
Originally Posted by Julian K. (Post 223841)
:DYes, of cause.


I tried the command

Code:
rhoCentralFoam | betterLogger 10000 myLog
and got the following output:

Code:
bash: betterLogger: command not found
Thus, I looked for a package named 'betterLogger' with YaST, but couldn't find anything. Sorry for my ignorance.
I think Henrik has defined betterLogger as an alias to superiorButYetUnwrittenLoggingProgramJUstForTheSak eOfAnExample on his machine ;)

henrik July 23, 2009 09:46
Bernhard,

you have little faith ;)

Code:
#!/usr/bin/perl

use strict;

my $nLines = 0;
my $maxLines = shift;
my $file = shift;

open(OF, "> $file");

while ( <> )
{
#    print $_; // Uncomment for betaTee

    if ( $nLines++ == $maxLines )
    {
        system("mv $file $file.last");
        open(OF, ">$file");
        $nLines = 1;
    }

    print OF $_;
}
and it is superior, if you sort out the programming language, and, yes, it was unwritten.

Julien: cut & paste the code into a text file and name is "betaLogger". Then execute chmod u+x betaLogger

Code:
julianFoam | betaLogger 1000 log
Henrik

Julian K. July 23, 2009 10:05
Quote:
Originally Posted by henrik (Post 223848)
Bernhard,

you have little faith ;)

Code:
#!/usr/bin/perl

use strict;

my $nLines = 0;
my $maxLines = shift;
my $file = shift;

open(OF, "> $file");

while ( <> )
{
#    print $_; // Uncomment for betaTee

    if ( $nLines++ == $maxLines )
    {
        system("mv $file $file.last");
        open(OF, ">$file");
        $nLines = 1;
    }

    print OF $_;
}
and it is superior, if you sort out the programming language, and, yes, it was unwritten.

Julien: cut & paste the code into a text file and name is "betaLogger". Then execute chmod u+x betaLogger

Code:
julianFoam | betaLogger 1000 log
Henrik
Yes, that's it! Thank you very much! :)

hiuluom June 12, 2015 00:44
Hello,

I did following step above as Henrik guide, but I met error

Code:
betaLogger: command not found
How to fix it?

Thanks

faab November 30, 2015 06:43
Hi,

@Hiuluom Have you tried to execute the script directly by using julianFoam | ./betaLogger 1000 log instead ?

I have a question for Henrik. First of all thank you for the script. I have a problem when I use it for parallel runs, it returns a blank file. Any idea why ?
The command I run is: runParallel nameofsolver nbProcs | betaLogger 1000 log

Fischlurch May 31, 2016 13:52
Hi,

thanks to Henrik and Bernhard for this nice post and script, it was really helpful.

I do not know if someone is still interested in this, but to run the "betaLogger" in a bash script (as faab has done I assume), I just wrote instead the "runParallel" command the proper line:

Code:
mpirun -np $np solverName -parallel | ./betaLogger 1000 logFileName
Another way is to change the $WM_PROJECT_DIR/bin/tools/RunFunctions script.

Cheers

Hauke

gaza February 14, 2019 13:04
Quote:
Originally Posted by gschaider (Post 223844)
I think Henrik has defined betterLogger as an alias to superiorButYetUnwrittenLoggingProgramJUstForTheSak eOfAnExample on his machine ;)

Hi
Can you rewrite it into bash?
I am running my cases on remote machine and there is no perl

gschaider February 14, 2019 13:50
Quote:
Originally Posted by gaza (Post 724800)
Hi
Can you rewrite it into bash?
I am running my cases on remote machine and there is no perl

Me? Rewrite what into bash?


I'm sorry: no


Reasons:

a) I only use bash to glue commands together. That's what it was designed for. Not "real" programming
b) there is a reason that log-analysis is usually done in proper programming languages
c) I don't need it and you're not a customer. These are usually the reasons why I write stuff and that is enough to fill my time

gaza February 15, 2019 05:56
Ok
I thought that code of betaLogger is easy to rewrite into bash
thnks for reply


All times are GMT -4. The time now is 03:15.