CFD Online Discussion Forums

CFD Online Discussion Forums (http://www.cfd-online.com/Forums/)
-   OpenFOAM Running, Solving & CFD (http://www.cfd-online.com/Forums/openfoam-solving/)
-   -   Log File too Large (http://www.cfd-online.com/Forums/openfoam-solving/66749-log-file-too-large.html)

Julian K. July 23, 2009 06:21

Log File too Large
 
Hallo,

I do have the following, rather banal problem: I'd like to direct the residuals of my simulation to a log file, in order to be able to plot them at some time. However, since my time step is very small, the log file expands quickly. I estimated that, if I run my simulation for 24 hours, the log file will have a size of approx. 3 gigabyte.

So, how do you handle the size of your log files?
Is there a possibility to restrict the size of the file to a certain limit, so that, for instance, only the last 1000 residuals are written?



Cheers,

Julian

gschaider July 23, 2009 07:14

Quote:

Originally Posted by Julian K. (Post 223797)
Hallo,

I do have the following, rather banal problem: I'd like to direct the residuals of my simulation to a log file, in order to be able to plot them at some time. However, since my time step is very small, the log file expands quickly. I estimated that, if I run my simulation for 24 hours, the log file will have a size of approx. 3 gigabyte.

So, how do you handle the size of your log files?
Is there a possibility to restrict the size of the file to a certain limit, so that, for instance, only the last 1000 residuals are written?



Cheers,

Julian

Possible solutions:

julianFoam | gzip >theLog.gz

or

julianFoam | tail -10000 >theLog

henrik July 23, 2009 07:21

Bernhard,

not sure that the second option will help. I think this will simply collect the first 10000 lines, write them into theLog and then stop.

However, Julian want to have the last 10000 lines dynamically. So 10-10010 which implies rewriting the file from the start. I think the system tools for log-rotation will be helpful here, but don't know more than that.

Henrik

gschaider July 23, 2009 08:01

Quote:

Originally Posted by henrik (Post 223809)
Bernhard,

not sure that the second option will help. I think this will simply collect the first 10000 lines, write them into theLog and then stop.

I think you're confusing heads with tails. What you mean would have been

julianFoam | head -10000 >theLog

But you're right. tail won't know the last 10k lines before the simulation ended

Quote:

Originally Posted by henrik (Post 223809)
However, Julian want to have the last 10000 lines dynamically. So 10-10010 which implies rewriting the file from the start. I think the system tools for log-rotation will be helpful here, but don't know more than that.

I think even those tools would have a hard time truncating a file that is still being written.

A workaround (beware: advertisement) that could reduce the amount of data written (but would still write all residuals so it would still be a lot of data) would be to run the solver with "pyFoamRunner.py --no-log". That would not write a log-file but extract the residuals from the standard-output and write them to the .analyzed-directory. Similar things might also work with the foamLog-etc-scripts that come with OF

Bernhard

Julian K. July 23, 2009 08:11

Thank you Bernhard and Henrik for your quick reply.

I tried both commands that you posted

Quote:

gschaider:
julianFoam | gzip >theLog.gz
Quote:

gschaider:
julianFoam | tail -10000 >theLog
However, both did not work out. They solely created a file named 'theLog' and 'theLog.gz', respectively. The files are empty and there is no computation process started.

Where's the Problem?

Quote:

henrik:
I think the system tools for log-rotation will be helpful here, but don't know more than that.
After a quick research with google, I found this page http://www.linux-praxis.de/lpic1/man...logrotate.html (in German)/http://linuxcommand.org/man_pages/logrotate8.html (in English). I will go through it and see if this is an option for me.

henrik July 23, 2009 08:23

Bernhard,

you simply write myLog and once it contains more than x lines you flush the handle, move the file to myLog.last and start a fresh myLog.

We must not use use a pipe at the exit! So something like this
Code:

julianFoam | betterLogger 10000 myLog
When finished do

Code:

cat myLog.last myLog > myLog
and you are guaranteed to have at least x+1 lines.

Actually, it should not be hard to write such a shell-filter or pyFoam?

Henrik

gschaider July 23, 2009 09:05

Quote:

Originally Posted by Julian K. (Post 223820)
Thank you Bernhard and Henrik for your quick reply.

I tried both commands that you posted

However, both did not work out. They solely created a file named 'theLog' and 'theLog.gz', respectively. The files are empty and there is no computation process started.

Where's the Problem?

You DID use an existing solver instead of julianFoam?

Quote:

Originally Posted by Julian K. (Post 223820)
After a quick research with google, I found this page http://www.linux-praxis.de/lpic1/man...logrotate.html (in German)/http://linuxcommand.org/man_pages/logrotate8.html (in English). I will go through it and see if this is an option for me.

In my experience logrotate usually temporarily stops the daemon whose logs it rotates for "busy" logfiles. Which of course is not an option for a solver

Julian K. July 23, 2009 09:09

Quote:

You DID use an existing solver instead of julianFoam?
:DYes, of cause.


I tried the command

Code:

rhoCentralFoam | betterLogger 10000 myLog
and got the following output:

Code:

bash: betterLogger: command not found
Thus, I looked for a package named 'betterLogger' with YaST, but couldn't find anything. Sorry for my ignorance.

gschaider July 23, 2009 09:11

Quote:

Originally Posted by henrik (Post 223825)
Bernhard,

you simply write myLog and once it contains more than x lines you flush the handle, move the file to myLog.last and start a fresh myLog.

We must not use use a pipe at the exit! So something like this
Code:

julianFoam | betterLogger 10000 myLog
When finished do

Code:

cat myLog.last myLog > myLog
and you are guaranteed to have at least x+1 lines.

Actually, it should not be hard to write such a shell-filter or pyFoam?

Not really. I'll think about it.

What it already has is --compress-option for PyFoamRunner.py that writes the logfile as a gz (the compression ratio is not optimal). I'm just not sure whether it is already in the last released version

gschaider July 23, 2009 09:17

Quote:

Originally Posted by Julian K. (Post 223841)
:DYes, of cause.


I tried the command

Code:

rhoCentralFoam | betterLogger 10000 myLog
and got the following output:

Code:

bash: betterLogger: command not found
Thus, I looked for a package named 'betterLogger' with YaST, but couldn't find anything. Sorry for my ignorance.

I think Henrik has defined betterLogger as an alias to superiorButYetUnwrittenLoggingProgramJUstForTheSak eOfAnExample on his machine ;)

henrik July 23, 2009 09:46

Bernhard,

you have little faith ;)

Code:

#!/usr/bin/perl

use strict;

my $nLines = 0;
my $maxLines = shift;
my $file = shift;

open(OF, "> $file");

while ( <> )
{
#    print $_; // Uncomment for betaTee

    if ( $nLines++ == $maxLines )
    {
        system("mv $file $file.last");
        open(OF, ">$file");
        $nLines = 1;
    }

    print OF $_;
}

and it is superior, if you sort out the programming language, and, yes, it was unwritten.

Julien: cut & paste the code into a text file and name is "betaLogger". Then execute chmod u+x betaLogger

Code:

julianFoam | betaLogger 1000 log
Henrik

Julian K. July 23, 2009 10:05

Quote:

Originally Posted by henrik (Post 223848)
Bernhard,

you have little faith ;)

Code:

#!/usr/bin/perl

use strict;

my $nLines = 0;
my $maxLines = shift;
my $file = shift;

open(OF, "> $file");

while ( <> )
{
#    print $_; // Uncomment for betaTee

    if ( $nLines++ == $maxLines )
    {
        system("mv $file $file.last");
        open(OF, ">$file");
        $nLines = 1;
    }

    print OF $_;
}

and it is superior, if you sort out the programming language, and, yes, it was unwritten.

Julien: cut & paste the code into a text file and name is "betaLogger". Then execute chmod u+x betaLogger

Code:

julianFoam | betaLogger 1000 log
Henrik

Yes, that's it! Thank you very much! :)


All times are GMT -4. The time now is 09:47.