[Julian K.]

Hallo,

I do have the following, rather banal problem: I'd like to direct the residuals of my simulation to a log file, in order to be able to plot them at some time. However, since my time step is very small, the log file expands quickly. I estimated that, if I run my simulation for 24 hours, the log file will have a size of approx. 3 gigabyte.

So, how do you handle the size of your log files?
Is there a possibility to restrict the size of the file to a certain limit, so that, for instance, only the last 1000 residuals are written?


Cheers,

Julian

[gschaider]

Originally Posted by Julian K.

Hallo,

I do have the following, rather banal problem: I'd like to direct the residuals of my simulation to a log file, in order to be able to plot them at some time. However, since my time step is very small, the log file expands quickly. I estimated that, if I run my simulation for 24 hours, the log file will have a size of approx. 3 gigabyte.

So, how do you handle the size of your log files?
Is there a possibility to restrict the size of the file to a certain limit, so that, for instance, only the last 1000 residuals are written?


Cheers,

Julian

Possible solutions:

julianFoam | gzip >theLog.gz

or

julianFoam | tail -10000 >theLog

[henrik]

Bernhard,

not sure that the second option will help. I think this will simply collect the first 10000 lines, write them into theLog and then stop.

However, Julian want to have the last 10000 lines dynamically. So 10-10010 which implies rewriting the file from the start. I think the system tools for log-rotation will be helpful here, but don't know more than that.

Henrik

[gschaider]

Originally Posted by henrik

Bernhard,

not sure that the second option will help. I think this will simply collect the first 10000 lines, write them into theLog and then stop.

I think you're confusing heads with tails. What you mean would have been

julianFoam | head -10000 >theLog

But you're right. tail won't know the last 10k lines before the simulation ended

Originally Posted by henrik

However, Julian want to have the last 10000 lines dynamically. So 10-10010 which implies rewriting the file from the start. I think the system tools for log-rotation will be helpful here, but don't know more than that.

I think even those tools would have a hard time truncating a file that is still being written.

A workaround (beware: advertisement) that could reduce the amount of data written (but would still write all residuals so it would still be a lot of data) would be to run the solver with "pyFoamRunner.py --no-log". That would not write a log-file but extract the residuals from the standard-output and write them to the .analyzed-directory. Similar things might also work with the foamLog-etc-scripts that come with OF

Bernhard

[Julian K.]

Thank you Bernhard and Henrik for your quick reply.

I tried both commands that you posted

gschaider:
julianFoam | gzip >theLog.gz

gschaider:
julianFoam | tail -10000 >theLog

However, both did not work out. They solely created a file named 'theLog' and 'theLog.gz', respectively. The files are empty and there is no computation process started.

Where's the Problem?

henrik:
I think the system tools for log-rotation will be helpful here, but don't know more than that.

After a quick research with google, I found this page http://www.linux-praxis.de/lpic1/man...logrotate.html (in German)/http://linuxcommand.org/man_pages/logrotate8.html (in English). I will go through it and see if this is an option for me.

[henrik]

Bernhard,

you simply write myLog and once it contains more than x lines you flush the handle, move the file to myLog.last and start a fresh myLog.

We must not use use a pipe at the exit! So something like this

julianFoam | betterLogger 10000 myLog

When finished do

cat myLog.last myLog > myLog

and you are guaranteed to have at least x+1 lines.

Actually, it should not be hard to write such a shell-filter or pyFoam?

Henrik

[gschaider]

Originally Posted by Julian K.

Thank you Bernhard and Henrik for your quick reply.

I tried both commands that you posted

However, both did not work out. They solely created a file named 'theLog' and 'theLog.gz', respectively. The files are empty and there is no computation process started.

Where's the Problem?

You DID use an existing solver instead of julianFoam?

Originally Posted by Julian K.

After a quick research with google, I found this page http://www.linux-praxis.de/lpic1/man...logrotate.html (in German)/http://linuxcommand.org/man_pages/logrotate8.html (in English). I will go through it and see if this is an option for me.

In my experience logrotate usually temporarily stops the daemon whose logs it rotates for "busy" logfiles. Which of course is not an option for a solver

[Julian K.]

You DID use an existing solver instead of julianFoam?

Yes, of cause.


I tried the command

rhoCentralFoam | betterLogger 10000 myLog

and got the following output:

bash: betterLogger: command not found

Thus, I looked for a package named 'betterLogger' with YaST, but couldn't find anything. Sorry for my ignorance.

[gschaider]

Originally Posted by henrik

Bernhard,

you simply write myLog and once it contains more than x lines you flush the handle, move the file to myLog.last and start a fresh myLog.

We must not use use a pipe at the exit! So something like this

julianFoam | betterLogger 10000 myLog

When finished do

cat myLog.last myLog > myLog

and you are guaranteed to have at least x+1 lines.

Actually, it should not be hard to write such a shell-filter or pyFoam?

Not really. I'll think about it.

What it already has is --compress-option for PyFoamRunner.py that writes the logfile as a gz (the compression ratio is not optimal). I'm just not sure whether it is already in the last released version

[gschaider]

Originally Posted by Julian K.

Yes, of cause.


I tried the command

rhoCentralFoam | betterLogger 10000 myLog

and got the following output:

bash: betterLogger: command not found

Thus, I looked for a package named 'betterLogger' with YaST, but couldn't find anything. Sorry for my ignorance.

I think Henrik has defined betterLogger as an alias to superiorButYetUnwrittenLoggingProgramJUstForTheSak eOfAnExample on his machine

[henrik]

Bernhard,

you have little faith

#!/usr/bin/perl

use strict;

my $nLines = 0;
my $maxLines = shift;
my $file = shift;

open(OF, "> $file");

while ( <> )
{
#    print $_; // Uncomment for betaTee

    if ( $nLines++ == $maxLines )
    {
        system("mv $file $file.last");
        open(OF, ">$file");
        $nLines = 1;
    }

    print OF $_;
}

and it is superior, if you sort out the programming language, and, yes, it was unwritten.

Julien: cut & paste the code into a text file and name is "betaLogger". Then execute chmod u+x betaLogger

julianFoam | betaLogger 1000 log

Henrik

[Julian K.]

Originally Posted by henrik

Bernhard,

you have little faith

#!/usr/bin/perl

use strict;

my $nLines = 0;
my $maxLines = shift;
my $file = shift;

open(OF, "> $file");

while ( <> )
{
#    print $_; // Uncomment for betaTee

    if ( $nLines++ == $maxLines )
    {
        system("mv $file $file.last");
        open(OF, ">$file");
        $nLines = 1;
    }

    print OF $_;
}

and it is superior, if you sort out the programming language, and, yes, it was unwritten.

Julien: cut & paste the code into a text file and name is "betaLogger". Then execute chmod u+x betaLogger

julianFoam | betaLogger 1000 log

Henrik

Yes, that's it! Thank you very much!

[hiuluom]

Hello,

I did following step above as Henrik guide, but I met error

betaLogger: command not found

How to fix it?

Thanks

[faab]

Hi,

@Hiuluom Have you tried to execute the script directly by using julianFoam | ./betaLogger 1000 log instead ?

I have a question for Henrik. First of all thank you for the script. I have a problem when I use it for parallel runs, it returns a blank file. Any idea why ?
The command I run is: runParallel nameofsolver nbProcs | betaLogger 1000 log

[Fischlurch]

Hi,

thanks to Henrik and Bernhard for this nice post and script, it was really helpful.

I do not know if someone is still interested in this, but to run the "betaLogger" in a bash script (as faab has done I assume), I just wrote instead the "runParallel" command the proper line:

mpirun -np $np solverName -parallel | ./betaLogger 1000 logFileName

Another way is to change the $WM_PROJECT_DIR/bin/tools/RunFunctions script.

Cheers

Hauke

[gaza]

Originally Posted by gschaider

I think Henrik has defined betterLogger as an alias to superiorButYetUnwrittenLoggingProgramJUstForTheSak eOfAnExample on his machine

Hi
Can you rewrite it into bash?
I am running my cases on remote machine and there is no perl

[gschaider]

Originally Posted by gaza

Hi
Can you rewrite it into bash?
I am running my cases on remote machine and there is no perl

Me? Rewrite what into bash?


I'm sorry: no


Reasons:

a) I only use bash to glue commands together. That's what it was designed for. Not "real" programming
b) there is a reason that log-analysis is usually done in proper programming languages
c) I don't need it and you're not a customer. These are usually the reasons why I write stuff and that is enough to fill my time

[gaza]

Ok
I thought that code of betaLogger is easy to rewrite into bash
thnks for reply