CFD Online URL
[Sponsors]
Home > Forums > OpenFOAM Running, Solving & CFD

Log File too Large

Register Blogs Members List Search Today's Posts Mark Forums Read

Like Tree2Likes
  • 1 Post By gschaider
  • 1 Post By henrik

Reply
 
LinkBack Thread Tools Display Modes
Old   July 23, 2009, 07:21
Default Log File too Large
  #1
Member
 
Julian Krick
Join Date: May 2009
Location: Guelph
Posts: 88
Rep Power: 7
Julian K. is on a distinguished road
Hallo,

I do have the following, rather banal problem: I'd like to direct the residuals of my simulation to a log file, in order to be able to plot them at some time. However, since my time step is very small, the log file expands quickly. I estimated that, if I run my simulation for 24 hours, the log file will have a size of approx. 3 gigabyte.

So, how do you handle the size of your log files?
Is there a possibility to restrict the size of the file to a certain limit, so that, for instance, only the last 1000 residuals are written?



Cheers,

Julian
__________________
grid generation: ICEM CFD 13.0
solver: CFX 13.0
Julian K. is offline   Reply With Quote

Old   July 23, 2009, 08:14
Default
  #2
Assistant Moderator
 
Bernhard Gschaider
Join Date: Mar 2009
Posts: 3,889
Rep Power: 38
gschaider will become famous soon enoughgschaider will become famous soon enough
Quote:
Originally Posted by Julian K. View Post
Hallo,

I do have the following, rather banal problem: I'd like to direct the residuals of my simulation to a log file, in order to be able to plot them at some time. However, since my time step is very small, the log file expands quickly. I estimated that, if I run my simulation for 24 hours, the log file will have a size of approx. 3 gigabyte.

So, how do you handle the size of your log files?
Is there a possibility to restrict the size of the file to a certain limit, so that, for instance, only the last 1000 residuals are written?



Cheers,

Julian
Possible solutions:

julianFoam | gzip >theLog.gz

or

julianFoam | tail -10000 >theLog
mm.abdollahzadeh likes this.
gschaider is offline   Reply With Quote

Old   July 23, 2009, 08:21
Default
  #3
Senior Member
 
Henrik Rusche
Join Date: Mar 2009
Location: Braunschweig, Niedersachsen, Germany
Posts: 275
Rep Power: 8
henrik is on a distinguished road
Bernhard,

not sure that the second option will help. I think this will simply collect the first 10000 lines, write them into theLog and then stop.

However, Julian want to have the last 10000 lines dynamically. So 10-10010 which implies rewriting the file from the start. I think the system tools for log-rotation will be helpful here, but don't know more than that.

Henrik
henrik is offline   Reply With Quote

Old   July 23, 2009, 09:01
Default
  #4
Assistant Moderator
 
Bernhard Gschaider
Join Date: Mar 2009
Posts: 3,889
Rep Power: 38
gschaider will become famous soon enoughgschaider will become famous soon enough
Quote:
Originally Posted by henrik View Post
Bernhard,

not sure that the second option will help. I think this will simply collect the first 10000 lines, write them into theLog and then stop.
I think you're confusing heads with tails. What you mean would have been

julianFoam | head -10000 >theLog

But you're right. tail won't know the last 10k lines before the simulation ended

Quote:
Originally Posted by henrik View Post
However, Julian want to have the last 10000 lines dynamically. So 10-10010 which implies rewriting the file from the start. I think the system tools for log-rotation will be helpful here, but don't know more than that.
I think even those tools would have a hard time truncating a file that is still being written.

A workaround (beware: advertisement) that could reduce the amount of data written (but would still write all residuals so it would still be a lot of data) would be to run the solver with "pyFoamRunner.py --no-log". That would not write a log-file but extract the residuals from the standard-output and write them to the .analyzed-directory. Similar things might also work with the foamLog-etc-scripts that come with OF

Bernhard
gschaider is offline   Reply With Quote

Old   July 23, 2009, 09:11
Default
  #5
Member
 
Julian Krick
Join Date: May 2009
Location: Guelph
Posts: 88
Rep Power: 7
Julian K. is on a distinguished road
Thank you Bernhard and Henrik for your quick reply.

I tried both commands that you posted

Quote:
gschaider:
julianFoam | gzip >theLog.gz
Quote:
gschaider:
julianFoam | tail -10000 >theLog
However, both did not work out. They solely created a file named 'theLog' and 'theLog.gz', respectively. The files are empty and there is no computation process started.

Where's the Problem?

Quote:
henrik:
I think the system tools for log-rotation will be helpful here, but don't know more than that.
After a quick research with google, I found this page http://www.linux-praxis.de/lpic1/man...logrotate.html (in German)/http://linuxcommand.org/man_pages/logrotate8.html (in English). I will go through it and see if this is an option for me.
__________________
grid generation: ICEM CFD 13.0
solver: CFX 13.0
Julian K. is offline   Reply With Quote

Old   July 23, 2009, 09:23
Default
  #6
Senior Member
 
Henrik Rusche
Join Date: Mar 2009
Location: Braunschweig, Niedersachsen, Germany
Posts: 275
Rep Power: 8
henrik is on a distinguished road
Bernhard,

you simply write myLog and once it contains more than x lines you flush the handle, move the file to myLog.last and start a fresh myLog.

We must not use use a pipe at the exit! So something like this
Code:
julianFoam | betterLogger 10000 myLog
When finished do

Code:
cat myLog.last myLog > myLog
and you are guaranteed to have at least x+1 lines.

Actually, it should not be hard to write such a shell-filter or pyFoam?

Henrik
henrik is offline   Reply With Quote

Old   July 23, 2009, 10:05
Default
  #7
Assistant Moderator
 
Bernhard Gschaider
Join Date: Mar 2009
Posts: 3,889
Rep Power: 38
gschaider will become famous soon enoughgschaider will become famous soon enough
Quote:
Originally Posted by Julian K. View Post
Thank you Bernhard and Henrik for your quick reply.

I tried both commands that you posted

However, both did not work out. They solely created a file named 'theLog' and 'theLog.gz', respectively. The files are empty and there is no computation process started.

Where's the Problem?
You DID use an existing solver instead of julianFoam?

Quote:
Originally Posted by Julian K. View Post
After a quick research with google, I found this page http://www.linux-praxis.de/lpic1/man...logrotate.html (in German)/http://linuxcommand.org/man_pages/logrotate8.html (in English). I will go through it and see if this is an option for me.
In my experience logrotate usually temporarily stops the daemon whose logs it rotates for "busy" logfiles. Which of course is not an option for a solver
gschaider is offline   Reply With Quote

Old   July 23, 2009, 10:09
Default
  #8
Member
 
Julian Krick
Join Date: May 2009
Location: Guelph
Posts: 88
Rep Power: 7
Julian K. is on a distinguished road
Quote:
You DID use an existing solver instead of julianFoam?
Yes, of cause.


I tried the command

Code:
rhoCentralFoam | betterLogger 10000 myLog
and got the following output:

Code:
bash: betterLogger: command not found
Thus, I looked for a package named 'betterLogger' with YaST, but couldn't find anything. Sorry for my ignorance.
__________________
grid generation: ICEM CFD 13.0
solver: CFX 13.0
Julian K. is offline   Reply With Quote

Old   July 23, 2009, 10:11
Default
  #9
Assistant Moderator
 
Bernhard Gschaider
Join Date: Mar 2009
Posts: 3,889
Rep Power: 38
gschaider will become famous soon enoughgschaider will become famous soon enough
Quote:
Originally Posted by henrik View Post
Bernhard,

you simply write myLog and once it contains more than x lines you flush the handle, move the file to myLog.last and start a fresh myLog.

We must not use use a pipe at the exit! So something like this
Code:
julianFoam | betterLogger 10000 myLog
When finished do

Code:
cat myLog.last myLog > myLog
and you are guaranteed to have at least x+1 lines.

Actually, it should not be hard to write such a shell-filter or pyFoam?
Not really. I'll think about it.

What it already has is --compress-option for PyFoamRunner.py that writes the logfile as a gz (the compression ratio is not optimal). I'm just not sure whether it is already in the last released version
gschaider is offline   Reply With Quote

Old   July 23, 2009, 10:17
Default
  #10
Assistant Moderator
 
Bernhard Gschaider
Join Date: Mar 2009
Posts: 3,889
Rep Power: 38
gschaider will become famous soon enoughgschaider will become famous soon enough
Quote:
Originally Posted by Julian K. View Post
Yes, of cause.


I tried the command

Code:
rhoCentralFoam | betterLogger 10000 myLog
and got the following output:

Code:
bash: betterLogger: command not found
Thus, I looked for a package named 'betterLogger' with YaST, but couldn't find anything. Sorry for my ignorance.
I think Henrik has defined betterLogger as an alias to superiorButYetUnwrittenLoggingProgramJUstForTheSak eOfAnExample on his machine
gschaider is offline   Reply With Quote

Old   July 23, 2009, 10:46
Default
  #11
Senior Member
 
Henrik Rusche
Join Date: Mar 2009
Location: Braunschweig, Niedersachsen, Germany
Posts: 275
Rep Power: 8
henrik is on a distinguished road
Bernhard,

you have little faith

Code:
#!/usr/bin/perl

use strict;

my $nLines = 0;
my $maxLines = shift;
my $file = shift;

open(OF, "> $file");

while ( <> )
{
#    print $_; // Uncomment for betaTee

    if ( $nLines++ == $maxLines )
    {
        system("mv $file $file.last");
        open(OF, ">$file");
        $nLines = 1;
    }

    print OF $_;
}
and it is superior, if you sort out the programming language, and, yes, it was unwritten.

Julien: cut & paste the code into a text file and name is "betaLogger". Then execute chmod u+x betaLogger

Code:
julianFoam | betaLogger 1000 log
Henrik
Hanzo likes this.
henrik is offline   Reply With Quote

Old   July 23, 2009, 11:05
Default
  #12
Member
 
Julian Krick
Join Date: May 2009
Location: Guelph
Posts: 88
Rep Power: 7
Julian K. is on a distinguished road
Quote:
Originally Posted by henrik View Post
Bernhard,

you have little faith

Code:
#!/usr/bin/perl

use strict;

my $nLines = 0;
my $maxLines = shift;
my $file = shift;

open(OF, "> $file");

while ( <> )
{
#    print $_; // Uncomment for betaTee

    if ( $nLines++ == $maxLines )
    {
        system("mv $file $file.last");
        open(OF, ">$file");
        $nLines = 1;
    }

    print OF $_;
}
and it is superior, if you sort out the programming language, and, yes, it was unwritten.

Julien: cut & paste the code into a text file and name is "betaLogger". Then execute chmod u+x betaLogger

Code:
julianFoam | betaLogger 1000 log
Henrik
Yes, that's it! Thank you very much!
__________________
grid generation: ICEM CFD 13.0
solver: CFX 13.0
Julian K. is offline   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Convection discretization schemes for LES sek OpenFOAM Running, Solving & CFD 22 July 11, 2012 04:18
Changing the grid on the same set-up Katya FLUENT 12 October 8, 2009 17:31
Paraview command not found hardy OpenFOAM Paraview & paraFoam 7 September 18, 2008 05:59
DxFoam reader update hjasak OpenFOAM Post-Processing 69 April 24, 2008 02:24
Importing I-DEAS unv file hbetb FLUENT 1 August 10, 2001 14:22


All times are GMT -4. The time now is 07:44.