Logging help

[koranthala]

Hi,
Is it possible somehow to have the logging module rotate the files
every time I start it.
Basically, I can automatically rotate using RotatingFileHandler;
Now I want it rotated every time I start the program too.
Ex: The logging file - log.txt
Now, rotatingfilehandler goes and updates to log.txt.1, log.txt.2,
log.txt.3 etc.
Now, I also want to rotate it automatically I start the program -
i.e.
The second time I start the program - log.txt -> log.txt.1, log.txt.
1 -> log.txt.2 etc.

I can write up a simple script to move it by 1 digit every time.
My question is - is it possible automatically via logging
functionality? This is needed because say TimedRotatingFileHandler etc
wont work that well otherwise.

It does look like twisted.python.logfile.DailyLogFile seems to be
an Ok option, but I want to use the logging module in the Python
distribution, since it is more powerful.

 

[Chris Rebert]

Hi,
Is it possible somehow to have the logging module rotate the files
every time I start it.
Basically, I can automatically rotate using RotatingFileHandler;
Now I want it rotated every time I start the program too.

Just call the .doRollover() method of the RotatingFileHandler at the
start of the program.
Reading The Fine Documentation for the module is helpful.

Cheers,
Chris

 

[koranthala]

Just call the .doRollover() method of the RotatingFileHandler at the
start of the program.
Reading The Fine Documentation for the module is helpful.

Cheers,
Chris

My question was poorly framed. I will try to explain the issue a
little more.
Current doRollover method is not very helpful to rolling over every
day if the process starts and stops many times.
For example:
TimedRotatingFileHandler - when='D' Interval=1. I call
handler.doRollover everytime process starts.

First run - the output files are
log.txt
----------
Second run - files are
log.txt, log.txt.2009-01-20
----------
Till now, fine.
Third run - files are
log.txt, log.txt.2009-01-20 ***But the earlier file is overwritten***

The doRollover method does not append to the earlier file. Rather, it
creates a new file with the same name.
Due to this the earlier logs are all gone.

 

[Gabriel Genellina]

Current doRollover method is not very helpful to rolling over every
day if the process starts and stops many times.
For example:
TimedRotatingFileHandler - when='D' Interval=1. I call
^^^^^^^^^^^^^^^^^^^^^^^^ handler.doRollover everytime process starts.

Why the move to TimedRotatingFileHandler?

 

[Gabriel Genellina]

Current doRollover method is not very helpful to rolling over every
day if the process starts and stops many times.
For example:
TimedRotatingFileHandler - when='D' Interval=1. I call
^^^^^^^^^^^^^^^^^^^^^^^^ handler.doRollover everytime process starts.

Why the move to TimedRotatingFileHandler?

 

[Vinay Sajip]

The doRollover method does not append to the earlier file. Rather, it
creates a new file with the same name.

Err... that's what rollover means - switching to a new log file (and
renaming the old ones). If it just appended to the old one, why would
it be called doRollover ? ;-)

Regards,

Vinay Sajip

 

[koranthala]

Err... that's what rollover means - switching to a new log file (and
renaming the old ones). If it just appended to the old one, why would
it be called doRollover ? ;-)

Regards,

Vinay Sajip

I understand Vinay. But my point is that I wanted a mechanism to
rotate the log files based on data - i.e. today one log, tomorrow
another. This is easier because when trouble tickets are raised, users
mention that X failed at this time.
Now, timedrotatingfilehandler does it - but only if the program is
running the whole length of time.
My tool is invoked by a cron job - so the program runs, then stops
again and again.
When I posted the question here, I was forwarded to the doRollover
mechanism as a solution.
I was just mentioning that doRollover, due to overwriting the files I
would lose everything that is stored before.
What I did now was to copy the file, and then append to the rolledover
file - which does serve the purpose.
But it is kludgy, and I dont think my situation is unique. I did see
other people asking for the same functionality.

So I was wondering whether it would be good idea to have this also in
the next version.

 

[Vinay Sajip]

I understand Vinay. But my point is that I wanted a mechanism to
rotate the log files based on data - i.e. today one log, tomorrow

Did you mean "based on date"?

another. This is easier because when trouble tickets are raised, users
mention that X failed at this time.
Now, timedrotatingfilehandler does it - but only if the program is
running the whole length of time.
My tool is invoked by a cron job - so the program runs, then stops
again and again.

If you just want a log file whose name is date-based, you don't need a
rotating file handler. Compute the file name from the date and use the
API to create a FileHandler specifying that file name, and add it to
your logger. For example:

import logging, time

logging.basicConfig(level=logging.DEBUG, filename=time.strftime("/path/
to/my/logs/myapp-%Y-%m-%d-%H%M.log", time.localtime()), filemode="w")
logging.debug("Hello, world!")

Hopefully you can adapt this snippet to your needs.

Regards,

Vinay Sajip

 

[koranthala]

Did you mean "based on date"?


If you just want a log file whose name is date-based, you don't need a
rotating file handler. Compute the file name from the date and use the
API to create a FileHandler specifying that file name, and add it to
your logger. For example:

import logging, time

logging.basicConfig(level=logging.DEBUG, filename=time.strftime("/path/
to/my/logs/myapp-%Y-%m-%d-%H%M.log", time.localtime()), filemode="w")
logging.debug("Hello, world!")

Hopefully you can adapt this snippet to your needs.

Regards,

Vinay Sajip

Thank you very much Vinay. You have been extremely helpful.
This was my first design - but then I found that log system was taking
up quite a huge chunk of the memory.
That is why I went to rotating file handler.
Anyways, now I have modified doRollover to append if file is there,
so, for me it is working.
What I was thinking was such an option in the basic logging system
might be of good help.
Again, Vinay, Thank you very much. You were extremely helpful.

 

[Vinay Sajip]

Thank you very much Vinay. You have been extremely helpful.
This was my first design - but then I found that log system was taking
up quite a huge chunk of the memory.
That is why I went to rotating file handler.

Using just plain FileHandler takes up less memory than using
RotatingFileHandler (because using the latter imports the "handlers"
module into memory, whereas using the former does not). I'm not aware
of any problem where logging takes up a huge chunk of memory (under
normal usage) - how did you measure the memory usage which was due to
logging?

Anyways, now I have modified doRollover to append if file is there,
so, for me it is working.
What I was thinking was such an option in the basicloggingsystem
might be of good help.

The solution I suggested is IMO better than using a patched
RotatiingFileHandler, both because it avoids importing an extra module
(if memory really is a concern) and because you don't need to change
your code across Python version updates. How does my suggestion to use
FileHandler not meet your use case?

Again, Vinay, Thank you very much. You were extremely helpful.

You're welcome.

Best wishes,

Vinay Sajip

 

[koranthala]

Using just plain FileHandler takes up less memory than using
RotatingFileHandler (because using the latter imports the "handlers"
module into memory, whereas using the former does not). I'm not aware
of any problem where logging takes up a huge chunk of memory (under
normal usage) - how did you measure the memory usage which was due to
logging?


The solution I suggested is IMO better than using a patched
RotatiingFileHandler, both because it avoids importing an extra module
(if memory really is a concern) and because you don't need to change
your code across Python version updates. How does my suggestion to use
FileHandler not meet your use case?


You're welcome.

Best wishes,

Vinay Sajip

oops - again I was very unclear in my post.
What i meant was that due to the log getting very big - i.e. the log
was in 100MB range (since it is a new product - a lot of debugging is
done via logs). Not RAM.
Having a rotating file handler helped there - because I could say only
5 such logs are needed.

 

[Aahz]

Is it possible somehow to have the logging module rotate the files
every time I start it.

If you're on Linux, why not use logrotate?