RotatingFileHandler Fails

[nac]

The RotatingFileHandler running on win 7 64-bit; py 2.7 is failing
when the script launches a process using subprocess.Popen. Works fine
if the subprocess is not launched

The exception thrown
Traceback (most recent call last):
File "C:\Python27\lib\logging\handlers.py", line 78, in emit
self.doRollover()
File "C:\Python27\lib\logging\handlers.py", line 141, in doRollover
os.rename(self.baseFilename, dfn)
WindowsError: [Error 32] The process cannot access the file because it
is being used by another process

Anyone have an idea how to fix this?


import os, sys
import logging
import logging.handlers
import subprocess

def chomp(s):
"remove trailing carriage return"
if s[-1:]=='\n': return s[:-1]
else: return s

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %
(name)-2s %(levelname)-8s %(threadName)-12s %(message)s')
q5Logger = logging.getLogger('Q5')
logFileHandler = logging.handlers.RotatingFileHandler(filename='c:\
\logger\\q5.log', mode= 'a', maxBytes=100, backupCount=5)
logFileHandler.setFormatter(logging.Formatter('%(asctime)s %(name)-2s %
(levelname)-8s %(threadName)-12s %(message)s'))
logFileHandler.setLevel(logging.DEBUG)
q5Logger.addHandler(logFileHandler)

progOutput = subprocess.Popen(r'dir *.*', shell=True, bufsize=1000,
stdout=subprocess.PIPE).stdout
line = progOutput.readline()
while (line) != "":
q5Logger.info( chomp(line))
line = progOutput.readline()
rc = progOutput.close()

 

[Jean-Michel Pichavant]

The RotatingFileHandler running on win 7 64-bit; py 2.7 is failing
when the script launches a process using subprocess.Popen. Works fine
if the subprocess is not launched

The exception thrown
Traceback (most recent call last):
File "C:\Python27\lib\logging\handlers.py", line 78, in emit
self.doRollover()
File "C:\Python27\lib\logging\handlers.py", line 141, in doRollover
os.rename(self.baseFilename, dfn)
WindowsError: [Error 32] The process cannot access the file because it
is being used by another process

Anyone have an idea how to fix this?


import os, sys
import logging
import logging.handlers
import subprocess

def chomp(s):
"remove trailing carriage return"
if s[-1:]=='\n': return s[:-1]
else: return s

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %
(name)-2s %(levelname)-8s %(threadName)-12s %(message)s')
q5Logger = logging.getLogger('Q5')
logFileHandler = logging.handlers.RotatingFileHandler(filename='c:\
\logger\\q5.log', mode= 'a', maxBytes=100, backupCount=5)
logFileHandler.setFormatter(logging.Formatter('%(asctime)s %(name)-2s %
(levelname)-8s %(threadName)-12s %(message)s'))
logFileHandler.setLevel(logging.DEBUG)
q5Logger.addHandler(logFileHandler)

progOutput = subprocess.Popen(r'dir *.*', shell=True, bufsize=1000,
stdout=subprocess.PIPE).stdout
line = progOutput.readline()
while (line) != "":
q5Logger.info( chomp(line))
line = progOutput.readline()
rc = progOutput.close()

I don't think you can open a file from 2 different processes, hence the
error message. The only exception would be that one of them opens it in
read only mode which won't happen for log file.

You cannot fix it, it is the operand system that blocks the operation.
Using thread may allow you to log into the same file, but could bring a
bunch of other problems.

I know some ppl use a log server, your processes using a socket (client)
logger. You could do that on your local machine. There's a tutorial on
that in the logging documentation.

JM

 

[nac]

The RotatingFileHandler running on win 7 64-bit; py 2.7 is failing
when the script launches a process using subprocess.Popen. Works fine
if the subprocess is not launched

The exception thrown
Traceback (most recent call last):
  File "C:\Python27\lib\logging\handlers.py", line 78, in emit
    self.doRollover()
  File "C:\Python27\lib\logging\handlers.py", line 141, in doRollover
    os.rename(self.baseFilename, dfn)
WindowsError: [Error 32] The process cannot access the file because it
is being used by another process

Anyone have an idea how to fix this?

import os, sys
import logging
import logging.handlers
import subprocess

def chomp(s):
      "remove trailing carriage return"
      if s[-1:]=='\n': return s[:-1]
      else: return s

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %
(name)-2s %(levelname)-8s %(threadName)-12s %(message)s')
q5Logger = logging.getLogger('Q5')
logFileHandler = logging.handlers.RotatingFileHandler(filename='c:\
\logger\\q5.log', mode= 'a', maxBytes=100, backupCount=5)
logFileHandler.setFormatter(logging.Formatter('%(asctime)s %(name)-2s %
(levelname)-8s %(threadName)-12s %(message)s'))
logFileHandler.setLevel(logging.DEBUG)
q5Logger.addHandler(logFileHandler)

progOutput = subprocess.Popen(r'dir *.*', shell=True, bufsize=1000,
stdout=subprocess.PIPE).stdout
line = progOutput.readline()
while (line) != "":
      q5Logger.info( chomp(line))
      line = progOutput.readline()
rc = progOutput.close()

I don't think you can open a file from 2 different processes, hence the
error message. The only exception would be that one of them opens it in
read only mode which won't happen for log file.

You cannot fix it, it is the operand system that blocks the operation.
Using thread may allow you to log into the same file, but could bring a
bunch of other problems.

I know some ppl use a log server, your processes using a socket (client)
logger. You could do that on your local machine. There's a tutorial on
that in the logging documentation.

JM

Thanks for taking the time to look at the problem. In case you are
interested I was able to get this to work with the code at URL below.
Seemed to work because the subprocess I am running is not actually
using the log file but the OS make it inherit a handle to the log
file. The code below disables that inheritance. Do you see any
problems with the approach?

http://bugs.python.org/file14420/NTSafeLogging.py

 

[arun1]

Hi nac,

NTSafeLogging.py is working fine without any errors, but its not rotating
the log files as rotatingfilehandler does.
Do you have any working sample with NTSafeLogging which rotates the log
file.

logging issue with subprocess.Popen can be solved using this code

import threading
lock = threading.RLock()
def acquire_lock():
lock.acquire()

def release_lock():
lock.release()

call the acquire_lock() at the begining of method and release_lock() at
the end

 

[arun1]

Hi,

Actually NonInheritedRotatingFileHandler is rotating the log files but some
times it falis and showing I/O errors while the log file limit reaches the
given size.

Thanks

Arun