unpickling derived LogRecord in python 2.7 from python2.6

[ivdneut]

Hello all,

I have a service that runs in python 2.6.4. This service sends
LogRecords to a log monitoring app on my workstation running python
2.7. The LogRecord class is derived:

class LogRecord(logging.LogRecord):

def __init__(self, name, level, fn, lno, user, hostname, msg,
args, exc_info, func=None):

if sys.version_info[1] > 4:
logging.LogRecord.__init__(self, name, level, fn, lno,
msg, args, exc_info, func)
else:
logging.LogRecord.__init__(self, name, level, fn, lno,
msg, args, exc_info)

Now when I try to unpickle it:

record = cPickle.loads(data)

I get a TypeError exception:

TypeError: ('__init__() takes at least 8 arguments (1 given)', <class
'...gRecord'>, ())

I've searched the web and this group, but most results are old. It
worked when my workstation still ran python 2.6.

Thank you,

Ian.

 

[Peter Otten]

I have a service that runs in python 2.6.4. This service sends
LogRecords to a log monitoring app on my workstation running python
2.7. The LogRecord class is derived:

class LogRecord(logging.LogRecord):

def __init__(self, name, level, fn, lno, user, hostname, msg,
args, exc_info, func=None):

if sys.version_info[1] > 4:
logging.LogRecord.__init__(self, name, level, fn, lno,
msg, args, exc_info, func)
else:
logging.LogRecord.__init__(self, name, level, fn, lno,
msg, args, exc_info)

Now when I try to unpickle it:

record = cPickle.loads(data)

I get a TypeError exception:

TypeError: ('__init__() takes at least 8 arguments (1 given)', <class
'...gRecord'>, ())

I've searched the web and this group, but most results are old. It
worked when my workstation still ran python 2.6.

The Problem is that as of Python 2.7 logging.LogRecord has become a newstyle
class which is pickled/unpickled differently. I don't know if there is an
official way to do the conversion, but here's what I've hacked up.
The script can read pickles written with 2.6 in 2.7, but not the other way
round.

$ cat pickle_logrec.py
import sys
import pickle
import logging

class LogRecord(logging.LogRecord):

def __init__(self, name, level, fn, lno, user, hostname, msg, args,
exc_info, func=None):

if sys.version_info[1] > 4:
logging.LogRecord.__init__(self, name, level, fn, lno, msg,
args, exc_info, func)
else:
logging.LogRecord.__init__(self, name, level, fn, lno, msg,
args, exc_info)

def makeLogRecord():
return LogRecord(*[None]*9)

if issubclass(LogRecord, object):
print "LogRecord is a newstyle class"
class MyUnpickler(pickle.Unpickler):
def find_class(self, *args):
klass = pickle.Unpickler.find_class(self, *args)
if klass is LogRecord:
return makeLogRecord
return klass
else:
print "LogRecord is an oldstyle class"
MyUnpickler = pickle.Unpickler

if __name__ == "__main__":
if "--load" in sys.argv:
print "loading"
with open("tmp.pickle") as f:
restored = MyUnpickler(f).load()
print restored
else:
print "dumping"
with open("tmp.pickle", "w") as f:
f.write(pickle.dumps(LogRecord("yadda", *[None]*8)))


$ python2.6 pickle_logrec.py
LogRecord is an oldstyle class
dumping
$ python2.6 pickle_logrec.py --load
LogRecord is an oldstyle class
loading
<LogRecord: yadda, None, None, None, "None">
$ python2.7 pickle_logrec.py --load
LogRecord is a newstyle class
loading
<LogRecord: yadda, None, None, None, "None">

No warranty, use at your own risk.

 

[Vinay Sajip]

The Problem is that as of Python 2.7logging.LogRecord has become a newstyle
class which is pickled/unpickled differently. I don't know if there is an
official way to do the conversion, but here's what I've hacked up.
The script can read pickles written with 2.6 in 2.7, but not the other way
round.
[code snipped]

I don't know about "official", but another way of doing this is to
pickle just the LogRecord's __dict__ and send that over the wire. The
logging package contains a function makeLogRecord(d) where d is a
dict.

This is the approach used by the examples in the library documentation
which pickle events for sending across a network:

http://docs.python.org/howto/loggin...and-receiving-logging-events-across-a-network

The built-in SocketHandler pickles the LogRecord's __dict__ rather
than the LogRecord itself, precisely because of the improved
interoperability over pickling the instance directly.

Regards,

Vinay Sajip

 

[Peter Otten]

The Problem is that as of Python 2.7logging.LogRecord has become a
newstyle class which is pickled/unpickled differently. I don't know if
there is an official way to do the conversion, but here's what I've
hacked up. The script can read pickles written with 2.6 in 2.7, but not
the other way round.
[code snipped]

I don't know about "official", but another way of doing this is to
pickle just the LogRecord's __dict__ and send that over the wire. The
logging package contains a function makeLogRecord(d) where d is a
dict.

You are right, my approach is too complicated and only needed when the OP
cannot modify the sending script -- which is unlikely.

This is the approach used by the examples in the library documentation
which pickle events for sending across a network:

http://docs.python.org/howto/logging-cookbook.html#sending-and-receiving- logging-events-across-a-network

The built-in SocketHandler pickles the LogRecord's __dict__ rather
than the LogRecord itself, precisely because of the improved
interoperability over pickling the instance directly.

As a minimal change ensuring that the logging.LogRecord subclass used by the
OP is a newstyle class in 2.6 with

class LogRecord(logging.LogRecord, object):
#...

should work, too.

 

[ivdneut]

The Problem is that as of Python 2.7logging.LogRecord has become a
newstyle class which is pickled/unpickled differently. I don't know if
there is an official way to do the conversion, but here's what I've
hacked up. The script can read pickles written with 2.6 in 2.7, but not
the other way round.
[code snipped]

I don't know about "official", but another way of doing this is to
pickle just the LogRecord's __dict__ and send that over the wire. The
logging package contains a function makeLogRecord(d) where d is a
dict.

You are right, my approach is too complicated and only needed when the OP
cannot modify the sending script -- which is unlikely.

This is the approach used by the examples in the library documentation
which pickle events for sending across a network:

logging-events-across-a-network



The built-in SocketHandler pickles the LogRecord's __dict__ rather
than the LogRecord itself, precisely because of the improved
interoperability over pickling the instance directly.

As a minimal change ensuring that the logging.LogRecord subclass used by the
OP is a newstyle class in 2.6 with

class LogRecord(logging.LogRecord, object):
    #...

should work, too.

I tried this, but it didn't work. Pickling the __dict__ and then use
makeLogRecord does the trick.

Thank you very much for the excellent help,

Ian.