handlers.SocketHandler and exceptions

Discussion in 'Python' started by writeson, Jan 16, 2008.

  1. writeson

    writeson Guest

    Hi all,

    On our Linux systems at work I've written a Twisted logging server
    that receives log messages from multiple servers/processes to post
    them to a log file, essentially serializing all the process log
    messages. This works well, that is until I tried this test code:

    try:
    t = 10 / 0
    except Exception, e:
    log.exception("divide by zero")

    where log is the logger instance retreived from a call to getLogger().
    The problem is the handlers.SocketHandler tries to cPickle.dump() the
    log record, which in this case contains an exc_info tuple, the last
    item of which is a Traceback object. The pickling fails with an
    "unpickleable error" and that's that.

    Does anyone have any ideas how to handle this situation? I'd hate to
    have to give up using the log.exception(...) call as it's useful to
    get strack trace information in the log file.

    Thanks in advance,
    Doug Farrell
     
    writeson, Jan 16, 2008
    #1
    1. Advertising

  2. writeson

    Mark Tolonen Guest

    "writeson" <> wrote in message
    news:...
    > Hi all,
    >
    > On our Linux systems at work I've written a Twisted logging server
    > that receives log messages from multiple servers/processes to post
    > them to a log file, essentially serializing all the process log
    > messages. This works well, that is until I tried this test code:
    >
    > try:
    > t = 10 / 0
    > except Exception, e:
    > log.exception("divide by zero")
    >
    > where log is the logger instance retreived from a call to getLogger().
    > The problem is the handlers.SocketHandler tries to cPickle.dump() the
    > log record, which in this case contains an exc_info tuple, the last
    > item of which is a Traceback object. The pickling fails with an
    > "unpickleable error" and that's that.
    >
    > Does anyone have any ideas how to handle this situation? I'd hate to
    > have to give up using the log.exception(...) call as it's useful to
    > get strack trace information in the log file.
    >
    > Thanks in advance,
    > Doug Farrell


    Check out the traceback module. It can translate the traceback into a
    variety of formats (such as a string) that can be pickled.

    --Mark
     
    Mark Tolonen, Jan 17, 2008
    #2
    1. Advertising

  3. writeson

    writeson Guest

    Mark,

    >
    > Check out the traceback module. It can translate the traceback into a
    > variety of formats (such as a string) that can be pickled.
    >
    > --Mark


    Thanks for the reply. I was looking at the traceback module and
    thinking along the same lines you are. The problem I'm having with
    that is how to modify the behavior of the SocketHandler code so it
    would call the traceback module functions. The point at which the
    handlers.SocketHandler code fails is in the method makePickle(), and
    I'm not sure how to overload/override that method. I tried creating my
    own class:

    class MySocketHandler(handlers.SocketHandler):
    def makePickle(self, record):
    # perform new code that transforms a Traceback object into a
    string

    but so far I haven't figured out how to get the logging module to use
    my class. In my logging configuration file I tried something like
    this:

    [handler_local_server]
    class=mydirectory.MySocketHandler
    level=DEBUG
    formatter=general
    args=("localhost", handlers.DEFAULT_TCP_LOGGING_PORT + 1)

    but I can't seem to get the logging module to include mydirectory in
    its search path for modules.

    So that's where I'm stuck now.

    Again, thanks for your response,
    Doug
     
    writeson, Jan 17, 2008
    #3
  4. writeson

    Vinay Sajip Guest

    On Jan 17, 1:47 pm, writeson <> wrote:
    > Mark,
    >
    >
    >
    > > Check out the traceback module. It can translate the traceback into a
    > > variety of formats (such as a string) that can be pickled.

    >
    > > --Mark

    >
    > Thanks for the reply. I was looking at the traceback module and
    > thinking along the same lines you are. The problem I'm having with
    > that is how to modify the behavior of the SocketHandler code so it
    > would call the traceback module functions. The point at which the
    > handlers.SocketHandler code fails is in the method makePickle(), and
    > I'm not sure how to overload/override that method. I tried creating my
    > own class:
    >
    > class MySocketHandler(handlers.SocketHandler):
    > def makePickle(self, record):
    > # perform new code that transforms a Traceback object into a
    > string
    >
    > but so far I haven't figured out how to get theloggingmodule to use
    > my class. In myloggingconfiguration file I tried something like
    > this:
    >
    > [handler_local_server]
    > class=mydirectory.MySocketHandler
    > level=DEBUG
    > formatter=general
    > args=("localhost", handlers.DEFAULT_TCP_LOGGING_PORT + 1)
    >
    > but I can't seem to get theloggingmodule to include mydirectory in
    > its search path for modules.
    >
    > So that's where I'm stuck now.
    >
    > Again, thanks for your response,
    > Doug


    What version of Python are you running? Recent versions should have a
    fix for this. Here's the makePickle method of SocketHandler:

    def makePickle(self, record):
    """
    Pickles the record in binary format with a length prefix, and
    returns it ready for transmission across the socket.
    """
    ei = record.exc_info
    if ei:
    dummy = self.format(record) # just to get traceback text
    into record.exc_text
    record.exc_info = None # to avoid Unpickleable error
    s = cPickle.dumps(record.__dict__, 1)
    if ei:
    record.exc_info = ei # for next handler
    slen = struct.pack(">L", len(s))
    return slen + s

    Notice the code to avoid the Unpickleable error.

    Best regards,

    Vinay Sajip
     
    Vinay Sajip, Jan 17, 2008
    #4
  5. writeson

    writeson Guest

    Vinay,

    Thanks for your reply, very interesting. We're currently running
    Python2.3 (though we are getting ready to move to Python2.5), so I'm
    guessing the code you're showing comes from Python2.5? I'm wondering
    if I can edit the handlers.py code in my Python2.3 installation, make
    the changes you show above, and have things work? Any thoughts on
    this?

    Thanks for the help!!
    Doug
     
    writeson, Jan 17, 2008
    #5
  6. writeson

    Vinay Sajip Guest

    On Jan 17, 5:50 pm, writeson <> wrote:
    > Vinay,
    >
    > Thanks for your reply, very interesting. We're currently running
    > Python2.3 (though we are getting ready to move to Python2.5), so I'm
    > guessing the code you're showing comes from Python2.5? I'm wondering
    > if I can edit the handlers.py code in my Python2.3 installation, make
    > the changes you show above, and have things work? Any thoughts on
    > this?
    >
    > Thanks for the help!!
    > Doug


    It shouldn't be a problem if you edit your copy [make a backup first,
    of course :)] , but make sure you also edit __init__.py which has the
    part that caches the exception text in record.exc_text. Just search
    for exc_text in the logging code to find all the places to change.

    Best regards,


    Vinay
     
    Vinay Sajip, Jan 17, 2008
    #6
  7. writeson

    writeson Guest

    On Jan 17, 2:45 pm, Vinay Sajip <> wrote:

    Vinay,

    Again, thanks for your very timely help! I was just editing the
    handlers.py code, and didn't really understand how that was going to
    work, and of course it didn't. I was just about to write to you again,
    and voila, you'd already responded with what I needed to know. I would
    have been floundering around for quite awhile before I'd have found
    (if ever) the change you mentioned to __init__.py. I made the changes
    and my logging server is working as I expected! Exceptions are being
    placed in the log file, complete with their tracebacks.

    Again, thanks very much for your help, greatly appreciated!
    Doug
     
    writeson, Jan 17, 2008
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ahmed Moustafa
    Replies:
    5
    Views:
    30,123
    Chris Smith
    Jul 14, 2004
  2. Paul Miller
    Replies:
    3
    Views:
    1,064
    Alex Martelli
    Nov 12, 2003
  3. oj
    Replies:
    9
    Views:
    1,090
  4. Vinay Sajip
    Replies:
    1
    Views:
    383
    Vinay Sajip
    Jul 15, 2008
  5. writeson

    logging.handlers.SocketHandler

    writeson, Mar 5, 2009, in forum: Python
    Replies:
    5
    Views:
    496
    Vinay Sajip
    Mar 9, 2009
Loading...

Share This Page