Wierd M2Crypto bug - phony "peer did not return certificate" error

Discussion in 'Python' started by John Nagle, Jan 10, 2007.

  1. John Nagle

    John Nagle Guest

    Here's a wierd problem:

    I have a little test case for M2Crypto, which just opens up SSL connections to
    web servers and reads their certificates. This works fine.

    But if I execute

    socket.setdefaulttimeout(5.0)

    so that the sockets don't wait too long if there's no SSL server, I get
    a "peer did not return certificate" exception every time.

    Environment:
    Windows 2000 SP 4
    M2Crypto: M2Crypto-0.17.win32-py2.4.exe
    Python: Python 2.4.4 (#71, Oct 18 2006, 08:34:43)
    OpenSSL: Win32 OpenSSL v0.9.8d

    Notes:
    Running M2Crypto does NOT use the version of OpenSSL that comes
    with Python. You have to install OpenSSL separately, or
    M2Crypto's DLL won't find the OpenSSL DLLS.

    There's a good chance that this may be related to:

    [ python-Bugs-1098618 ] socket.setdefaulttimeout() breaks smtplib.starttls()

    But that should have been fixed in Python 2.4.4, right? Or did that patch
    not make it in?

    John Nagle

    (And no, I can't run Python 2.5, because MySQLdb support doesn't work for
    2.5 yet.)
     
    John Nagle, Jan 10, 2007
    #1
    1. Advertising

  2. Re: Wierd M2Crypto bug - phony "peer did not return certificate"error

    John Nagle wrote:
    > I have a little test case for M2Crypto, which just opens up SSL
    > connections to
    > web servers and reads their certificates. This works fine.
    >
    > But if I execute
    >
    > socket.setdefaulttimeout(5.0)


    Yup, this is a known problem, this breaks all M2Crypto code that uses
    sockets. The Twisted wrapper part still works, as it hands the network
    activity to Twisted.

    If you can make do without setdefaulttimeout you should be fine.

    This is bug https://bugzilla.osafoundation.org/show_bug.cgi?id=2341

    The bug has a patch that works on Linux, but it would need Windows and
    Mac specific parts before it can be checked in.

    --
    Heikki Toivonen
     
    Heikki Toivonen, Jan 10, 2007
    #2
    1. Advertising

  3. John Nagle

    Guest

    That's a problem for me. I need short timeouts; I'm accessing sites
    that might or might not have SSL support, and I need to quickly time
    out when there's no SSL server.

    PyOpenSSL handles timeout correctly, but M2Crypto does not. On the
    other hand, M2Crypto actually checks certificates, which PyOpenSSL does
    not. So we have two broken implementations.

    Python needs a merge here. Read Guido van Rossum's rant on M2Crypto:

    http://www.artima.com/weblogs/viewpost.jsp?thread=95863

    Both of these packages contain wrappers for OpenSSL, but both wrappers
    are incompatible and buggy. M2Crypto also has some Python components.
    The PyOpenSSL wrapper, which ships with Python, doesn't expose enough
    of the OpenSSL API. M2Crypto exposes more of the API, but doesn't work
    as well.

    What's probably needed is to implement the additional API functions of
    the M2Crypto wrapper in the PyOpenSSL wrapper, so that the M2Crypto
    components written in Python could be used with it. Then we'd have one
    good implementation instead of two broken ones.

    John Nagle


    (News feed broken, using Google Groups as backup.)
     
    , Jan 11, 2007
    #3
  4. Re: Wierd M2Crypto bug - phony "peer did not return certificate"error

    wrote:
    > That's a problem for me. I need short timeouts; I'm accessing sites
    > that might or might not have SSL support, and I need to quickly time
    > out when there's no SSL server.


    You should be able to do short timeouts, just not using the global
    setdefaulttimeout. Have you tried Connection.set_socket_read/write_timeout?

    Also like I mentioned before, if you use the Twisted wrapper and let
    Twisted handle network stuff you should be fine.

    > Python needs a merge here. Read Guido van Rossum's rant on M2Crypto:
    >
    > http://www.artima.com/weblogs/viewpost.jsp?thread=95863


    That's old news, I believe I have fixed all of the issues mentioned
    there already in the 0.16 release.

    > What's probably needed is to implement the additional API functions of
    > the M2Crypto wrapper in the PyOpenSSL wrapper, so that the M2Crypto
    > components written in Python could be used with it. Then we'd have one
    > good implementation instead of two broken ones.


    M2Crypto and pyOpenSSL (and pyOpenSSL-extended, which you might want to
    take a look at as well) are implemented pretty differently, so merging
    seems unlikely. They are pretty small code wise, though, so it is not an
    impossible task.

    Personally I think I'd prefer if Python stdlib contained a better SSL
    module that did at least all the checks required for safe SSL
    connection. (Yeah, yeah, maybe I need to write it myself if nobody else
    gets to it;)

    --
    Heikki Toivonen
     
    Heikki Toivonen, Jan 11, 2007
    #4
  5. John Nagle

    John Nagle Guest

    Re: Wierd M2Crypto bug - phony "peer did not return certificate"error

    Heikki Toivonen wrote:
    > wrote:
    >
    >> That's a problem for me. I need short timeouts; I'm accessing sites
    >>that might or might not have SSL support, and I need to quickly time
    >>out when there's no SSL server.

    >
    >
    > You should be able to do short timeouts, just not using the global
    > setdefaulttimeout. Have you tried Connection.set_socket_read/write_timeout?


    Yes. That does not affect the connect timeout; it's only effective once
    the connection has been opened. And adjusting the session timeout
    just recreates the blocking/non blocking problem.

    Incidentally, "get_socket_read_timeout()" doesn't work. Generates
    "EXCEPTION at socket level: unpack str size does not match format",
    every time, at least with Python 2.4 on Windows. The lower level
    function returns one number as a string, like "7200", and the unpack
    function tries to unpack it as "ll", which fails.

    > Also like I mentioned before, if you use the Twisted wrapper and let
    > Twisted handle network stuff you should be fine.


    That would mean struggling with Twisted and dealing with its bugs.
    (For example, has the MySQLdb mess been resolved for Twisted's API?)
    This isn't a long-running application; it usually runs as a CGI program.
    So Twisted is inappropriate.

    > M2Crypto and pyOpenSSL (and pyOpenSSL-extended, which you might want to
    > take a look at as well) are implemented pretty differently, so merging
    > seems unlikely. They are pretty small code wise, though, so it is not an
    > impossible task.


    That seened good idea, and so I downloaded the source and tried to build
    it on a Windows machine to run with Python 2.4. But that build needs
    (exactly) Visual Studio 7.1, which I don't have. It also wants a specific
    version of OpenSSL, and has a build file which seems to assume a 1998 version
    of Visual C++. The last change was in 1995, and it's still at an 0.x version,
    so it's effectively abandonware. I have some doubts that it really works
    any more. I saw some Python 2.2/2.3 specific code in there.

    I could try building on a Linux system, but it's useful to me if Python
    works on both Windows and Linux.
    >
    > Personally I think I'd prefer if Python stdlib contained a better SSL
    > module that did at least all the checks required for safe SSL
    > connection. (Yeah, yeah, maybe I need to write it myself if nobody else
    > gets to it;)


    What I need is proper SSL operation, certificate chain validation,
    useful exception info when a connection fails (including why), and
    read acess to the certificate itself in some reasonably sane form.
    And I need to time out an SSL connection if it can't connect and
    verify within 4 seconds. It's embarassing that this doesn't work,
    despite four different wrappers for OpenSSL.

    I'm willing to spend a few hundred dollars towards making this happen.

    John Nagle
     
    John Nagle, Jan 12, 2007
    #5
  6. John Nagle

    John Nagle Guest

    Re: Wierd M2Crypto bug - phony "peer did not return certificate"error

    OK, I have a one-line fix.

    To Connection.py of M2Crypto:

    def connect(self, addr):
    self.socket.connect(addr)
    self.addr = addr
    self.socket.settimeout(None) # Back to normal timeout (NEW)
    self.setup_ssl()
    self.set_connect_state()
    ret = self.connect_ssl()
    check = getattr(self, 'postConnectionCheck', self.clientPostConnectionCheck)
    if check is not None:
    if not check(self.get_peer_cert(), self.addr[0]):
    raise Checker.SSLVerificationError, 'post connection check failed'
    return ret


    After the socket is connected, we turn its timeout off.

    The effect is that if the caller sets

    socket.setdefaulttimeout(timeoutsecs)

    before creating the Connection object, that will set the connection
    timeout. The socket will be created as non-blocking, but before
    any reads or writes are done, we clear the timeout, making it blocking
    again. So the connection will time out as requested, but we won't
    get errors because the socket is non-blocking and is being used by
    code that expects it to block.

    Gets usable connection timeouts, and prevents bogus "peer did not
    return certificate" errors.

    John Nagle
    Animats
     
    John Nagle, Jan 12, 2007
    #6
  7. Re: Wierd M2Crypto bug - phony "peer did not return certificate"error

    John Nagle wrote:
    > def connect(self, addr):
    > self.socket.connect(addr)
    > self.addr = addr
    > self.socket.settimeout(None) # Back to normal timeout (NEW)


    I am not sure if this is always appropriate. In fact, doesn't this just
    eliminate the timeout completely from this connection?

    In your case you could achieve what you want by calling
    connection_object.socket.settimeout(4.0) method just before calling
    connect, without needing to edit M2Crypto itself.

    setdefaulttimeout, socket.settimeout and socket.setblocking are pretty
    confusing...

    --
    Heikki Toivonen
     
    Heikki Toivonen, Jan 13, 2007
    #7
  8. John Nagle

    John Nagle Guest

    Re: Wierd M2Crypto bug - phony "peer did not return certificate"error

    Heikki Toivonen wrote:
    > John Nagle wrote:
    >
    >>def connect(self, addr):
    >> self.socket.connect(addr)
    >> self.addr = addr
    >> self.socket.settimeout(None) # Back to normal timeout (NEW)

    >
    >
    > I am not sure if this is always appropriate. In fact, doesn't this just
    > eliminate the timeout completely from this connection?
    >
    > In your case you could achieve what you want by calling
    > connection_object.socket.settimeout(4.0) method just before calling
    > connect, without needing to edit M2Crypto itself.


    No, this actually works right. It seems that the default timeout
    controls the timeout on TCP connect, but puts the socket into
    non-blocking mode. So, if as soon as a TCP connection is
    opened, but before the SSL handshake starts, the socket timeout
    is set to None, then the socket returns to blocking mode before
    its first read.

    At least on Windows. More later on how this works on Linux.

    John Nagle
     
    John Nagle, Jan 13, 2007
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Buddy Ackerman

    creating Phony links

    Buddy Ackerman, Mar 9, 2005, in forum: ASP .Net
    Replies:
    2
    Views:
    444
    Buddy Ackerman
    Mar 9, 2005
  2. Sean
    Replies:
    8
    Views:
    964
  3. Jean-Paul Calderone
    Replies:
    0
    Views:
    372
    Jean-Paul Calderone
    Jan 19, 2007
  4. John Nagle

    M2Crypto possible intermittent bug

    John Nagle, Oct 10, 2007, in forum: Python
    Replies:
    5
    Views:
    331
    John Nagle
    Oct 12, 2007
  5. medicalfraud
    Replies:
    0
    Views:
    533
    medicalfraud
    Apr 12, 2009
Loading...

Share This Page