with statement and context managers

Discussion in 'Python' started by Steven D'Aprano, Aug 3, 2011.

  1. I'm not greatly experienced with context managers and the with statement, so
    I would like to check my logic.

    Somebody (doesn't matter who, or where) stated that they frequently use this
    idiom:

    spam = MyContextManager(*args)
    for ham in my_iter:
    with spam:
    # do stuff


    but to me that looks badly wrong. Surely the spam context manager object
    will exit after the first iteration, and always raise an exception on the
    second? But I don't quite understand context managers enough to be sure.


    I've tested it with two examples:

    # Simple example using built-in file context manager.

    >>> spam = open('aaa')
    >>> for ham in range(5):

    .... with spam:
    .... print ham
    ....
    0
    Traceback (most recent call last):
    File "<stdin>", line 2, in <module>
    ValueError: I/O operation on closed file


    # Slightly more complex example.

    >>> from contextlib import closing
    >>> import urllib
    >>> spam = closing(urllib.urlopen('http://www.python.org'))
    >>> for ham in range(5):

    .... with spam as page:
    .... print ham, sum(len(line) for line in page)
    ....
    0 18486
    1
    Traceback (most recent call last):
    File "<stdin>", line 3, in <module>
    File "<stdin>", line 3, in <genexpr>
    File "/usr/local/lib/python2.7/socket.py", line 528, in next
    line = self.readline()
    File "/usr/local/lib/python2.7/socket.py", line 424, in readline
    recv = self._sock.recv
    AttributeError: 'NoneType' object has no attribute 'recv'




    Am I right to expect that the above idiom cannot work? If not, what sort of
    context managers do work as shown?




    --
    Steven
    Steven D'Aprano, Aug 3, 2011
    #1
    1. Advertising

  2. On Tue, Aug 2, 2011 at 10:15 PM, Steven D'Aprano
    <> wrote:
    > I'm not greatly experienced with context managers and the with statement,so
    > I would like to check my logic.
    >
    > Somebody (doesn't matter who, or where) stated that they frequently use this
    > idiom:
    >
    > spam = MyContextManager(*args)
    > for ham in my_iter:
    >    with spam:
    >         # do stuff
    >

    [snip]
    > # Simple example using built-in file context manager.
    >
    >>>> spam = open('aaa')
    >>>> for ham in range(5):

    > ...     with spam:
    > ...             print ham
    > ...
    > 0
    > Traceback (most recent call last):
    >  File "<stdin>", line 2, in <module>
    > ValueError: I/O operation on closed file


    file_context = lambda: open('aaa')
    for i in range(3):
    with file_context():
    print "hello"

    ... but if the context is short it is clearer and time saving to _not_
    alias it. If the context is sufficiently complicated then it is worth
    it to make the complex code into a first class context manager -
    contextlib.contextmanager makes this very easy and extremely readable.

    -Jack
    Jack Diederich, Aug 3, 2011
    #2
    1. Advertising

  3. Steven D'Aprano

    Nobody Guest

    On Wed, 03 Aug 2011 12:15:44 +1000, Steven D'Aprano wrote:

    > I'm not greatly experienced with context managers and the with statement, so
    > I would like to check my logic.
    >
    > Somebody (doesn't matter who, or where) stated that they frequently use this
    > idiom:
    >
    > spam = MyContextManager(*args)
    > for ham in my_iter:
    > with spam:
    > # do stuff
    >
    >
    > but to me that looks badly wrong. Surely the spam context manager object
    > will exit after the first iteration, and always raise an exception on the
    > second? But I don't quite understand context managers enough to be sure.


    It depends upon the implementation of MyContextManager. If it's
    implemented using the contextlib.contextmanager decorator, then you're
    correct: you can only use it once. OTOH, if you implement your own class
    with __enter__ and __exit__ methods, you can use the same context manager
    object multiple times.
    Nobody, Aug 3, 2011
    #3
  4. Am 03.08.2011 04:15 schrieb Steven D'Aprano:

    > I'm not greatly experienced with context managers and the with
    > statement, so I would like to check my logic.
    >
    > Somebody (doesn't matter who, or where) stated that they frequently
    > use this idiom:
    >
    > spam = MyContextManager(*args)
    > for ham in my_iter:
    > with spam:
    > # do stuff
    >
    >
    > but to me that looks badly wrong. Surely the spam context manager
    > object will exit after the first iteration, and always raise an
    > exception on the second? But I don't quite understand context
    > managers enough to be sure.


    Depends on the implementation. As already stated, a
    contextlib.contextmanager will only run once. But it is easy to turn it
    into a persistent one - which internally initializes as often as needed.

    class PersistentCM(object):
    def __init__(self, target):
    self.target = target
    self.current = []
    def __enter__(self):
    cm = self.target()
    cm.__enter__()
    self.current.append(cm)
    def __exit__(self, *e):
    return self.current.pop(-1).__exit__(*e)

    (untested)

    This would allow for a CM to be used multiple times. It internally
    generates a new one on every __enter__() call and disposes of it on
    every __exit__(). As __enter__() and __exit__() are supposed to be used
    symmetrical, it should(!) work this way.

    Many CMs don't allow your shown use, though, partly due to obvious reasons:

    * The ...ing named ones do the action they are named after in the
    __exit__() part and do nothing in the __enter__() part. (E.g. closing.)

    * The ...ed named ones do the named action in __enter__() and undo it in
    __exit__(). They may or may not work multiple times.

    For example, take threading.Lock et al., which you can have locked
    several times.


    >>>> spam = open('aaa')
    >>>> for ham in range(5):

    > ... with spam:
    > ... print ham
    > ...
    > 0
    > Traceback (most recent call last):
    > File "<stdin>", line 2, in<module>
    > ValueError: I/O operation on closed file


    To be honest, this one I don't understand as well. __exit__() closes the
    file spam, but even before "print ham" can be executed for a second
    time, there is a check if the file is still open. What does __enter__()
    do here? Just check if it is closed, or is it more? Nevertheless, it
    makes sense.


    > # Slightly more complex example.
    >
    >>>> from contextlib import closing
    >>>> import urllib
    >>>> spam = closing(urllib.urlopen('http://www.python.org'))
    >>>> for ham in range(5):

    > ... with spam as page:
    > ... print ham, sum(len(line) for line in page)
    > ...
    > 0 18486
    > 1
    > Traceback (most recent call last):
    > File "<stdin>", line 3, in<module>
    > File "<stdin>", line 3, in<genexpr>
    > File "/usr/local/lib/python2.7/socket.py", line 528, in next
    > line = self.readline()
    > File "/usr/local/lib/python2.7/socket.py", line 424, in readline
    > recv = self._sock.recv
    > AttributeError: 'NoneType' object has no attribute 'recv'


    Here the above said applies: after closing happens in __exit__() and
    _sock gets set to None, the object is not usable any longer.


    > Am I right to expect that the above idiom cannot work? If not, what sort of
    > context managers do work as shown?


    As said, take threading.Lock as an example: they can be acquire()d and
    release()d as often as you want, and these actions happen in __enter__()
    and __exit__(), respectively.


    HTH,

    Thomas
    Thomas Rachel, Aug 3, 2011
    #4
  5. Thomas Rachel wrote:

    > Am 03.08.2011 04:15 schrieb Steven D'Aprano:

    [...]
    > > but to me that looks badly wrong. Surely the spam context manager
    > > object will exit after the first iteration, and always raise an
    > > exception on the second? But I don't quite understand context
    > > managers enough to be sure.

    >
    > Depends on the implementation. As already stated, a
    > contextlib.contextmanager will only run once. But it is easy to turn it
    > into a persistent one - which internally initializes as often as needed.


    Thanks, that's exactly the sort of information I was after.

    Thank you to everyone who answered.


    --
    Steven
    Steven D'Aprano, Aug 5, 2011
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    3
    Views:
    278
    Bjoern Schliessmann
    Jan 12, 2007
  2. Terry Reedy
    Replies:
    1
    Views:
    222
    alex23
    May 5, 2010
  3. Chris Withers

    bug? context managers vs ImportErrors

    Chris Withers, Aug 19, 2010, in forum: Python
    Replies:
    1
    Views:
    232
    Steven D'Aprano
    Aug 19, 2010
  4. Ethan Furman
    Replies:
    2
    Views:
    270
    Ethan Furman
    Aug 19, 2010
  5. Jason
    Replies:
    0
    Views:
    232
    Jason
    Nov 30, 2010
Loading...

Share This Page