Re: threading support in python

Discussion in 'Python' started by Jean-Paul Calderone, Sep 6, 2006.

  1. On 05 Sep 2006 17:31:11 -0700, Paul Rubin <"http://phr.cx"@nospam.invalid> wrote:
    >
    > def func(filename):
    > with open(filename) as f:
    > do_something_with(f)
    > # f definitely gets closed when the "with" block exits
    >
    >which more explicitly shows the semantics actually desired. Not that
    >"huge" a benefit as far as I can tell. Lisp programmers have gotten
    >along fine without it for 40+ years...


    Uh yea. No lisp programmer has ever written a with-* function... ever.

    Jean-Paul
    Jean-Paul Calderone, Sep 6, 2006
    #1
    1. Advertising

  2. Jean-Paul Calderone

    Paul Rubin Guest

    Jean-Paul Calderone <> writes:
    > >which more explicitly shows the semantics actually desired. Not that
    > >"huge" a benefit as far as I can tell. Lisp programmers have gotten
    > >along fine without it for 40+ years...

    >
    > Uh yea. No lisp programmer has ever written a with-* function... ever.


    The context was Lisp programmers have gotten along fine without
    counting on the refcounting GC semantics that sjdevnull advocates
    Python stay with. GC is supposed to make it look like every object
    stays around forever, and any finalizer that causes an explicit
    internal state change in an extant object (like closing a file or
    socket) is not in the GC spirit to begin with.
    Paul Rubin, Sep 6, 2006
    #2
    1. Advertising

  3. Jean-Paul Calderone

    Guest

    Paul Rubin wrote:
    > Jean-Paul Calderone <> writes:
    > > >which more explicitly shows the semantics actually desired. Not that
    > > >"huge" a benefit as far as I can tell. Lisp programmers have gotten
    > > >along fine without it for 40+ years...

    > >
    > > Uh yea. No lisp programmer has ever written a with-* function... ever.

    >
    > The context was Lisp programmers have gotten along fine without
    > counting on the refcounting GC semantics that sjdevnull advocates
    > Python stay with. GC is supposed to make it look like every object
    > stays around forever, and any finalizer that causes an explicit
    > internal state change in an extant object (like closing a file or
    > socket) is not in the GC spirit to begin with.


    I disagree, strongly. If you want "every object stays around forever"
    semantics, you can just not free anything. GC is actually supposed to
    free things that are unreachable at least when memory becomes tight,
    and nearly every useful garbage collected language allows destructors
    that could have effects visible to the rest of the program. Reference
    counting allows more deterministic semantics that can eliminate
    repeating scope information multiple times.
    , Sep 6, 2006
    #3
  4. Jean-Paul Calderone

    Steve Holden Guest

    wrote:
    > Paul Rubin wrote:
    >
    >>Jean-Paul Calderone <> writes:
    >>
    >>>>which more explicitly shows the semantics actually desired. Not that
    >>>>"huge" a benefit as far as I can tell. Lisp programmers have gotten
    >>>>along fine without it for 40+ years...
    >>>
    >>>Uh yea. No lisp programmer has ever written a with-* function... ever.

    >>
    >>The context was Lisp programmers have gotten along fine without
    >>counting on the refcounting GC semantics that sjdevnull advocates
    >>Python stay with. GC is supposed to make it look like every object
    >>stays around forever, and any finalizer that causes an explicit
    >>internal state change in an extant object (like closing a file or
    >>socket) is not in the GC spirit to begin with.

    >
    >
    > I disagree, strongly. If you want "every object stays around forever"
    > semantics, you can just not free anything. GC is actually supposed to
    > free things that are unreachable at least when memory becomes tight,
    > and nearly every useful garbage collected language allows destructors
    > that could have effects visible to the rest of the program. Reference
    > counting allows more deterministic semantics that can eliminate
    > repeating scope information multiple times.
    >

    Clearly you guys are determined to disagree. It seemed obvious to me
    that Paul's reference to making it "look like every object stays around
    forever" doesn't exclude their being garbage-collected once the program
    no longer contains any reference to them.

    You simplify the problems involved with GC-triggered destructors to the
    point of triviality. There are exceedingly subtle and difficult issues
    here: read some of the posts to the python-dev list about such issues
    and then see if you still feel the same way.

    regards
    Steve
    --
    Steve Holden +44 150 684 7255 +1 800 494 3119
    Holden Web LLC/Ltd http://www.holdenweb.com
    Skype: holdenweb http://holdenweb.blogspot.com
    Recent Ramblings http://del.icio.us/steve.holden
    Steve Holden, Sep 6, 2006
    #4
  5. Jean-Paul Calderone

    Guest

    Steve Holden wrote:
    > wrote:
    > > Paul Rubin wrote:
    > >
    > >>Jean-Paul Calderone <> writes:
    > >>
    > >>>>which more explicitly shows the semantics actually desired. Not that
    > >>>>"huge" a benefit as far as I can tell. Lisp programmers have gotten
    > >>>>along fine without it for 40+ years...
    > >>>
    > >>>Uh yea. No lisp programmer has ever written a with-* function... ever.
    > >>
    > >>The context was Lisp programmers have gotten along fine without
    > >>counting on the refcounting GC semantics that sjdevnull advocates
    > >>Python stay with. GC is supposed to make it look like every object
    > >>stays around forever, and any finalizer that causes an explicit
    > >>internal state change in an extant object (like closing a file or
    > >>socket) is not in the GC spirit to begin with.

    > >
    > >
    > > I disagree, strongly. If you want "every object stays around forever"
    > > semantics, you can just not free anything. GC is actually supposed to
    > > free things that are unreachable at least when memory becomes tight,
    > > and nearly every useful garbage collected language allows destructors
    > > that could have effects visible to the rest of the program. Reference
    > > counting allows more deterministic semantics that can eliminate
    > > repeating scope information multiple times.
    > >

    > Clearly you guys are determined to disagree. It seemed obvious to me
    > that Paul's reference to making it "look like every object stays around
    > forever" doesn't exclude their being garbage-collected once the program
    > no longer contains any reference to them.
    >
    > You simplify the problems involved with GC-triggered destructors to the
    > point of triviality. There are exceedingly subtle and difficult issues
    > here: read some of the posts to the python-dev list about such issues


    No doubt that it's hard. On the other hand, current CPython captures
    programmer-friendly behavior quite well. My main assertions are that:
    1. Saying that GC is just freeing memory after it won't be referenced
    anymore is disingenuous; it is _already_ common practice in Python (and
    other languages) for destructors to close files, sockets, and otherwise
    deallocate non-memory resources.
    2. The ref-counting semantics are extremely valuable to the programmer.
    Throwing them out without careful consideration is a bad
    idea--ref-counting is _not_ simply one GC implementation among many, it
    actually offers useful semantics and the cost of giving up those
    semantics should be considered before throwing out refcounting.

    I'm actually willing to be convinced on (2); I think that what
    ref-counting offers is a massive improvement over nondeterministic GC,
    and it seems that refcounting has historically been supportable, but if
    there are real tangible benefits to python programmers from eliminating
    it that outweigh the niceties of deterministic GC, then I'd be okay
    with sacrificing it. It just seems like people are very cavalier about
    giving up something that is a very nice feature in order to make other
    implementations simpler.

    (1) I think is here to stay, if you're going to tell programmers that
    their destructors can't make program-visible changes (e.g. closing the
    database connection when a dbconn is destroyed), that's a _huge_ change
    from current practice that needs serious debate.
    , Sep 6, 2006
    #5
  6. Jean-Paul Calderone

    Paul Rubin Guest

    "" <> writes:
    > Throwing them out without careful consideration is a bad
    >idea--ref-counting is _not_ simply one GC implementation among many, it
    >actually offers useful semantics and the cost of giving up those
    >semantics should be considered before throwing out refcounting.


    It's too late to consider anything before throwing out refcounting.
    Refcounting has already been thrown out (in Jython, IronPython, and
    maybe PyPy). It's just an implementation artifact of CPython and MANY
    other language implementations have gotten along perfectly well
    without it.

    > (1) I think is here to stay, if you're going to tell programmers that
    > their destructors can't make program-visible changes (e.g. closing the
    > database connection when a dbconn is destroyed), that's a _huge_ change
    > from current practice that needs serious debate.


    We had that debate already (PEP 343). Yes, there is some sloppy
    current practice by CPython users that relies on the GC to close the
    db conn. That practice already fails in several other Python
    implementations and with PEP 343, we now have a clean way to fix it.
    I don't understand why you're so fixated on keeping the sloppy method
    around. The benefit is marginal at best. If you want stack-like
    deallocation of something, ask for it explicitly.
    Paul Rubin, Sep 6, 2006
    #6
  7. Jean-Paul Calderone

    Guest

    Paul Rubin wrote:
    > "" <> writes:
    > > (1) I think is here to stay, if you're going to tell programmers that
    > > their destructors can't make program-visible changes (e.g. closing the
    > > database connection when a dbconn is destroyed), that's a _huge_ change
    > > from current practice that needs serious debate.

    >
    > We had that debate already (PEP 343). Yes, there is some sloppy
    > current practice by CPython users that relies on the GC to close the
    > db conn.


    This point is unrelated to with or ref-counting. Even the standard
    library will close file objects when they are GC'd. If this is not
    acceptable, it's a major change; that's why I say (1) is here to stay.
    But I think we're misunderstanding each other somehow on this point (I
    don't think you're saying that the standard library is sloppily coded
    in this regard), I just don't know how.
    , Sep 6, 2006
    #7
  8. Jean-Paul Calderone

    Paul Rubin Guest

    "" <> writes:
    > > We had that debate already (PEP 343). Yes, there is some sloppy
    > > current practice by CPython users that relies on the GC to close the
    > > db conn.

    >
    > This point is unrelated to with or ref-counting. Even the standard
    > library will close file objects when they are GC'd.


    I don't think so. AFAIK, there is no such thing as "when they are
    GC'd" in the language spec. There is, at best, "if they are GC'd",
    not "when". You are not guaranteed that GC ever takes place. At
    best, those destructors run when the application totally shuts down,
    as the process is about to exit, but I'm not sure whether the spec
    promises even that.

    > If this is not acceptable, it's a major change;


    It is guaranteeing GC running at any particular time that would be a
    major change. There is no such guarantee right now; there is just an
    implementation artifact in CPython that has led to careless habits
    among some users.
    Paul Rubin, Sep 6, 2006
    #8
  9. Jean-Paul Calderone

    Guest

    Paul Rubin wrote:
    > "" <> writes:
    > > > We had that debate already (PEP 343). Yes, there is some sloppy
    > > > current practice by CPython users that relies on the GC to close the
    > > > db conn.

    > >
    > > This point is unrelated to with or ref-counting. Even the standard
    > > library will close file objects when they are GC'd.

    >
    > I don't think so. AFAIK, there is no such thing as "when they are
    > GC'd" in the language spec.


    If they don't get GC'd, then "when they are GC'd" is never. The point
    is that the standard library _does_ close files and take other
    program-visible actions in __del__ methods; I'm unclear on if you think
    that doing so is actually sloppy practice (as opposed to users of the
    standard library relying on the GC running in some deterministic manner
    or even at all, which you clearly do think is sloppy practice).

    I originally thought that was what you meant when you said that "GC is
    supposed to make it look like every object stays around forever, and
    any finalizer that causes an explicit internal state change in an
    extant object (like closing a file or socket) is not in the GC spirit
    to begin with." but going back and reading it I'm not sure.
    , Sep 6, 2006
    #9
  10. Jean-Paul Calderone

    Paul Rubin Guest

    "" <> writes:
    > If they don't get GC'd, then "when they are GC'd" is never. The point
    > is that the standard library _does_ close files and take other
    > program-visible actions in __del__ methods; I'm unclear on if you think
    > that doing so is actually sloppy practice


    I don't know if I'd say "sloppy"; it's certainly messy and can lead to
    complex bugs, but there are pragmatic reasons for wanting to do it in
    some situations and it can be used to good purpose if you're careful.
    But I think people sometimes expect too much from it, as Steve Holden
    indicated.

    It should not be compared with C++ destructors, since C++ deallocation
    is explicit. I'm not sure what other languages with actual GC support
    anything like it. I implemented it in Lisp once but found it
    disconcerting since some of the stuff removed by GC in the system I
    was working on were no-longer-referenced graphics objects on the
    screen, which would thereby wink out of existence at surprising times.
    I concluded that the application should keep track of stuff like that
    instead of tossing them into the air for the GC to clean up.

    > I originally thought that was what you meant when you said that "GC is
    > supposed to make it look like every object stays around forever, and
    > any finalizer that causes an explicit internal state change in an
    > extant object (like closing a file or socket) is not in the GC spirit
    > to begin with." but going back and reading it I'm not sure.


    GC is supposed to just release resources that are no longer referenced
    (since you can't tell whether they're still around, you can act as if
    they are always stay around). For files, maybe closing them is
    ok--the file abstraction mimics synchronous i/o and in the idealized
    version, closing them is a do-nothing and the closure is undetectable.
    For sockets (which are supposed to send particular messages over the
    wire when the socket shuts down) it's less appealing.
    Paul Rubin, Sep 6, 2006
    #10
  11. On 2006-09-06, <> wrote:
    > Paul Rubin wrote:
    >> "" <> writes:
    >> > (1) I think is here to stay, if you're going to tell programmers that
    >> > their destructors can't make program-visible changes (e.g. closing the
    >> > database connection when a dbconn is destroyed), that's a _huge_ change
    >> > from current practice that needs serious debate.

    >>
    >> We had that debate already (PEP 343). Yes, there is some sloppy
    >> current practice by CPython users that relies on the GC to close the
    >> db conn.

    >
    > This point is unrelated to with or ref-counting. Even the standard
    > library will close file objects when they are GC'd.


    This is not totally true. My experience is that if you use the
    tarfile module any tarfile that was opened for appending or
    writing risks being corrupted if it isn't closed explicedly.

    --
    Antoon Pardon
    Antoon Pardon, Sep 7, 2006
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. km
    Replies:
    41
    Views:
    1,132
    lcaamano
    Sep 6, 2006
  2. Replies:
    4
    Views:
    321
    Steve Holden
    Sep 6, 2006
  3. Replies:
    9
    Views:
    1,036
    Mark Space
    Dec 29, 2007
  4. Steven Woody
    Replies:
    0
    Views:
    406
    Steven Woody
    Jan 9, 2009
  5. Jure Erznožnik
    Replies:
    51
    Views:
    1,616
    Hendrik van Rooyen
    Jun 22, 2009
Loading...

Share This Page