Re: Is python is fully thread safe

Discussion in 'Python' started by jepler@unpythonic.net, Jul 22, 2004.

  1. Guest

    No. Not all Python programs are thread safe. For instance, here is a
    Python program that has a problem due to "threads"


    s = """Gur Mra bs Clguba, ol Gvz Crgref

    Ornhgvshy vf orggre guna htyl.
    Rkcyvpvg vf orggre guna vzcyvpvg.
    Fvzcyr vf orggre guna pbzcyrk.
    Pbzcyrk vf orggre guna pbzcyvpngrq.
    Syng vf orggre guna arfgrq.
    Fcnefr vf orggre guna qrafr.
    Ernqnovyvgl pbhagf.
    Fcrpvny pnfrf nera'g fcrpvny rabhtu gb oernx gur ehyrf.
    Nygubhtu cenpgvpnyvgl orngf chevgl.
    Reebef fubhyq arire cnff fvyragyl.
    Hayrff rkcyvpvgyl fvyraprq.
    Va gur snpr bs nzovthvgl, ershfr gur grzcgngvba gb thrff.
    Gurer fubhyq or bar-- naq cersrenoyl bayl bar --boivbhf jnl gb qb vg.
    Nygubhtu gung jnl znl abg or boivbhf ng svefg hayrff lbh'er Qhgpu.
    Abj vf orggre guna arire.
    Nygubhtu arire vf bsgra orggre guna *evtug* abj.
    Vs gur vzcyrzragngvba vf uneq gb rkcynva, vg'f n onq vqrn.
    Vs gur vzcyrzragngvba vf rnfl gb rkcynva, vg znl or n tbbq vqrn.
    Anzrfcnprf ner bar ubaxvat terng vqrn -- yrg'f qb zber bs gubfr!"""

    d = {}
    for c in (65, 97):
    for i in range(26):
    d[chr(i+c)] = chr((i+13) % 26 + c)

    threads

    print "".join([d.get(c, c) for c in s])
     
    , Jul 22, 2004
    #1
    1. Advertising

  2. Peter Maas Guest

    schrieb:
    > No. Not all Python programs are thread safe. For instance, here is a
    > Python program that has a problem due to "threads"
    >
    >
    > s = """Gur Mra bs Clguba, ol Gvz Crgref
    >

    [...]

    :))

    --
    -------------------------------------------------------------------
    Peter Maas, M+R Infosysteme, D-52070 Aachen, Tel +49-241-93878-0
    E-mail 'cGV0ZXIubWFhc0BtcGx1c3IuZGU=\n'.decode('base64')
    -------------------------------------------------------------------
     
    Peter Maas, Jul 22, 2004
    #2
    1. Advertising

  3. Jon Perez Guest

    Someone please explain to me where is the wit in this...

    wrote:

    > No. Not all Python programs are thread safe. For instance, here is a
    > Python program that has a problem due to "threads"
    >
    >
    > s = """Gur Mra bs Clguba, ol Gvz Crgref
    >
    > Ornhgvshy vf orggre guna htyl.
    > Rkcyvpvg vf orggre guna vzcyvpvg.
    > Fvzcyr vf orggre guna pbzcyrk.
    > Pbzcyrk vf orggre guna pbzcyvpngrq.
    > Syng vf orggre guna arfgrq.
    > Fcnefr vf orggre guna qrafr.
    > Ernqnovyvgl pbhagf.
    > Fcrpvny pnfrf nera'g fcrpvny rabhtu gb oernx gur ehyrf.
    > Nygubhtu cenpgvpnyvgl orngf chevgl.
    > Reebef fubhyq arire cnff fvyragyl.
    > Hayrff rkcyvpvgyl fvyraprq.
    > Va gur snpr bs nzovthvgl, ershfr gur grzcgngvba gb thrff.
    > Gurer fubhyq or bar-- naq cersrenoyl bayl bar --boivbhf jnl gb qb vg.
    > Nygubhtu gung jnl znl abg or boivbhf ng svefg hayrff lbh'er Qhgpu.
    > Abj vf orggre guna arire.
    > Nygubhtu arire vf bsgra orggre guna *evtug* abj.
    > Vs gur vzcyrzragngvba vf uneq gb rkcynva, vg'f n onq vqrn.
    > Vs gur vzcyrzragngvba vf rnfl gb rkcynva, vg znl or n tbbq vqrn.
    > Anzrfcnprf ner bar ubaxvat terng vqrn -- yrg'f qb zber bs gubfr!"""
    >
    > d = {}
    > for c in (65, 97):
    > for i in range(26):
    > d[chr(i+c)] = chr((i+13) % 26 + c)
    >
    > threads
    >
    > print "".join([d.get(c, c) for c in s])
     
    Jon Perez, Jul 23, 2004
    #3
  4. On Fri, 23 Jul 2004 08:27:19 +0800, Jon Perez <> wrote:

    > Someone please explain to me where is the wit in this...


    Run the program if you can't see it and you will understand why it has a
    problem with "threads" 8o)
     
    Marco Aschwanden, Jul 23, 2004
    #4
  5. Donn Cave Guest

    Quoth Marco Aschwanden <>:
    | On Fri, 23 Jul 2004 08:27:19 +0800, Jon Perez <> wrote:
    |
    | > Someone please explain to me where is the wit in this...
    |
    | Run the program if you can't see it and you will understand why it has a
    | problem with "threads" 8o)

    I believe he got that, but doesn't appreciate the humor behind it.
    Oh well - ask a stupid question, get a stupid answer.

    Donn
     
    Donn Cave, Jul 24, 2004
    #5
  6. [Donn Cave]
    > [Marco Aschwanden]
    > > [Jon Perez]


    > > > Someone please explain to me where is the wit in this...


    > > Run the program if you can't see it and you will understand why it
    > > has a problem with "threads" 8o)


    > I believe he got that, but doesn't appreciate the humor behind it. Oh
    > well - ask a stupid question, get a stupid answer.


    I did not closely follow the thread, but the `Subject:' line is not
    stating a stupid question in my opinion. On this list at least, to
    questions like "Is this or that standard Python module thread-safe?", a
    far too common reply is "Try it and you'll see...".

    Well, bugs occurring out of lack of thread-safety might, in many
    circumstances, hide for long before popping out, and hide again for long
    afterwards. They are not easy to reproduce. "Testing reveals presence
    of bugs, not their absence", as a proverb, is especially true in the
    context of thread-safety.

    Unless I'm missing a point (which is very possible), it does not seem
    to me that Python documentation is especially collaborative at teaching
    where threads are safe or not to use, so I'm left with the unpleasant
    feeling that my threaded Python applications should (ideally) never be
    allowed to go in real production. It's OK if I'm the only one to suffer
    from unexpected problems.

    I would much like if Python documentation had more definite and
    definitive answers about thread-safety of Python, everywhere.

    --
    Fran├žois Pinard http://www.iro.umontreal.ca/~pinard
     
    =?iso-8859-1?Q?Fran=E7ois?= Pinard, Jul 24, 2004
    #6
  7. Donn Cave Guest

    Quoth Franois Pinard <>:
    ....
    | Well, bugs occurring out of lack of thread-safety might, in many
    | circumstances, hide for long before popping out, and hide again for long
    | afterwards. They are not easy to reproduce. "Testing reveals presence
    | of bugs, not their absence", as a proverb, is especially true in the
    | context of thread-safety.
    |
    | Unless I'm missing a point (which is very possible), it does not seem
    | to me that Python documentation is especially collaborative at teaching
    | where threads are safe or not to use, so I'm left with the unpleasant
    | feeling that my threaded Python applications should (ideally) never be
    | allowed to go in real production. It's OK if I'm the only one to suffer
    | from unexpected problems.
    |
    | I would much like if Python documentation had more definite and
    | definitive answers about thread-safety of Python, everywhere.

    Well, I too am skeptical about multithreaded programs in production -
    written in Python or anything else. That is not to say it can't work,
    but concurrency is just fraught with potential for trouble. I don't
    imagine that's going to stop people from using them, but they really
    need to be able to think somewhat analytically about the issues.
    I think you understand that it's a little more complex than "Hot dog,
    the documentation says Python is thread safe!"

    Ironically, Python is actually a little ahead here because it doesn't
    strictly speaking support concurrency. That makes Python internals
    effectively the same as thread safe, so thread safety issues would be
    confined to application level issues and external functions that run
    after releasing the interpreter lock. I've had no particular trouble
    in my programs using a multithreaded API where others reportedly did
    have trouble, and for all I know it may be just because I was using
    Python and they weren't.

    Donn Cave,
     
    Donn Cave, Jul 25, 2004
    #7
  8. Dave Brueck Guest

    Donn Cave wrote:

    > Quoth Franois Pinard <>:
    > ...
    > | Well, bugs occurring out of lack of thread-safety might, in many
    > | circumstances, hide for long before popping out, and hide again for long
    > | afterwards. They are not easy to reproduce. "Testing reveals presence
    > | of bugs, not their absence", as a proverb, is especially true in the
    > | context of thread-safety.
    > |
    > | Unless I'm missing a point (which is very possible), it does not seem
    > | to me that Python documentation is especially collaborative at teaching
    > | where threads are safe or not to use, so I'm left with the unpleasant
    > | feeling that my threaded Python applications should (ideally) never be
    > | allowed to go in real production. It's OK if I'm the only one to suffer
    > | from unexpected problems.
    > |
    > | I would much like if Python documentation had more definite and
    > | definitive answers about thread-safety of Python, everywhere.
    >
    > Well, I too am skeptical about multithreaded programs in production -
    > written in Python or anything else. That is not to say it can't work,
    > but concurrency is just fraught with potential for trouble. I don't
    > imagine that's going to stop people from using them, but they really
    > need to be able to think somewhat analytically about the issues.
    > I think you understand that it's a little more complex than "Hot dog,
    > the documentation says Python is thread safe!"
    >
    > Ironically, Python is actually a little ahead here because it doesn't
    > strictly speaking support concurrency. That makes Python internals
    > effectively the same as thread safe, so thread safety issues would be
    > confined to application level issues and external functions that run
    > after releasing the interpreter lock. I've had no particular trouble
    > in my programs using a multithreaded API where others reportedly did
    > have trouble, and for all I know it may be just because I was using
    > Python and they weren't.


    This is quite true - automatic cleanup of resources on thread
    termination combined with the fact that multithreaded Python code won't
    corrupt the interpreter internals makes Python a fairly safe platform
    for multi-threading - you certainly end up with a bigger safety net
    than, say, C++.

    As an aside, I think the documentation wrt to multi-threaded safety
    could be improved, but I'm not sure how exactly to improve it because
    "thread safety" doesn't always have a clear definition. For example, two
    threads can safely access and even modify the same Python object (such
    as a dictionary) without explicit locking and the interpreter will not
    crash, so at that low-level you could say it is thread safe. But at the
    semantic level of your particular application, a function that modifies
    the same dictionary by multiple threads may or may not need explicit
    locking around the dictionary modification (depending on what is being
    done), so at that level the access may not be considered thread safe.

    As to what the Python documentation needs to say (and, in the end,
    perhaps it already does this well enough), I'd think it needs to make a
    few things clear:

    1) Accessing the same Python object from multiple threads from Python
    code won't corrupt the internals of the interpreter.

    2) Modules/functions that are just plain not thread-safe should be
    marked as such (e.g. the fpectl module).

    Beyond that, can you think of anything else? I doubt that the work
    required to document the low-level thread safety of each and every
    function is really worth it (not to mention the effort required to keep
    it up to date over time) - partly because of #1, above, partly because
    in decent multi-threaded programs there aren't too many shared objects
    and the points at which they are shared are fairly well-defined, and
    partly because when in doubt, it's fairly simple to add a mutex around a
    resource.

    -Dave
     
    Dave Brueck, Jul 26, 2004
    #8
  9. Donn Cave Guest

    Quoth Dave Brueck <>:
    ....
    | As an aside, I think the documentation wrt to multi-threaded safety
    | could be improved, but I'm not sure how exactly to improve it because
    | "thread safety" doesn't always have a clear definition. For example, two
    | threads can safely access and even modify the same Python object (such
    | as a dictionary) without explicit locking and the interpreter will not
    | crash, so at that low-level you could say it is thread safe. But at the
    | semantic level of your particular application, a function that modifies
    | the same dictionary by multiple threads may or may not need explicit
    | locking around the dictionary modification (depending on what is being
    | done), so at that level the access may not be considered thread safe.
    |
    | As to what the Python documentation needs to say (and, in the end,
    | perhaps it already does this well enough), I'd think it needs to make a
    | few things clear:
    |
    | 1) Accessing the same Python object from multiple threads from Python
    | code won't corrupt the internals of the interpreter.
    |
    | 2) Modules/functions that are just plain not thread-safe should be
    | marked as such (e.g. the fpectl module).
    |
    | Beyond that, can you think of anything else? I doubt that the work
    | required to document the low-level thread safety of each and every
    | function is really worth it (not to mention the effort required to keep
    | it up to date over time) - partly because of #1, above, partly because
    | in decent multi-threaded programs there aren't too many shared objects
    | and the points at which they are shared are fairly well-defined, and
    | partly because when in doubt, it's fairly simple to add a mutex around a
    | resource.

    Well, in the anything else category, we can consider the point that
    came up in another thread, where Paul Moore pointed out that the
    dictionary access that's atomic with core dictionaries is not atomic
    for Python instance objects that implement setitem. I know you can
    dismiss that as irrelevant because it doesn't break internal structures,
    but at least it underscores the point - we can't really say much more
    about thread safety than this, we think the interpreter will survive.
    The rest is up to the programmer.

    Donn Cave,
     
    Donn Cave, Jul 28, 2004
    #9
  10. [Dave Brueck]

    > As to what the Python documentation needs to say (and, in the end,
    > perhaps it already does this well enough), I'd think it needs to make
    > a few things clear:


    > 1) Accessing the same Python object from multiple threads from Python
    > code won't corrupt the internals of the interpreter.


    > 2) Modules/functions that are just plain not thread-safe should be
    > marked as such (e.g. the fpectl module).


    > Beyond that, can you think of anything else?


    If the absence of an explicit notification for a library module was
    meaning that the module is thread-safe, it would be a great relief
    indeed. (Of course, this convention itself should be clearly stated,
    along with your note '1)' above.)

    For those being anal about thread-safety of small operations, maybe that
    the documentation might usefully mention the usual `import dis' trickery
    to explore how many op-codes implement various operations. With maybe
    some due warning that implementation may (or may not) change for some
    particular operations.

    Best would be that the reference manual present thread-safety notes, for
    operations, in the same way that the library manual present them for
    modules. That is, any operation with no explicit notification would
    mean that this operation is thread-safe (unless overridden, of course).

    --
    Fran├žois Pinard http://www.iro.umontreal.ca/~pinard
     
    =?iso-8859-1?Q?Fran=E7ois?= Pinard, Jul 31, 2004
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Shane Hathaway
    Replies:
    3
    Views:
    332
    Martin v. =?iso-8859-15?q?L=F6wis?=
    Sep 12, 2003
  2. Harri Pesonen
    Replies:
    32
    Views:
    831
    Christopher A. Craig
    Sep 17, 2003
  3. Christopher A. Craig
    Replies:
    2
    Views:
    311
    Shane Hathaway
    Sep 12, 2003
  4. Gabriel Rossetti
    Replies:
    0
    Views:
    1,323
    Gabriel Rossetti
    Aug 29, 2008
  5. John Nagle
    Replies:
    5
    Views:
    472
    John Nagle
    Mar 12, 2012
Loading...

Share This Page