Recommended exception for objects that can't be pickled

S

Stefan Schwarzer

Hi,

Recently, I got a request [1] to support pickling of
`FTPHost` instances in my `ftplib` library.

I explained in the ticket why I think it's a bad idea and
now want to make explicit that `FTPHost` objects can't be
pickled. The usual way to do this seems to be defining a
`__getstate__` method and raise an exception there.

Now the question is "which exception?" and my research left
me a bit confused. I didn't find a recommendation for this
on the web, not even in the Python documentation for the
`pickle` module [2]. The only hint is that the documentation
states:

"""
The pickle module defines three exceptions:

exception pickle.PickleError

Common base class for the other pickling exceptions. It
inherits Exception.

exception pickle.PicklingError

Error raised when an unpicklable object is encountered
by Pickler. It inherits PickleError.

Refer to What can be pickled and unpickled? to learn
what kinds of objects can be pickled.

exception pickle.UnpicklingError

Error raised when there is a problem unpickling an
object, such as a data corruption or a security
violation. It inherits PickleError.

Note that other exceptions may also be raised during
unpickling, including (but not necessarily limited to)
AttributeError, EOFError, ImportError, and IndexError.
"""

This sounds like unpicklable objects should raise a
`PicklingError`. Indeed, if I do this, `pickle.dumps`
gives me (my own) `PicklingError`:

Python 3.3.2 (default, Nov 8 2013, 13:38:57)
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.... def __getstate__(self):
... raise pickle.PicklingError("can't pickle X objects")
...Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __getstate__
_pickle.PicklingError: can't pickle X objects

What now confuses me is that most, if not all, objects from
the standard library that aren't picklable raise a
`TypeError` when I try to pickle them:
Traceback (most recent call last):
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python3.3/socket.py", line 116, in __getstate__
raise TypeError("Cannot serialize socket object")
TypeError: Cannot serialize socket object

So the documentation for the `pickle` module (to me) implies
I should raise a `PicklingError` while the standard library
usually seems to use a `TypeError`. When I grep through the
library files for `PicklingError`, I get very few hits, most
of them in `pickle.py`:

$ find /usr/lib64/python3.3 -name "*.py" -exec grep -H PicklingError {} \;
/usr/lib64/python3.3/site-packages/numpy/numarray/session.py: except (pickle.PicklingError, TypeError, SystemError):
/usr/lib64/python3.3/pickle.py:__all__ = ["PickleError", "PicklingError", "UnpicklingError", "Pickler",
/usr/lib64/python3.3/pickle.py:class PicklingError(PickleError):
/usr/lib64/python3.3/pickle.py: raise PicklingError("Pickler.__init__() was not called by "
/usr/lib64/python3.3/pickle.py: raise PicklingError("Can't pickle %r object: %r" %
/usr/lib64/python3.3/pickle.py: raise PicklingError("%s must return string or tuple" % reduce)
/usr/lib64/python3.3/pickle.py: raise PicklingError("Tuple returned by %s must have "
/usr/lib64/python3.3/pickle.py: raise PicklingError("args from save_reduce() should be a tuple")
/usr/lib64/python3.3/pickle.py: raise PicklingError("func from save_reduce() should be callable")
/usr/lib64/python3.3/pickle.py: raise PicklingError(
/usr/lib64/python3.3/pickle.py: raise PicklingError(
/usr/lib64/python3.3/pickle.py: raise PicklingError(
/usr/lib64/python3.3/pickle.py: raise PicklingError(
/usr/lib64/python3.3/pickle.py: raise PicklingError(
/usr/lib64/python3.3/idlelib/rpc.py: except pickle.PicklingError:

Which exception would you raise for an object that can't be
pickled and why?

[1] http://ftputil.sschwarzer.net/trac/ticket/75
[2] https://docs.python.org/3.4/library/pickle.html

Best regards,
Stefan
 
T

Terry Reedy

Hi,

Recently, I got a request [1] to support pickling of
`FTPHost` instances in my `ftplib` library.

I explained in the ticket why I think it's a bad idea and
now want to make explicit that `FTPHost` objects can't be
pickled. The usual way to do this seems to be defining a
`__getstate__` method and raise an exception there.

Now the question is "which exception?" and my research left
me a bit confused. I didn't find a recommendation for this
on the web, not even in the Python documentation for the
`pickle` module [2]. The only hint is that the documentation
states:

"""
The pickle module defines three exceptions:

exception pickle.PickleError

Common base class for the other pickling exceptions. It
inherits Exception.

exception pickle.PicklingError

Error raised when an unpicklable object is encountered
by Pickler. It inherits PickleError.
Click to expand...

I am going to read this as 'unpicklable as determined by Pickler',
possibly due to a bug in the objects methods.
Refer to What can be pickled and unpickled? to learn
what kinds of objects can be pickled.

exception pickle.UnpicklingError

Error raised when there is a problem unpickling an
object, such as a data corruption or a security
violation. It inherits PickleError.

Note that other exceptions may also be raised during
unpickling, including (but not necessarily limited to)
AttributeError, EOFError, ImportError, and IndexError.
"""

This sounds like unpicklable objects should raise a
`PicklingError`. Indeed, if I do this, `pickle.dumps`
gives me (my own) `PicklingError`:

Python 3.3.2 (default, Nov 8 2013, 13:38:57)
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.... def __getstate__(self):
... raise pickle.PicklingError("can't pickle X objects")
...Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __getstate__
_pickle.PicklingError: can't pickle X objects

What now confuses me is that most, if not all, objects from
the standard library that aren't picklable raise a
`TypeError` when I try to pickle them:
Traceback (most recent call last):
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python3.3/socket.py", line 116, in __getstate__
raise TypeError("Cannot serialize socket object")
TypeError: Cannot serialize socket object
Click to expand...

I would just copy the socket.__getstate__, with a revised message.
So the documentation for the `pickle` module (to me) implies
I should raise a `PicklingError` while the standard library
usually seems to use a `TypeError`. When I grep through the
library files for `PicklingError`, I get very few hits, most
of them in `pickle.py`:
Click to expand...

This suggests that the exception is intended for use by Pickler, and not
by user code. The reason is pretty subtle and not explained in the doc.
See below.
$ find /usr/lib64/python3.3 -name "*.py" -exec grep -H PicklingError {} \;
/usr/lib64/python3.3/site-packages/numpy/numarray/session.py: except (pickle.PicklingError, TypeError, SystemError):
/usr/lib64/python3.3/pickle.py:__all__ = ["PickleError", "PicklingError", "UnpicklingError", "Pickler",
/usr/lib64/python3.3/pickle.py:class PicklingError(PickleError):
/usr/lib64/python3.3/pickle.py: raise PicklingError("Pickler.__init__() was not called by "
/usr/lib64/python3.3/pickle.py: raise PicklingError("Can't pickle %r object: %r" %
/usr/lib64/python3.3/pickle.py: raise PicklingError("%s must return string or tuple" % reduce)
/usr/lib64/python3.3/pickle.py: raise PicklingError("Tuple returned by %s must have "
/usr/lib64/python3.3/pickle.py: raise PicklingError("args from save_reduce() should be a tuple")
/usr/lib64/python3.3/pickle.py: raise PicklingError("func from save_reduce() should be callable")
Click to expand...

Most of the above look like bugs in code intended to work with Pickle.
In other words, if you user gets a TypeError, the user has made a
mistake by trying to pickle your object. If you tried to make your
object picklable, and users got PicklingError, that would indicate a bug
in your class code, not the users' use of your class.
/usr/lib64/python3.3/pickle.py: raise PicklingError(
/usr/lib64/python3.3/pickle.py: raise PicklingError(
/usr/lib64/python3.3/pickle.py: raise PicklingError(
/usr/lib64/python3.3/pickle.py: raise PicklingError(
/usr/lib64/python3.3/pickle.py: raise PicklingError(
/usr/lib64/python3.3/idlelib/rpc.py: except pickle.PicklingError:

Which exception would you raise for an object that can't be
pickled and why?
Click to expand...

TypeError, as explained above.
[1] http://ftputil.sschwarzer.net/trac/ticket/75
[2] https://docs.python.org/3.4/library/pickle.html
Click to expand...
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

why can't I pickle a class containing this dispatch dictionary? 6
pickle error: can't pickle instancemethod objects 2
Why can't you pickle instancemethods? 5
TypeError: can't pickle HASH objects? 10
Exception classes don't follow pickle protocol, problems unpickling 3
pickling extension class 4
doctest + shelve question 0
slice object unpickable 1

Members online

No members online now.
Total: 1,739 (members: 0, guests: 1,739)
Robots: 105

Forum statistics

Threads
474,156
Messages
2,570,877
Members
47,402
Latest member
xXBabooXx

Latest Threads

Top