storing pickles in sql data base

[David Bear]

I need to store pickled objects in postgresql. I reading through the pickle
docs it says to always open a file in binary mode because you can't be sure
if the pickled data is binary or text. So I have 2 question. Can I set the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?

 

[Gabriel Genellina]

I need to store pickled objects in postgresql. I reading through the
pickle
docs it says to always open a file in binary mode because you can't be
sure
if the pickled data is binary or text. So I have 2 question. Can I set
the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?

I'd use a binary datatype (raw, blob, binary, whatever postgres calls it).
Text columns might be converted or reencoded in some way, binary data
should never be modified in any way.

 

[Philippe Bouige]

I need to store pickled objects in postgresql.

You have this :

http://blog.melhase.net/articles/2006/06/06/
a-tale-of-postgresql-types-and-python

http://blog.melhase.net/articles/2006/06/13/
auto-postgresql-pickles

Philippe Bouige

 

[Nick Craig-Wood]

I need to store pickled objects in postgresql. I reading through the pickle
docs it says to always open a file in binary mode because you can't be sure
if the pickled data is binary or text. So I have 2 question. Can I set the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?

You could always encode it into text form, eg

>>> from cPickle import dumps, loads
>>> a = range(10)
>>> a [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> b = dumps(a).encode("zip").encode("base64").strip()
>>> b 'eJzTyCkw5PI04Er0NARiIyA2BmITIDYFYjMgNgdiCyC25ErUAwD5DQqD'
>>> loads(b.decode("base64").decode("zip")) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>

 

[Chris Mellon]

I need to store pickled objects in postgresql. I reading through the pickle
docs it says to always open a file in binary mode because you can't be sure
if the pickled data is binary or text. So I have 2 question. Can I set the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?

You could always encode it into text form, eg

from cPickle import dumps, loads
a = range(10)
a [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
b = dumps(a).encode("zip").encode("base64").strip()
b 'eJzTyCkw5PI04Er0NARiIyA2BmITIDYFYjMgNgdiCyC25ErUAwD5DQqD'
loads(b.decode("base64").decode("zip")) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file. base64 encoding will work on
protocol 1 and 2 (which are binary protocols and not text-safe), but
doing that removes the main benefit of the higher protocols, which is
smaller pickle data.

 

[Marc 'BlackJack' Rintsch]

[pickle]

Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file.

It's not really a text protocol it's more a binary protocol that uses just
the ASCII range of byte values. You have to write and read the "text"
files in binary mode or they break if taken across platform boundaries
because of the different line endings in Linux and Windows for instance.

Ciao,
Marc 'BlackJack' Rintsch

 

[Chris Mellon]

[pickle]

Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file.

It's not really a text protocol it's more a binary protocol that uses just
the ASCII range of byte values. You have to write and read the "text"
files in binary mode or they break if taken across platform boundaries
because of the different line endings in Linux and Windows for instance.

It's fine as long as you use universal line endings mode.

 

[Gabriel Genellina]

[pickle]

Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file.

It's not really a text protocol it's more a binary protocol that uses
just
the ASCII range of byte values. You have to write and read the "text"
files in binary mode or they break if taken across platform boundaries
because of the different line endings in Linux and Windows for instance.

It's fine as long as you use universal line endings mode.

Neither. Won't work for Unicode objects then. See bug#1724366
<http://python.org/sf/1724366>