tarfile's tar.extractfile() file-like object incompatible with pickle.load()?

Discussion in 'Python' started by Matt Doucleff, Aug 26, 2004.

  1. Hi everyone! I must be doing something wrong here :) I have a
    tarball that contains a single file whose contents are a pickled
    object. I would like to unpickle the object directly from the tarball
    using the file-like object provided by extractfile(). Attempts to do
    this result in EOFError. However if I first extract to a temporary
    file, then unpickle from there, it works. The below code reproduces
    the problem (on my machine at least). I'm running Python 2.3.4,
    manually installed on Debian Woody (original python removed). Thanks!

    This sample code creates (and then removes) files in the tmp directory
    and in the current working directory.

    # demonstrates extractfile/unpickle failure (bug?)

    # pickle a dict to a temp file
    # create tar file, add temp file to it, close tar file
    # open tar file for reading
    # obtain file-like object for pickled file using extractfile()
    # attempt to unpickle dict from file-like object
    # fails with EOFError exception

    import tarfile
    import pickle
    import tempfile
    import os

    if __name__ == '__main__':
    try:
    hashtopickle = { 'a' : 1, 'b' : 2 }

    # pickle to temp file
    (fd, tmpfilename) = tempfile.mkstemp()
    tmpfile = os.fdopen(fd, 'w')
    pickle.dump(hashtopickle, tmpfile)
    tmpfile.close()

    # create tar; add temp file
    tar = tarfile.open('tarpickle.tar', 'w')
    tar.add(tmpfilename, 'pickledhash')
    tar.close()

    # remove temp file
    os.remove(tmpfilename)

    # open tarfile for reading, get filelike
    tar = tarfile.open('tarpickle.tar', 'r')
    filelike = tar.extractfile('pickledhash')

    # fails
    hashcopy = pickle.load(filelike)

    finally:
    # cleanup
    os.remove('tarpickle.tar')
     
    Matt Doucleff, Aug 26, 2004
    #1
    1. Advertising

  2. Matt Doucleff

    Tom B. Guest

    "Matt Doucleff" <> wrote in message
    news:...
    > Hi everyone! I must be doing something wrong here :) I have a
    > tarball that contains a single file whose contents are a pickled
    > object. I would like to unpickle the object directly from the tarball
    > using the file-like object provided by extractfile(). Attempts to do
    > this result in EOFError. However if I first extract to a temporary
    > file, then unpickle from there, it works. The below code reproduces
    > the problem (on my machine at least). I'm running Python 2.3.4,
    > manually installed on Debian Woody (original python removed). Thanks!
    >
    > This sample code creates (and then removes) files in the tmp directory
    > and in the current working directory.
    >
    > # demonstrates extractfile/unpickle failure (bug?)
    >
    > # pickle a dict to a temp file
    > # create tar file, add temp file to it, close tar file
    > # open tar file for reading
    > # obtain file-like object for pickled file using extractfile()
    > # attempt to unpickle dict from file-like object
    > # fails with EOFError exception
    >
    > import tarfile
    > import pickle
    > import tempfile
    > import os
    >
    > if __name__ == '__main__':
    > try:
    > hashtopickle = { 'a' : 1, 'b' : 2 }
    >
    > # pickle to temp file
    > (fd, tmpfilename) = tempfile.mkstemp()
    > tmpfile = os.fdopen(fd, 'w')
    > pickle.dump(hashtopickle, tmpfile)
    > tmpfile.close()
    >
    > # create tar; add temp file
    > tar = tarfile.open('tarpickle.tar', 'w')
    > tar.add(tmpfilename, 'pickledhash')
    > tar.close()
    >
    > # remove temp file
    > os.remove(tmpfilename)
    >
    > # open tarfile for reading, get filelike
    > tar = tarfile.open('tarpickle.tar', 'r')
    > filelike = tar.extractfile('pickledhash')
    >
    > # fails
    > hashcopy = pickle.load(filelike)
    >
    > finally:
    > # cleanup
    > os.remove('tarpickle.tar')


    Maby you should,

    import StringIO

    hashcopy = pickle.load(StringIO.StringIO(filelike))

    but im not sure,

    Tom
     
    Tom B., Aug 27, 2004
    #2
    1. Advertising

  3. Matt Doucleff

    Tom B. Guest

    "Matt Doucleff" <> wrote in message
    news:...
    > Hi everyone! I must be doing something wrong here :) I have a
    > tarball that contains a single file whose contents are a pickled
    > object. I would like to unpickle the object directly from the tarball
    > using the file-like object provided by extractfile(). Attempts to do
    > this result in EOFError. However if I first extract to a temporary
    > file, then unpickle from there, it works. The below code reproduces
    > the problem (on my machine at least). I'm running Python 2.3.4,
    > manually installed on Debian Woody (original python removed). Thanks!
    >
    > This sample code creates (and then removes) files in the tmp directory
    > and in the current working directory.
    >
    > # demonstrates extractfile/unpickle failure (bug?)
    >
    > # pickle a dict to a temp file
    > # create tar file, add temp file to it, close tar file
    > # open tar file for reading
    > # obtain file-like object for pickled file using extractfile()
    > # attempt to unpickle dict from file-like object
    > # fails with EOFError exception
    >
    > import tarfile
    > import pickle
    > import tempfile
    > import os
    >
    > if __name__ == '__main__':
    > try:
    > hashtopickle = { 'a' : 1, 'b' : 2 }
    >
    > # pickle to temp file
    > (fd, tmpfilename) = tempfile.mkstemp()
    > tmpfile = os.fdopen(fd, 'w')
    > pickle.dump(hashtopickle, tmpfile)
    > tmpfile.close()
    >
    > # create tar; add temp file
    > tar = tarfile.open('tarpickle.tar', 'w')
    > tar.add(tmpfilename, 'pickledhash')
    > tar.close()
    >
    > # remove temp file
    > os.remove(tmpfilename)
    >
    > # open tarfile for reading, get filelike
    > tar = tarfile.open('tarpickle.tar', 'r')
    > filelike = tar.extractfile('pickledhash')
    >
    > # fails
    > hashcopy = pickle.load(filelike)
    >
    > finally:
    > # cleanup
    > os.remove('tarpickle.tar')


    It occurs to me that you need to do,

    hashcopy = pickle.loads(filelike)

    if filelike is a string.

    Tom
    P.S. have a look at pickle.dumps()
     
    Tom B., Aug 27, 2004
    #3
  4. "Tom B." <> wrote in message news:<>...
    > "Matt Doucleff" <> wrote in message
    > news:...
    > > Hi everyone! I must be doing something wrong here :) I have a
    > > tarball that contains a single file whose contents are a pickled
    > > object. I would like to unpickle the object directly from the tarball
    > > using the file-like object provided by extractfile(). Attempts to do
    > > this result in EOFError. However if I first extract to a temporary
    > > file, then unpickle from there, it works. The below code reproduces
    > > the problem (on my machine at least). I'm running Python 2.3.4,
    > > manually installed on Debian Woody (original python removed). Thanks!
    > >
    > > This sample code creates (and then removes) files in the tmp directory
    > > and in the current working directory.
    > >
    > > # demonstrates extractfile/unpickle failure (bug?)
    > >
    > > # pickle a dict to a temp file
    > > # create tar file, add temp file to it, close tar file
    > > # open tar file for reading
    > > # obtain file-like object for pickled file using extractfile()
    > > # attempt to unpickle dict from file-like object
    > > # fails with EOFError exception
    > >
    > > import tarfile
    > > import pickle
    > > import tempfile
    > > import os
    > >
    > > if __name__ == '__main__':
    > > try:
    > > hashtopickle = { 'a' : 1, 'b' : 2 }
    > >
    > > # pickle to temp file
    > > (fd, tmpfilename) = tempfile.mkstemp()
    > > tmpfile = os.fdopen(fd, 'w')
    > > pickle.dump(hashtopickle, tmpfile)
    > > tmpfile.close()
    > >
    > > # create tar; add temp file
    > > tar = tarfile.open('tarpickle.tar', 'w')
    > > tar.add(tmpfilename, 'pickledhash')
    > > tar.close()
    > >
    > > # remove temp file
    > > os.remove(tmpfilename)
    > >
    > > # open tarfile for reading, get filelike
    > > tar = tarfile.open('tarpickle.tar', 'r')


    > > filelike = tar.extractfile('pickledhash')
    > >
    > > # fails
    > > hashcopy = pickle.load(filelike)
    > >
    > > finally:
    > > # cleanup
    > > os.remove('tarpickle.tar')

    >
    > It occurs to me that you need to do,
    >
    > hashcopy = pickle.loads(filelike)
    >
    > if filelike is a string.
    >
    > Tom
    > P.S. have a look at pickle.dumps()


    The tarfile.extractfile() method does not read the contents
    of the encapsulated file into a string, but constructs a new
    object that implements file operations (it is like a file)
    and is intended to be used as if you had simply opened the
    tar-encapsulated file directly.

    Matt
     
    Matt Doucleff, Aug 27, 2004
    #4
  5. Matt Doucleff

    Tom B. Guest

    "Matt Doucleff" <> wrote in message
    news:...
    > "Tom B." <> wrote in message

    news:<>...
    > > "Matt Doucleff" <> wrote in message
    > > news:...
    > > > Hi everyone! I must be doing something wrong here :) I have a
    > > > tarball that contains a single file whose contents are a pickled
    > > > object. I would like to unpickle the object directly from the tarball
    > > > using the file-like object provided by extractfile(). Attempts to do
    > > > this result in EOFError. However if I first extract to a temporary
    > > > file, then unpickle from there, it works. The below code reproduces
    > > > the problem (on my machine at least). I'm running Python 2.3.4,
    > > > manually installed on Debian Woody (original python removed). Thanks!
    > > >
    > > > This sample code creates (and then removes) files in the tmp directory
    > > > and in the current working directory.
    > > >
    > > > # demonstrates extractfile/unpickle failure (bug?)
    > > >
    > > > # pickle a dict to a temp file
    > > > # create tar file, add temp file to it, close tar file
    > > > # open tar file for reading
    > > > # obtain file-like object for pickled file using extractfile()
    > > > # attempt to unpickle dict from file-like object
    > > > # fails with EOFError exception
    > > >
    > > > import tarfile
    > > > import pickle
    > > > import tempfile
    > > > import os
    > > >
    > > > if __name__ == '__main__':
    > > > try:
    > > > hashtopickle = { 'a' : 1, 'b' : 2 }
    > > >
    > > > # pickle to temp file
    > > > (fd, tmpfilename) = tempfile.mkstemp()
    > > > tmpfile = os.fdopen(fd, 'w')
    > > > pickle.dump(hashtopickle, tmpfile)
    > > > tmpfile.close()
    > > >
    > > > # create tar; add temp file
    > > > tar = tarfile.open('tarpickle.tar', 'w')
    > > > tar.add(tmpfilename, 'pickledhash')
    > > > tar.close()
    > > >
    > > > # remove temp file
    > > > os.remove(tmpfilename)
    > > >
    > > > # open tarfile for reading, get filelike
    > > > tar = tarfile.open('tarpickle.tar', 'r')

    >
    > > > filelike = tar.extractfile('pickledhash')
    > > >
    > > > # fails
    > > > hashcopy = pickle.load(filelike)
    > > >
    > > > finally:
    > > > # cleanup
    > > > os.remove('tarpickle.tar')

    > >
    > > It occurs to me that you need to do,
    > >
    > > hashcopy = pickle.loads(filelike)
    > >
    > > if filelike is a string.
    > >
    > > Tom
    > > P.S. have a look at pickle.dumps()

    >
    > The tarfile.extractfile() method does not read the contents
    > of the encapsulated file into a string, but constructs a new
    > object that implements file operations (it is like a file)
    > and is intended to be used as if you had simply opened the
    > tar-encapsulated file directly.
    >
    > Matt


    How about,

    filelike = tar.extractfile('pickledhash')
    filetext = filelike.read()
    hashcopy = pickle.load(StringIO.StringIO(filetext ))

    Tom
     
    Tom B., Aug 27, 2004
    #5
  6. Matt Doucleff

    Tom B. Guest

    "Matt Doucleff" <> wrote in message
    news:...
    > "Tom B." <> wrote in message

    news:<>...
    > > "Matt Doucleff" <> wrote in message
    > > news:...
    > > > Hi everyone! I must be doing something wrong here :) I have a
    > > > tarball that contains a single file whose contents are a pickled
    > > > object. I would like to unpickle the object directly from the tarball
    > > > using the file-like object provided by extractfile(). Attempts to do
    > > > this result in EOFError. However if I first extract to a temporary
    > > > file, then unpickle from there, it works. The below code reproduces
    > > > the problem (on my machine at least). I'm running Python 2.3.4,
    > > > manually installed on Debian Woody (original python removed). Thanks!
    > > >
    > > > This sample code creates (and then removes) files in the tmp directory
    > > > and in the current working directory.
    > > >
    > > > # demonstrates extractfile/unpickle failure (bug?)
    > > >
    > > > # pickle a dict to a temp file
    > > > # create tar file, add temp file to it, close tar file
    > > > # open tar file for reading
    > > > # obtain file-like object for pickled file using extractfile()
    > > > # attempt to unpickle dict from file-like object
    > > > # fails with EOFError exception
    > > >
    > > > import tarfile
    > > > import pickle
    > > > import tempfile
    > > > import os


    Now that I take a closer look at your program it should read,


    > > > if __name__ == '__main__':
    > > > try:
    > > > hashtopickle = { 'a' : 1, 'b' : 2 }
    > > >
    > > > # pickle to temp file
    > > > (fd, tmpfilename) = tempfile.mkstemp()
    > > > tmpfile = file(tmpfilename,'w+')
    > > > pickle.dump(hashtopickle, tmpfile)
    > > > tmpfile.close()
    > > >
    > > > # create tar; add temp file
    > > > tar = tarfile.open('tarpickle.tar', 'w')
    > > > tar.add(tmpfilename, 'pickledhash')
    > > > tar.close()
    > > >
    > > > # remove temp file
    > > > os.remove(tmpfilename)
    > > >
    > > > # open tarfile for reading, get filelike
    > > > tar = tarfile.open('tarpickle.tar', 'r')

    >
    > > > filelike = tar.extractfile('pickledhash')
    > > >
    > > > # fails
    > > > hashcopy = pickle.load(filelike)
    > > >
    > > > finally:
    > > > # cleanup
    > > > os.remove('tarpickle.tar')

    > >

    Your opening the tmpfilename in an odd way.

    Tom
     
    Tom B., Aug 27, 2004
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Claudio Grondi
    Replies:
    4
    Views:
    591
    Claudio Grondi
    Aug 20, 2005
  2. Replies:
    3
    Views:
    468
    =?iso-8859-1?q?Lars_Gust=E4bel?=
    Aug 28, 2005
  3. m_ahlenius
    Replies:
    2
    Views:
    307
    m_ahlenius
    Feb 8, 2010
  4. m_ahlenius
    Replies:
    9
    Views:
    569
    m_ahlenius
    Aug 21, 2010
  5. rudson alves
    Replies:
    1
    Views:
    238
    Dave Angel
    Aug 16, 2012
Loading...

Share This Page