Loading multiple versions of the same package at the same time

Discussion in 'Python' started by della, Nov 27, 2008.

  1. della

    della Guest

    Hi all,

    I've got some pickled files representing graphs (using networkx,
    http://networkx.lanl.gov if you're interested) that were produced
    using version 0.36 of the library.

    Now, they have released a new version of the library which is
    incompatible with respect to pickled files, so what I'd like to do is
    to write a script that loads the two versions of the library at once,
    unpickles with the old one, creates a new object with the new version
    and re-pickles it with the new version.

    So, I installed the two versions of the package using easy_install -m,
    but it looks like I can't import the two versions at once:

    >>> from pkg_resources import require
    >>> require('networkx==0.36')

    [networkx 0.36 (/usr/lib/python2.5/site-packages/networkx-0.36-
    py2.5.egg)]
    >>> import networkx as nx_old
    >>> require('networkx==1.0')

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626,
    in require
    needed = self.resolve(parse_requirements(requirements))
    File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 528,
    in resolve
    raise VersionConflict(dist,req) # XXX put more info here
    pkg_resources.VersionConflict: (networkx 0.36 (/usr/lib/python2.5/site-
    packages/networkx-0.36-py2.5.egg), Requirement.parse('networkx==1.0'))

    Any ideas?
    Thanks a lot
    matteo
    della, Nov 27, 2008
    #1
    1. Advertising

  2. della wrote:

    > Hi all,
    >
    > I've got some pickled files representing graphs (using networkx,
    > http://networkx.lanl.gov if you're interested) that were produced
    > using version 0.36 of the library.
    >
    > Now, they have released a new version of the library which is
    > incompatible with respect to pickled files, so what I'd like to do is
    > to write a script that loads the two versions of the library at once,
    > unpickles with the old one, creates a new object with the new version
    > and re-pickles it with the new version.
    >
    > So, I installed the two versions of the package using easy_install -m,
    > but it looks like I can't import the two versions at once:
    >
    >>>> from pkg_resources import require
    >>>> require('networkx==0.36')

    > [networkx 0.36 (/usr/lib/python2.5/site-packages/networkx-0.36-
    > py2.5.egg)]
    >>>> import networkx as nx_old
    >>>> require('networkx==1.0')

    > Traceback (most recent call last):
    > File "<stdin>", line 1, in <module>
    > File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626,
    > in require
    > needed = self.resolve(parse_requirements(requirements))
    > File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 528,
    > in resolve
    > raise VersionConflict(dist,req) # XXX put more info here
    > pkg_resources.VersionConflict: (networkx 0.36 (/usr/lib/python2.5/site-
    > packages/networkx-0.36-py2.5.egg), Requirement.parse('networkx==1.0'))
    >
    > Any ideas?


    You can't do that. How should python distinguish between the two modules
    with the same name?

    What you can do is

    - import the old package
    - unpickle
    - convert to some neutral exchange format, simple dicts, lists, tuples
    - pickle that

    - import the new package
    - unpickle the exchange data
    - stuff it into the new classes

    Depending on who's got control for what, you might consider overloading the
    pickle protocol to allow for this more automatic in the future.

    This whole ruckus is BTW one of the reasons I moved away from ZODB.

    Diez
    Diez B. Roggisch, Nov 27, 2008
    #2
    1. Advertising

  3. della

    della Guest

    On 27 Nov, 11:21, "Diez B. Roggisch" <> wrote:

    > You can't do that. How should python distinguish between the two modules
    > with the same name?


    That's why I was trying to import them with different names :)

    > What you can do is
    >
    >  - import the old package
    >  - unpickle
    >  - convert to some neutral exchange format, simple dicts, lists, tuples
    >  - pickle that
    >
    >  - import the new package
    >  - unpickle the exchange data
    >  - stuff it into the new classes


    Thanks, this is what I did.

    matteo
    della, Nov 27, 2008
    #3
  4. della wrote:

    > On 27 Nov, 11:21, "Diez B. Roggisch" <> wrote:
    >
    >> You can't do that. How should python distinguish between the two modules
    >> with the same name?

    >
    > That's why I was trying to import them with different names :)


    You weren't. The "as" creates just a local alias. It's roughly equivalent to


    import foo as bar

    <->

    import foo
    bar = foo
    del foo


    But in the interpreters module dict, foo it is, and stays.

    Diez
    Diez B. Roggisch, Nov 27, 2008
    #4
  5. della

    Terry Reedy Guest

    Diez B. Roggisch wrote:
    > della wrote:
    >
    >> On 27 Nov, 11:21, "Diez B. Roggisch" <> wrote:
    >>
    >>> You can't do that. How should python distinguish between the two modules
    >>> with the same name?

    >> That's why I was trying to import them with different names :)

    >
    > You weren't. The "as" creates just a local alias. It's roughly equivalent to
    >
    >
    > import foo as bar
    >
    > <->
    >
    > import foo
    > bar = foo
    > del foo
    >
    >
    > But in the interpreters module dict, foo it is, and stays.


    But giving the modules different names on the disk should work, no?
    Terry Reedy, Nov 27, 2008
    #5
  6. della

    della Guest

    On 27 Nov, 21:34, Terry Reedy <> wrote:

    > > But in the interpreters module dict, foo it is, and stays.

    >
    > But giving the modules different names on the disk should work, no?


    Yes, but -- for what I've understood -- that wouldn't solve my
    original problem with pickle, since I would need to carry around a
    module with the new name forever :)

    matteo
    della, Nov 28, 2008
    #6
  7. della

    Peter Otten Guest

    della wrote:

    > On 27 Nov, 21:34, Terry Reedy <> wrote:
    >
    >> > But in the interpreters module dict, foo it is, and stays.

    >>
    >> But giving the modules different names on the disk should work, no?

    >
    > Yes, but -- for what I've understood -- that wouldn't solve my
    > original problem with pickle, since I would need to carry around a
    > module with the new name forever :)


    You can rename the globals within the pickle. A starting point, not tested
    beyond running the demo script:

    $ cat fixpickle.py
    import pickle
    import pickletools

    def ops(d):
    prevpos = None
    previnfo = None
    for info, arg, pos in pickletools.genops(d):
    if prevpos is not None:
    yield d[prevpos:pos]
    prevpos = pos
    previnfo = info
    yield d[prevpos:]


    def tocode(dotted):
    parts = tuple(dotted.rsplit(".", 1))
    return "c%s\n%s\n" % parts

    def rename_globals(d, pairs):
    updates = dict((tocode(old), tocode(new)) for old, new in pairs)
    return "".join(updates.get(o, o) for o in ops(d))

    $ cat alpha.py
    class A(object):
    def __init__(self, x, y):
    self.x = x
    self.y = y
    def __str__(self):
    return "A(%s, %s)" % (self.x, self.y)

    $ cat beta.py
    class B(object):
    def __str__(self):
    return "B(%s, %s)" % (self.x, self.y)

    $ cat demo.py
    import alpha
    import fixpickle
    import pickle

    if __name__ == "__main__":
    a = alpha.A(1, 2)
    print a
    d = pickle.dumps(a)
    d = fixpickle.rename_globals(d, [("alpha.A", "beta.B")])
    b = pickle.loads(d)
    print b

    Peter
    Peter Otten, Nov 28, 2008
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John Davison
    Replies:
    3
    Views:
    764
    Amitabh
    Jul 16, 2004
  2. Replies:
    2
    Views:
    3,847
    Fuzzyman
    Feb 24, 2006
  3. Cowmix
    Replies:
    3
    Views:
    446
  4. Edward Diener
    Replies:
    36
    Views:
    1,124
    Thomas Jollans
    Jul 26, 2010
  5. Yves Petinot
    Replies:
    9
    Views:
    279
    Brian McCauley
    Jun 30, 2004
Loading...

Share This Page