Loading multiple versions of the same package at the same time

D

della

Hi all,

I've got some pickled files representing graphs (using networkx,
http://networkx.lanl.gov if you're interested) that were produced
using version 0.36 of the library.

Now, they have released a new version of the library which is
incompatible with respect to pickled files, so what I'd like to do is
to write a script that loads the two versions of the library at once,
unpickles with the old one, creates a new object with the new version
and re-pickles it with the new version.

So, I installed the two versions of the package using easy_install -m,
but it looks like I can't import the two versions at once:
[networkx 0.36 (/usr/lib/python2.5/site-packages/networkx-0.36-
py2.5.egg)]Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626,
in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 528,
in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (networkx 0.36 (/usr/lib/python2.5/site-
packages/networkx-0.36-py2.5.egg), Requirement.parse('networkx==1.0'))

Any ideas?
Thanks a lot
matteo
 
D

Diez B. Roggisch

della said:
Hi all,

I've got some pickled files representing graphs (using networkx,
http://networkx.lanl.gov if you're interested) that were produced
using version 0.36 of the library.

Now, they have released a new version of the library which is
incompatible with respect to pickled files, so what I'd like to do is
to write a script that loads the two versions of the library at once,
unpickles with the old one, creates a new object with the new version
and re-pickles it with the new version.

So, I installed the two versions of the package using easy_install -m,
but it looks like I can't import the two versions at once:
[networkx 0.36 (/usr/lib/python2.5/site-packages/networkx-0.36-
py2.5.egg)]Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626,
in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 528,
in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (networkx 0.36 (/usr/lib/python2.5/site-
packages/networkx-0.36-py2.5.egg), Requirement.parse('networkx==1.0'))

Any ideas?

You can't do that. How should python distinguish between the two modules
with the same name?

What you can do is

- import the old package
- unpickle
- convert to some neutral exchange format, simple dicts, lists, tuples
- pickle that

- import the new package
- unpickle the exchange data
- stuff it into the new classes

Depending on who's got control for what, you might consider overloading the
pickle protocol to allow for this more automatic in the future.

This whole ruckus is BTW one of the reasons I moved away from ZODB.

Diez
 
D

della

You can't do that. How should python distinguish between the two modules
with the same name?

That's why I was trying to import them with different names :)
What you can do is

 - import the old package
 - unpickle
 - convert to some neutral exchange format, simple dicts, lists, tuples
 - pickle that

 - import the new package
 - unpickle the exchange data
 - stuff it into the new classes

Thanks, this is what I did.

matteo
 
D

Diez B. Roggisch

della said:
That's why I was trying to import them with different names :)

You weren't. The "as" creates just a local alias. It's roughly equivalent to


import foo as bar

<->

import foo
bar = foo
del foo


But in the interpreters module dict, foo it is, and stays.

Diez
 
T

Terry Reedy

Diez said:
You weren't. The "as" creates just a local alias. It's roughly equivalent to


import foo as bar

<->

import foo
bar = foo
del foo


But in the interpreters module dict, foo it is, and stays.

But giving the modules different names on the disk should work, no?
 
D

della

But giving the modules different names on the disk should work, no?

Yes, but -- for what I've understood -- that wouldn't solve my
original problem with pickle, since I would need to carry around a
module with the new name forever :)

matteo
 
P

Peter Otten

della said:
Yes, but -- for what I've understood -- that wouldn't solve my
original problem with pickle, since I would need to carry around a
module with the new name forever :)

You can rename the globals within the pickle. A starting point, not tested
beyond running the demo script:

$ cat fixpickle.py
import pickle
import pickletools

def ops(d):
prevpos = None
previnfo = None
for info, arg, pos in pickletools.genops(d):
if prevpos is not None:
yield d[prevpos:pos]
prevpos = pos
previnfo = info
yield d[prevpos:]


def tocode(dotted):
parts = tuple(dotted.rsplit(".", 1))
return "c%s\n%s\n" % parts

def rename_globals(d, pairs):
updates = dict((tocode(old), tocode(new)) for old, new in pairs)
return "".join(updates.get(o, o) for o in ops(d))

$ cat alpha.py
class A(object):
def __init__(self, x, y):
self.x = x
self.y = y
def __str__(self):
return "A(%s, %s)" % (self.x, self.y)

$ cat beta.py
class B(object):
def __str__(self):
return "B(%s, %s)" % (self.x, self.y)

$ cat demo.py
import alpha
import fixpickle
import pickle

if __name__ == "__main__":
a = alpha.A(1, 2)
print a
d = pickle.dumps(a)
d = fixpickle.rename_globals(d, [("alpha.A", "beta.B")])
b = pickle.loads(d)
print b

Peter
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,904
Latest member
HealthyVisionsCBDPrice

Latest Threads

Top