Writing multiple files with with-context

[Shunichi Wakabayashi]

To write onto multiple files on the same time (a number of files are variable),
I'd like to code as follows, for example, IF I can do,

LIST_LEN = 4
with [ open('list_%d.txt' % i, 'w') for i in range(LIST_LEN) ] as fobjlist:
for i in range(1000):
fobjlist[random.randrange(LIST_LEN)].write(str(i)+"\n")

or using a dict object,

DICT_KEYS = ('foo', 'bar', 'baz')
with { k: open('dict_%s.txt' % k, 'w') for k in DICT_KEYS } as fobjdict:
for i in range(1000):
fobjdict[random.choice(DICT_KEYS)].write(str(i)+"\n")

However, list and dict don't has __exit__ method and so they cannot run.
One idea is using contextlib.nested(),

from contextlib import nested

with nested(*[open('list_%d.txt' % i, 'w') for i in range(LIST_LEN)]) as fobjlist:
for i in range(1000):
fobjlist[random.randrange(LIST_LEN)].write(str(i)+"\n")

with nested(*[open('dict_%s.txt' % k, 'w') for k in DICT_KEYS]) as fobjlist:
fobjdict = dict(zip(DICT_KEYS, fobjlist)) #convert list to dict
for i in range(1000):
fobjdict[random.choice(DICT_KEYS)].write(str(i)+"\n")

On Python2.x, this is OK. but 3.x warns that nested() is deprecated.
Moreover, on using dict, it is required to convert list to dict.

Another idea is to make container classes having __exit__() myself.

class MyList(list):
def __enter__(self):
return [ v.__enter__() for v in self ]
def __exit__(self, exc_type, exc_value, traceback):
ret = False
for v in self:
if v.__exit__(exc_type, exc_value, traceback):
ret = True
exc_type = exc_value = traceback = None
return ret

class MyDict(dict):
def __enter__(self):
return { k: v.__enter__() for k, v in self.items() }
def __exit__(self, exc_type, exc_value, traceback):
ret = False
for v in self.values():
if v.__exit__(exc_type, exc_value, traceback):
ret = True
exc_type = exc_value = traceback = None
return ret

with MyList( open('list_%d.txt' % i, 'w') for i in range(LIST_LEN) ) as fobjlist:
for i in range(1000):
fobjlist[random.randrange(LIST_LEN)].write(str(i)+"\n")

with MyDict( (k, open('dict_%s.txt' % k, 'w')) for k in DICT_KEYS ) as fobjdict:
for i in range(1000):
fobjdict[random.choice(DICT_KEYS)].write(str(i)+"\n")

I think this is smarter a little than others,
but it cannot guaranteed to call __exit__() of members in containers
if members are changed during with-context.

So, do you have another, more smart and pythonic way?

Thanks

 

[Ian Kelly]

One idea is using contextlib.nested(),

from contextlib import nested

with nested(*[open('list_%d.txt' % i, 'w') for i in range(LIST_LEN)]) as fobjlist:
 for i in range(1000):
   fobjlist[random.randrange(LIST_LEN)].write(str(i)+"\n")

with nested(*[open('dict_%s.txt' % k, 'w') for k in DICT_KEYS]) as fobjlist:
 fobjdict = dict(zip(DICT_KEYS, fobjlist)) #convert list to dict
 for i in range(1000):
   fobjdict[random.choice(DICT_KEYS)].write(str(i)+"\n")

On Python2.x, this is OK. but 3.x warns that nested() is deprecated.

Not merely deprecated. It has already been removed in 3.2.

Another idea is to make container classes having __exit__() myself.

class MyList(list):
 def __enter__(self):
   return [ v.__enter__() for v in self ]
 def __exit__(self, exc_type, exc_value, traceback):
   ret = False
   for v in self:
     if v.__exit__(exc_type, exc_value, traceback):
       ret = True
       exc_type = exc_value = traceback = None
   return ret

This has a number of subtle bugs in it:

1) Each context manager's __exit__ method is not loaded before the
corresponding __enter__ method is invoked.
2) If the second context manager's __enter__ method raises an
exception, the first context manager's __exit__ method is never
called, breaking the with statement guarantee.
3) The __exit__ methods are called in the same order that the
__enter__ methods were called. Since they form a stack, they should
be called in the reverse order.

These highlight the complexity of handling context managers correctly,
which I think suggests that a custom implementation is probably a bad
idea.

So, do you have another, more smart and pythonic way?

Copy the implementation of contextlib.nested to your own custom module
and use that. The last revision prior to its removal is here:

http://hg.python.org/cpython/file/45506be44514/Lib/contextlib.py