Overriding iadd for dictionary like objects

[RunThePun]

I'd like to build a database wrapper using DictMixin and allow items
to be appended by my own code. The problem is += is always understood
as setitem and getitem plainly.

d = MyDict()
d['a'] = 1

# this is the problem code that's I'd like to override. It's always
setitem('a', getitem('a') + 3)
d['a'] += 3
# i wanted to do something like my own 'appenditem' function which for
example could be useful if getitem is an expensive operation which can
be avoided.

I hope that was clear enough of a request, it's really late at night
here...

thanks,

RunPun

 

[Robert Kern]

I'd like to build a database wrapper using DictMixin and allow items
to be appended by my own code. The problem is += is always understood
as setitem and getitem plainly.

d = MyDict()
d['a'] = 1

# this is the problem code that's I'd like to override. It's always
setitem('a', getitem('a') + 3)
d['a'] += 3
# i wanted to do something like my own 'appenditem' function which for
example could be useful if getitem is an expensive operation which can
be avoided.

I hope that was clear enough of a request, it's really late at night
here...

I'm sorry, this is just part of the syntax of Python. You cannot override it.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Jan Kaliszewski]

I'd like to build a database wrapper using DictMixin and allow items
to be appended by my own code. The problem is += is always understood
as setitem and getitem plainly.

d = MyDict()
d['a'] = 1

# this is the problem code that's I'd like to override. It's always
setitem('a', getitem('a') + 3)
d['a'] += 3
# i wanted to do something like my own 'appenditem' function which for
example could be useful if getitem is an expensive operation which can
be avoided.

I hope that was clear enough of a request, it's really late at night
here...

I'm sorry, this is just part of the syntax of Python. You cannot
override it.

Though
d['a'] = 3
is equivalent to:
d.__setitem__('a', 3)

The
d['a'] += 3
*is not* equivalent to:
d.__setitem__('a', d.__getitem__('a') + 3)
*but is* equivalent to:
d.__getitem__('a').__iadd__(3)

Then you can override __getitem__() of MyDict in such a way that it
returns prepared (wrapped) object with overriden __iadd__() as you
want to.

How could I now it:

1 import collections
2 import functools
3 import itertools
4
5
6 def verbose_func(func):
7 'Function decorator that makes a function "verbose"'
8
9 @functools.wraps(func, assigned=('__name__', '__doc__'))
10 def func_wrapper(*args, **kwargs):
11 iargs = (map(str, args))
12 ikwargs = ('{0}={1}'.format(key, value)
13 for key, value in kwargs.items())
14 func_args = ', '.join(itertools.chain(iargs, ikwargs))
15 print('{0}({1})'.format(func.__name__, func_args))
16 return func(*args, **kwargs)
17
18 return func_wrapper
19
20
21 def verbose_cls(base):
22 'Class decorator that makes callable attributes "verbose"'
23
24 quiet = ('__new__', '__repr__', '__str__')
25
26 def cls_wrapper(cls):
27 for name in vars(base):
28 attr = getattr(cls, name)
29 if isinstance(attr, collections.Callable) and name not in
quiet:
30 setattr(cls, name, verbose_func(attr))
31 return cls
32
33 return cls_wrapper
34
35
36 @verbose_cls(dict)
37 class VerboseDict(dict):
38 pass
39
40
41 @verbose_cls(int)
42 class MyInt(int):
43
44 @verbose_func
45 def __iadd__(self, other):
46 int.__add__(self, other) # can do something more interesting
47
48
49 if __name__ == '__main__':
50 d = VerboseDict()
51
52 print("d['a'] = 3")
53 d['a'] = MyInt(3)
54
55 print("d['a'] += 3")
56 d['a'] += MyInt(3)

*j

 

[Jan Kaliszewski]

I'd like to build a database wrapper using DictMixin and allow items
to be appended by my own code. The problem is += is always understood
as setitem and getitem plainly.

d = MyDict()
d['a'] = 1

# this is the problem code that's I'd like to override. It's always
setitem('a', getitem('a') + 3)
d['a'] += 3
# i wanted to do something like my own 'appenditem' function which for
example could be useful if getitem is an expensive operation which can
be avoided.

I hope that was clear enough of a request, it's really late at night
here...

I'm sorry, this is just part of the syntax of Python. You cannot
override it.

Though
d['a'] = 3
is equivalent to:
d.__setitem__('a', 3)

The
d['a'] += 3
*is not* equivalent to:
d.__setitem__('a', d.__getitem__('a') + 3)
*but is* equivalent to:
d.__setitem__('a', d.__getitem__('a').__iadd__(3))

Then you can override __getitem__() of MyDict in such a way that it
returns prepared (wrapped) object with overriden __iadd__() as you
want to.

How could I now it:

1 import collections
2 import functools
3 import itertools
4
5
6 def verbose_func(func):
7 'Function decorator that makes a function "verbose"'
8
9 @functools.wraps(func, assigned=('__name__', '__doc__'))
10 def func_wrapper(*args, **kwargs):
11 iargs = (map(str, args))
12 ikwargs = ('{0}={1}'.format(key, value)
13 for key, value in kwargs.items())
14 func_args = ', '.join(itertools.chain(iargs, ikwargs))
15 print('{0}({1})'.format(func.__name__, func_args))
16 return func(*args, **kwargs)
17
18 return func_wrapper
19
20
21 def verbose_cls(base):
22 'Class decorator that makes callable attributes "verbose"'
23
24 quiet = ('__new__', '__repr__', '__str__')
25
26 def cls_wrapper(cls):
27 for name in vars(base):
28 attr = getattr(cls, name)
29 if isinstance(attr, collections.Callable) and name not in
quiet:
30 setattr(cls, name, verbose_func(attr))
31 return cls
32
33 return cls_wrapper
34
35
36 @verbose_cls(dict)
37 class VerboseDict(dict):
38 pass
39
40
41 @verbose_cls(int)
42 class MyInt(int):
43
44 @verbose_func
45 def __iadd__(self, other):
46 int.__add__(self, other) # can do something more interesting
47
48
49 if __name__ == '__main__':
50 d = VerboseDict()
51
52 print("d['a'] = 3")
53 d['a'] = MyInt(3)
54
55 print("d['a'] += 3")
56 d['a'] += MyInt(3)

*j

 

[Robert Kern]

I'd like to build a database wrapper using DictMixin and allow items
to be appended by my own code. The problem is += is always understood
as setitem and getitem plainly.

d = MyDict()
d['a'] = 1

# this is the problem code that's I'd like to override. It's always
setitem('a', getitem('a') + 3)
d['a'] += 3
# i wanted to do something like my own 'appenditem' function which for
example could be useful if getitem is an expensive operation which can
be avoided.

I hope that was clear enough of a request, it's really late at night
here...

I'm sorry, this is just part of the syntax of Python. You cannot
override it.

Though
d['a'] = 3
is equivalent to:
d.__setitem__('a', 3)

The
d['a'] += 3
*is not* equivalent to:
d.__setitem__('a', d.__getitem__('a') + 3)
*but is* equivalent to:
d.__getitem__('a').__iadd__(3)

Then you can override __getitem__() of MyDict in such a way that it
returns prepared (wrapped) object with overriden __iadd__() as you
want to.

You could, but then you will almost certainly run into problems using the
wrapped object in places that really expect the true object.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[RunThePun]

On 2009-08-26 17:16 PM, RunThePun wrote:
I'd like to build a database wrapper using DictMixin and allow items
to be appended by my own code. The problem is += is always understood
as setitem and getitem plainly.
d = MyDict()
d['a'] = 1
# this is the problem code that's I'd like to override. It's always
setitem('a', getitem('a') + 3)
d['a'] += 3
# i wanted to do something like my own 'appenditem' function which for
example could be useful if getitem is an expensive operation which can
be avoided.
I hope that was clear enough of a request, it's really late at night
here...
I'm sorry, this is just part of the syntax of Python. You cannot
override it.

Though
d['a'] = 3
is equivalent to:
d.__setitem__('a', 3)

The
d['a'] += 3
*is not* equivalent to:
d.__setitem__('a', d.__getitem__('a') + 3)
*but is* equivalent to:
d.__getitem__('a').__iadd__(3)

Then you can override __getitem__() of MyDict in such a way that it
returns prepared (wrapped) object with overriden __iadd__() as you
want to.

You could, but then you will almost certainly run into problems using the
wrapped object in places that really expect the true object.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

Exactly my problem Robert. I'm actually going to be using this dict
for storing and retrieving strings so wrapping str and replacing the
iadd would cause alot of craziness.

Anybody have any more ideas? I think python should/could havev a
syntax for overriding this behaviour, i mean, obviously the complexity
of supporting all operators with the getitem syntax could introduce
alot of clutter. But maybe there's an elegant solution out there...

---RP

 

[Robert Kern]

Anybody have any more ideas? I think python should/could havev a
syntax for overriding this behaviour, i mean, obviously the complexity
of supporting all operators with the getitem syntax could introduce
alot of clutter. But maybe there's an elegant solution out there...

I would recommend just adding a method to MyDict that does exactly what you want.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Aahz]

Anybody have any more ideas? I think python should/could havev a
syntax for overriding this behaviour, i mean, obviously the complexity
of supporting all operators with the getitem syntax could introduce
alot of clutter. But maybe there's an elegant solution out there...

Any solution you create must support this:

L = [3]
L[0] += 7
L

[10]

 

[Carl Banks]

Anybody have any more ideas? I think python should/could havev a
syntax for overriding this behaviour, i mean, obviously the complexity
of supporting all operators with the getitem syntax could introduce
alot of clutter. But maybe there's an elegant solution out there...

I don't think it needs a syntax for that, but I'm not so sure a method
to modify a value in place with a single key lookup wouldn't
occasioanally be useful.

For instance:

def increment(value):
return value+1
d = { 'a': 1 }
d.apply_to_value('a',increment)
print d

and d['a'] would be 2. The difference is that only one lookup
occurs. (The dictionary would be locked to modifications while the
callback function is called, same as if it were being iterated over.)

Thing is, there's no way to get that ability except subclass dict in
C; can't be done from Python. Which is why even though it'd be a
pretty rare need it at least deserves a bit of consideration before
being tabled.

As a workaround, if lookups are expensive, you can add an indirection
to the dict. For example, you could let each value in the dict be a
singleton list which is permanently bound to the key. Then you only
have look up the key once to modify the value.

d = { 'a': [1] }
ston = d['a']
ston[0] += 3

and now d['a'][0] is 4, and there was only one dict lookup. Downside
is that you have the [0] to carry around, but that is the price of
indirection. A wrapper class might help.


Carl Banks

 

[Terry Reedy]

I don't think it needs a syntax for that, but I'm not so sure a method
to modify a value in place with a single key lookup wouldn't
occasioanally be useful.

Augmented assignment does that.

For instance:

def increment(value):
return value+1
d = { 'a': 1 }
d.apply_to_value('a',increment)
print d

and d['a'] would be 2. The difference is that only one lookup
occurs.

Like this?

>>> d={'a': 2}
>>> d['a'] += 2
>>> d['a']

4

This does not cover all replacements, but certainly the most common.

[snip]

As a workaround, if lookups are expensive,

But they are not. Because (C)Python is heavily based on dict name lookup
for builtins and global names and attributes, as well as overt dict
lookup, must effort has gone into optimizing dict lookup.

> you can add

something even slower ;-). In particular, a method lookup + method
call, as you suggest above.

One can always avoid calculating the key object twice if that is expensive.

Terry Jan Reedy

 

[Robert Kern]

Augmented assignment does that.

No, it uses one __getitem__ and one __setitem__ thus two key lookups.

For instance:

def increment(value):
return value+1
d = { 'a': 1 }
d.apply_to_value('a',increment)
print d

and d['a'] would be 2. The difference is that only one lookup
occurs.

Like this?

d={'a': 2}
d['a'] += 2
d['a']

4

This does not cover all replacements, but certainly the most common.

Take look farther up in the thread for the actual point at issue. The OP knows
that augmented assignment works for the common cases. He wants to override the
behavior on the container to handle some uncommon cases, and this is not
possible because Python breaks up the operation into three orthogonal actions.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Carl Banks]

Augmented assignment does that.

Internally uses two lookups, one for getting, and one for setting. I
think this is an unavoidable given Python's semantics. Look at the
traceback:

.... d['a'] += 1
.... 2 0 LOAD_GLOBAL 0 (d)
3 LOAD_CONST 1 ('a')
6 DUP_TOPX 2
9 BINARY_SUBSCR
10 LOAD_CONST 2 (1)
13 INPLACE_ADD
14 ROT_THREE
15 STORE_SUBSCR
16 LOAD_CONST 0 (None)
19 RETURN_VALUE

But they are not. Because (C)Python is heavily based on dict name lookup
for builtins and global names and attributes, as well as overt dict
lookup, must effort has gone into optimizing dict lookup.

The actual lookup algorithm Python dicts use is well-optimized, yes,
but the dict could contain keys that have expensive comparison and
hash-code calculation, in which case lookup is going to be slow.


Carl Banks

 

[Steven D'Aprano]

Apparently you're defining "key lookup" some other way than as `what
__getitem__ does'.

What exactly does "key lookup" mean to you?

I've always understood it as `retrieving the value associated with a
key', which obviously isn't required for assignment--otherwise it
wouldn't be possible to add new keys to a mapping.

When you retrieve a value from a dictionary using __getitem__, e.g.:

dict["K"]

the dict has to search the hash table for the record with key "K". This a
key lookup.

(I use the term "search", but of course for hash tables this is usually
very fast. For Python dicts, you can assume it will usually be a constant
time, independent of the key or the size of the dict.)

When you store a value in a dictionary using __setitem__, e.g.:

dict["K"] = 42

the dict has to search the hash table for the correct place to store a
record with key "K". It obviously can't place the record in some
arbitrary place, it has to be in the correct place for future lookups to
find it. This is also a key lookup.

 

[Carl Banks]

Internally uses two lookups, one for getting, and one for setting.

I think this is an unavoidable given Python's semantics.  Look at
the traceback:

def x():

...     d['a'] += 1
...

dis.dis(x)

  2           0 LOAD_GLOBAL              0 (d)
              3 LOAD_CONST               1 ('a')
              6 DUP_TOPX                 2
              9 BINARY_SUBSCR

OK, there's one lookup, but...

             10 LOAD_CONST               2 (1)
             13 INPLACE_ADD
             14 ROT_THREE
             15 STORE_SUBSCR
             16 LOAD_CONST               0 (None)
             19 RETURN_VALUE

... I don't see anything in there that retrieves the value a second time.....

STORE_SUBSCR has to look up the position in the hash table to store
the value, hence the second lookup.

I'll like the originator correct me if I've made a mistake, but I read
"lookup" as actually meaning "lookup", not "value-comparison".

This has nothing to do with value comparison. I was talking about key
comparison, which happens when looking up a position in a hash table.
I was the first person to use the word "lookup" in this thread and I
specifically meant hash-table position lookup.

At least in part because the question, as it was posed, specifically
related to a wrapper-class (providing a mapping ("dict like") interface)
around a database of some sort other than Python's dict class per se.

How do the details of Python's native dict-type's internal (hashtable)
algorithm matter when they're explicitly /not/ being used?

Well it doesn't apply specifically to the OP's problem. I changed the
topic a bit by making it specific to dicts. Is that ok with you? Was
that not allowed?

The OP can add a method like apply_to_value to his own class, but one
can't do that for dicts. Ergo why something like apply_to_value()
would be useful enough in rare circumstances where lookup is very slow
to merit a moments consideration before being rejected.

(If dict did have a method like that, the OP would at least know which
method to override.)


Carl Banks

 

[RunThePun]

Carl Banks wrote:
I don't think it needs a syntax for that, but I'm not so sure a method
to modify a value in place with a single key lookup wouldn't
occasioanally be useful.
Augmented assignment does that.
Internally uses two lookups, one for getting, and one for setting.
I think this is an unavoidable given Python's semantics.  Look at
the traceback:
def x():
...     d['a'] += 1
...
dis.dis(x)
  2           0 LOAD_GLOBAL              0 (d)
              3 LOAD_CONST               1 ('a')
              6 DUP_TOPX                 2
              9 BINARY_SUBSCR

OK, there's one lookup, but...

... I don't see anything in there that retrieves the value a second time....

STORE_SUBSCR has to look up the position in the hash table to store
the value, hence the second lookup.

I'll like the originator correct me if I've made a mistake, but I read
"lookup" as actually meaning "lookup", not "value-comparison".

This has nothing to do with value comparison.  I was talking about key
comparison, which happens when looking up a position in a hash table.
I was the first person to use the word "lookup" in this thread and I
specifically meant hash-table position lookup.

At least in part because the question, as it was posed, specifically
related to a wrapper-class (providing a mapping ("dict like") interface)
around a database of some sort other than Python's dict class per se.

How do the details of Python's native dict-type's internal (hashtable)
algorithm matter when they're explicitly /not/ being used?

Well it doesn't apply specifically to the OP's problem.  I changed the
topic a bit by making it specific to dicts.  Is that ok with you?  Was
that not allowed?

The OP can add a method like apply_to_value to his own class, but one
can't do that for dicts.  Ergo why something like apply_to_value()
would be useful enough in rare circumstances where lookup is very slow
to merit a moments consideration before being rejected.

(If dict did have a method like that, the OP would at least know which
method to override.)

Carl Banks

First of all I'd like to say thanks for this discussion, you guys are
awesome.

I probably should have explained my problem better to begin with and I
apologize for that. So now I'll start from the top:

I made a DictMixin where the keys are filenames and the values are the
file contents. It was very simple and easy to do thanks to DictMixin.

For example this code writes "abc" in a file named "temp.txt" and
prints the contents of the file named "swallow", these files are
looked up/created/deleted in the directory "spam":

d = FilesDict('spam')
d['temp.txt'] = 'abc'
print(d['swallow'])

This was very convenient for me because I wanted to use a simple DB
which could be read and edited by shell scripts and non-pythonistas,
without the heavy ORM. Also, if in the future an online full featured
DB would be needed, I could easily convert the DictMixin methods. So
up to here I had a good solution.

My problem arose when I wanted to append a string to a file which
using open(..., 'ab') would have been miles more efficient because I
wouldn't have to read the entire file (__getitem__) and then write the
entire file back (__setitem__). The files are expected to be as big as
600 KB which will be appended 30 bytes at a time about 3 times a
second. Performance-wise the system would probably work without open
(..., 'ab') but it would be a real thrashing so the current solution
uses a method "AddTo" as Robert suggested, sacrificing the neat
getitem/setitem syntax.

Just so I don't leave out any information, I actually also made a
DirectoryDict for having multiple 'tables'. In this DictMixin, keys
are directory names and values are FilesDict instances. So to write
"European or African" to the file "/root/tmp/velocity" one would be
just:

d = DirectoryDict("/root")
d["tmp"]["velocity"] = "European or African"

So now I hope it's clearer why and how I wanted a special
__item_iadd__ for the dictionary syntax,

RunThePun

 

[Aahz]

I made a DictMixin where the keys are filenames and the values are the
file contents. It was very simple and easy to do thanks to DictMixin.

For example this code writes "abc" in a file named "temp.txt" and
prints the contents of the file named "swallow", these files are
looked up/created/deleted in the directory "spam":

d =3D FilesDict('spam')
d['temp.txt'] =3D 'abc'
print(d['swallow'])

My problem arose when I wanted to append a string to a file which
using open(..., 'ab') would have been miles more efficient because I
wouldn't have to read the entire file (__getitem__) and then write the
entire file back (__setitem__). The files are expected to be as big as
600 KB which will be appended 30 bytes at a time about 3 times a
second. Performance-wise the system would probably work without open
(..., 'ab') but it would be a real thrashing so the current solution
uses a method "AddTo" as Robert suggested, sacrificing the neat
getitem/setitem syntax.

You can do mostly what you want, I think, by having __setitem__()
convert string values into FileProxy() objects that have an appropriate
__iadd__() method. That brings a whole new set of problems, of course.

 

[RunThePun]

I made a DictMixin where the keys are filenames and the values are the
file contents. It was very simple and easy to do thanks to DictMixin.

For example this code writes "abc" in a file named "temp.txt" and
prints the contents of the file named "swallow", these files are
looked up/created/deleted in the directory "spam":

d =3D FilesDict('spam')
d['temp.txt'] =3D 'abc'
print(d['swallow'])

My problem arose when I wanted to append a string to a file which
using open(..., 'ab') would have been miles more efficient because I
wouldn't have to read the entire file (__getitem__) and then write the
entire file back (__setitem__). The files are expected to be as big as
600 KB which will be appended 30 bytes at a time about 3 times a
second. Performance-wise the system would probably work without open
(..., 'ab') but it would be a real thrashing so the current solution
uses a method "AddTo" as Robert suggested, sacrificing the neat
getitem/setitem syntax.

You can do mostly what you want, I think, by having __setitem__()
convert string values into FileProxy() objects that have an appropriate
__iadd__() method.  That brings a whole new set of problems, of course.

I'm guessing you meant __getitem__, which is what Jan Kaliszewski
suggested, but as you noted, would be a bit cumbersome in this case.

 

[Aahz]

I made a DictMixin where the keys are filenames and the values are the
file contents. It was very simple and easy to do thanks to DictMixin.

For example this code writes "abc" in a file named "temp.txt" and
prints the contents of the file named "swallow", these files are
looked up/created/deleted in the directory "spam":
d =3D3D FilesDict('spam')
d['temp.txt'] =3D3D 'abc'
print(d['swallow'])

My problem arose when I wanted to append a string to a file which
using open(..., 'ab') would have been miles more efficient because I
wouldn't have to read the entire file (__getitem__) and then write the
entire file back (__setitem__). The files are expected to be as big as
600 KB which will be appended 30 bytes at a time about 3 times a
second. Performance-wise the system would probably work without open
(..., 'ab') but it would be a real thrashing so the current solution
uses a method "AddTo" as Robert suggested, sacrificing the neat
getitem/setitem syntax.

You can do mostly what you want, I think, by having __setitem__()
convert string values into FileProxy() objects that have an appropriate
__iadd__() method. =A0That brings a whole new set of problems, of course.

I'm guessing you meant __getitem__, which is what Jan Kaliszewski
suggested, but as you noted, would be a bit cumbersome in this case.

Actually, what I meant was __setitem__. The idea is that you create the
proxy item when you add the data to the dict (wrapping the original
data), and the proxy has an __iadd__ method, which would allow you to do
the file append.

 

[RunThePun]

I made a DictMixin where the keys are filenames and the values are the
file contents. It was very simple and easy to do thanks to DictMixin.
For example this code writes "abc" in a file named "temp.txt" and
prints the contents of the file named "swallow", these files are
looked up/created/deleted in the directory "spam":
d =3D3D FilesDict('spam')
d['temp.txt'] =3D3D 'abc'
print(d['swallow'])
My problem arose when I wanted to append a string to a file which
using open(..., 'ab') would have been miles more efficient because I
wouldn't have to read the entire file (__getitem__) and then write the
entire file back (__setitem__). The files are expected to be as big as
600 KB which will be appended 30 bytes at a time about 3 times a
second. Performance-wise the system would probably work without open
(..., 'ab') but it would be a real thrashing so the current solution
uses a method "AddTo" as Robert suggested, sacrificing the neat
getitem/setitem syntax.
You can do mostly what you want, I think, by having __setitem__()
convert string values into FileProxy() objects that have an appropriate
__iadd__() method. =A0That brings a whole new set of problems, of course.

I'm guessing you meant __getitem__, which is what Jan Kaliszewski
suggested, but as you noted, would be a bit cumbersome in this case.

Actually, what I meant was __setitem__.  The idea is that you create the
proxy item when you add the data to the dict (wrapping the original
data), and the proxy has an __iadd__ method, which would allow you to do
the file append.

But you do mean that __getitem__ would return a wrapped object as
well, right? Otherwise I don't see how the iadd would be relevant
because:
d['a'] += 3
is equivalent to:
d.__setitem__('a', d.__getitem__('a').__iadd__(3))

 

[Aahz]

On Aug 30, 10:33=3DA0pm, (e-mail address removed) (Aahz) wrote:

.com>,
RunThePun =3DA0<[email protected]> wrote:

I made a DictMixin where the keys are filenames and the values are the
file contents. It was very simple and easy to do thanks to DictMixin.

For example this code writes "abc" in a file named "temp.txt" and
prints the contents of the file named "swallow", these files are
looked up/created/deleted in the directory "spam":
d =3D3D3D FilesDict('spam')
d['temp.txt'] =3D3D3D 'abc'
print(d['swallow'])

My problem arose when I wanted to append a string to a file which
using open(..., 'ab') would have been miles more efficient because I
wouldn't have to read the entire file (__getitem__) and then write the
entire file back (__setitem__). The files are expected to be as big as
600 KB which will be appended 30 bytes at a time about 3 times a
second. Performance-wise the system would probably work without open
(..., 'ab') but it would be a real thrashing so the current solution
uses a method "AddTo" as Robert suggested, sacrificing the neat
getitem/setitem syntax.

You can do mostly what you want, I think, by having __setitem__()
convert string values into FileProxy() objects that have an appropriat= e
__iadd__() method. =3DA0That brings a whole new set of problems, of co=

urse.

I'm guessing you meant __getitem__, which is what Jan Kaliszewski
suggested, but as you noted, would be a bit cumbersome in this case.

Actually, what I meant was __setitem__. =A0The idea is that you create th= e
proxy item when you add the data to the dict (wrapping the original
data), and the proxy has an __iadd__ method, which would allow you to do
the file append.

But you do mean that __getitem__ would return a wrapped object as
well, right? Otherwise I don't see how the iadd would be relevant
because:
d['a'] +=3D 3
is equivalent to:
d.__setitem__('a', d.__getitem__('a').__iadd__(3))

The __getitem__ returns the proxy object created by the first __setitem__
when you did

d['a'] = 'something'

The __iadd__ of the proxy object returns self, so it's the same object,
which means that the second __setitem__ (called by +=) can check to see
that you already have a proxy object and it does not need to rewrap it.

If this still doesn't make sense, I suggest you go ahead and try the
experiment to prove that it does work. ;-)