huge slowdown on append

[ianaré]

Hey all,

I have a for loop which included the line:

items_ren.append(join(newPath,renamedItem))

I changed it to this:
items_ren.append([join(newPath,renamedItem), False])

And processing speed is now much much slower. For 5780 items the old
function would take 9.5 seconds (there is other stuff going on
obviously), after changing that single line, speed is now 55 seconds
in the same conditions!

Can anyone give me some pointers please? I would like to keep the same
structure if possible, as it allows the code in other places to be
much faster. TIA

BTW, the 'join' function is this:
def join(newPath,renamedItem):
return unicode(os.path.join(newPath,renamedItem))

 

[ianaré]

Hey all,

I have a for loop which included the line:

items_ren.append(join(newPath,renamedItem))

I changed it to this:
items_ren.append([join(newPath,renamedItem), False])

And processing speed is now much much slower. For 5780 items the old
function would take 9.5 seconds (there is other stuff going on
obviously), after changing that single line, speed is now 55 seconds
in the same conditions!

Can anyone give me some pointers please? I would like to keep the same
structure if possible, as it allows the code in other places to be
much faster. TIA

BTW, the 'join' function is this:
def join(newPath,renamedItem):
return unicode(os.path.join(newPath,renamedItem))

Seems like I found a way ... Use the old way, then at the end of the
for loop add the 'False'.
def addStatus(x):
return (x,False)
items_ren = map(addStatus, items_ren)

9.5 seconds wOOt!

However, if anyone has a faster way, let me know!

 

[Fredrik Lundh]

I changed it to this:
items_ren.append([join(newPath,renamedItem), False])

And processing speed is now much much slower. For 5780 items the old
function would take 9.5 seconds (there is other stuff going on
obviously), after changing that single line, speed is now 55 seconds
in the same conditions!

have you checked the process size? short lists are relatively expensive
to create, both performance-wise, and more importantly, memory-wise.
try using tuples instead.

or better, move the flags to a separate data structure (either a list,
or, if possible, a set or dictionary keyed on filenames).

</F>

 

[Peter Otten]

Hey all,

I have a for loop which included the line:

items_ren.append(join(newPath,renamedItem))

I changed it to this:
items_ren.append([join(newPath,renamedItem), False])

And processing speed is now much much slower. For 5780 items the old
function would take 9.5 seconds (there is other stuff going on
obviously), after changing that single line, speed is now 55 seconds
in the same conditions!

Can anyone give me some pointers please? I would like to keep the same
structure if possible, as it allows the code in other places to be
much faster. TIA

BTW, the 'join' function is this:
def join(newPath,renamedItem):
return unicode(os.path.join(newPath,renamedItem))

Seems like I found a way ... Use the old way, then at the end of the
for loop add the 'False'.
def addStatus(x):
return (x,False)
items_ren = map(addStatus, items_ren)

9.5 seconds wOOt!

However, if anyone has a faster way, let me know!

Maybe

import itertools as it
items_ren = zip(items_ren, it.repeat(False))

Or if you want to start earlier on

def gen_items():
for ...
yield newPath, renamedItem

items_ren = zip(it.imap(unicode, it.starmap(os.path.join, gen_items())),
it.repeat(False))

Probably buggy, but you should get the idea...

Peter