Python Generators

[mpc]

HI all,
I am trying to write a while loop that will iterate over generators to
capture all the headers of FFCache directories. However, the
generators embedded within the argument of another generator do not
seem to re-initiate. the example below loops through and initiates the
generator embedded in the argument only once. Can anyone explain while
the generator will not re-initiate, and suggest a simple fix?

#!/usr/bin/env python
import os,struct,time
def generate_headers(cachedir):
for name, blocksize in cachefiles:
pathname = os.path.join(cachedir,name)
f = open(pathname,"rb")
f.seek(4096)
while True:
header = f.read(36)
if not header: break
fields = struct.unpack(">9I",header)
if fields[0] == 0x00010008: yield f, fields
fp = f.tell()
offset = fp % blocksize
if offset:
f.seek(blocksize - offset,1)
f.close()
def concatenate(sequences):
for seq in sequences:
for item in seq:
yield item

all_caches = (path for path,dirlist,filelist in os.walk("/Users/") if
'_CACHE_MAP_' in filelist)
cachefiles = [('_CACHE_001_',256),('_CACHE_002_',1024),('_CACHE_003_',
4096)]
n = 0
while True:
n += 1
time.sleep(0.5)
headers = concatenate(generate_headers(path) for path in
all_caches)
for h in headers:
print h,n
# this doesn't work either

while True:
n += 1
time.sleep(0.5)
for path in all_caches:
headers = generate_headers(path)
for h in headers:
print h,n
# however if I hard code the path
path = "FFCache"
while True:
n += 1
headers = generate_headers(path)
for h in headers:
print h,n
#but of course i do not wish to hard code the path.

 

[sturlamolden]

generator embedded in the argument only once. Can anyone explain while
the generator will not re-initiate, and suggest a simple fix?

I am not sure what you are trying to do, but it seems a bit confused.

for s in seq: yield s
h = concat(s for s in seq)
for i in h: print i,n


0 0
1 0
2 0
0 1
1 1
2 1
0 2
1 2
2 2
0 3
1 3
2 3
0 4
1 4
2 4

Generators work they way they should. Even when one is used as
argument for another.

 

[Peter Otten]

I am trying to write a while loop that will iterate over generators to
capture all the headers of FFCache directories. However, the
generators embedded within the argument of another generator do not
seem to re-initiate. the example below loops through and initiates the
generator embedded in the argument only once. Can anyone explain while
the generator will not re-initiate, and suggest a simple fix?

A generator or generator expression does indeed only run once.
.... print "---", i, "---"
.... for k in gen: print k,
.... print
....
--- 0 ---
1 3
--- 1 ---

--- 2 ---

The fix is to use a list or list comprehension, or make a new generator
every time you need one:
.... print "---", i, "---"
.... gen = (i for i in range(5) if i%2)
.... for k in gen: print k,
.... print
....
--- 0 ---
1 3
--- 1 ---
1 3
--- 2 ---
1 3

At first glance I would guess that in your case all_caches is the culprit
that has to be moved into the 'while True: ...' loop.

while True:
n += 1
time.sleep(0.5)

all_caches = (path for path,dirlist,filelist in os.walk("/Users/") if
'_CACHE_MAP_' in filelist)

for path in all_caches:
headers = generate_headers(path)
for h in headers:
print h,n

(Untested, because you didn't bother to provide a self-contained example)

Peter

 

[Peter Otten]

I am trying to write a while loop that will iterate over generators to
capture all the headers of FFCache directories. However, the
generators embedded within the argument of another generator do not
seem to re-initiate. the example below loops through and initiates the
generator embedded in the argument only once. Can anyone explain while
the generator will not re-initiate, and suggest a simple fix?

A generator or generator expression does indeed only run once.
.... print "---", i, "---"
.... for k in gen: print k,
.... print
....
--- 0 ---
1 3
--- 1 ---

--- 2 ---

The fix is to use a list or list comprehension, or make a new generator
every time you need one:
.... print "---", i, "---"
.... gen = (i for i in range(5) if i%2)
.... for k in gen: print k,
.... print
....
--- 0 ---
1 3
--- 1 ---
1 3
--- 2 ---
1 3

At first glance I would guess that in your case all_caches is the culprit
that has to be moved into the 'while True: ...' loop.

while True:
n += 1
time.sleep(0.5)

all_caches = (path for path,dirlist,filelist in os.walk("/Users/") if
'_CACHE_MAP_' in filelist)

for path in all_caches:
headers = generate_headers(path)
for h in headers:
print h,n

(Untested, because you didn't bother to provide a self-contained example)

Peter

 

[Matt Nordhoff]

mpc wrote:

def concatenate(sequences):
for seq in sequences:
for item in seq:
yield item

You should check out itertools.chain(). It does this. You call it like
"chain(seq1, seq2, ...)" instead of "chain(sequences)" though, which may
be a problem for you.

The rest of itertools might be interesting too:

<http://docs.python.org/lib/module-itertools.html>

<snip>
--

 

[Marius Gedminas]

You should check out itertools.chain(). It does this. You call it like
"chain(seq1, seq2, ...)" instead of "chain(sequences)" though, which may
be a problem for you.

Solved rather easily by chain(*sequences):
... return chain(*sequences)
...

concat([[1,2], [3, 4], [5], [6, 7, 8]])

list(concat([[1,2], [3, 4], [5], [6, 7, 8]]))

[1, 2, 3, 4, 5, 6, 7, 8]


wondering if google groups will add a .sig or not-ly,
Marius Gedminas