Infinite loops and synchronization

[Vincent Gulinao]

lst = list()

(lst populated by async twisted deferred callbacks)

while True:
if len(lst) == SOME_NUMBER:
return lst

Q1: is this a common OK practice? I'm worried infinite loops hogs memory.
Q2: operating on list from threads (mostly appends) must be safe,
right (synchronization)?

 

[Piet van Oostrum]

VG> lst = list()
VG> (lst populated by async twisted deferred callbacks)

VG> while True:
VG> if len(lst) == SOME_NUMBER:
VG> return lst

VG> Q1: is this a common OK practice? I'm worried infinite loops hogs memory.
VG> Q2: operating on list from threads (mostly appends) must be safe,
VG> right (synchronization)?

I am not familiar enough with twisted, but I think the principle is
independent from twisted.

This loop will not hog memory but it will hog CPU time.

You should use a synchronisation construct like threading.Condition or a
Semaphore. Here is my suggestion with a Condition:

Global somewhere:
lst_cond = Condition()

In your loop:

lst = list() # Why not lst = []?

while True: # strange while/if combo
if len(lst) == SOME_NUMBER:
return lst

Make that:

with lst_cond:
while len(lst) < SOME_NUMBER:
lst_cond.wait()
return lst

In the callback:

with lst_cond:
lst.append(new_value)
lst_cond.notify()

In case you don't have a python that supports the with statement (not
even `from future') you should use:

lst_cond.acquire()
try:
.....
finally:
lst_cond.release()

I think the solution with a semaphore is less elegant.

global: sem = Semaphore()

loop:
for i in range(SOME_NUMBER):
sem.acquire()
return lst

In callback:

lst.append(new_value)
sem.release()

*Be careful: I haven't tested this code (not even syntax checked). So
consider it pseudo code.*

 

[pdpi]

lst = list()

(lst populated by async twisted deferred callbacks)

while True:
        if len(lst) == SOME_NUMBER:
                return lst

Q1: is this a common OK practice? I'm worried infinite loops hogs memory.
Q2: operating on list from threads (mostly appends) must be safe,
right (synchronization)?

Q1: I'll answer your question with another. What's so fundamentally
different between your infinite loop and this one:

while len(lst) != SOME_NUMBER:
pass
return lst

which is not an "infinite loop"[1]. Why would yours be any worse in
terms of memory than mine? Are you allocating anything that would hog
memory? Of course, like Piet said, it *will* hog your CPU, so you want
a time.sleep(.1) in there, at the least. Of course, the question is:
why aren't you using a semaphore to let you know you can proceed, and
make the provider increment the semaphore?

[1] -- well, it can be, if len(lst) == SOME_NUMBER never comes about,
and I'd hazard a guess that that's pretty much where your fear of
memory hogging comes from: it's easy to allocate stuff and not
deallocate it within a cycle, only to find the bounds on that cycle
going awry.

 

[Lawrence D'Oliveiro]

Q1: is this a common OK practice? I'm worried infinite loops hogs memory.

The problem is not that the loop is infinite, but that it busy-waits,
hogging CPU.

 

[elshwaaf]

ÈÓã Çááå ÇáÑÍãä ÇáÑÍíã
æÈå äÓÊÚíä

 

[Aahz]

lst = list()

while True:
if len(lst) == SOME_NUMBER:
return lst

Q2: operating on list from threads (mostly appends) must be safe,
right (synchronization)?

What do you mean by "safe"? Python certainly won't crash, but there's
no guarantee that the list will be consistent from *your* perspective.
Consider what happens if len(lst) == SOME_NUMBER - 1 and some other part
of your code adds two elements to lst. You'll skip right over your if
condition.