List comprehension vs generator expression memory allocation

[candide]

Let's code a function allowing access to the multiples of a given
integer (say m) in the range from a to b where a and b are two given
integers. For instance, with data input

a,b,m=17, 42, 5

the function allows access to :

20 25 30 35 40


Each of the following two functions mult1() and mult2() solves the
question :


# -----------------------------------------
def mult1(a,b,m):
return (x for x in range(a,b)[(m-a%m)%m:b:m])

def mult2(a,b,m):
return range(a,b)[(m-a%m)%m:b:m]
# -----------------------------------------


mult2() function returns a list and obviously mult2() needs Python to
allocate memory for this list. What I was wondering is if the same might
be said about mult1(). More precisely, does Python allocate memory for
the whole target list range(a,b)[m-a%m:b:m]?

 

[Jon Clements]

Let's code a function allowing access to the multiples of a given
integer (say m) in the range from a to b where a and b are two given
integers. For instance, with data input

a,b,m=17, 42, 5

the function allows access to :

20 25 30 35 40

Each of the following two functions mult1() and mult2() solves the
question :

# -----------------------------------------
def mult1(a,b,m):
    return (x for x in range(a,b)[(m-a%m)%m:b:m])

def mult2(a,b,m):
    return range(a,b)[(m-a%m)%m:b:m]
# -----------------------------------------

mult2() function returns a list and obviously mult2() needs Python to
allocate memory for this list. What I was wondering is if the same might
be said about mult1(). More precisely, does Python allocate memory for
the whole target list range(a,b)[m-a%m:b:m]?

Yes - range always creates a list (in the 2.x series) -- mult1 creates
a list, then returns a generator, so list(mult1(a,b,m)) is identical
to mult2(a,b,m).

Jon.

 

[Dave Angel]

Let's code a function allowing access to the multiples of a given
integer (say m) in the range from a to b where a and b are two given
integers. For instance, with data input

a,b,m, 42, 5

the function allows access to :

20 25 30 35 40

Each of the following two functions mult1() and mult2() solves the
question :

# -----------------------------------------
def mult1(a,b,m):
return (x for x in range(a,b)[(m-a%m)%m:b:m])

def mult2(a,b,m):
return range(a,b)[(m-a%m)%m:b:m]
# -----------------------------------------

mult2() function returns a list and obviously mult2() needs Python to
allocate memory for this list. What I was wondering is if the same might
be said about mult1(). More precisely, does Python allocate memory for
the whole target list range(a,b)[m-a%m:b:m]?

Yes - range always creates a list (in the 2.x series) -- mult1 creates
a list, then returns a generator, so list(mult1(a,b,m)) is identical
to mult2(a,b,m).

Jon.

First, you're clearly using python 2.x, since in version 3, you'd get an
error in each of the definitions, doing the slice on a generator.

Both of those functions create the identical list (after first creating
a bigger one!). Then mult1() creates a generator object that owns the
list, and passes back the generator object. The list isn't freed till
the calling code releases the generator object.

The code in mult2() just passes the list back directly, and it'll get
freed when the calling code stops using it (eg. reuses the attribute
which references it).

So mult1() adds extra complexity without saving any memory. If you want
to save memory, try (not thoroughly tested):

def mult3(a, b, m):
start = m*((a-1)/m + 1)
end = m*((b-1)/m + 1)
return xrange(start, end, m)

 

[candide]

Duncan Booth a écrit :

Why are you slicing the result of range? Why not just pass appropriate
arguments to range or xrange directly?

Why ? Guilty ignorance

def f(a,b,m):
return xrange((a+m-1)//m*m, b, m)

Nice code, furthermore giving the best execution time, thanks.