Re: Problem of function calls from map()

Discussion in 'Python' started by Dasn, Aug 22, 2006.

  1. Dasn

    Dasn Guest

    Thanks for your reply.

    Well, please drop a glance at my current profile report:

    #------------------------ test.py ---------------------
    import os, sys, profile

    print os.uname()
    print sys.version

    # size of 'dict.txt' is about 3.6M, 154563 lines
    f = open('dict.txt', 'r')
    print "Reading lines..."
    lines = f.readlines()
    print "Done."

    def splitUsing(chars):
    def tmp(s):
    return s.split(chars)
    return tmp


    def sp0(lines):
    """====> sp0() -- Normal 'for' loop"""
    l = []
    for line in lines:
    l.append(line.split('\t'))
    return l

    def sp1(lines):
    """====> sp1() -- List-comprehension"""
    return [s.split('\t') for s in lines]

    def sp2(lines):
    """====> sp2() -- Map with lambda function"""
    return map(lambda s: s.split('\t'), lines)

    def sp3(lines):
    """====> sp3() -- Map with splitUsing() function"""
    return map(splitUsing('\t'), lines)

    def sp4(lines):
    """====> sp4() -- Not correct, but very fast"""
    return map(str.split, lines)

    for num in xrange(5):
    fname = 'sp%(num)s' % locals()
    print eval(fname).__doc__
    profile.run(fname+'(lines)')

    #---------------------------End of test.py ----------------

    $ python test.py

    ('OpenBSD', 'Compaq', '3.9', 'kernel#1', 'i386')
    2.4.2 (#1, Mar 2 2006, 14:17:22)
    [GCC 3.3.5 (propolice)]
    Reading lines...
    Done.
    ====> sp0() -- Normal 'for' loop
    309130 function calls in 20.510 CPU seconds

    Ordered by: standard name

    ncalls tottime percall cumtime percall filename:lineno(function)
    154563 4.160 0.000 4.160 0.000 :0(append)
    1 0.010 0.010 0.010 0.010 :0(setprofile)
    154563 6.490 0.000 6.490 0.000 :0(split)
    1 0.380 0.380 20.500 20.500 <string>:1(?)
    0 0.000 0.000 profile:0(profiler)
    1 0.000 0.000 20.510 20.510 profile:0(sp0(lines))
    1 9.470 9.470 20.120 20.120 test.py:20(sp0)


    ====> sp1() -- List-comprehension
    154567 function calls in 12.240 CPU seconds

    Ordered by: standard name

    ncalls tottime percall cumtime percall filename:lineno(function)
    1 0.000 0.000 0.000 0.000 :0(setprofile)
    154563 6.740 0.000 6.740 0.000 :0(split)
    1 0.380 0.380 12.240 12.240 <string>:1(?)
    0 0.000 0.000 profile:0(profiler)
    1 0.000 0.000 12.240 12.240 profile:0(sp1(lines))
    1 5.120 5.120 11.860 11.860 test.py:27(sp1)


    ====> sp2() -- Map with lambda function
    309131 function calls in 20.480 CPU seconds

    Ordered by: standard name

    ncalls tottime percall cumtime percall filename:lineno(function)
    1 4.600 4.600 20.100 20.100 :0(map)
    1 0.000 0.000 0.000 0.000 :0(setprofile)
    154563 7.320 0.000 7.320 0.000 :0(split)
    1 0.370 0.370 20.470 20.470 <string>:1(?)
    0 0.000 0.000 profile:0(profiler)
    1 0.010 0.010 20.480 20.480 profile:0(sp2(lines))
    1 0.000 0.000 20.100 20.100 test.py:31(sp2)
    154563 8.180 0.000 15.500 0.000 test.py:33(<lambda>)


    ====> sp3() -- Map with splitUsing() function
    309132 function calls in 21.900 CPU seconds

    Ordered by: standard name

    ncalls tottime percall cumtime percall filename:lineno(function)
    1 5.540 5.540 21.520 21.520 :0(map)
    1 0.000 0.000 0.000 0.000 :0(setprofile)
    154563 7.100 0.000 7.100 0.000 :0(split)
    1 0.380 0.380 21.900 21.900 <string>:1(?)
    0 0.000 0.000 profile:0(profiler)
    1 0.000 0.000 21.900 21.900 profile:0(sp3(lines))
    1 0.000 0.000 0.000 0.000 test.py:14(splitUsing)
    154563 8.880 0.000 15.980 0.000 test.py:15(tmp)
    1 0.000 0.000 21.520 21.520 test.py:35(sp3)


    ====> sp4() -- Not correct, but very fast
    5 function calls in 3.090 CPU seconds

    Ordered by: standard name

    ncalls tottime percall cumtime percall filename:lineno(function)
    1 2.660 2.660 2.660 2.660 :0(map)
    1 0.000 0.000 0.000 0.000 :0(setprofile)
    1 0.430 0.430 3.090 3.090 <string>:1(?)
    0 0.000 0.000 profile:0(profiler)
    1 0.000 0.000 3.090 3.090 profile:0(sp4(lines))
    1 0.000 0.000 2.660 2.660 test.py:39(sp4)


    The problem is the default behavior of str.split should be more complex
    than str.split('\t'). If we could use the str.split('\t') in map(), the
    result would be witty. What do u guys think?
     
    Dasn, Aug 22, 2006
    #1
    1. Advertising

  2. Dasn

    Peter Otten Guest

    Dasn wrote:

    > # size of 'dict.txt' is about 3.6M, 154563 lines
    > f = open('dict.txt', 'r')


    > lines = f.readlines()


    > def sp0(lines):
    >         """====> sp0() -- Normal 'for' loop"""
    >         l = []
    >         for line in lines:
    >                 l.append(line.split('\t'))
    >         return l


    Where do you get the data from in the "real world"? If it's a file you might
    get the best overall performance if you skip the readlines() call:

    sp0(f)

    Anyway, here's another variant you can try:

    from itertools import izip, starmap, repeat

    def splitlines(lines):
    return list(starmap(str.split, izip(lines, repeat("\t"))))

    Peter
     
    Peter Otten, Aug 22, 2006
    #2
    1. Advertising

  3. Dasn <> wrote:
    ># size of 'dict.txt' is about 3.6M, 154563 lines
    >f = open('dict.txt', 'r')
    >print "Reading lines..."
    >lines = f.readlines()
    >print "Done."

    [ ... ]
    >def sp1(lines):
    > """====> sp1() -- List-comprehension"""
    > return [s.split('\t') for s in lines]

    [ ... ]
    >def sp4(lines):
    > """====> sp4() -- Not correct, but very fast"""
    > return map(str.split, lines)
    >
    >for num in xrange(5):
    > fname = 'sp%(num)s' % locals()
    > print eval(fname).__doc__
    > profile.run(fname+'(lines)')


    >====> sp1() -- List-comprehension
    > 154567 function calls in 12.240 CPU seconds

    [ ... ]
    >====> sp4() -- Not correct, but very fast
    > 5 function calls in 3.090 CPU seconds

    [ ... ]
    >The problem is the default behavior of str.split should be more complex
    >than str.split('\t'). If we could use the str.split('\t') in map(), the
    >result would be witty. What do u guys think?


    I think there's something weird going on -- sp4 should be making
    154563 calls to str.split. So no wonder it goes faster -- it's
    not doing any work.

    How does [s.split() for s in lines] compare to sp2's
    [s.split('\t') for s in lines] ?

    --
    \S -- -- http://www.chaos.org.uk/~sion/
    ___ | "Frankly I have no feelings towards penguins one way or the other"
    \X/ | -- Arthur C. Clarke
    her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump
     
    Sion Arrowsmith, Aug 22, 2006
    #3
  4. Sion Arrowsmith wrote:

    > I think there's something weird going on -- sp4 should be making
    > 154563 calls to str.split. So no wonder it goes faster -- it's not doing
    > any work.


    of course it does:

    >>> lines = ["line\tone", "line\ttwo"]
    >>> [s.split("\t") for s in lines]

    [['line', 'one'], ['line', 'two']]
    >>> map(str.split, lines)

    [['line', 'one'], ['line', 'two']]

    the problem is that he's using a Python-level profiler to benchmark things written
    in C.

    (you cannot really use "profile" to *benchmark* things written in Python either; the
    profiler tells you where a given program spends the time, not how fast it is in com-
    parision with other programs)

    </F>
     
    Fredrik Lundh, Aug 22, 2006
    #4
  5. Fredrik Lundh <> wrote:
    >Sion Arrowsmith wrote:
    >> I think there's something weird going on -- sp4 should be making
    >> 154563 calls to str.split. So no wonder it goes faster -- it's not doing
    >> any work.

    >the problem is that he's using a Python-level profiler to benchmark things written
    >in C.
    >
    >(you cannot really use "profile" to *benchmark* things written in Python either; the
    >profiler tells you where a given program spends the time, not how fast it is in com-
    >parision with other programs)


    Hmm. Playing around with timeit suggests that although split() *is*
    faster than split("\t"), it's fractional, rather than the OP's four
    times faster. Is the overhead of profile keeping track of calls in
    Python getting in the way? (Not having used profile -- hence my
    confusion.) And why can map() keep everything at the C level when
    the list comprehension can't?

    --
    \S -- -- http://www.chaos.org.uk/~sion/
    ___ | "Frankly I have no feelings towards penguins one way or the other"
    \X/ | -- Arthur C. Clarke
    her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump
     
    Sion Arrowsmith, Aug 22, 2006
    #5
  6. Sion Arrowsmith wrote:

    >> (you cannot really use "profile" to *benchmark* things written in Python either; the
    >> profiler tells you where a given program spends the time, not how fast it is in com-
    >> parision with other programs)

    >
    > Hmm. Playing around with timeit suggests that although split() *is*
    > faster than split("\t"), it's fractional, rather than the OP's four
    > times faster. Is the overhead of profile keeping track of calls in
    > Python getting in the way?


    correct.

    > And why can map() keep everything at the C level when the list com-
    > prehension can't?


    map is called with two Python objects (the str.split callable and the
    sequence object), while the list comprehension is turned into a byte-
    code loop that evaluates s.split for each item in the sequence; compare
    and contrast:

    >>> def func(a):

    .... return map(str.split, a)
    ....
    >>> dis.dis(func)

    2 0 LOAD_GLOBAL 0 (map)
    3 LOAD_GLOBAL 1 (str)
    6 LOAD_ATTR 2 (split)
    9 LOAD_FAST 0 (a)
    12 CALL_FUNCTION 2
    15 RETURN_VALUE

    >>> def func(a):

    .... return [s.split() for s in a]
    ....
    >>> dis.dis(func)

    2 0 BUILD_LIST 0
    3 DUP_TOP
    4 STORE_FAST 1 (_[1])
    7 LOAD_FAST 0 (a)
    10 GET_ITER
    >> 11 FOR_ITER 19 (to 33)

    14 STORE_FAST 2 (s)
    17 LOAD_FAST 1 (_[1])
    20 LOAD_FAST 2 (s)
    23 LOAD_ATTR 0 (split)
    26 CALL_FUNCTION 0
    29 LIST_APPEND
    30 JUMP_ABSOLUTE 11
    >> 33 DELETE_FAST 1 (_[1])

    36 RETURN_VALUE

    (LOAD_GLOBAL and LOAD_ATTR does full name lookups, while LOAD_FAST loads
    a local variable using an integer index)

    </F>
     
    Fredrik Lundh, Aug 22, 2006
    #6
  7. Dasn

    Steve Holden Guest

    Fredrik Lundh wrote:
    > Sion Arrowsmith wrote:
    >
    >
    >>>(you cannot really use "profile" to *benchmark* things written in Python either; the
    >>>profiler tells you where a given program spends the time, not how fast it is in com-
    >>>parision with other programs)

    >>
    >>Hmm. Playing around with timeit suggests that although split() *is*
    >>faster than split("\t"), it's fractional, rather than the OP's four
    >>times faster. Is the overhead of profile keeping track of calls in
    >>Python getting in the way?

    >
    >
    > correct.
    >
    >
    >>And why can map() keep everything at the C level when the list com-

    >
    > > prehension can't?

    >
    > map is called with two Python objects (the str.split callable and the
    > sequence object), while the list comprehension is turned into a byte-
    > code loop that evaluates s.split for each item in the sequence; compare
    > and contrast:
    >
    > >>> def func(a):

    > .... return map(str.split, a)
    > ....
    > >>> dis.dis(func)

    > 2 0 LOAD_GLOBAL 0 (map)
    > 3 LOAD_GLOBAL 1 (str)
    > 6 LOAD_ATTR 2 (split)
    > 9 LOAD_FAST 0 (a)
    > 12 CALL_FUNCTION 2
    > 15 RETURN_VALUE
    >
    > >>> def func(a):

    > .... return [s.split() for s in a]
    > ....
    > >>> dis.dis(func)

    > 2 0 BUILD_LIST 0
    > 3 DUP_TOP
    > 4 STORE_FAST 1 (_[1])
    > 7 LOAD_FAST 0 (a)
    > 10 GET_ITER
    > >> 11 FOR_ITER 19 (to 33)

    > 14 STORE_FAST 2 (s)
    > 17 LOAD_FAST 1 (_[1])
    > 20 LOAD_FAST 2 (s)
    > 23 LOAD_ATTR 0 (split)
    > 26 CALL_FUNCTION 0
    > 29 LIST_APPEND
    > 30 JUMP_ABSOLUTE 11
    > >> 33 DELETE_FAST 1 (_[1])

    > 36 RETURN_VALUE
    >
    > (LOAD_GLOBAL and LOAD_ATTR does full name lookups, while LOAD_FAST loads
    > a local variable using an integer index)
    >
    > </F>
    >

    Well I guess if people wanted to argue for keeping the functionals this
    should be on the list ...

    regards
    Steve
    --
    Steve Holden +44 150 684 7255 +1 800 494 3119
    Holden Web LLC/Ltd http://www.holdenweb.com
    Skype: holdenweb http://holdenweb.blogspot.com
    Recent Ramblings http://del.icio.us/steve.holden
     
    Steve Holden, Aug 24, 2006
    #7
  8. Steve Holden wrote:

    > Well I guess if people wanted to argue for keeping the functionals this
    > should be on the list ...


    who's arguing ?

    is this perhaps a little like the "now that we have lexical scoping, the default
    argument object binding trick is no longer needed" myth ?

    </F>
     
    Fredrik Lundh, Aug 24, 2006
    #8
  9. Dasn

    Steve Holden Guest

    Fredrik Lundh wrote:
    > Steve Holden wrote:
    >
    >
    >>Well I guess if people wanted to argue for keeping the functionals this
    >>should be on the list ...

    >
    >
    > who's arguing ?
    >

    Please note that word "if", but you are surely aware that there havebeen
    suggestions that Python's functional programming aspects aren't
    mainstream and should therefore be ditched.

    > is this perhaps a little like the "now that we have lexical scoping, the default
    > argument object binding trick is no longer needed" myth ?
    >

    Could be. Removing that feature would also be a mistake.

    regards
    Steve
    --
    Steve Holden +44 150 684 7255 +1 800 494 3119
    Holden Web LLC/Ltd http://www.holdenweb.com
    Skype: holdenweb http://holdenweb.blogspot.com
    Recent Ramblings http://del.icio.us/steve.holden
     
    Steve Holden, Aug 28, 2006
    #9
  10. Dasn

    Dasn Guest

    On Tue, Aug 22, 2006 at 04:50:39PM +0200, Fredrik Lundh wrote:
    > (you cannot really use "profile" to *benchmark* things written in
    > Python either; the profiler tells you where a given program spends the
    > time, not how fast it is in com- parision with other programs)


    Thanks for your indication.
     
    Dasn, Aug 28, 2006
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Honne Gowda A
    Replies:
    2
    Views:
    894
    Karl Heinz Buchegger
    Oct 31, 2003
  2. amit kumar
    Replies:
    5
    Views:
    6,151
    velthuijsen
    May 18, 2004
  3. Replies:
    2
    Views:
    944
    Bengt Richter
    Aug 1, 2005
  4. Dasn
    Replies:
    6
    Views:
    431
    Paul McGuire
    Aug 22, 2006
  5. Bob
    Replies:
    5
    Views:
    272
Loading...

Share This Page