module import performance question

Discussion in 'Python' started by Jeff Sykes, Nov 11, 2003.

  1. Jeff Sykes

    Jeff Sykes Guest

    I have a cgi script that imports modules based on a user config file.
    I am suffering some performance problems when I import these modules.
    Some logging revealed that it seems to be taking about 1.3 seconds to
    import these modules. This is running on a Windows 2000 box with a
    Pentium II 400 processor, Python 2.1, Apache 1.3.19 as the web server.
    I don't need screaming performance numbers, but this time is
    excessive. Is it me, or does that number seem kind of slow? Any code
    optimizations or recommendations? I would like to keep the
    architecture the same, just standard cgi scripts, so mod_python looks
    like it's a bit more involved than I would like.

    Here is a code snip:

    dyn_blocks = {} # set up a dict of the dynamic imports
    if confok: # was the configuration present?
    blocks = getBlocks(config) # get the application info from config
    for block in blocks: # each block indictates the ap & pg (module)
    ap = string.split(string.strip(block[1:len(block)]))[0]
    pg = string.split(string.strip(block[1:len(block)]))[1]
    sys.path.append(getAppPath(ap)) # add module dir to sys.path
    if (block[0] == '~'):
    try:
    dyn_blocks[ap + "_" + pg] = __import__(pg)
    except:
    Log.stacktrace(Log.ERROR) # log problem importing module

    Any help is appreciated.
     
    Jeff Sykes, Nov 11, 2003
    #1
    1. Advertising

  2. "Jeff Sykes" <> wrote in message
    news:...
    > I have a cgi script that imports modules based on a user config

    file.
    > I am suffering some performance problems when I import these

    modules.
    > Some logging revealed that it seems to be taking about 1.3 seconds

    to
    > import these modules. This is running on a Windows 2000 box with a
    > Pentium II 400 processor, Python 2.1, Apache 1.3.19 as the web

    server.
    > I don't need screaming performance numbers, but this time is
    > excessive. Is it me, or does that number seem kind of slow?


    How many modules are being imported?

    > Any code
    > optimizations or recommendations?


    Don't append to sys.path. Try sys.path.insert(0,addlpath). Then the
    import will get an immediate hit and always the right module rather
    than stepping through all the preceeding directories.

    HTH,


    Emile van Sebille
     
    Emile van Sebille, Nov 11, 2003
    #2
    1. Advertising

  3. Jeff Sykes

    Jeff Sykes Guest

    Thanks, Emile, that's a good idea. I found another issue, too. See my
    response to another poster below:

    >> I think there was a different culprit, though. After some more
    >> dilligent logging, I realized that immediately after import I
    >> called upon an attribute of the imported module that wouldn't
    >> _necessarily_ be present. When the attribute was not present,
    >> it raised an exception, which I caught. This exception handling
    >> was a real performance anchor! I now check hasattr() before I
    >> execute this block of code, and this has helped performance
    >> significantly.
    >>
    >> Here's a snip of the block that occured right after import:
    >>
    >> try:
    >> # next line was not there before
    >> if hasattr(dyn_blocks[key], "BUFFER_ON"):
    >> context.setBufferFlag(dyn_blocks[key].BUFFER_ON)
    >> except:
    >> Log.stacktrace(Log.ERROR)
    >>
    >> I guess the lesson I learned is don't use exception handling
    >> unnecessarily. Which I know not to do anyway. Sigh. Back to
    >> school for me.


    "Emile van Sebille" <> wrote in message news:<boqutc$1gl3q2$-berlin.de>...
    > "Jeff Sykes" <> wrote in message
    > news:...
    > > I have a cgi script that imports modules based on a user config

    > file.
    > > I am suffering some performance problems when I import these

    > modules.
    > > Some logging revealed that it seems to be taking about 1.3 seconds

    > to
    > > import these modules. This is running on a Windows 2000 box with a
    > > Pentium II 400 processor, Python 2.1, Apache 1.3.19 as the web

    > server.
    > > I don't need screaming performance numbers, but this time is
    > > excessive. Is it me, or does that number seem kind of slow?

    >
    > How many modules are being imported?
    >
    > > Any code
    > > optimizations or recommendations?

    >
    > Don't append to sys.path. Try sys.path.insert(0,addlpath). Then the
    > import will get an immediate hit and always the right module rather
    > than stepping through all the preceeding directories.
    >
    > HTH,
    >
    >
    > Emile van Sebille
    >
     
    Jeff Sykes, Nov 11, 2003
    #3
  4. Jeff Sykes

    Peter Abel Guest

    (Jeff Sykes) wrote in message news:<>...
    ....
    ....
    ....

    > Here is a code snip:
    >
    > dyn_blocks = {} # set up a dict of the dynamic imports
    > if confok: # was the configuration present?
    > blocks = getBlocks(config) # get the application info from config
    > for block in blocks: # each block indictates the ap & pg (module)


    doen't know how long is len(blocks), if it's very long you could
    accelerate the *for loop* be replacing the following two lines with:

    (ap,pg) = string.split(string.strip(block[1:]))
    **Remark: block[1:len(block)] is the same as block[1:], but without
    calulating len(block) everytime, further more the split
    and the strip is done only once

    > ap = string.split(string.strip(block[1:len(block)]))[0]
    > pg = string.split(string.strip(block[1:len(block)]))[1]


    ....
    ....
    ....

    Regards
    Peter
     
    Peter Abel, Nov 11, 2003
    #4
  5. (Jeff Sykes) wrote in message news:<>...
    > ...
    > Here's a snip of the block that occured right after import:
    >
    > try:
    > # next line was not there before
    > if hasattr(dyn_blocks[key], "BUFFER_ON"):
    > context.setBufferFlag(dyn_blocks[key].BUFFER_ON)
    > except:
    > Log.stacktrace(Log.ERROR)
    > I guess the lesson I learned is don't use exception handling
    > unnecessarily. Which I know not to do anyway. Sigh. Back to
    > school for me.


    Wrong lesson. Python wants you to use exception handling.
    The time sink above code is the unnecessary uses of:
    > Log.stacktrace(Log.ERROR)


    A couple of Alternatives:
    ## smaller, not quite careful enough for my taste
    try:
    context.setBufferFlag(dyn_blocks[key].BUFFER_ON)
    except AttributeError:
    pass
    except:
    Log.stacktrace(Log.ERROR)

    ## better, more to my taste
    try:
    try:
    flag = dyn_blocks[key].BUFFER_ON
    except AttributeError:
    pass
    else:
    context.setBufferFlag(flag)
    except:
    Log.stacktrace(Log.ERROR)

    -Scott
     
    Scott D. Daniels, Nov 11, 2003
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.

Share This Page