how to optimize zipimport

Discussion in 'Python' started by Coonay, Mar 26, 2009.

  1. Coonay

    Coonay Guest

    in my mudule ,i import another mudule from a zip ,

    when i call my mudle method,the module in the zip will be import from
    the zip every time,that slow down the operation,

    i try to search the some workarodnd or solution but i don't get one,

    could you help me ?
    Coonay, Mar 26, 2009
    #1
    1. Advertising

  2. Coonay

    Coonay Guest

    Re: how to optimize zipimport

    see,the zipimort takes nearly 0.5 second

    D 03-25 07:53PM 21.374
    Loading __main__
    I 03-25 07:53PM 21.455
    zipimporter('/base/data/home/apps/coonay/1.332322118600950324/
    django.zip', '')
    W 03-25 07:53PM 22.046
    appengine_django module






    On Mar 26, 10:41 am, Coonay <> wrote:


    > in my mudule ,i import another mudule from a zip ,
    >
    > when i call  my mudle method,the module in the zip will be import from
    > the zip every time,that slow down the operation,
    >
    > i try to search the some workarodnd or solution but i don't get one,
    >
    > could you help me ?
    Coonay, Mar 26, 2009
    #2
    1. Advertising

  3. Coonay

    John Machin Guest

    Re: how to optimize zipimport

    On Mar 26, 2:06 pm, Coonay <> wrote:
    > see,the zipimort takes nearly 0.5 second
    >
    > D 03-25 07:53PM 21.374
    > Loading __main__
    > I 03-25 07:53PM 21.455
    > zipimporter('/base/data/home/apps/coonay/1.332322118600950324/
    > django.zip', '')


    ?? zipimport.Zipimporter() has one arg, not two.

    > W 03-25 07:53PM 22.046
    > appengine_django module
    >
    > On Mar 26, 10:41 am, Coonay <> wrote:
    >
    > > in my mudule ,i import another mudule from a zip ,

    >
    > > when i call  my mudle method,the module in the zip will be import from
    > > the zip every time,that slow down the operation,

    >
    > > i try to search the some workarodnd or solution but i don't get one,


    Why are you explicitly using the zipimport module? In other words, why
    don't you put the full path of the zip file in sys.path and then just
    use the normal "import module_in_the_zip" mechanism? Note that this
    can/should be done once at the top of your calling module. In other
    words, the import code is executed when the calling module is imported
    by your main script, not every time you call a method in your calling
    module.

    Alternatively, if you feel you *must* use zipimport, what's to stop
    you doing that once at the top of the calling module?

    Note: it may help when you reply to show a stripped-down version of
    your calling module, sufficient to back up your description of what is
    happening.

    HTH,
    John
    John Machin, Mar 26, 2009
    #3
  4. Coonay

    Coonay Guest

    Re: how to optimize zipimport

    On Mar 26, 1:38 pm, John Machin <> wrote:
    > On Mar 26, 2:06 pm, Coonay <> wrote:
    >
    > > see,the zipimort takes nearly 0.5 second

    >
    > > D 03-25 07:53PM 21.374
    > > Loading __main__
    > > I 03-25 07:53PM 21.455
    > > zipimporter('/base/data/home/apps/coonay/1.332322118600950324/
    > > django.zip', '')

    >
    > ?? zipimport.Zipimporter() has one arg, not two.
    >
    > > W 03-25 07:53PM 22.046
    > > appengine_django module

    >
    > > On Mar 26, 10:41 am, Coonay <> wrote:

    >
    > > > in my mudule ,i import another mudule from a zip ,

    >
    > > > when i call  my mudle method,the module in the zip will be import from
    > > > the zip every time,that slow down the operation,

    >
    > > > i try to search the some workarodnd or solution but i don't get one,

    >

    i don't use zipimport directly,just like u said i put the zip in the
    sys.path


    > Why are you explicitly using the zipimport module? In other words, why
    > don't you put the full path of the zip file in sys.path and then just
    > use the normal "import module_in_the_zip" mechanism? Note that this


    i found the the module in the zip reloaded everytime the code is
    called,i mean, say ,first time it take me 1 second to call a method in
    the zip,but it take another 1 second to call the mothod for another
    time ,and so on


    > can/should be done once at the top of your calling module. In other
    > words, the import code is executed when the calling module is imported
    > by your main script, not every time you call a method in your calling
    > module.
    >

    In my developing environment ,there is a limitation of file number
    that i can deploy,such as i 'm only allowed to deploy 100 python
    files,so i have to zip some module


    > Alternatively, if you feel you *must* use zipimport, what's to stop
    > you doing that once at the top of the calling module?
    >
    > Note: it may help when you reply to show a stripped-down version of
    > your calling module, sufficient to back up your description of what is
    > happening.
    >
    > HTH,
    > John
    Coonay, Mar 30, 2009
    #4
  5. Re: how to optimize zipimport

    En Mon, 30 Mar 2009 09:39:20 -0300, Coonay <> escribió:
    > On Mar 26, 1:38 pm, John Machin <> wrote:


    >> Why are you explicitly using the zipimport module? In other words, why
    >> don't you put the full path of the zip file in sys.path and then just
    >> use the normal "import module_in_the_zip" mechanism? Note that this

    >
    > i found the the module in the zip reloaded everytime the code is
    > called,i mean, say ,first time it take me 1 second to call a method in
    > the zip,but it take another 1 second to call the mothod for another
    > time ,and so on


    Uh? Which Python version?
    Once a module is imported, it isn't loaded again, even from a zip file.
    Surely there is another reason for the delay, not the import itself.

    --
    Gabriel Genellina
    Gabriel Genellina, Apr 2, 2009
    #5
  6. Coonay

    John Machin Guest

    Re: how to optimize zipimport

    On Mar 30, 11:39 pm, Coonay <> wrote:
    > On Mar 26, 1:38 pm, John Machin <> wrote:
    >
    > > On Mar 26, 2:06 pm, Coonay <> wrote:

    >


    > > > On Mar 26, 10:41 am, Coonay <> wrote:

    >
    > > > > in my mudule ,i import another mudule from a zip ,

    >
    > > > > when i call  my mudle method,the module in the zip will be import from
    > > > > the zip every time,that slow down the operation,

    >
    > > > > i try to search the some workarodnd or solution but i don't get one,

    >
    > i don't use zipimport directly,just like u said i put the zip in the
    > sys.path
    >
    >
    > i found the the module in the zip reloaded  everytime the code is
    > called,i mean, say ,first time it take me 1 second to call a method in
    > the zip,but it take another 1 second to call the mothod for another
    > time ,and so on
    >


    How do you know that it is being reloaded each time a methos is
    called?



    > > Note: it may help when you reply to show a stripped-down version of
    > > your calling module, sufficient to back up your description of what is
    > > happening.


    We can only guess if you don't show us what is actually happening.
    Vague descriptions are not very useful. Show us how exactly you are
    importing the modules, how you are calling methods, with timing
    calls ...

    By the way, have you read this part of the documentation:
    "Note that if an archive only contains .py files, Python will not
    attempt to modify the archive by adding the corresponding .pyc or .pyo
    file, meaning that if a ZIP archive doesn't contain .pyc files,
    importing may be rather slow."
    ?

    Do you have .pyc files in your ZIP file? If not that would make the
    first import slower than it should be.

    However second, third, ... imports (if you do any) should be
    instantaneous -- unless you are messing about deleting modules from
    sys.modules.

    And *method* calls should be instantaneous.

    Does your app run at the same speed when you have all the modules in a
    directory instead of inside a zip file?

    Apart from code executed once at app start to inject the path to the
    zip file into sys.path, have you made any other changes to accommodate
    using import from zip? If so, what changes?

    Have you tried running your app from the shell (Windows translation:
    in a Command Prompt window) like this:

    python -v app_start.py

    This will show you all the imports as they happen.

    HTH ... but a clear description from you (plus code samples) of
    exactly what you are doing woul dhelp a whole lot more.

    John
    John Machin, Apr 2, 2009
    #6
  7. Re: how to optimize zipimport

    On Apr 2, 6:26 pm, John Machin <> wrote:
    > On Mar 30, 11:39 pm, Coonay <> wrote:
    >
    >
    >
    > > On Mar 26, 1:38 pm, John Machin <> wrote:

    >
    > > > On Mar 26, 2:06 pm, Coonay <> wrote:

    >
    > > > > On Mar 26, 10:41 am, Coonay <> wrote:

    >
    > > > > > in my mudule ,i import another mudule from a zip ,

    >
    > > > > > when i call  my mudle method,the module in the zip will be import from
    > > > > > the zip every time,that slow down the operation,

    >
    > > > > > i try to search the some workarodnd or solution but i don't get one,

    >
    > > i don't use zipimport directly,just like u said i put the zip in the
    > > sys.path

    >
    > > i found the the module in the zip reloaded  everytime the code is
    > > called,i mean, say ,first time it take me 1 second to call a method in
    > > the zip,but it take another 1 second to call the mothod for another
    > > time ,and so on

    >
    > How do you know that it is being reloaded each time a methos is
    > called?
    >
    > > > Note: it may help when you reply to show a stripped-down version of
    > > > your calling module, sufficient to back up your description of what is
    > > > happening.

    >
    > We can only guess if you don't show us what is actually happening.
    > Vague descriptions are not very useful. Show us how exactly you are
    > importing the modules, how you are calling methods, with timing
    > calls ...


    yes , i put the pyc file with py source file in the zip
    >
    > By the way, have you read this part of the documentation:
    > "Note that if an archive only contains .py files, Python will not
    > attempt to modify the archive by adding the corresponding .pyc or .pyo
    > file, meaning that if a ZIP archive doesn't contain .pyc files,
    > importing may be rather slow."
    > ?
    >
    > Do you have .pyc files in your ZIP file? If not that would make the
    > first import slower than it should be.
    >
    > However second, third, ... imports (if you do any) should be
    > instantaneous -- unless you are messing about deleting modules from
    > sys.modules.
    >
    > And *method* calls should be instantaneous.




    it run much faster when i unzip the lib and put then in a directory

    >
    > Does your app run at the same speed when you have all the modules in a
    > directory instead of inside a zip file?
    >
    > Apart from code executed once at app start to inject the path to the
    > zip file into sys.path, have you made any other changes to accommodate
    > using import from zip? If so, what changes?
    >
    > Have you tried running your app from the shell (Windows translation:
    > in a Command Prompt window) like this:
    >
    > python -v app_start.py
    >
    > This will show you all the imports as they happen.
    >
    > HTH ... but a clear description from you (plus code samples) of
    > exactly what you are doing woul dhelp a whole lot more.
    >
    > John
    Coonay.appspot, Apr 4, 2009
    #7
  8. Coonay

    John Machin Guest

    Re: how to optimize zipimport

    On Apr 4, 7:06 pm, "Coonay.appspot" <> wrote:
    > On Apr 2, 6:26 pm, John Machin <> wrote:

    [snip]
    > > > i found the the module in the zip reloaded  everytime the code is
    > > > called,i mean, say ,first time it take me 1 second to call a method in
    > > > the zip,but it take another 1 second to call the mothod for another
    > > > time ,and so on

    >
    > > How do you know that it is being reloaded each time a methos is
    > > called?


    You haven't answered the above question.




    > > > > Note: it may help when you reply to show a stripped-down version of
    > > > > your calling module, sufficient to back up your description of what is
    > > > > happening.

    >
    > > We can only guess if you don't show us what is actually happening.
    > > Vague descriptions are not very useful. Show us how exactly you are
    > > importing the modules, how you are calling methods, with timing
    > > calls ...


    No answer.

    [snip]

    > > Apart from code executed once at app start to inject the path to the
    > > zip file into sys.path, have you made any other changes to accommodate
    > > using import from zip? If so, what changes?


    No answer.

    > > Have you tried running your app from the shell (Windows translation:
    > > in a Command Prompt window) like this:

    >
    > > python -v app_start.py

    >
    > > This will show you all the imports as they happen.


    No answer.

    > > HTH ... but a clear description from you (plus code samples) of
    > > exactly what you are doing woul dhelp a whole lot more.


    No answer.

    Conclusion: you don't actually want any help.
    John Machin, Apr 4, 2009
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Bob Swerdlow
    Replies:
    1
    Views:
    359
  2. Gabriele *Darkbard* Farina

    zipimport

    Gabriele *Darkbard* Farina, May 4, 2005, in forum: Python
    Replies:
    1
    Views:
    376
    Jonathan Brady
    May 4, 2005
  3. Replies:
    2
    Views:
    269
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=
    Jan 28, 2006
  4. shabda raaj
    Replies:
    1
    Views:
    2,789
    Diez B. Roggisch
    Sep 10, 2007
  5. Replies:
    0
    Views:
    266
Loading...

Share This Page