Re: Restricting import file lookup for pyd, dll, ...

Discussion in 'Python' started by Bernard Lebel, Oct 20, 2006.

  1. Hi,

    That's because I'm using Python through another application, via the
    pywin32 extensions. When that other application starts, it performs
    several thousands of file requests (we're talking 4,500, roughly) in
    the Python installation, locations where there are Python files, and
    in some other locations that don't make sense. This adds considerable
    time to the startup time of the application, we're talking between 2
    and 9 seconds.

    This a problem with the application, not Python. But I'm looking for
    ways to minimize this overhead so the users of this application waste
    less time waiting after the startup.


    Thanks
    Bernard



    On 10/19/06, Gabriel Genellina <> wrote:
    > You can't; part of the standard library are .pyd/.dll files.
    > Those existence checks should be fairly fast - why are you worried about them?
    Bernard Lebel, Oct 20, 2006
    #1
    1. Advertising

  2. Bernard Lebel

    Magnus Lycka Guest

    Bernard Lebel wrote:
    > Hi,
    >
    > That's because I'm using Python through another application, via the
    > pywin32 extensions. When that other application starts, it performs
    > several thousands of file requests (we're talking 4,500, roughly) in
    > the Python installation, locations where there are Python files, and
    > in some other locations that don't make sense. This adds considerable
    > time to the startup time of the application, we're talking between 2
    > and 9 seconds.


    Sounds like a broken (networked?) file system. The only time I've
    had that kind of problems with python startup is when I've had really
    slow anti-virus programs that scanned all the files I opened. But then
    it wasn't file requests that mattered, but actually opening them...
    It still wasn't anywhere near 9 seconds though...
    Magnus Lycka, Oct 27, 2006
    #2
    1. Advertising

  3. Magnus Lycka wrote:

    >> That's because I'm using Python through another application, via the
    >> pywin32 extensions. When that other application starts, it performs
    >> several thousands of file requests (we're talking 4,500, roughly) in
    >> the Python installation, locations where there are Python files, and
    >> in some other locations that don't make sense. This adds considerable
    >> time to the startup time of the application, we're talking between 2
    >> and 9 seconds.

    >
    > Sounds like a broken (networked?) file system. The only time I've
    > had that kind of problems with python startup is when I've had really
    > slow anti-virus programs that scanned all the files I opened. But then
    > it wasn't file requests that mattered, but actually opening them...
    > It still wasn't anywhere near 9 seconds though...


    if anyone finds out more about this issue, feel free to add a note to
    this FAQ entry:

    http://www.effbot.org/pyfaq/why-does-python-sometimes-take-so-long-to-start.htm

    </F>
    Fredrik Lundh, Oct 28, 2006
    #3
  4. Hello,

    It's been almost two months since I last investigated this issue, so
    now I wish to revive this conversation.

    To answer some of the questions raised by contributors....


    [Gabriel Genellina] Try to shorten the PYTHONPATH to the really
    required directories
    (deleting those locations "that don't make sense").

    [Bernard] I customized the PYTHONPATH using a pth file. The pth file
    contains this:
    \\Linuxserver\ANIMATION\XSI\WORKGROUP_ANIMATION\Data\Scripts
    \\Linuxserver\ANIMATION\DB\MT\MT_WORKGROUP\Data\Scripts
    \\Linuxserver\ANIMATION\DB\TS\TS_WORKGROUP\Data\Scripts
    \\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE

    That's it. I could hardly shorten these paths, unless the ressources
    exposed through these paths were copied locally on every computer. Atm
    I do not have management tools at my disposal to handle such a setup.

    When print the paths:

    C:\WINDOWS\system32\python24.zip
    C:\Documents and Settings\blebel
    C:\Python24\DLLs
    C:\Python24\lib
    C:\Python24\lib\plat-win
    C:\Python24\lib\lib-tk
    C:\Python24
    d:\bernard\work\workgroups\workgroup_animation\data\scripts
    d:\bernard\work\workgroups\mt_workgroup\data\scripts
    d:\bernard\work\workgroups\ts_workgroup\data\scripts
    \\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE
    c:\users\blebel\Softimage\XSI_5.11\Data\Scripts
    C:\Python24\lib\site-packages
    C:\Python24\lib\site-packages\win32
    C:\Python24\lib\site-packages\win32\lib
    C:\Python24\lib\site-packages\Pythonwin

    If by "shortening the PYTHONPATH" you meant having less paths, I
    hardly see how I could do less than what I do with the pth. All these
    paths seem to be built in.

    Now what would be REALLY nice is to have the ability to specify in the
    paths the location of *specific files*. Is this possible? What happens
    is that Python is visiting all kinds of locations to find some files,
    like os, ntpath, and many more. If I could tell Python right away
    where are these files located, I hope I could gain something.




    [Fredrik Lundh] a plain Python 2.4 interpreter can start, execute a
    command, and shut
    down in about 0.13 seconds on my machine. 2.5 does the same thing in
    0.10 seconds.

    [Bernard] The problem is not firing up the standalone Python
    interpreter. The problem is firing the application that embeds a
    Python interpreter.

    As explained above, when I start this application, many files are
    looked in many different places. All PYTHONPATH locations are
    traversed, where pyd-dll-py-pyw-pyc files are searched. Many of these
    files are searched in highly improbable locations.




    [Fredrik Lundh] are you sure you're benchmarking *Python's* start up
    time, and not the
    time it takes to load all the modules used by your application, or the
    time it takes for "filemon" to print all those 4500 requests to the
    monitor window?

    [Bernard] The Filemon overhead consistently clocks at 1 second, not
    matter the test I perform. Since I ran all tests with Filemon running,
    I feel safe to ignore this constant.

    Now, as to the first sentence, to be fair, no, I'm not exactly sure. I
    don't know all the details of the Python's embedding in this software,
    and this knowledge is out of my reach atm.

    Here is the relevant Filemon log:
    http://www.bernardlebel.com/img_remote/3D/XSI/python/Filemon_XSIPython.LOG
    (710k file)



    [Magnus Lycka] Sounds like a broken (networked?) file system. The only time I've
    had that kind of problems with python startup is when I've had really
    slow anti-virus programs that scanned all the files I opened. But then
    it wasn't file requests that mattered, but actually opening them...
    It still wasn't anywhere near 9 seconds though...

    [Bernard] This is not entirely impossible, but I get similar results
    on every computer I perform this test. By "every computer", I mean
    different locations, with different hardwares and different network
    setups. Even when the files are all local on the computer I get this.



    Thanks
    Bernard



    On 10/28/06, Fredrik Lundh <> wrote:
    > Magnus Lycka wrote:
    >
    > >> That's because I'm using Python through another application, via the
    > >> pywin32 extensions. When that other application starts, it performs
    > >> several thousands of file requests (we're talking 4,500, roughly) in
    > >> the Python installation, locations where there are Python files, and
    > >> in some other locations that don't make sense. This adds considerable
    > >> time to the startup time of the application, we're talking between 2
    > >> and 9 seconds.

    > >
    > > Sounds like a broken (networked?) file system. The only time I've
    > > had that kind of problems with python startup is when I've had really
    > > slow anti-virus programs that scanned all the files I opened. But then
    > > it wasn't file requests that mattered, but actually opening them...
    > > It still wasn't anywhere near 9 seconds though...

    >
    > if anyone finds out more about this issue, feel free to add a note to
    > this FAQ entry:
    >
    > http://www.effbot.org/pyfaq/why-does-python-sometimes-take-so-long-to-start.htm
    >
    > </F>
    >
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >
    Bernard Lebel, Dec 11, 2006
    #4
  5. Oops, sorry for the inconsistency. The pth file rather looks like this:

    d:\bernard\work\workgroups\workgroup_animation\data\scripts
    d:\bernard\work\workgroups\mt_workgroup\data\scripts
    d:\bernard\work\workgroups\ts_workgroup\data\scripts
    \\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE
    c:\users\blebel\Softimage\XSI_5.11\Data\Scripts


    Bernard



    On 12/11/06, Bernard Lebel <> wrote:
    > Hello,
    >
    > It's been almost two months since I last investigated this issue, so
    > now I wish to revive this conversation.
    >
    > To answer some of the questions raised by contributors....
    >
    >
    > [Gabriel Genellina] Try to shorten the PYTHONPATH to the really
    > required directories
    > (deleting those locations "that don't make sense").
    >
    > [Bernard] I customized the PYTHONPATH using a pth file. The pth file
    > contains this:
    > \\Linuxserver\ANIMATION\XSI\WORKGROUP_ANIMATION\Data\Scripts
    > \\Linuxserver\ANIMATION\DB\MT\MT_WORKGROUP\Data\Scripts
    > \\Linuxserver\ANIMATION\DB\TS\TS_WORKGROUP\Data\Scripts
    > \\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE
    >
    > That's it. I could hardly shorten these paths, unless the ressources
    > exposed through these paths were copied locally on every computer. Atm
    > I do not have management tools at my disposal to handle such a setup.
    >
    > When print the paths:
    >
    > C:\WINDOWS\system32\python24.zip
    > C:\Documents and Settings\blebel
    > C:\Python24\DLLs
    > C:\Python24\lib
    > C:\Python24\lib\plat-win
    > C:\Python24\lib\lib-tk
    > C:\Python24
    > d:\bernard\work\workgroups\workgroup_animation\data\scripts
    > d:\bernard\work\workgroups\mt_workgroup\data\scripts
    > d:\bernard\work\workgroups\ts_workgroup\data\scripts
    > \\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE
    > c:\users\blebel\Softimage\XSI_5.11\Data\Scripts
    > C:\Python24\lib\site-packages
    > C:\Python24\lib\site-packages\win32
    > C:\Python24\lib\site-packages\win32\lib
    > C:\Python24\lib\site-packages\Pythonwin
    >
    > If by "shortening the PYTHONPATH" you meant having less paths, I
    > hardly see how I could do less than what I do with the pth. All these
    > paths seem to be built in.
    >
    > Now what would be REALLY nice is to have the ability to specify in the
    > paths the location of *specific files*. Is this possible? What happens
    > is that Python is visiting all kinds of locations to find some files,
    > like os, ntpath, and many more. If I could tell Python right away
    > where are these files located, I hope I could gain something.
    >
    >
    >
    >
    > [Fredrik Lundh] a plain Python 2.4 interpreter can start, execute a
    > command, and shut
    > down in about 0.13 seconds on my machine. 2.5 does the same thing in
    > 0.10 seconds.
    >
    > [Bernard] The problem is not firing up the standalone Python
    > interpreter. The problem is firing the application that embeds a
    > Python interpreter.
    >
    > As explained above, when I start this application, many files are
    > looked in many different places. All PYTHONPATH locations are
    > traversed, where pyd-dll-py-pyw-pyc files are searched. Many of these
    > files are searched in highly improbable locations.
    >
    >
    >
    >
    > [Fredrik Lundh] are you sure you're benchmarking *Python's* start up
    > time, and not the
    > time it takes to load all the modules used by your application, or the
    > time it takes for "filemon" to print all those 4500 requests to the
    > monitor window?
    >
    > [Bernard] The Filemon overhead consistently clocks at 1 second, not
    > matter the test I perform. Since I ran all tests with Filemon running,
    > I feel safe to ignore this constant.
    >
    > Now, as to the first sentence, to be fair, no, I'm not exactly sure. I
    > don't know all the details of the Python's embedding in this software,
    > and this knowledge is out of my reach atm.
    >
    > Here is the relevant Filemon log:
    > http://www.bernardlebel.com/img_remote/3D/XSI/python/Filemon_XSIPython.LOG
    > (710k file)
    >
    >
    >
    > [Magnus Lycka] Sounds like a broken (networked?) file system. The only time I've
    > had that kind of problems with python startup is when I've had really
    > slow anti-virus programs that scanned all the files I opened. But then
    > it wasn't file requests that mattered, but actually opening them...
    > It still wasn't anywhere near 9 seconds though...
    >
    > [Bernard] This is not entirely impossible, but I get similar results
    > on every computer I perform this test. By "every computer", I mean
    > different locations, with different hardwares and different network
    > setups. Even when the files are all local on the computer I get this.
    >
    >
    >
    > Thanks
    > Bernard
    >
    >
    >
    > On 10/28/06, Fredrik Lundh <> wrote:
    > > Magnus Lycka wrote:
    > >
    > > >> That's because I'm using Python through another application, via the
    > > >> pywin32 extensions. When that other application starts, it performs
    > > >> several thousands of file requests (we're talking 4,500, roughly) in
    > > >> the Python installation, locations where there are Python files, and
    > > >> in some other locations that don't make sense. This adds considerable
    > > >> time to the startup time of the application, we're talking between 2
    > > >> and 9 seconds.
    > > >
    > > > Sounds like a broken (networked?) file system. The only time I've
    > > > had that kind of problems with python startup is when I've had really
    > > > slow anti-virus programs that scanned all the files I opened. But then
    > > > it wasn't file requests that mattered, but actually opening them...
    > > > It still wasn't anywhere near 9 seconds though...

    > >
    > > if anyone finds out more about this issue, feel free to add a note to
    > > this FAQ entry:
    > >
    > > http://www.effbot.org/pyfaq/why-does-python-sometimes-take-so-long-to-start.htm
    > >
    > > </F>
    > >
    > > --
    > > http://mail.python.org/mailman/listinfo/python-list
    > >

    >
    Bernard Lebel, Dec 11, 2006
    #5
  6. At Monday 11/12/2006 19:11, Bernard Lebel wrote:

    >If by "shortening the PYTHONPATH" you meant having less paths, I
    >hardly see how I could do less than what I do with the pth. All these
    >paths seem to be built in.


    I mean, delete the ones that actually don't exist. sys.path is a
    simple list; you can traverse it (perhaps backwards is easier) in
    site.py or sitecustomize.py and delete unexistant entries.
    Moving the network related paths (\\linuxserver\...) towards the end
    may help (if you don't get conflicts by finding the wrong module in
    the wrong place)

    [Fredrik Lundh] a plain Python 2.4 interpreter can start, execute a
    >command, and shut
    >down in about 0.13 seconds on my machine. 2.5 does the same thing in
    >0.10 seconds.
    >
    >[Bernard] The problem is not firing up the standalone Python
    >interpreter. The problem is firing the application that embeds a
    >Python interpreter.
    >
    >As explained above, when I start this application, many files are
    >looked in many different places. All PYTHONPATH locations are
    >traversed, where pyd-dll-py-pyw-pyc files are searched. Many of these
    >files are searched in highly improbable locations.


    But the whole process takes about 3sec - as seen on your log.
    (enabling ms resolution on FileMon may help too, because the timing
    is now too much rough)

    >[Fredrik Lundh] are you sure you're benchmarking *Python's* start up
    >time, and not the
    >time it takes to load all the modules used by your application, or the
    >time it takes for "filemon" to print all those 4500 requests to the
    >monitor window?
    >
    >[Bernard] The Filemon overhead consistently clocks at 1 second, not
    >matter the test I perform. Since I ran all tests with Filemon running,
    >I feel safe to ignore this constant.
    >
    >Now, as to the first sentence, to be fair, no, I'm not exactly sure. I
    >don't know all the details of the Python's embedding in this software,
    >and this knowledge is out of my reach atm.


    Have you tried running the interpreter alone? Not your embedded application.
    Try invoking python with the -v flag. For an embedded application,
    define PYTHONVERBOSE=1 in the environment before running.
    Perhaps it's not Python which slows down your startup, but other
    parts of the app.


    --
    Gabriel Genellina
    Softlab SRL

    __________________________________________________
    Correo Yahoo!
    Espacio para todos tus mensajes, antivirus y antispam ¡gratis!
    ¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar
    Gabriel Genellina, Dec 12, 2006
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John Keeling

    import pyd from zip file

    John Keeling, Sep 2, 2004, in forum: Python
    Replies:
    1
    Views:
    391
    David Bolen
    Sep 2, 2004
  2. wen
    Replies:
    1
    Views:
    480
    Daniel Dittmar
    Aug 23, 2005
  3. Bernard Lebel
    Replies:
    0
    Views:
    263
    Bernard Lebel
    Oct 19, 2006
  4. jrh

    import dll instead of pyd

    jrh, Jul 25, 2008, in forum: Python
    Replies:
    2
    Views:
    408
    Fredrik Lundh
    Jul 25, 2008
  5. Replies:
    5
    Views:
    163
    Junze Liu
    Jan 24, 2013
Loading...

Share This Page