Re: Populating a dictionary, fast

Discussion in 'Python' started by Michael Bacarella, Nov 11, 2007.

  1. > > This would seem to implicate the line id2name[id] = name as being
    excruciatingly slow.
    >
    > As others have pointed out there is no way that this takes 45
    > minutes.Must be something with your system or setup.
    >
    > A functionally equivalent code for me runs in about 49 seconds!
    > (it ends up using about twice as much ram as the data on disk)


    You can download the list of keys from here, it's 43M gzipped:
    http://www.sendspace.com/file/9530i7

    and see it take about 45 minutes with this:

    $ cat cache-keys.py
    #!/usr/bin/python
    v = {}
    for line in open('keys.txt'):
    v[long(line.strip())] = True
     
    Michael Bacarella, Nov 11, 2007
    #1
    1. Advertising

  2. On Sun, 11 Nov 2007 08:51:37 -0800, Michael Bacarella wrote:

    >> As others have pointed out there is no way that this takes 45
    >> minutes.Must be something with your system or setup.
    >>
    >> A functionally equivalent code for me runs in about 49 seconds!
    >> (it ends up using about twice as much ram as the data on disk)

    >
    > You can download the list of keys from here, it's 43M gzipped:
    > http://www.sendspace.com/file/9530i7
    >
    > and see it take about 45 minutes with this:
    >
    > $ cat cache-keys.py
    > #!/usr/bin/python
    > v = {}
    > for line in open('keys.txt'):
    > v[long(line.strip())] = True


    Takes about 40 seconds here.

    bj@s8n:~$ time python test.py

    real 0m38.758s
    user 0m25.290s
    sys 0m1.580s

    Ciao,
    Marc 'BlackJack' Rintsch
     
    Marc 'BlackJack' Rintsch, Nov 11, 2007
    #2
    1. Advertising

  3. On Nov 11, 11:51 am, Michael Bacarella <> wrote:

    > and see it take about 45 minutes with this:
    >
    > $ cat cache-keys.py
    > #!/usr/bin/python
    > v = {}
    > for line in open('keys.txt'):
    > v[long(line.strip())] = True


    On my system (windows vista) your code (using your data) runs in:

    36 seconds with python 2.4
    25 seconds with python 2.5
    39 seconds with python 3000


    i.
     
    Istvan Albert, Nov 12, 2007
    #3
  4. > > and see it take about 45 minutes with this:
    > >
    > > $ cat cache-keys.py
    > > #!/usr/bin/python
    > > v = {}
    > > for line in open('keys.txt'):
    > > v[long(line.strip())] = True

    >
    > On my system (windows vista) your code (using your data) runs in:
    >
    > 36 seconds with python 2.4
    > 25 seconds with python 2.5
    > 39 seconds with python 3000


    *wow*

    The win32 Python or the cygwin Python?

    What CPU architecture?
     
    Michael Bacarella, Nov 12, 2007
    #4
  5. On Nov 12, 12:39 pm, "Michael Bacarella" <> wrote:

    > The win32 Python or the cygwin Python?
    >
    > What CPU architecture?


    it is the win32 version, a dual core laptop with T5220 Core 2
     
    Istvan Albert, Nov 12, 2007
    #5
  6. Michael Bacarella

    MrJean1 Guest

    On MacOS** your version

    #!/usr/bin/python
    v = {}
    for line in open('keys.txt'):
    v[long(line.strip())] = True

    using these <http://www.sendspace.com/file/9530i7> keys takes

    24.9 secs with ActivePython 2.5.1
    and
    212.3 secs with Apple's Python 2.3.5.


    However, this version

    #!/usr/bin/python
    v = {}
    for line in open('keys.txt'):
    v[line] = True

    takes
    16.5 secs with ActivePython 2.5.1
    and
    18.7 secs with Apple's Python 2.3.5.


    /Jean Brouwers

    **) MacOS X 10.4.10 on Mini Mac, 1.83 GHz Intel Core Duo, 2 GB RAM.
     
    MrJean1, Nov 12, 2007
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Michael Bacarella

    Populating a dictionary, fast

    Michael Bacarella, Nov 10, 2007, in forum: Python
    Replies:
    4
    Views:
    526
    Istvan Albert
    Nov 11, 2007
  2. Michael Bacarella

    Re: Populating a dictionary, fast

    Michael Bacarella, Nov 11, 2007, in forum: Python
    Replies:
    6
    Views:
    543
    Hrvoje Niksic
    Nov 12, 2007
  3. Michael Bacarella

    Re: Populating a dictionary, fast

    Michael Bacarella, Nov 11, 2007, in forum: Python
    Replies:
    3
    Views:
    277
    Ben Finney
    Nov 11, 2007
  4. Michael Bacarella

    Re: Populating a dictionary, fast

    Michael Bacarella, Nov 11, 2007, in forum: Python
    Replies:
    2
    Views:
    466
    Jeffrey Froman
    Nov 12, 2007
  5. Michael Bacarella

    Re: Populating a dictionary, fast

    Michael Bacarella, Nov 11, 2007, in forum: Python
    Replies:
    16
    Views:
    707
    Francesc Altet
    Nov 21, 2007
Loading...

Share This Page