urllib2 script slowing and stopping

Discussion in 'Python' started by Dantium, Oct 11, 2010.

  1. Dantium

    Dantium Guest

    I have a small script that reads several CSV files in a directory and
    puts the data in a DB using Django.

    There are about 1.7 million records in 120 CSV files, I am running the
    script on a VPS with about 512mb of memory python 2.6.5 on ubuntu
    10.04.

    The script gets slow and seems to lock after about 870000 records.
    running top show that the memory is all being used up y the python
    process, is there someway I can improve on this script?


    class Command(BaseCommand):

    def handle(self, *args, **options):
    count = 0
    d = urllib2.urlopen(postcode_dir).read()
    postcodefiles = re.findall('<a href="(.*?\.csv)">', d)
    nprog = 0

    for n in range(nprog, len(postcodefiles)):
    fl = postcodefiles[n]
    print 'Processing %d %s ...' % (n, fl)
    s = urllib2.urlopen(postcode_dir + fl)
    c = csv.reader(s.readlines())
    for row in c:
    postcode = row[0]
    location = Point(map(float, row[10:12]))
    Postcode.objects.create(code=postcode,
    location=location)
    count += 1
    if count % 10000 == 0:
    print "Imported %d" % count
    s.close()
    nprog = n+1



    Thanks

    -Dan
     
    Dantium, Oct 11, 2010
    #1
    1. Advertising

  2. Dantium

    Ian Guest

    On Oct 11, 2:48 pm, Dantium <> wrote:
    > I have a small script that reads several CSV files in a directory and
    > puts the data in a DB using Django.
    >
    > There are about 1.7 million records in 120 CSV files, I am running the
    > script on a VPS with about 512mb of memory python 2.6.5 on ubuntu
    > 10.04.
    >
    > The script gets slow and seems to lock after about 870000 records.
    > running top show that the memory is all being used up y the python
    > process, is there someway I can improve on this script?


    Probably you have "DEBUG = True" in your Django settings.py file. In
    debug mode, Django records every query that is executed in
    django.db.connection.queries. To fix it, either disable debug mode or
    just have your script go in and clear out that list from time to time.

    HTH,
    Ian
     
    Ian, Oct 11, 2010
    #2
    1. Advertising

  3. Dantium

    Dantium Guest

    On Oct 11, 10:07 pm, Ian <> wrote:
    > On Oct 11, 2:48 pm, Dantium <> wrote:
    >
    > > I have a small script that reads several CSV files in a directory and
    > > puts the data in a DB using Django.

    >
    > > There are about 1.7 million records in 120 CSV files, I am running the
    > > script on a VPS with about 512mb of memory python 2.6.5 on ubuntu
    > > 10.04.

    >
    > > The script gets slow and seems to lock after about 870000 records.
    > > running top show that the memory is all being used up y the python
    > > process, is there someway I can improve on this script?

    >
    > Probably you have "DEBUG = True" in your Django settings.py file.  In
    > debug mode, Django records every query that is executed in
    > django.db.connection.queries.  To fix it, either disable debug mode or
    > just have your script go in and clear out that list from time to time.
    >
    > HTH,
    > Ian


    Yeah thanks that helped!

    It was still running really low on memory by the end though but they
    all got added.
     
    Dantium, Oct 11, 2010
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Andy Oakey

    ASP.NET application slowing down

    Andy Oakey, Aug 12, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    400
    S. Justin Gengo
    Aug 12, 2003
  2. moondaddy
    Replies:
    4
    Views:
    539
    Steven Cheng[MSFT]
    May 6, 2004
  3. Bruce Hodge

    Slow Connections Slowing Down Site.

    Bruce Hodge, Nov 16, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    360
    =?Utf-8?B?RURGIFNvZnR3YXJl?=
    Nov 29, 2004
  4. =?Utf-8?B?QmFobWFu?=

    page refresh slowing down

    =?Utf-8?B?QmFobWFu?=, Dec 21, 2005, in forum: ASP .Net
    Replies:
    3
    Views:
    381
    bhawin13
    Dec 22, 2005
  5. Josef Cihal
    Replies:
    0
    Views:
    883
    Josef Cihal
    Sep 5, 2005
Loading...

Share This Page