parse/slice/...

Discussion in 'Python' started by rcmn, Jan 6, 2009.

  1. rcmn

    rcmn Guest

    I'm not sure how to call it sorry for the subject description.
    Here what i'm trying to accomplish.
    the script i'm working on, take a submitted list (for line in file)
    and generate thread for it. unfortunately winxp has a limit of 500
    thread . So I have to parse/slice the file by chunk of 500 and loop
    until the list is done.
    I though i would of done it in no time but i can't get started for
    some reason.
    And i can't find a good way to do it efficiently . Does anyone have
    something similar to this.

    thank you
    rcmn, Jan 6, 2009
    #1
    1. Advertising

  2. rcmn

    Guest

    rcmn> So I have to parse/slice the file by chunk of 500 and loop until
    rcmn> the list is done. I though i would of done it in no time but i
    rcmn> can't get started for some reason.

    Check the itertools module. Probably itertools.islice.

    --
    Skip Montanaro - - http://smontanaro.dyndns.org/
    , Jan 6, 2009
    #2
    1. Advertising

  3. On Tue, 2009-01-06 at 11:23 -0800, rcmn wrote:
    > I'm not sure how to call it sorry for the subject description.
    > Here what i'm trying to accomplish.
    > the script i'm working on, take a submitted list (for line in file)
    > and generate thread for it. unfortunately winxp has a limit of 500
    > thread . So I have to parse/slice the file by chunk of 500 and loop
    > until the list is done.
    > I though i would of done it in no time but i can't get started for
    > some reason.
    > And i can't find a good way to do it efficiently . Does anyone have
    > something similar to this.


    If your thread is smart enough to handle the file you can do something
    like:

    class Worker(threading.Thread):
    def __init__(self, file_object):
    self.file_object = file_object
    threading.Thread.__init__(self)

    def run(self):
    while True:
    line = self.file_object.readline()
    if line == '':
    return
    # process line

    NUM_WORKERS = 500
    bigfile = open('somefile', 'r')
    workers = [Worker(bigfile) for i in range(NUM_WORKERS)]
    for worker in workers:
    worker.start()
    Albert Hopkins, Jan 6, 2009
    #3
  4. rcmn

    Tim Arnold Guest

    "rcmn" <> wrote in message
    news:...
    > I'm not sure how to call it sorry for the subject description.
    > Here what i'm trying to accomplish.
    > the script i'm working on, take a submitted list (for line in file)
    > and generate thread for it. unfortunately winxp has a limit of 500
    > thread . So I have to parse/slice the file by chunk of 500 and loop
    > until the list is done.
    > I though i would of done it in no time but i can't get started for
    > some reason.
    > And i can't find a good way to do it efficiently . Does anyone have
    > something similar to this.
    >
    > thank you


    Here's how I work on a list a bunch of items (100 by default) ata time:

    def drain_list(tlist,step=None):
    if not step:
    step = 100
    j=0
    for i in range(step,len(tlist), step):
    yield tlist[j:i]
    j = i
    if j < len(tlist):
    yield tlist[j:]

    --Tim
    Tim Arnold, Jan 7, 2009
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. walala
    Replies:
    1
    Views:
    793
  2. Nicolas Matringe
    Replies:
    9
    Views:
    716
    Mike Treseler
    Jun 14, 2004
  3. Brandon

    Null slice? Synthesis in XST?

    Brandon, Aug 31, 2005, in forum: VHDL
    Replies:
    0
    Views:
    557
    Brandon
    Aug 31, 2005
  4. Replies:
    19
    Views:
    1,106
    Daniel Vallstrom
    Mar 15, 2005
  5. 7stud --

    optparse: parse v. parse! ??

    7stud --, Feb 20, 2008, in forum: Ruby
    Replies:
    3
    Views:
    175
    7stud --
    Feb 20, 2008
Loading...

Share This Page