Speed Freak

Discussion in 'Perl' started by Dave Bee, Dec 29, 2003.

  1. Dave Bee

    Dave Bee Guest

    This is a conceptual question rather than a specific coding one, but
    hopefully someone might have played around with something similar. In
    a nutshell, I have around 10 million information entries with lots of
    data points. My current script has two stages - the first organises
    certain data points of the data into large (huge) hashes, and the
    second stage forks off lots of children and does the subsequent
    processing to produce ldifs, using the information in the hashes
    (thanks to copy-on-write, and the fact that the children don't need to
    update the hashes this doesn't use a great deal of memory).

    My current problem is with stage one - it is, by current necessity, a
    single process, since it needs to refer to information within the
    hashes as it builds them, and the processing required by the single
    processor is the choke point here. I would like to cut down the
    current time it takes to do the first stage processing (~50 minutes)
    and I am at liberty to use any interesting techniques in order to do
    so - my hardware is somewhat above spec (24 CPU 6800, 48G RAM etc),
    and can be dedicated 100% to the script when it runs, so unusual and
    incredibly memory / CPU wasteful techniques are more than welcome.

    I've thought of threading (no real experience, but I could probably
    figure something out), parent hash-controller with multiple forked
    children etc, I'm just curious if anyone has done something similar
    and already knows the most efficient way of doing this.

    Dave
    Dave Bee, Dec 29, 2003
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ham

    I need speed Mr .Net....speed

    Ham, Oct 28, 2004, in forum: ASP .Net
    Replies:
    6
    Views:
    2,312
    Antony Baula
    Oct 29, 2004
  2. efiedler
    Replies:
    1
    Views:
    2,011
    Tim Ward
    Oct 9, 2003
  3. Replies:
    2
    Views:
    2,265
    Howard
    Apr 28, 2004
  4. Replies:
    2
    Views:
    330
    Christopher Benson-Manica
    Apr 28, 2004
  5. Daniel Pitts

    You know you're a Java freak when...

    Daniel Pitts, Dec 6, 2007, in forum: Java
    Replies:
    9
    Views:
    407
    Andreas Leitgeb
    Dec 9, 2007
Loading...

Share This Page