PLEASE HELP ABOUT HASH!

Discussion in 'Perl Misc' started by news.hinet.net, Mar 2, 2004.

  1. I have the vast data in mysql about (1.5G)
    I got some data from machine and want to search some field
    that does not exist in mysql. i just want to insert these data that
    does not exist in mysql before.

    I use hash to handle these data.

    First i load data into hash from mysql and search data if it is exist.
    but i use the vast memory at the same time

    how to solve this program??

    if i insert all data by setup db field data is unique. this will spend a lot
    of time.


    give me some suggest!!
    ps:the data is more than 500M one time insert process
     
    news.hinet.net, Mar 2, 2004
    #1
    1. Advertising

  2. On Tue, 02 Mar 2004 10:57:54 +0800, news.hinet.net wrote:

    > I have the vast data in mysql about (1.5G)
    > I got some data from machine and want to search some field
    > that does not exist in mysql. i just want to insert these data that
    > does not exist in mysql before.
    >
    > I use hash to handle these data.
    >
    > First i load data into hash from mysql and search data if it is exist.
    > but i use the vast memory at the same time
    >
    > how to solve this program??
    >
    > if i insert all data by setup db field data is unique. this will spend a lot
    > of time.
    >
    >
    > give me some suggest!!
    > ps:the data is more than 500M one time insert process


    This is all very interesting, but where's the code you use at present?
    That *may* shed some light on this most wonderful issue.

    Right now, the only thing I could suggest is write to a file instead of a
    hash and then parse the file - one line at a time.

    OTOH, maybe you need to use SQL more efficently.

    Don't know without the code.

    --
    Jim

    Copyright notice: all code written by the author in this post is
    released under the GPL. http://www.gnu.org/licenses/gpl.txt
    for more information.

    a fortune quote ...
    Pascal, n.: A programming language named after a man who would
    <turn over in his grave if he knew about it.
     
    James Willmore, Mar 2, 2004
    #2
    1. Advertising

  3. news.hinet.net

    Guest

    "news.hinet.net" <> wrote:
    > I have the vast data in mysql about (1.5G)
    > I got some data from machine and want to search some field
    > that does not exist in mysql.


    I think you mean that the field exists, and you wish to identify values for
    that field which do not exist.

    > i just want to insert these data that
    > does not exist in mysql before.


    Is the "insert ignore..." MySQL-specific extension to sql what you want?

    > I use hash to handle these data.
    >
    > First i load data into hash from mysql and search data if it is exist.
    > but i use the vast memory at the same time
    >
    > how to solve this program??


    Huge hashes take huge amounts of memory. There is no ready solution to
    that problem.

    >
    > if i insert all data by setup db field data is unique. this will spend a
    > lot of time.


    Please explain in code what you mean by this, and specify in numbers the
    amount of time it would take (and in what an acceptable time would be).

    > give me some suggest!!
    > ps:the data is more than 500M one time insert process


    It is unlikely that the programming, debugging, and run-time of a
    sophisticated one-time only solution will be less than the run-time of the
    naive "insert ignore..." solution.



    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Mar 2, 2004
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    4
    Views:
    565
    Chris Uppal
    May 5, 2005
  2. KK
    Replies:
    2
    Views:
    706
    Big Brian
    Oct 14, 2003
  3. MuZZy
    Replies:
    7
    Views:
    1,823
    Mike Hewson
    Jan 7, 2005
  4. rp
    Replies:
    1
    Views:
    582
    red floyd
    Nov 10, 2011
  5. Srijayanth Sridhar
    Replies:
    19
    Views:
    670
    David A. Black
    Jul 2, 2008
Loading...

Share This Page