huge hash creates Illegal instruction (core dumped)

Discussion in 'Perl Misc' started by Hauke Juhls, Jul 3, 2003.

  1. Hauke Juhls

    Hauke Juhls Guest

    Hi,
    I hope anyone out there can help me... I have the following problem:
    I create a huge hash of lists (~32000 keys in the hash) and the list
    contain ~1000 different strings each. Unfortunately this seems to be
    too large - when I watch the process increasing in memory it freezes
    after occupying ~256MB of RAM - then the perl script crashes with
    "Illegal instruction (core dumped)".
    I am using perl version 5.005_03 on an RS6000 running AIX 4.
    Anyone else had that problem before? I'm thankful for any suggestion
    you make!

    Thanks,
    Hauke!
    Hauke Juhls, Jul 3, 2003
    #1
    1. Advertising

  2. Hauke Juhls wrote:
    > Hi,
    > I hope anyone out there can help me... I have the following problem:
    > I create a huge hash of lists (~32000 keys in the hash) and the list
    > contain ~1000 different strings each. Unfortunately this seems to be
    > too large - when I watch the process increasing in memory it freezes
    > after occupying ~256MB of RAM - then the perl script crashes with
    > "Illegal instruction (core dumped)".
    > I am using perl version 5.005_03 on an RS6000 running AIX 4.
    > Anyone else had that problem before? I'm thankful for any suggestion
    > you make!


    Read the section in AIX Developers Guide which refers to "Large Program
    Support". Basically, AIX divides user process space into 256 MB
    segments. Unless you take special measures at link time, your
    executable is inherently limited to 256MB stack + data.

    You can either relink the Perl binary using the '-bmaxdata:0xn0000000'
    switch (replacing 'n' with the number of 256MB segments to be permitted,
    where 1 <= n <= 8) or modify the binary by direct patching.

    Attached is a little script I use to muck directly with the binary.

    <obligatory_warning>

    Messing directly with a critical binary file is inherently dangerous,
    and may require root access to the system. Make a backup copy and work
    carefully. If things break, you get to keep all the pieces!!

    </obligatory_warning>

    Steve

    #! /bin/ksh

    # This script modifies 32-bit XCOFF variable o_maxdata (+4C).
    # Non-zero o_maxdata uses the AIX "Large Address Space Model."
    # See "AIX: General Programming Concepts, Writing and Debugging Pgms."

    # exactly two parameters?
    if [[ $# = 0 ]]
    then
    print 'usage: "maxdata filename [ 0-8 ]"' ;
    print '\tThe second parameter (if present) specifies units of 256M.' ;
    print '\tIf omitted, the current value of maxdata is displayed.' ;
    print '\te.g.: "maxdata /bin/xldb 4" sets /bin/xldb maxdata to 1G.' ;
    exit ;
    fi

    # first parm a 32-bit xcoff?
    file $1 | grep -q 'executable (RISC System/6000) or object module' ;
    if [[ $? -ne 0 ]]
    then
    print "$1 isn't a readable 32-bit XCOFF file." ;
    exit ;
    fi

    if [[ $# = 1 ]]; then
    # Just display current setting.

    amount=`od -HAn -N4 -j76 $1 | head -1 | sed -e 's/^[ ]*//g'`
    case $amount in
    0* ) print "$1 is not using a large address model"; exit ;;
    1* ) int=256 ;;
    2* ) int=512 ;;
    3* ) int=768 ;;
    4* ) int=1024 ;;
    5* ) int=1269 ;;
    6* ) int=1525 ;;
    7* ) int=1781 ;;
    8* ) int=2048 ;;
    esac
    print "$1 currently has maxdata setting of ${int}M"

    elif [[ $# = 2 ]]; then
    # Set a value

    # ... and writable?
    if [[ ! -w $1 ]]
    then
    print "$1 isn't a writable and executable file." ;
    exit ;
    fi

    # second parm equals 0,1,2,...,8 ?
    case ${2:-4} in
    [0-8] ) typeset -i8 d=${2:-4}*16 ;
    typeset -i10 m=${2:-4}*256 ;;
    * ) print 'The second parameter has to be an integer [0-8].' ;
    exit 1 ;;
    esac

    # Do it:
    print -n "$1 o_maxdata before: " ; od -HAx -N4 -j76 $1 | head -1 ;

    eval "print -n '\\0'${d#*#}'\\0\\0\\0' |
    dd of=$1 bs=4 count=1 seek=19 conv=notrunc 2>/dev/null" ;

    print -n "$1 o_maxdata after: " ; od -HAx -N4 -j76 $1 | head -1 ;

    print "Maxdata in $1 now set to ${m}M." ;
    fi
    Steven N. Hirsch, Jul 4, 2003
    #2
    1. Advertising

  3. Hauke Juhls

    Hauke Juhls Guest

    Thanks Steven, that helped me a lot!

    "Steven N. Hirsch" <> wrote in message news:<YIgNa.26977$>...
    > #! /bin/ksh
    >
    > # This script modifies 32-bit XCOFF variable o_maxdata (+4C).
    > # Non-zero o_maxdata uses the AIX "Large Address Space Model."
    > # See "AIX: General Programming Concepts, Writing and Debugging Pgms."
    >
    > # exactly two parameters?
    > if [[ $# = 0 ]]
    > then
    > print 'usage: "maxdata filename [ 0-8 ]"' ;
    > print '\tThe second parameter (if present) specifies units of 256M.' ;
    > print '\tIf omitted, the current value of maxdata is displayed.' ;
    > print '\te.g.: "maxdata /bin/xldb 4" sets /bin/xldb maxdata to 1G.' ;
    > exit ;
    > fi
    >
    > # first parm a 32-bit xcoff?
    > file $1 | grep -q 'executable (RISC System/6000) or object module' ;
    > if [[ $? -ne 0 ]]
    > then
    > print "$1 isn't a readable 32-bit XCOFF file." ;
    > exit ;
    > fi
    >
    > if [[ $# = 1 ]]; then
    > # Just display current setting.
    >
    > amount=`od -HAn -N4 -j76 $1 | head -1 | sed -e 's/^[ ]*//g'`
    > case $amount in
    > 0* ) print "$1 is not using a large address model"; exit ;;
    > 1* ) int=256 ;;
    > 2* ) int=512 ;;
    > 3* ) int=768 ;;
    > 4* ) int=1024 ;;
    > 5* ) int=1269 ;;
    > 6* ) int=1525 ;;
    > 7* ) int=1781 ;;
    > 8* ) int=2048 ;;
    > esac
    > print "$1 currently has maxdata setting of ${int}M"
    >
    > elif [[ $# = 2 ]]; then
    > # Set a value
    >
    > # ... and writable?
    > if [[ ! -w $1 ]]
    > then
    > print "$1 isn't a writable and executable file." ;
    > exit ;
    > fi
    >
    > # second parm equals 0,1,2,...,8 ?
    > case ${2:-4} in
    > [0-8] ) typeset -i8 d=${2:-4}*16 ;
    > typeset -i10 m=${2:-4}*256 ;;
    > * ) print 'The second parameter has to be an integer [0-8].' ;
    > exit 1 ;;
    > esac
    >
    > # Do it:
    > print -n "$1 o_maxdata before: " ; od -HAx -N4 -j76 $1 | head -1 ;
    >
    > eval "print -n '\\0'${d#*#}'\\0\\0\\0' |
    > dd of=$1 bs=4 count=1 seek=19 conv=notrunc 2>/dev/null" ;
    >
    > print -n "$1 o_maxdata after: " ; od -HAx -N4 -j76 $1 | head -1 ;
    >
    > print "Maxdata in $1 now set to ${m}M." ;
    > fi
    >
    > --
    Hauke Juhls, Jul 8, 2003
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Nish
    Replies:
    1
    Views:
    298
    red floyd
    Sep 3, 2004
  2. Pieter Droogendijk

    segmentation fault (core dumped)

    Pieter Droogendijk, Dec 20, 2003, in forum: C Programming
    Replies:
    13
    Views:
    4,529
    no_name
    Jan 5, 2004
  3. su
    Replies:
    5
    Views:
    346
  4. DanielJohnson

    Segmentation Fault (core dumped)

    DanielJohnson, Feb 18, 2007, in forum: C Programming
    Replies:
    29
    Views:
    837
    Mark McIntyre
    Feb 22, 2007
  5. Glen Hendry
    Replies:
    2
    Views:
    283
    James Willmore
    Oct 9, 2003
Loading...

Share This Page