Re: FileChannel.map() gives 'cannot allocate memory'

Discussion in 'Java' started by Roedy Green, Aug 13, 2003.

  1. Roedy Green

    Roedy Green Guest

    On Wed, 13 Aug 2003 15:38:37 -0400, Ian Soboroff <>
    wrote or quoted :

    > But mapping 1MB
    >'pages' causes me to get an IOException from the native map0() call
    >saying 'cannot allocate memory' at some random point in my program.


    The JVM has only a 2 gig address space all told. I would hazard a
    guess that it allocates virtual RAM for whacking huge native map
    chunks say 10+ MB in a way different from small ones. I would guess
    it allocates outside the heap for huge ones, and uses the standard gc
    heap for small ones.


    Allocating a 1 MB contiguous object is going to tax the limits of the
    JVM. Try giving it some extra virtual ram on your java.exe command
    line. See http://mindprod.com/jgloss/javaexe.html

    --
    Canadian Mind Products, Roedy Green.
    Coaching, problem solving, economical contract programming.
    See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
     
    Roedy Green, Aug 13, 2003
    #1
    1. Advertising

  2. Roedy Green

    Ian Soboroff Guest

    Roedy Green <> writes:

    > Allocating a 1 MB contiguous object is going to tax the limits of the
    > JVM. Try giving it some extra virtual ram on your java.exe command
    > line. See http://mindprod.com/jgloss/javaexe.html


    Try running the sample code a few times. It allocates a 1MB map just
    fine. Allocate and re-allocate over and over and at some point the VM
    says it can't allocate memory for it.

    Note the program doesn't keep the old map.

    But, trying your suggestion:

    $ java -Xms256m -Xmx512m -cp . Test j2sdk-1_4_2-linux-i586.rpm 10000 1000000
    Mapping /usr/local/src/Java/Sun/1.4/1.4.2/j2sdk-1_4_2-linux-i586.rpm
    size of file: 34653233
    size of page: 1000000
    iterations : 10000
    Exception occurred at count 1510 trying to map from 30000000 to 31000000
    (1461 faults so far)
    Exception in thread "main" java.io.IOException: Cannot allocate memory
    at sun.nio.ch.FileChannelImpl.map0(Native Method)
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:705)
    at Test.main(Test.java:58)


    Ian
     
    Ian Soboroff, Aug 13, 2003
    #2
    1. Advertising

  3. Roedy Green

    Ian Soboroff Guest

    Roedy Green <> writes:

    > On Wed, 13 Aug 2003 17:18:38 -0400, Ian Soboroff <>
    > wrote or quoted :
    >
    >>Try running the sample code a few times. It allocates a 1MB map just
    >>fine. Allocate and re-allocate over and over and at some point the VM
    >>says it can't allocate memory for it.

    >
    > What you want to find out is:
    >
    > 1. how much free ram is there at each stage. Is something packratting
    > the old allocations?
    >
    > 2. Just before you die is the problem lack of ram or lack of
    > contiguous ram?
    >
    > See http://mindprod.com/jgloss/packratting.html


    Tricky to find out, but I'm running the VM on the test code with a
    start size of 256MB and a max heap of 512MB. The file I'm mapping is
    about 35MB, and I'm only mapping a 1MB region of it.

    As for packratting, MappedByteBuffers don't have a dispose() method.
    Setting the reference to null before remapping doesn't help. The
    exception is thrown from a native method
    (sun.nio.ch.FileChannelImpl.map0(Native Method)), so if memory leakage
    is happening it's on the C level.

    In fact, I'm pretty sure it's leakage in the native method side. If
    I divide the map size by a factor of 10, but increase the number of
    iterations by 10, I get a crash after 10x iterations (where x is the
    number if iterations before I get an exception in the first case):

    $ java -Xms256m -Xmx512m -cp . Test j2sdk-1_4_2-linux-i586.rpm 10000 1000000
    Mapping j2sdk-1_4_2-linux-i586.rpm
    size of file: 34653233
    size of page: 1000000
    iterations : 10000
    Exception occurred at count 1500 trying to map from 0 to 1000000
    (1460 faults so far)
    Exception in thread "main" java.io.IOException: Cannot allocate memory
    at sun.nio.ch.FileChannelImpl.map0(Native Method)
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:705)
    at Test.main(Test.java:59)

    $ java -Xms256m -Xmx512m -cp . Test j2sdk-1_4_2-linux-i586.rpm 100000 100000
    Mapping j2sdk-1_4_2-linux-i586.rpm
    size of file: 34653233
    size of page: 100000
    iterations : 100000
    Exception occurred at count 14071 trying to map from 22400000 to 22500000
    (14036 faults so far)
    Exception in thread "main" java.io.IOException: Cannot allocate memory
    at sun.nio.ch.FileChannelImpl.map0(Native Method)
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:705)
    at Test.main(Test.java:59)

    It's something related to map overhead, but I would have expected that
    to be constant per map setup/teardown, rather than related to the size
    of the mapping.

    Ian
     
    Ian Soboroff, Aug 14, 2003
    #3
  4. Roedy Green

    Roedy Green Guest

    On Thu, 14 Aug 2003 08:29:00 -0400, Ian Soboroff <>
    wrote or quoted :

    >
    >It's something related to map overhead, but I would have expected that
    >to be constant per map setup/teardown, rather than related to the size
    >of the mapping.


    Look up http://mindprod.com/jgloss/profiler.html

    You need a tool that will let you find out what flavours of objects
    were gobbling ram just before you die. That may be a clue if it is
    your fault or Sun's.

    I get the impression there may be some JNI involved. Is it possible
    to temporarily turn it off or roughly simulate it with some Java code.

    --
    Canadian Mind Products, Roedy Green.
    Coaching, problem solving, economical contract programming.
    See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
     
    Roedy Green, Aug 14, 2003
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Spendius
    Replies:
    4
    Views:
    2,311
    Harald Hein
    Jul 4, 2003
  2. Paul J. Lucas
    Replies:
    12
    Views:
    1,490
    Paul J. Lucas
    Aug 19, 2005
  3. rveloso

    popen cannot allocate memory

    rveloso, Apr 4, 2006, in forum: C Programming
    Replies:
    1
    Views:
    1,174
    Keith Thompson
    Apr 4, 2006
  4. Replies:
    8
    Views:
    1,930
    Csaba
    Feb 18, 2006
  5. Andreas
    Replies:
    1
    Views:
    1,694
    Piet van Oostrum
    Mar 18, 2009
Loading...

Share This Page