/var/adm/wtmp, malloc, and ENOMEM

Discussion in 'C Programming' started by Maja, Dec 10, 2004.

  1. Maja

    Maja Guest

    Greetings,

    /var/adm/wtmp files were getting out of hand
    and I wanted to filter the file rather than zeroing it out.
    I found some code that would only keep the last x days
    but it required creating a temporary file in /tmp
    and then copying it back to /var/adm. I rewrote the code
    to load the contents of the wtmp file into memory and then
    write the filtered records in memory directly back to /var/adm/wtmp,
    eliminating the need for a temporary file.

    This worked great until I encountered a wtmp file that was
    370 MB. malloc fails setting errno to ENOMEM (not enough
    storage space). My system has 16GB of memory and 9GB of paging
    (33% used). So, I should have plenty of memory.

    My question: is there a workaround for this type of malloc error.
    Thanks,
    Maja
    Maja, Dec 10, 2004
    #1
    1. Advertising

  2. Maja

    dandelion Guest

    "Maja" <> wrote in message
    news:...
    <snip>

    > This worked great until I encountered a wtmp file that was
    > 370 MB. malloc fails setting errno to ENOMEM (not enough
    > storage space). My system has 16GB of memory and 9GB of paging
    > (33% used). So, I should have plenty of memory.


    Hmmm... By the sound of it (you do not provide the actual call to malloc
    which fails), i'd say you're right and this is a bug somewhere. However, in
    order to make a specific diagnosis, i'd need to know which implementation of
    'malloc' you are using (there are quite a few around) and check it out.
    Something i'm not planning to do.

    The call *should* succeed, AFAICT, so if it doesn't, it's an implementation
    issue,you can best pass on to the vendor of your compiler or ask this
    question again on a newsgroup that is dedicated to your OS/compiler.

    HTH (a tiny bit)

    regards,

    dandelion.
    dandelion, Dec 10, 2004
    #2
    1. Advertising

  3. Maja wrote:

    > Greetings,
    >
    > /var/adm/wtmp files were getting out of hand
    > and I wanted to filter the file rather than zeroing it out.
    > I found some code that would only keep the last x days
    > but it required creating a temporary file in /tmp
    > and then copying it back to /var/adm. I rewrote the code
    > to load the contents of the wtmp file into memory and then
    > write the filtered records in memory directly back to /var/adm/wtmp,
    > eliminating the need for a temporary file.
    >
    > This worked great until I encountered a wtmp file that was
    > 370 MB. malloc fails setting errno to ENOMEM (not enough
    > storage space). My system has 16GB of memory and 9GB of paging
    > (33% used). So, I should have plenty of memory.
    >
    > My question: is there a workaround for this type of malloc error.
    > Thanks,
    > Maja
    >


    Your answer is system specific. You probably would get a better answer
    in (looking at what you have) comp.unix.programmer. That said, I think
    the temporary file is good, because you may some day hit a file big
    enough (say, > 2 or 4 GB) that you'd likely have problems with, at least
    on a 32 bit machine.

    [OT] -- I can reproduce the behavior on linux if I set "ulimit -v" to
    something less than the amount I'm trying to malloc. Perhaps
    you have some limits imposed that way... If that doesn't help,
    perhaps you have some issues in your standard library malloc...

    -David
    David Resnick, Dec 10, 2004
    #3
  4. On Fri, 10 Dec 2004 06:56:55 -0800, Maja wrote:

    > Greetings,
    >
    > /var/adm/wtmp files were getting out of hand
    > and I wanted to filter the file rather than zeroing it out.
    > I found some code that would only keep the last x days
    > but it required creating a temporary file in /tmp
    > and then copying it back to /var/adm. I rewrote the code
    > to load the contents of the wtmp file into memory and then
    > write the filtered records in memory directly back to /var/adm/wtmp,
    > eliminating the need for a temporary file.
    >
    > This worked great until I encountered a wtmp file that was
    > 370 MB. malloc fails setting errno to ENOMEM (not enough
    > storage space). My system has 16GB of memory and 9GB of paging
    > (33% used). So, I should have plenty of memory.


    The limitations on what malloc() can allocate are down to your platform,
    well beyond the scope of the C language itself. Perhaps there are
    configurable per-process limits etc. Also, if you are only storing
    filtered records in memory the memory used would most likely be less,
    perhaps a lot less, than 370MB. Perhaps you have a bug in your code and
    you are requesting much more memory than you thought. That's easy to log.

    > My question: is there a workaround for this type of malloc error.


    Request less memory or configure your system to allow more memory to be
    requested.

    However I question the whole approach of storing the data in memory. If
    your program has a problem after it starts to write data back to wtmp the
    data not written will be lost. Writing to a temporary file is a much
    better approach. Just write to a temporary file in /var/adm and rename the
    file back to wtmp when complete. This could well be MORE efficient than
    storing large amounts of data in memory before writing, if efficiency is
    your concern.

    Lawrence
    Lawrence Kirby, Dec 10, 2004
    #4
  5. "Maja" <> writes:
    [...]
    > This worked great until I encountered a wtmp file that was
    > 370 MB. malloc fails setting errno to ENOMEM (not enough
    > storage space). My system has 16GB of memory and 9GB of paging
    > (33% used). So, I should have plenty of memory.


    The amount of memory available on your system isn't necessarily the
    same as the amount of memory available to your program.

    --
    Keith Thompson (The_Other_Keith) <http://www.ghoti.net/~kst>
    San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
    We must do something. This is something. Therefore, we must do this.
    Keith Thompson, Dec 10, 2004
    #5
  6. Maja

    Maja Guest

    It should have occured to me I might be up against
    a system limit. There's a -bmaxdata compile flag
    I used to allow the program to allocate memory over
    256 meg.

    Many thanks for pointing me in the right direction.
    Maja
    Maja, Dec 10, 2004
    #6
  7. In article <41b9be33$0$51241$4all.nl>, "dandelion" <> writes:
    >
    > "Maja" <> wrote in message
    > news:...
    >
    > > This worked great until I encountered a wtmp file that was
    > > 370 MB. malloc fails setting errno to ENOMEM (not enough
    > > storage space). My system has 16GB of memory and 9GB of paging
    > > (33% used). So, I should have plenty of memory.

    >
    > Hmmm... By the sound of it (you do not provide the actual call to malloc
    > which fails), i'd say you're right and this is a bug somewhere.


    Nonsense. malloc is allowed to fail for any reason. Assuming a bug
    here is like assuming that someone stole your car if you can't find
    your keys.

    Off topic: The reference to wtmp implies this is a POSIX system. Most
    POSIX systems (maybe all; I'm not going to check the standard to see
    if this is required) provide a mechanism to limit how much memory a
    process can allocate. This is a Good Thing, and has been part of Unix
    for a long time. Consult the documentation for the setrlimit system
    call and the ulimit command (really commands, since various shells
    provide various flavors of ulimit builtins).

    > The call *should* succeed, AFAICT,


    You can't tell. None of us can; we don't have nearly enough
    information. Certainly the amount of physical and virtual memory in
    the system doesn't suffice. (In fact, depending on the OS, it may
    not even be relevant.)

    And, once again, we have an excellent demonstration why answering OT
    questions here is a bad idea.

    > so if it doesn't, it's an implementation issue,


    Or it isn't, since this is a hosted implementation, and it may just
    be reporting - correctly - that the hosting OS has denied its request.

    > you can best pass on to the vendor of your compiler or ask this
    > question again on a newsgroup that is dedicated to your OS/compiler.


    Indeed.

    --
    Michael Wojcik

    The antics which have been drawn together in this book are huddled here
    for mutual protection like sheep. If they had half a wit apiece each
    would bound off in many directions, to unsimplify the target. -- Walt Kelly
    Michael Wojcik, Dec 10, 2004
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. thomson
    Replies:
    10
    Views:
    2,469
    Eliyahu Goldin
    Jun 20, 2005
  2. thomson
    Replies:
    0
    Views:
    358
    thomson
    Jun 20, 2005
  3. Alex Vinokur
    Replies:
    4
    Views:
    461
    Ron Natalie
    Sep 24, 2003
  4. Roger Pack
    Replies:
    5
    Views:
    177
    Roger Pack
    Nov 16, 2009
  5. Tom Shealy
    Replies:
    1
    Views:
    647
    Robert Klemme
    Mar 6, 2010
Loading...

Share This Page