Using malloc/free in a tight loop

Discussion in 'C Programming' started by Navaneeth, Nov 17, 2010.

  1. Navaneeth

    Navaneeth Guest

    My application will be looping for 5000 times. On each iteration, it
    has to create a string and pass that into other methods for
    processing. This string length will only be known at the runtime.

    So in each iteration, this is what happens.

    ptr = malloc
    process(ptr)
    free(ptr)

    I am wondering doing frequent malloc and free is ok? Will this
    fragment the memory?

    My development environment is GCC and Linux. But I'd be happy to know
    about the implementation in windows also.
     
    Navaneeth, Nov 17, 2010
    #1
    1. Advertising

  2. On Nov 17, 11:02 am, Navaneeth <> wrote:
    > My application will be looping for 5000 times. On each iteration, it
    > has to create a string and pass that into other methods for
    > processing. This string length will only be known at the runtime.
    >
    > So in each iteration, this is what happens.
    >
    > ptr = malloc
    > process(ptr)
    > free(ptr)
    >
    > I am wondering doing frequent malloc and free is ok? Will this
    > fragment the memory?
    >
    > My development environment is GCC and Linux. But I'd be happy to know
    > about the implementation in windows also.
    >

    Unlikely. When you create strings and delete strings at random you
    might fragment the memory, but if you are matching allocations to
    frees then the system will usually resuse the memory. (If you print
    out the pointers with %p you'll probably find that you get the same
    value time after time).

    Unless the strings are extremely long then a modern computer isn't
    going to notice allocating 5000 of them.
     
    Malcolm McLean, Nov 17, 2010
    #2
    1. Advertising

  3. Navaneeth

    Navaneeth Guest

    > Unless the strings are extremely long then a modern computer isn't
    > going to notice allocating 5000 of them.


    Thanks. My strings are not long. May be 20 characters.

    So in general, can malloc fragment the memory? I was expecting the
    modern implementations will be clever enough to reduce this problem.
    Isn't that true?
     
    Navaneeth, Nov 17, 2010
    #3
  4. On Wed, 17 Nov 2010 04:00:25 -0800 (PST), Navaneeth <>
    wrote:
    >> Unless the strings are extremely long then a modern computer isn't
    >> going to notice allocating 5000 of them.

    >
    > Thanks. My strings are not long. May be 20 characters.


    You know, if you have a very short maximum cap for the length, you
    could just allocate that and reuse the buffer. (You could also by
    default reuse the buffer and only reallocate a larger one as it
    becomes necessary.)

    That said, the first response was correct in that it's not likely to
    make much of a difference in this quite simple case. In general though,
    I would avoid unnecessary repeated allocations and frees in a tight
    loop.

    > So in general, can malloc fragment the memory?


    In general, it can indeed. Especially if your processing step here
    involves allocating space that _isn't_ freed, it can happen in your
    example too, but probably not to a large extent. (The malloc
    implementation just needs to be smart enough to reuse the space
    left by the previous freed buffer, in the common case that it's
    sufficiently large).

    --
    Mikko Rauhala <> - http://www.iki.fi/mjr/blog/
    The Finnish Pirate Party - http://piraattipuolue.fi/
    World Transhumanist Association - http://transhumanism.org/
    Singularity Institute - http://singinst.org/
     
    Mikko Rauhala, Nov 17, 2010
    #4
  5. Navaneeth

    JohnF Guest

    Navaneeth <> wrote:
    > My application will be looping for 5000 times. On each iteration, it
    > has to create a string and pass that into other methods for
    > processing. This string length will only be known at the runtime.
    > So in each iteration, this is what happens.
    > ptr = malloc
    > process(ptr)
    > free(ptr)
    > I am wondering doing frequent malloc and free is ok? Will this
    > fragment the memory?


    As per MM's reply, not a problem. Nevertheless, I usually do
    this kind of thing, when I very temporarily need lots of short
    strings, with a short function that allocates memory from a
    static array used as a wraparound buffer, something like
    as follows,
    #define BUFFSZ 1000 /* any size you like */
    void *tempalloc(int nbytes) {
    static unsigned char buffer[BUFFSZ]; /* buffer */
    static int index = 0; /*start at beginning of buffer*/
    void *p = NULL; /* returned ptr */
    if ( nbytes <=0 || nbytes > BUFFSZ ) goto end_of_job; /* error */
    if ( index+nbytes > BUFFSZ ) index=0; /* wrap buffer */
    p = (void *)(buffer+index); /* ptr returned to caller */
    index += nbytes; /* next ptr past returned nbytes */
    end_of_job: return(p); } /* back to caller */
    No need to free, or anything. You just have to be logically sure
    you won't still be using the memory when it's "realloc'ed" after
    the internal static buffer wraps around. (And the above code
    doesn't align returned memory on anything.)
    --
    John Forkosh ( mailto: where j=john and f=forkosh )
     
    JohnF, Nov 17, 2010
    #5
  6. Navaneeth

    Navaneeth Guest

    >    #define BUFFSZ 1000  /* any size you like */
    >    void *tempalloc(int nbytes) {
    >      static unsigned char buffer[BUFFSZ]; /* buffer */
    >      static int index = 0;      /*start at beginning of buffer*/
    >      void *p = NULL;            /* returned ptr */
    >      if ( nbytes <=0 || nbytes > BUFFSZ ) goto end_of_job; /* error */
    >      if ( index+nbytes > BUFFSZ ) index=0; /* wrap buffer */
    >      p = (void *)(buffer+index); /* ptr returned to caller */
    >      index += nbytes;           /* next ptr past returned nbytes */
    >      end_of_job: return(p); }   /* back to caller */


    John,

    Thanks. This seems to be a good idea. I will evaluate it.
     
    Navaneeth, Nov 17, 2010
    #6
  7. On Nov 17, 9:01 am, JohnF <> wrote:
    > Navaneeth <> wrote:
    > > My application will be looping for 5000 times. On each iteration, it
    > > has to create a string and pass that into other methods for
    > > processing. This string length will only be known at the runtime.
    > > So in each iteration, this is what happens.
    > >   ptr = malloc
    > >   process(ptr)
    > >   free(ptr)
    > > I am wondering doing frequent malloc and free is ok? Will this
    > > fragment the memory?

    >
    > As per MM's reply, not a problem. Nevertheless, I usually do
    > this kind of thing, when I very temporarily need lots of short
    > strings, with a short function that allocates memory from a
    > static array used as a wraparound buffer, something like
    > as follows,



    A variant that will deal with long
    strings (this is the method described
    by Mikko Rauhala). The idea is malloc once and reuse
    the buffer. Only realloc if you have to.

    (untested, no error checking)

       #define BUFFSZ 20  /* any size you like, 2 will do */
    #define MEM_INCREMENT 2 /* unless you have a good reason use 2 */

       void *tempalloc(int nbytes) {
        static void *p=malloc(BUFFSZ);
    static int currbufsize = BUFFSZ;
         

    while ( nbytes > currbufsize) {
    curbuffsize = currbufsize*MEM_INCREMENT;
    p==remlloc(p,currbuufsize);
    }

    return p;
    }


    This has the advantage of portable alignment and
    it will not fail if a large allocation is requested.
    It has the disadvantage that it
    requires the use of the library malloc.
    You can only use tempalloc safely if you are
    sure that you do not need the memory from
    a previous call to tempalloc any more.


    A
     
    William Hughes, Nov 17, 2010
    #7
  8. "Navaneeth" <> wrote in message
    news:...
    > My application will be looping for 5000 times. On each iteration, it
    > has to create a string and pass that into other methods for
    > processing. This string length will only be known at the runtime.
    >
    > So in each iteration, this is what happens.
    >
    > ptr = malloc
    > process(ptr)
    > free(ptr)
    >
    > I am wondering doing frequent malloc and free is ok? Will this
    > fragment the memory?
    >
    > My development environment is GCC and Linux. But I'd be happy to know
    > about the implementation in windows also.


    Sounds like a region allocator might help you out a bit:

    http://groups.google.com/group/comp.lang.c/browse_frm/thread/97a65e8d96f6007c
     
    Chris M. Thomasson, Nov 17, 2010
    #8
  9. "Chris M. Thomasson" <> wrote in message
    news:xeUEo.35278$...
    > "Navaneeth" <> wrote in message
    > news:...
    >> My application will be looping for 5000 times. On each iteration, it
    >> has to create a string and pass that into other methods for
    >> processing. This string length will only be known at the runtime.
    >>
    >> So in each iteration, this is what happens.
    >>
    >> ptr = malloc
    >> process(ptr)
    >> free(ptr)
    >>
    >> I am wondering doing frequent malloc and free is ok? Will this
    >> fragment the memory?
    >>
    >> My development environment is GCC and Linux. But I'd be happy to know
    >> about the implementation in windows also.

    >
    > Sounds like a region allocator might help you out a bit:



    Or, you might be able to do something really simple like:

    http://codepad.org/kTdySzAe
    ______________________________________________________________
    #include <stdio.h>
    #include <stdlib.h>




    #define ITERS 128U

    #define RAND() ((size_t)rand())

    #define STRING_MAX_SIZE 64U

    #define STRING_ALPHABET "ABCDEFGHIJKLMNOPQRSTUVWXYZ"

    #define STRING_ALPHABET_GET() \
    STRING_ALPHABET[RAND() % (sizeof(STRING_ALPHABET) - 1)]




    char*
    string_populate(
    char* string,
    size_t* psize
    ){
    size_t i;
    size_t size = RAND() % STRING_MAX_SIZE;
    *psize = size;

    string[size] = '\0';

    for (i = 0; i < size; ++i)
    {
    string = STRING_ALPHABET_GET();
    }

    return string;
    }


    void
    string_process(
    char const* string,
    size_t size
    ){
    size_t i;

    for (i = 0; i < size; ++i)
    {
    putchar(string);
    }

    puts("\n_______________________________________________\n");
    }




    int
    main(void)
    {
    char buffer[STRING_MAX_SIZE];
    size_t i;

    for (i = 0; i < ITERS; ++i)
    {
    size_t size;
    char* string = string_populate(buffer, &size);
    string_process(string, size);
    }

    puts("\n\n___________________________________________\n"
    "The program has completed.\n");

    getchar();

    return 0;
    }

    ______________________________________________________________




    No need to call `malloc()/free()' and friends... ;^)
     
    Chris M. Thomasson, Nov 18, 2010
    #9
  10. Navaneeth

    Nobody Guest

    On Wed, 17 Nov 2010 04:00:25 -0800, Navaneeth wrote:

    > So in general, can malloc fragment the memory? I was expecting the
    > modern implementations will be clever enough to reduce this problem.
    > Isn't that true?


    They will try to minimise it, but it isn't always possible.

    E.g.:

    for (n = n0; n <= n1; n++) {
    void *p = malloc(n);
    void *q = malloc(1);
    ...
    free(p);
    }

    Each new allocation will be 1 byte too large to fit into the space
    for the previous allocation. Some of them will fit due to the previous
    allocation having been oversized, but they can't all fit.

    If you really need to prevent fragmentation, you need a relocating
    allocator, which implies a different interface. With malloc(), any
    returned pointer remains valid until free()d; the implementation can't
    just move blocks as it chooses.
     
    Nobody, Nov 19, 2010
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Rajshekhar
    Replies:
    5
    Views:
    2,337
    Jonathan Bartlett
    Mar 29, 2005
  2. Joseph

    Nice a tight loop?

    Joseph, Jul 7, 2006, in forum: C Programming
    Replies:
    16
    Views:
    719
    Ian Collins
    Jul 7, 2006
  3. gry

    performance of tight loop

    gry, Dec 14, 2010, in forum: Python
    Replies:
    8
    Views:
    314
    Peter Otten
    Dec 14, 2010
  4. Richard A. DeVenezia

    Update a status object while in a tight loop ?

    Richard A. DeVenezia, Apr 3, 2004, in forum: Javascript
    Replies:
    1
    Views:
    121
    Richard Cornford
    Apr 4, 2004
  5. Isaac Won
    Replies:
    9
    Views:
    444
    Ulrich Eckhardt
    Mar 4, 2013
Loading...

Share This Page