allocators and string

Discussion in 'C++' started by Christopher, Dec 13, 2011.

  1. Christopher

    Christopher Guest

    Admittedly, I've never had to worry about making a custom allocator. I
    think that is what I need. I need to replace a broken buffer, where it
    looks like the only real goal was to allocate large chunks of storage
    at a time instead of every insertion.

    Where do I start?

    It isn't standard or documented how much a string allocates when an
    insertion causes it to run out of space, is it?
    I've heard things like "double the current."

    If I want to the string to allocate space for 50,000 elements at a
    time, every time? How do I go about that.
     
    Christopher, Dec 13, 2011
    #1
    1. Advertising

  2. Christopher

    Christopher Guest

    On Dec 13, 5:08 pm, Paavo Helde <> wrote:

    >
    > I think STL allocators do not decide how much to allocate, they just
    > allocate.
    >
    > You can wrap std::string in a custom class and use the capacity() and
    > reserve() member functions for checking and enlarging the buffer.
    > However, from your description it sounds like std::string might not be
    > the best fit here. What problem are you trying to resolve?



    Well the broken implementation that I am trying to replace is just a
    buffer to grab XML from an http response. The author evidently tried
    to make it faster by allocating configurable sized chunks of chars and
    adding them to a linked list as they come in. Then allocating one
    after calculating the total size of all and copying on when
    extracting. It doesn't seem to make much of a difference and throws
    exceptions which I've given up trying to fix. I read something similar
    called "simple segregation" when looking up allocators.

    The things I know of that would effect design are
    * I get variable sized char data from the http library.
    * I get the data in multiple calls and need to append often
    * The total size of the data tends to be very large ~50,000 characters
    worth of xml document on average, but can be tiny up to a gig of data.
    * I need to give the caller whom wishes to extract the entire contents
    in contiguous memory at once
     
    Christopher, Dec 14, 2011
    #2
    1. Advertising

  3. Christopher

    Christopher Guest

    P.S. I could just use std::wstring as is, but if speed by eliminating
    allocations is possible, it would be a good time for me to learn how
    to do it instead of always depending on default STL behavior.
     
    Christopher, Dec 14, 2011
    #3
  4. Christopher

    Goran Guest

    On Dec 14, 8:40 am, Christopher <> wrote:
    > On Dec 13, 5:08 pm, Paavo Helde <> wrote:
    >
    >
    >
    > > I think STL allocators do not decide how much to allocate, they just
    > > allocate.

    >
    > > You can wrap std::string in a custom class and use the capacity() and
    > > reserve() member functions for checking and enlarging the buffer.
    > > However, from your description it sounds like std::string might not be
    > > the best fit here. What problem are you trying to resolve?

    >
    > Well the broken implementation that I am trying to replace is just a
    > buffer to grab XML from an http response. The author evidently tried
    > to make it faster by allocating configurable sized chunks of chars and
    > adding them to a linked list as they come in. Then allocating one
    > after calculating the total size of all and copying on when
    > extracting. It doesn't seem to make much of a difference and throws
    > exceptions which I've given up trying to fix. I read something similar
    > called "simple segregation" when looking up allocators.
    >
    > The things I know of that would effect design are
    > * I get variable sized char data from the http library.
    > * I get the data in multiple calls and need to append often
    > * The total size of the data tends to be very large ~50,000 characters
    > worth of xml document on average, but can be tiny up to a gig of data.
    > * I need to give the caller whom wishes to extract the entire contents
    > in contiguous memory at once


    That's a shame. Normally, XML parsers know how to receive some sort of
    a stream and work off that. Streams get rid of all your
    considerations.

    That said, if you know approx. max size, perhaps, as Paavo said, you
    should play with reserve(). std::vector might be a better fit, too.

    Goran.
     
    Goran, Dec 14, 2011
    #4
  5. Christopher

    Christopher Guest

    On Dec 14, 12:43 pm, Paavo Helde <> wrote:
    > Christopher <> wrote in news:f20c0d81-495c-4ffc-88da-
    > :
    >
    > > The things I know of that would effect design are
    > > * I get variable sized char data from the http library.
    > > * I get the data in multiple calls and need to append often
    > > * The total size of the data tends to be very large ~50,000 characters
    > > worth of xml document on average, but can be tiny up to a gig of data.

    >
    > Wow, 1G XML files! I hope you have 64-bit compilations.


    Yea, tell me about it. I've never seen a project where anything and
    everything is represented as xml rather than having a domain and
    classes for representing business objects. It is out of my hands. It's
    ironic that there is an issue with speed and a lesser issue with
    memory usage.

    > I assume that the size of the packet is not known in advance, otherwise
    > it would be easy to allocate the right buffer in the beginning.


    Correct.

    > So what was the problem again? Are you concerned about speed, memory
    > usage or something else?


    Speed primarily. Memory also, but not as much as speed.

    > Have you actually verified that standard
    > std::string append functionality does not meet your needs? The
    > exponential growth of the buffer is specifically designed to give
    > amortized linear complexity in such usage scenarios, without knowing the
    > required buffer size in advance it would be hard to compete with that.


    Well, who is to say that std::string doesn't suit my needs until there
    is something to compare it against. It is more of an investigation
    into whether or not a better speed could be achieved. The original
    author of the broken buffer claimed to achieve better speed, but the
    performace tests are skewed and incorrect anyway.
     
    Christopher, Dec 14, 2011
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ross Ridge
    Replies:
    2
    Views:
    891
    Dhruv
    Jul 7, 2003
  2. Andy Venikov

    std containers and allocators.

    Andy Venikov, Jul 11, 2005, in forum: C++
    Replies:
    4
    Views:
    1,362
    Andy Venikov
    Jul 13, 2005
  3. Jef Driesen
    Replies:
    1
    Views:
    304
  4. Diego Martins
    Replies:
    11
    Views:
    622
    Jeff F
    Aug 29, 2007
  5. sheam
    Replies:
    7
    Views:
    365
    Frank Birbacher
    Jun 12, 2008
Loading...

Share This Page