Problems allocating memory

Discussion in 'C++' started by Neclepsio, Mar 9, 2007.

  1. Neclepsio

    Neclepsio Guest

    Hi everyone.
    I've made a class Matrix, which contains a pointer to the data and some
    methods which all return a copy of the matrix modified in some way. The
    program works quite well for small matrices, but as matrix dimension
    grows up it crashes in deterministic locations depending on data,
    machine and OS.
    In particular, this happens when I test it with 3000x3000 matrices
    (which take up more than 30MB) under three different machines/OSes. The
    crash happens when much memory is still available (for example, under
    cygwin, more than 1GB of virtual memory available), and I cannot think
    there is no contiguous space for allocating the matrix.
    I always use references and never pointers to matrices, so Matrix
    objects are not dynamically allocated (the data, however, is allocated
    by the constructor). Could this be a problem with stack/heap limitations?
    Can someone help me with some idea on how to further investigate the
    problem?

    Thank you,
    Ignazio
    Neclepsio, Mar 9, 2007
    #1
    1. Advertising

  2. "Neclepsio" <> wrote in message
    news:...
    : Hi everyone.
    : I've made a class Matrix, which contains a pointer to the data and
    some
    : methods which all return a copy of the matrix modified in some way.
    The
    : program works quite well for small matrices, but as matrix dimension
    : grows up it crashes in deterministic locations depending on data,
    : machine and OS.
    : In particular, this happens when I test it with 3000x3000 matrices
    : (which take up more than 30MB) under three different machines/OSes.
    The
    : crash happens when much memory is still available (for example, under
    : cygwin, more than 1GB of virtual memory available), and I cannot think
    : there is no contiguous space for allocating the matrix.
    : I always use references and never pointers to matrices, so Matrix
    : objects are not dynamically allocated (the data, however, is allocated
    : by the constructor). Could this be a problem with stack/heap
    limitations?
    : Can someone help me with some idea on how to further investigate the
    : problem?

    The first thing to check is that you have provided correct
    implementations of the destructor, the copy-constructor, and the
    copy-assignment operators.

    But to make your life easier in the first place, you should consider
    using std::vector instead of a raw pointer to store the matrix data.

    hth -Ivan
    --
    http://ivan.vecerina.com/contact/?subject=NG_POST <- email contact form
    Brainbench MVP for C++ <> http://www.brainbench.com
    Ivan Vecerina, Mar 9, 2007
    #2
    1. Advertising

  3. Neclepsio

    Neclepsio Guest

    Thank you for replying.

    Ivan Vecerina wrote:
    > The first thing to check is that you have provided correct
    > implementations of the destructor, the copy-constructor, and the
    > copy-assignment operators.


    Everything is ok. The second?

    > But to make your life easier in the first place, you should consider
    > using std::vector instead of a raw pointer to store the matrix data.


    I needed raw pointers to use the BLAS library.

    Thank you,
    Ignazio
    Neclepsio, Mar 9, 2007
    #3
  4. Neclepsio wrote:
    > Hi everyone.
    > I've made a class Matrix, which contains a pointer to the data and some
    > methods which all return a copy of the matrix modified in some way. The
    > program works quite well for small matrices, but as matrix dimension
    > grows up it crashes in deterministic locations depending on data,
    > machine and OS.
    > In particular, this happens when I test it with 3000x3000 matrices
    > (which take up more than 30MB) under three different machines/OSes. The
    > crash happens when much memory is still available (for example, under
    > cygwin, more than 1GB of virtual memory available), and I cannot think
    > there is no contiguous space for allocating the matrix.
    > I always use references and never pointers to matrices, so Matrix
    > objects are not dynamically allocated (the data, however, is allocated
    > by the constructor). Could this be a problem with stack/heap limitations?


    The fact that it crashes under three different platforms suggests not.

    > Can someone help me with some idea on how to further investigate the
    > problem?


    If you can reproduce the problem in a small program then post the code
    here. Plently will be willing to take a look.

    >
    > Thank you,
    > Ignazio


    john
    John Harrison, Mar 9, 2007
    #4
  5. Neclepsio wrote:
    > Hi everyone.
    > I've made a class Matrix, which contains a pointer to the data and some
    > methods which all return a copy of the matrix modified in some way. The
    > program works quite well for small matrices, but as matrix dimension
    > grows up it crashes in deterministic locations depending on data,
    > machine and OS.
    > In particular, this happens when I test it with 3000x3000 matrices
    > (which take up more than 30MB) under three different machines/OSes. The
    > crash happens when much memory is still available (for example, under
    > cygwin, more than 1GB of virtual memory available), and I cannot think
    > there is no contiguous space for allocating the matrix.
    > I always use references and never pointers to matrices, so Matrix
    > objects are not dynamically allocated (the data, however, is allocated
    > by the constructor). Could this be a problem with stack/heap limitations?


    There is no "heap limitation", unless you impose it yourself
    (hint: try "ulimit -a"). The stack is definitely limited, in
    the order of a few MB usually (try "ulimit -s") -- but you only
    hit this limitation if you allocate too many/much _local_ variables.

    > Can someone help me with some idea on how to further investigate the
    > problem?


    If this is an x86 platform, I highly recommend valgrind,
    this a utility that will point out many bugs to you, and does
    not require a recompile. Other than that you may try
    electric fence ("efence"), though this requires a recompile
    and is not as useful as valgrind.

    And the obvious way out -- recompile your code with debug
    information on (usually the -g option), run it under a
    debugger (like gdb), let it crash, then backtrace and
    check where the problem occurs. Put tons of debug printouts
    there and hopefully all will become clear.

    HTH,
    - J.
    Jacek Dziedzic, Mar 9, 2007
    #5
  6. >
    >> But to make your life easier in the first place, you should consider
    >> using std::vector instead of a raw pointer to store the matrix data.

    >
    >
    > I needed raw pointers to use the BLAS library.
    >


    std::vector does not prevent you from getting a raw pointer to the data.

    john
    John Harrison, Mar 9, 2007
    #6
  7. Neclepsio

    Neclepsio Guest

    Jacek Dziedzic wrote:
    > The stack is definitely limited, in
    > the order of a few MB usually (try "ulimit -s") -- but you only
    > hit this limitation if you allocate too many/much _local_ variables.


    Yes, I know. In my despair, I was wondering if something like:

    Matrix A(3000, 3000);
    Matrix B(3000, 3000);
    Matrix C(3000, 3000);

    or

    Matrix B = A.Transpose() * B.SubMatrix(Vector::All, Vector::All);
    //(each method returns a new Matrix object)

    Would take up much stack space... Anyway, since I get a bad_alloc in
    Matrix constructor (which dynamically allocates the space), I don't
    think it's the point.

    > If this is an x86 platform, I highly recommend valgrind,
    > this a utility that will point out many bugs to you, and does
    > not require a recompile. Other than that you may try
    > electric fence ("efence"), though this requires a recompile
    > and is not as useful as valgrind.


    Thank you, I'm downloading it.

    > And the obvious way out -- recompile your code with debug
    > information on (usually the -g option), run it under a
    > debugger (like gdb), let it crash, then backtrace and
    > check where the problem occurs. Put tons of debug printouts
    > there and hopefully all will become clear.


    Yes, I've done... On the cygwin machine (the most accessable for me) it
    crashes on the line:

    Matrix temp = P.Transpose();

    which just allocates and fills a new Matrix with the same dimensions as
    caller (just swapped).



    Thank you,
    Ignazio
    Neclepsio, Mar 9, 2007
    #7
  8. Neclepsio

    Neclepsio Guest

    John Harrison wrote:
    > If you can reproduce the problem in a small program then post the code
    > here. Plently will be willing to take a look.


    I'll try to write it as small as I can.

    Thank you,
    Ignazio
    Neclepsio, Mar 9, 2007
    #8
  9. Neclepsio

    Neclepsio Guest

    John Harrison wrote:
    > std::vector does not prevent you from getting a raw pointer to the data.


    So, if new can't allocate the memory, why should std::vector be able to?

    Thank you,
    Ignazio
    Neclepsio, Mar 9, 2007
    #9
  10. * Neclepsio:
    > John Harrison wrote:
    >> std::vector does not prevent you from getting a raw pointer to the data.

    >
    > So, if new can't allocate the memory, why should std::vector be able to?


    The std::vector code is more likely to be correct.

    --
    A: Because it messes up the order in which people normally read text.
    Q: Why is it such a bad thing?
    A: Top-posting.
    Q: What is the most annoying thing on usenet and in e-mail?
    Alf P. Steinbach, Mar 9, 2007
    #10
  11. * Alf P. Steinbach:
    > The std::vector code is more likely to be correct.


    Care to elaborate on this hollow claim?

    stl::vector is an array with all kinds of nasty strings attached - much
    like the rest of this troublesome library. It's an array with an identity
    crisis, it's an array with an attitude, but in the end it is still an array.
    If you can't new that array yourself, then STL surely can't either. STL *will*
    end up doing the new itself.

    There's an enormous amount of uncalled for STL zealotry going on, with people
    "suggesting" to use STL without even the *slightest* bit of validation to
    their claims. The STL is *not* The One Answer. In fact, often, it's exactly
    the WRONG answer.

    --
    Martijn van Buul -
    Martijn van Buul, Mar 9, 2007
    #11
  12. * Martijn van Buul:
    > * Alf P. Steinbach:
    >> The std::vector code is more likely to be correct.

    >
    > Care to elaborate


    no.

    --
    A: Because it messes up the order in which people normally read text.
    Q: Why is it such a bad thing?
    A: Top-posting.
    Q: What is the most annoying thing on usenet and in e-mail?
    Alf P. Steinbach, Mar 9, 2007
    #12
  13. Neclepsio wrote:
    > John Harrison wrote:
    >
    >> std::vector does not prevent you from getting a raw pointer to the data.

    >
    >
    > So, if new can't allocate the memory, why should std::vector be able to?
    >
    > Thank you,
    > Ignazio


    You are changing the subject, you said that you couldn't use std::vector
    because you needed raw pointers. I pointed out that this wasn't true. I
    didn't say std::vector would cure your problem.

    john
    John Harrison, Mar 9, 2007
    #13
  14. Neclepsio

    Neclepsio Guest

    John Harrison wrote:
    > You are changing the subject, you said that you couldn't use std::vector
    > because you needed raw pointers. I pointed out that this wasn't true. I
    > didn't say std::vector would cure your problem.


    Ah, ok, I misunderstood your point. Thank you.
    Neclepsio, Mar 9, 2007
    #14
  15. Neclepsio

    Neclepsio Guest

    Alf P. Steinbach wrote:
    > The std::vector code is more likely to be correct.


    This would not solve my problem: understand where I'm wrong.

    Ignazio
    Neclepsio, Mar 9, 2007
    #15
  16. * Neclepsio:
    > Alf P. Steinbach wrote:
    >> The std::vector code is more likely to be correct.

    >
    > This would not solve my problem: understand where I'm wrong.


    See Ivan Vecerina's reply earlier in the thread.

    --
    A: Because it messes up the order in which people normally read text.
    Q: Why is it such a bad thing?
    A: Top-posting.
    Q: What is the most annoying thing on usenet and in e-mail?
    Alf P. Steinbach, Mar 9, 2007
    #16
  17. Neclepsio

    Kai-Uwe Bux Guest

    Neclepsio wrote:

    > Alf P. Steinbach wrote:
    >> The std::vector code is more likely to be correct.

    >
    > This would not solve my problem: understand where I'm wrong.


    As long as we don't see the code that is giving you trouble, we won't be
    able to help you solve _that_ problem. See the FAQ [5.8] for helpful hints
    on how to post about code that does not work.


    Best

    Kai-Uwe Bux
    Kai-Uwe Bux, Mar 9, 2007
    #17
  18. Neclepsio

    bnonaj Guest

    Neclepsio wrote:
    > Hi everyone.
    > I've made a class Matrix, which contains a pointer to the data and some
    > methods which all return a copy of the matrix modified in some way. The
    > program works quite well for small matrices, but as matrix dimension
    > grows up it crashes in deterministic locations depending on data,
    > machine and OS.
    > In particular, this happens when I test it with 3000x3000 matrices
    > (which take up more than 30MB) under three different machines/OSes. The
    > crash happens when much memory is still available (for example, under
    > cygwin, more than 1GB of virtual memory available), and I cannot think
    > there is no contiguous space for allocating the matrix.
    > I always use references and never pointers to matrices, so Matrix
    > objects are not dynamically allocated (the data, however, is allocated
    > by the constructor). Could this be a problem with stack/heap limitations?
    > Can someone help me with some idea on how to further investigate the
    > problem?
    >
    > Thank you,
    > Ignazio


    Have you got exception handling so failed allocation will cause a throw?
    Is the allocation and deallocation kept within each class? (reset
    pointer to NULL and throw if access tried, and have a test)
    Never keep more than one permanent pointer to the memory, (get the
    pointer only within the function requiring it)
    Try adding index checking via class operator[] overloading, and throw
    exceptions if outside the range)
    Always pass the classes that make up the matrix by reference, avoids a
    copy constructor and subsequent destructor which may free memory
    inappropriately
    Preferably use an STL class like vector which handles memory well
    already, (why reinvent the wheel)
    Good luck.

    JB
    bnonaj, Mar 9, 2007
    #18
  19. "Martijn van Buul" <> wrote in message
    news:...
    :* Alf P. Steinbach:
    : > The std::vector code is more likely to be correct.
    :
    : Care to elaborate on this hollow claim?
    :
    : stl::vector is an array with all kinds of nasty strings attached -
    much
    : like the rest of this troublesome library. It's an array with an
    identity
    : crisis, it's an array with an attitude, but in the end it is still an
    array.
    : If you can't new that array yourself, then STL surely can't either.
    : STL *will* end up doing the new itself.
    :
    : There's an enormous amount of uncalled for STL zealotry going on, with
    people
    : "suggesting" to use STL without even the *slightest* bit of validation
    to
    : their claims. The STL is *not* The One Answer. In fact, often, it's
    exactly
    : the WRONG answer.

    One should start by using well-designed and well-tested libraries
    before trying to reinvent the wheel. Especially when one is unable
    to correctly manages memory.

    This is just sound advice to first "get things working".
    Where do you see zealotrly in there?

    Until proven wrong ("show us the code..."), incorrect memory management
    in the OP's code remains the most likely hypothesis.

    Ivan
    --
    http://ivan.vecerina.com/contact/?subject=NG_POST <- email contact form
    Ivan Vecerina, Mar 10, 2007
    #19
  20. "Neclepsio" <> wrote in message
    news:45f19f53$...
    : Thank you for replying.
    :
    : Ivan Vecerina wrote:
    : > The first thing to check is that you have provided correct
    : > implementations of the destructor, the copy-constructor, and the
    : > copy-assignment operators.
    :
    : Everything is ok.
    How do we know?
    [ your final comment does nothing to make me trust a bare claim ]

    : The second?
    The second is: another source of Undefined Behavior.
    Third: you are actually allocating more memory than is available.

    : > But to make your life easier in the first place, you should consider
    : > using std::vector instead of a raw pointer to store the matrix data.
    :
    : I needed raw pointers to use the BLAS library.

    Sure: &vec.front() or &vec


    And of course, there are C++ equivalents to BLAS, as others pointed out.

    So there are several options to solve the problem.
    But if you want a diagnosis, please post the source code.


    Kind regards,
    Ivan
    --
    http://ivan.vecerina.com/contact/?subject=NG_POST <- email contact form
    Brainbench MVP for C++ <> http://www.brainbench.com
    Ivan Vecerina, Mar 10, 2007
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. hall
    Replies:
    4
    Views:
    440
  2. soni29
    Replies:
    6
    Views:
    406
    Kevin Goodsell
    Sep 5, 2003
  3. Axel
    Replies:
    1
    Views:
    729
    stephan beal
    Oct 27, 2003
  4. Sameer
    Replies:
    2
    Views:
    281
    David White
    Nov 3, 2003
  5. Rakesh Kumar
    Replies:
    5
    Views:
    680
    James Kanze
    Dec 21, 2007
Loading...

Share This Page