Suspected Memory Leak in Multithread queue implmenetation

Discussion in 'C++' started by tikcireviva, Jan 15, 2007.

  1. tikcireviva

    tikcireviva Guest

    Hi Guys,

    I've done a mulithread queue implementation on stl<queue>, my
    developement environment is on VC6 as well as FC3. Let's talks about
    the win32 side.

    The suspected memory leak is find after I've run through my unit test
    cases.

    Test Case:
    1. start 2 threads, thread A and thread B.
    2. thread A enqueue 10000 dummy objects, meanwhile thread B dequeue
    10000 dummy objects (delete object at the same time), these operations
    are called a "cycle".
    3. there is a while loop to loop the cycle, in between each cycle,
    there is a 5 seconds sleep time.
    4. run this for 10 minutes
    5. stop insert any dummy objects

    I've run this test case for 15 mins, and monitor its private_bytes,
    working_set and virtual_bytes with the windows performance monitor.

    My concern after running this test case is that,

    1. virtual_bytes increase double its size frequently (seems to me it is
    due to the queue memory allocation), and the virtual_bytes never drops
    into its original position (I mean at the beginning when the program is
    started)
    2. both private_bytes and working_set are increasing as well, just the
    fact that, duing the 5 seconds wait time, (step 3), these values drop
    to nearly zeros, but after the 5 seconds wait time, these values
    increase rapidly back to a state where it was dropped. and, of course
    keep increasing.
    3. during the (step 5), I can tell virtual_bytes is at its max, and
    private_bytes and working_set are set at a values (1/3 below the max
    point) and won't be able to drop any more.

    My questions are:

    1. shall windows OS handle the virtual_bytes, private_bytes and
    working_bytes dellocation?
    2. or is it because there is error on my implementation of multithread
    queue?

    Thank you very much. I am really appreciate yours help,

    Best wishes,

    Kit

    --------------- My queue --------------------------

    #ifndef __QUEUELOCK_HEADER__
    #define __QUEUELOCK_HEADER__

    #ifdef WIN32
    # define _WINSOCKAPI_
    # include <Windows.h>
    #else
    extern "C" {
    # include <pthread.h>
    };
    #endif

    #include <queue>
    #include <iostream>

    #define EXIT_IF(EXP) if (EXP) { cout << "Failed qlock: " #EXP "\n";
    exit(1); }

    #define QUEUE_MAX_SIZE 30000
    using namespace std;

    /* Queue with a lock, a template extending queue template
    */
    template <typename _Tp, typename _Sequence = deque<_Tp> >
    class qlock : public queue<_Tp,_Sequence> {
    typedef typename _Sequence::value_type value_type;
    typedef typename _Sequence::reference reference;
    typedef typename _Sequence::const_reference const_reference;
    CRITICAL_SECTION CriticalSection;// Lock for modification
    HANDLE hSynEvent; // Empty queue condition
    public:
    qlock() {
    // Initialize the critical section one time only.
    InitializeCriticalSection(&CriticalSection);
    hSynEvent = CreateSemaphore(NULL, 0, QUEUE_MAX_SIZE, NULL);
    if (hSynEvent == NULL)
    {
    printf("CreateEvent failed: %d\n", GetLastError());
    exit(1);
    }
    };
    ~qlock() {
    // Release resources used by the critical section object.
    DeleteCriticalSection(&CriticalSection);
    EXIT_IF(CloseHandle(hSynEvent));
    };
    void enqueue(const value_type& __x) {
    EnterCriticalSection(&CriticalSection); // Lock the queue
    if (this->size() > QUEUE_MAX_SIZE)
    {
    while(this->size() > 1) // Clear the queue
    {
    if(!this->empty()) {
    value_type r;
    r = this->front();
    this->pop();
    if(r != NULL) delete r;
    }
    }
    }
    this->push(__x);
    // Release Semaphore
    ReleaseSemaphore(hSynEvent, 1, NULL);
    LeaveCriticalSection(&CriticalSection); // Unlock the queue
    };
    //reference
    value_type dequeue(int i=0,int j=0) { // Blocking-call
    bool empty = true;
    value_type r = {0};
    EnterCriticalSection(&CriticalSection);
    empty = this->empty();
    if(!empty) {
    r = this->front();
    this->pop();
    }
    LeaveCriticalSection(&CriticalSection);
    while (empty) {
    // Wait until something is written into the queue
    if(WaitForSingleObject(hSynEvent, INFINITE) == WAIT_FAILED)
    {
    printf("WaitForSingleObject failed (%d)\n", GetLastError());
    exit(1);
    };
    EnterCriticalSection(&CriticalSection);
    empty = this->empty();
    if(!empty) {
    r = this->front();
    this->pop();
    }
    LeaveCriticalSection(&CriticalSection);
    };
    return r;
    }
    const_reference dequeue() const { // Blocking-call
    bool empty = true;
    const_reference r;
    EnterCriticalSection(&CriticalSection);
    empty = this->empty();
    if(!empty) {
    r = this->front();
    this->pop();
    }
    LeaveCriticalSection(&CriticalSection);

    while (empty) {
    // Wait until something is written into the queue
    if(WaitForSingleObject(hSynEvent, INFINITE) == WAIT_FAILED)
    {
    printf("WaitForSingleObject failed (%d)\n", GetLastError());
    LOG(LV_EMERG,"Wait for Single Queue Object failed.",1);
    exit(1);
    };
    EnterCriticalSection(&CriticalSection);
    empty = this->empty();
    if(!empty) {
    r = this->front();
    this->pop();
    }
    LeaveCriticalSection(&CriticalSection);
    };
    return r;
    }
    int qSize() {
    int size = 0;
    EnterCriticalSection(&CriticalSection);
    size = this->size();
    LeaveCriticalSection(&CriticalSection);
    return size;
    }
    };
    tikcireviva, Jan 15, 2007
    #1
    1. Advertising

  2. tikcireviva

    mlimber Guest

    tikcireviva wrote:
    [snip]
    > My questions are:
    >
    > 1. shall windows OS handle the virtual_bytes, private_bytes and
    > working_bytes dellocation?
    > 2. or is it because there is error on my implementation of multithread
    > queue?

    [snip]

    This newsgroup is for discussing the standard C++ language proper (see
    http://parashift.com/c -faq-lite/how-to-post.html#faq-5.9). In
    particular, we don't discuss Microsoft-specific questions or
    multithreading, which is not supported in standard C++ and is thus
    necessarily platform-specific. For the former, try
    microsoft.public.dotnet.languages.vc or similar, and for the latter,
    try comp.programming.threads or similar.

    Alternately, if you can rephrase your questions in terms of the
    standard C++ language, this would be the right place to ask.

    Cheers! --M
    mlimber, Jan 15, 2007
    #2
    1. Advertising

  3. tikcireviva

    tikcireviva Guest

    Thanks for the advance,

    Let me rephrase a little bit for my questions.

    1. The stl::queue double its memory for storage when it hits the
    maximun size, is there any method to explicitly delete the allocated
    memory from stl::queue? Since it consumes the virtual memory without
    releaseing it.
    2. Would you guys mind critique if I've done anything wrong on my
    stl::queue multithread implemenation?

    Best wishes,

    Kit

    mlimber ¼g¹D¡G

    > tikcireviva wrote:
    > [snip]
    > > My questions are:
    > >
    > > 1. shall windows OS handle the virtual_bytes, private_bytes and
    > > working_bytes dellocation?
    > > 2. or is it because there is error on my implementation of multithread
    > > queue?

    > [snip]
    >
    > This newsgroup is for discussing the standard C++ language proper (see
    > http://parashift.com/c -faq-lite/how-to-post.html#faq-5.9). In
    > particular, we don't discuss Microsoft-specific questions or
    > multithreading, which is not supported in standard C++ and is thus
    > necessarily platform-specific. For the former, try
    > microsoft.public.dotnet.languages.vc or similar, and for the latter,
    > try comp.programming.threads or similar.
    >
    > Alternately, if you can rephrase your questions in terms of the
    > standard C++ language, this would be the right place to ask.
    >
    > Cheers! --M
    tikcireviva, Jan 15, 2007
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. r_obert@REMOVE_THIS.hotmail.com

    Multithread or Multithread DLL?

    r_obert@REMOVE_THIS.hotmail.com, Nov 27, 2004, in forum: C++
    Replies:
    0
    Views:
    2,240
    r_obert@REMOVE_THIS.hotmail.com
    Nov 27, 2004
  2. Al Franz

    suspected cPickle memory leak

    Al Franz, May 13, 2005, in forum: Python
    Replies:
    0
    Views:
    429
    Al Franz
    May 13, 2005
  3. Russell Warren

    Is Queue.Queue.queue.clear() thread-safe?

    Russell Warren, Jun 22, 2006, in forum: Python
    Replies:
    4
    Views:
    681
    Russell Warren
    Jun 27, 2006
  4. Replies:
    1
    Views:
    393
    Andrew Tomazos
    May 10, 2009
  5. Kris
    Replies:
    0
    Views:
    480
Loading...

Share This Page