Information About A BlockingQueue

Discussion in 'Java' started by Jason Cavett, Apr 25, 2007.

  1. Jason Cavett

    Jason Cavett Guest

    I am currently in the process of using a BlockingQueue
    (LinkedBlockingQueue, to be exact) to help with a producer/consumer
    aspect of an application I'm developing. The user can continuously
    add items to the queue for processing (and this does work). However,
    I'm a little confused about how a BlockingQueue works.

    When I add an item to the BlockingQueue, is the items copied into the
    queue, or is it just a reference to the actual object?


    The reason I ask is because I notice that after adding 9 to 11 items
    into the queue, the application slows considerably (despite the fact
    that the "queue" stuff is running in a separate thread). After
    watching the output from the garbage collector, it appears as though I
    have a memory leak. I attempted to use hprof to profile my
    application to see where the memory leak is taking place, but I'm not
    fully understanding the output of hprof. When my application lags
    like this, it somewaht defeats the purpose of the use of the
    BlockingQueue since I want users to be able to queue up files to
    process.

    Can anybody clear up any possible issues or provide some insight with
    the way BlockingQueue works and any possible memory issues I should be
    keeping an eye out for (there isn't much information that I could find
    from my own Google searches).

    Thanks
    Jason Cavett, Apr 25, 2007
    #1
    1. Advertising

  2. On 25.04.2007 15:47, Jason Cavett wrote:
    > I am currently in the process of using a BlockingQueue
    > (LinkedBlockingQueue, to be exact) to help with a producer/consumer
    > aspect of an application I'm developing. The user can continuously
    > add items to the queue for processing (and this does work). However,
    > I'm a little confused about how a BlockingQueue works.
    >
    > When I add an item to the BlockingQueue, is the items copied into the
    > queue, or is it just a reference to the actual object?


    The blocking queue holds on to objects in the queue like any other
    collection does.

    > The reason I ask is because I notice that after adding 9 to 11 items
    > into the queue, the application slows considerably (despite the fact
    > that the "queue" stuff is running in a separate thread).


    What is an "item"?

    > After
    > watching the output from the garbage collector, it appears as though I
    > have a memory leak. I attempted to use hprof to profile my
    > application to see where the memory leak is taking place, but I'm not


    hprof is unlikely to help you with detecting memory leaks other than
    finding excessive creation of instances. But to get a leak you must
    also be holding on to those instances longer than intended which is
    something you do not get from hprof IIRC. You might want to look at one
    of the various profiler / memory analyzer tools around (JProbe,
    OptimizeIt, Eclipse TPTP).

    > fully understanding the output of hprof. When my application lags
    > like this, it somewaht defeats the purpose of the use of the
    > BlockingQueue since I want users to be able to queue up files to
    > process.
    >
    > Can anybody clear up any possible issues or provide some insight with
    > the way BlockingQueue works and any possible memory issues I should be
    > keeping an eye out for (there isn't much information that I could find
    > from my own Google searches).


    Here are some potential reasons for the slowdown:

    - your objects are large and you are tight on JVM memory so GC has to
    kick in more often or your JVM starts swapping to disk

    - your processing is the slow bit (too much concurrency for example,
    or it uses too much mem per processing instance)

    There are probably more but without more detail we can't really tell.

    Regards

    robert
    Robert Klemme, Apr 25, 2007
    #2
    1. Advertising

  3. Jason Cavett

    Jason Cavett Guest

    On Apr 25, 9:57 am, Robert Klemme <> wrote:
    > On 25.04.2007 15:47, Jason Cavett wrote:
    >
    > > I am currently in the process of using a BlockingQueue
    > > (LinkedBlockingQueue, to be exact) to help with a producer/consumer
    > > aspect of an application I'm developing. The user can continuously
    > > add items to the queue for processing (and this does work). However,
    > > I'm a little confused about how a BlockingQueue works.

    >
    > > When I add an item to the BlockingQueue, is the items copied into the
    > > queue, or is it just a reference to the actual object?

    >
    > The blocking queue holds on to objects in the queue like any other
    > collection does.
    >
    > > The reason I ask is because I notice that after adding 9 to 11 items
    > > into the queue, the application slows considerably (despite the fact
    > > that the "queue" stuff is running in a separate thread).

    >
    > What is an "item"?
    >
    > > After
    > > watching the output from the garbage collector, it appears as though I
    > > have a memory leak. I attempted to use hprof to profile my
    > > application to see where the memory leak is taking place, but I'm not

    >
    > hprof is unlikely to help you with detecting memory leaks other than
    > finding excessive creation of instances. But to get a leak you must
    > also be holding on to those instances longer than intended which is
    > something you do not get from hprof IIRC. You might want to look at one
    > of the various profiler / memory analyzer tools around (JProbe,
    > OptimizeIt, Eclipse TPTP).
    >
    > > fully understanding the output of hprof. When my application lags
    > > like this, it somewaht defeats the purpose of the use of the
    > > BlockingQueue since I want users to be able to queue up files to
    > > process.

    >
    > > Can anybody clear up any possible issues or provide some insight with
    > > the way BlockingQueue works and any possible memory issues I should be
    > > keeping an eye out for (there isn't much information that I could find
    > > from my own Google searches).

    >
    > Here are some potential reasons for the slowdown:
    >
    > - your objects are large and you are tight on JVM memory so GC has to
    > kick in more often or your JVM starts swapping to disk
    >
    > - your processing is the slow bit (too much concurrency for example,
    > or it uses too much mem per processing instance)
    >
    > There are probably more but without more detail we can't really tell.
    >
    > Regards
    >
    > robert


    Thanks for the tips. I checked into the "objects are large...so GC
    hast to kick in more often" Turns out this is the problem. I forgot
    I had done it, but I was making deep copies of objects to put onto the
    BlockingQueue so the user could continue to edit the file but the file
    (at that point in time) could also be processed.

    Of course, creating deep copies quickly fills up memory, so I quickly
    ran out of memory. I can easily fix this by putting the (reference
    to) the object on the queue directly instead of a copy of the object,
    but, unfortunately that raises the problem where users can continue
    editing objects and what they originally put in the queue will not be
    what's processed if they have continued to make changes to the file
    (the processing is done in a separate thread).

    Any suggestions on how to handle this problem?

    Anyway, even if there isn't (I've come up with a temporary solution
    for now), thank you for your helpful suggestions on the memory problem.
    Jason Cavett, Apr 26, 2007
    #3
  4. On 26.04.2007 14:57, Jason Cavett wrote:
    > On Apr 25, 9:57 am, Robert Klemme <> wrote:
    >> On 25.04.2007 15:47, Jason Cavett wrote:
    >>
    >>> I am currently in the process of using a BlockingQueue
    >>> (LinkedBlockingQueue, to be exact) to help with a producer/consumer
    >>> aspect of an application I'm developing. The user can continuously
    >>> add items to the queue for processing (and this does work). However,
    >>> I'm a little confused about how a BlockingQueue works.
    >>> When I add an item to the BlockingQueue, is the items copied into the
    >>> queue, or is it just a reference to the actual object?

    >> The blocking queue holds on to objects in the queue like any other
    >> collection does.
    >>
    >>> The reason I ask is because I notice that after adding 9 to 11 items
    >>> into the queue, the application slows considerably (despite the fact
    >>> that the "queue" stuff is running in a separate thread).

    >> What is an "item"?
    >>
    >>> After
    >>> watching the output from the garbage collector, it appears as though I
    >>> have a memory leak. I attempted to use hprof to profile my
    >>> application to see where the memory leak is taking place, but I'm not

    >> hprof is unlikely to help you with detecting memory leaks other than
    >> finding excessive creation of instances. But to get a leak you must
    >> also be holding on to those instances longer than intended which is
    >> something you do not get from hprof IIRC. You might want to look at one
    >> of the various profiler / memory analyzer tools around (JProbe,
    >> OptimizeIt, Eclipse TPTP).
    >>
    >>> fully understanding the output of hprof. When my application lags
    >>> like this, it somewaht defeats the purpose of the use of the
    >>> BlockingQueue since I want users to be able to queue up files to
    >>> process.
    >>> Can anybody clear up any possible issues or provide some insight with
    >>> the way BlockingQueue works and any possible memory issues I should be
    >>> keeping an eye out for (there isn't much information that I could find
    >>> from my own Google searches).

    >> Here are some potential reasons for the slowdown:
    >>
    >> - your objects are large and you are tight on JVM memory so GC has to
    >> kick in more often or your JVM starts swapping to disk
    >>
    >> - your processing is the slow bit (too much concurrency for example,
    >> or it uses too much mem per processing instance)
    >>
    >> There are probably more but without more detail we can't really tell.

    >
    > Thanks for the tips. I checked into the "objects are large...so GC
    > hast to kick in more often" Turns out this is the problem. I forgot
    > I had done it, but I was making deep copies of objects to put onto the
    > BlockingQueue so the user could continue to edit the file but the file
    > (at that point in time) could also be processed.
    >
    > Of course, creating deep copies quickly fills up memory, so I quickly
    > ran out of memory. I can easily fix this by putting the (reference
    > to) the object on the queue directly instead of a copy of the object,
    > but, unfortunately that raises the problem where users can continue
    > editing objects and what they originally put in the queue will not be
    > what's processed if they have continued to make changes to the file
    > (the processing is done in a separate thread).
    >
    > Any suggestions on how to handle this problem?


    Increase heap size. Or, since you are using a blocking queue anyway,
    use a lower max size for the queue. Deal with your "files" (whatever
    that means) more efficiently.

    > Anyway, even if there isn't (I've come up with a temporary solution
    > for now), thank you for your helpful suggestions on the memory problem.


    I'm glad you figured out.

    Kind regards

    robert
    Robert Klemme, Apr 26, 2007
    #4
  5. Jason Cavett wrote:
    > On Apr 25, 9:57 am, Robert Klemme <> wrote:
    >> On 25.04.2007 15:47, Jason Cavett wrote:
    >>
    >>> I am currently in the process of using a BlockingQueue
    >>> (LinkedBlockingQueue, to be exact) to help with a producer/consumer
    >>> aspect of an application I'm developing. The user can continuously
    >>> add items to the queue for processing (and this does work). However,
    >>> I'm a little confused about how a BlockingQueue works.
    >>> When I add an item to the BlockingQueue, is the items copied into the
    >>> queue, or is it just a reference to the actual object?

    >> The blocking queue holds on to objects in the queue like any other
    >> collection does.
    >>
    >>> The reason I ask is because I notice that after adding 9 to 11 items
    >>> into the queue, the application slows considerably (despite the fact
    >>> that the "queue" stuff is running in a separate thread).

    >> What is an "item"?
    >>
    >>> After
    >>> watching the output from the garbage collector, it appears as though I
    >>> have a memory leak. I attempted to use hprof to profile my
    >>> application to see where the memory leak is taking place, but I'm not

    >> hprof is unlikely to help you with detecting memory leaks other than
    >> finding excessive creation of instances. But to get a leak you must
    >> also be holding on to those instances longer than intended which is
    >> something you do not get from hprof IIRC. You might want to look at one
    >> of the various profiler / memory analyzer tools around (JProbe,
    >> OptimizeIt, Eclipse TPTP).
    >>
    >>> fully understanding the output of hprof. When my application lags
    >>> like this, it somewaht defeats the purpose of the use of the
    >>> BlockingQueue since I want users to be able to queue up files to
    >>> process.
    >>> Can anybody clear up any possible issues or provide some insight with
    >>> the way BlockingQueue works and any possible memory issues I should be
    >>> keeping an eye out for (there isn't much information that I could find
    >>> from my own Google searches).

    >> Here are some potential reasons for the slowdown:
    >>
    >> - your objects are large and you are tight on JVM memory so GC has to
    >> kick in more often or your JVM starts swapping to disk
    >>
    >> - your processing is the slow bit (too much concurrency for example,
    >> or it uses too much mem per processing instance)
    >>
    >> There are probably more but without more detail we can't really tell.
    >>
    >> Regards
    >>
    >> robert

    >
    > Thanks for the tips. I checked into the "objects are large...so GC
    > hast to kick in more often" Turns out this is the problem. I forgot
    > I had done it, but I was making deep copies of objects to put onto the
    > BlockingQueue so the user could continue to edit the file but the file
    > (at that point in time) could also be processed.
    >
    > Of course, creating deep copies quickly fills up memory, so I quickly
    > ran out of memory. I can easily fix this by putting the (reference
    > to) the object on the queue directly instead of a copy of the object,
    > but, unfortunately that raises the problem where users can continue
    > editing objects and what they originally put in the queue will not be
    > what's processed if they have continued to make changes to the file
    > (the processing is done in a separate thread).
    >
    > Any suggestions on how to handle this problem?


    Here are a couple of ideas, summary information and lazy copy.

    You might be able to pick out a subset of the data represented by the
    entire deep copy that covers everything that really needs to go through
    the blocking queue. Construct a new object that represents just that
    data, and put it on the queue.

    In a lazy copy approach you make a note of the fact that there is a
    virtual copy of the object, but just put the object reference on the
    queue. If the virtual copy makes it to the head of the queue without the
    original being modified, you just remove the note of its existence. You
    actually do the deep clone if, and only if, the user does something that
    changes the value of the original before you have finished with the
    virtual copy.

    Patricia
    Patricia Shanahan, Apr 26, 2007
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.

Share This Page