Fav. Memory Stream Impl.

Discussion in 'Java' started by Jan Burse, Nov 25, 2011.

  1. Jan Burse

    Jan Burse Guest

    Dear All,

    Would be interested in a good memory stream
    implementation. Requirement will be multiple
    readers / writers and positioning.

    Just found this code:

    http://www.mikepaul.ca/index.php?op...ream-for-javaj2me&catid=4:mobile-dev&Itemid=4
    ..Net style MemoryStream for Java/J2ME!
    Thursday, 27 May 2010 22:52 | Written by Michael Paul

    Although it has read / write and positioning,
    I don't think it is thread safe. Any other
    implementations around.

    Maybe some operations that provide InputStream /
    OutputStream factory methods.

    But
     
    Jan Burse, Nov 25, 2011
    #1
    1. Advertising

  2. Jan Burse

    markspace Guest

    On 11/25/2011 9:39 AM, Jan Burse wrote:
    > Dear All,
    >
    > Would be interested in a good memory stream implementation.


    What is a "memory stream?"

    Java already has bounded and unbounded queues, in thread safe and
    non-thread safe version. For special pruposes Java also has things like
    memory buffers for logging.

    What's the actual use model for this thing?

    (Esp. for those of us who don't know anything about C#/.Net)
     
    markspace, Nov 25, 2011
    #2
    1. Advertising

  3. Jan Burse

    Jan Burse Guest

    markspace schrieb:
    > For special pruposes Java also has things like memory buffers for logging.


    Which classes? Part of JDK or some utility?
     
    Jan Burse, Nov 25, 2011
    #3
  4. Jan Burse

    markspace Guest

    On 11/25/2011 11:59 AM, Jan Burse wrote:
    > markspace schrieb:
    >> For special pruposes Java also has things like memory buffers for
    >> logging.

    >
    > Which classes? Part of JDK or some utility?



    It wouldn't kill you to look through the Java API yourself, you know.
    Specifically for the logger, since that's what you seem to be interested in:

    <http://docs.oracle.com/javase/7/docs/api/java/util/logging/MemoryHandler.html>
     
    markspace, Nov 25, 2011
    #4
  5. Jan Burse

    Jan Burse Guest

    markspace schrieb:
    > On 11/25/2011 11:59 AM, Jan Burse wrote:
    >> markspace schrieb:
    >>> For special pruposes Java also has things like memory buffers for
    >>> logging.

    >>
    >> Which classes? Part of JDK or some utility?

    >
    >
    > It wouldn't kill you to look through the Java API yourself, you know.
    > Specifically for the logger, since that's what you seem to be interested
    > in:
    >
    > <http://docs.oracle.com/javase/7/docs/api/java/util/logging/MemoryHandler.html>
    >
    >
    >


    No the memory stream need not be cyclic. The Java/J2ME
    code reference I gave has maximally 256*chunks of size
    16384 each. And there is something with a cicular buffer
    in it.

    But I guess the .NET memory streams are not cyclic. I am
    looking for something like the .NET memory streams, so
    that I can get rid of temporary file names in my
    application.

    Bye
     
    Jan Burse, Nov 25, 2011
    #5
  6. Jan Burse

    Jan Burse Guest

    Jan Burse, Nov 25, 2011
    #6
  7. Jan Burse

    markspace Guest

    On 11/25/2011 12:23 PM, Jan Burse wrote:
    > Jan Burse schrieb:
    >>
    >> But I guess the .NET memory streams are not cyclic. I am
    >> looking for something like the .NET memory streams, so
    >> that I can get rid of temporary file names in my
    >> application.

    >
    > http://msdn.microsoft.com/en-us/library/system.io.memorystream.aspx


    <http://docs.oracle.com/javase/7/docs/api/java/io/ByteArrayOutputStream.html>

    <http://docs.oracle.com/javase/7/docs/api/java/io/ByteArrayInputStream.html>
     
    markspace, Nov 25, 2011
    #7
  8. Jan Burse

    Jan Burse Guest

    markspace schrieb:
    > On 11/25/2011 12:23 PM, Jan Burse wrote:
    >> Jan Burse schrieb:
    >>>
    >>> But I guess the .NET memory streams are not cyclic. I am
    >>> looking for something like the .NET memory streams, so
    >>> that I can get rid of temporary file names in my
    >>> application.

    >>
    >> http://msdn.microsoft.com/en-us/library/system.io.memorystream.aspx

    >
    > <http://docs.oracle.com/javase/7/docs/api/java/io/ByteArrayOutputStream.html>
    >
    >
    > <http://docs.oracle.com/javase/7/docs/api/java/io/ByteArrayInputStream.html>
    >
    >
    >


    The memory stream should be sharable.
    Thats why I wrote:

    > Requirement will be multiple readers / writers


    And also seekable.
    Thats why I wrote.

    > and positioning.


    To my knowledge ByteArrayOutputStream and ByteArrayInputStream
    do not satisfy these requirements.

    Bye
     
    Jan Burse, Nov 25, 2011
    #8
  9. Jan Burse

    Jan Burse Guest

    Jan Burse schrieb:
    > The memory stream should be sharable.


    Well to some extend ByteArrayOutputStream and
    ByteArrayInputStream are shareble. They are
    at least thread safe, I find:

    public synchronized void write(int b) {

    So the same output object can be used by multiple
    threads. And the same input object can be used
    by multiple threads.

    But I would like to be able to do the same as with
    a temporary file. Some threads open it for read,
    and some threads open it for write.

    Bye
     
    Jan Burse, Nov 25, 2011
    #9
  10. Jan Burse

    Arne Vajhøj Guest

    On 11/25/2011 12:39 PM, Jan Burse wrote:
    > Would be interested in a good memory stream
    > implementation. Requirement will be multiple
    > readers / writers and positioning.
    >
    > Just found this code:
    >
    > http://www.mikepaul.ca/index.php?op...ream-for-javaj2me&catid=4:mobile-dev&Itemid=4
    >
    > .Net style MemoryStream for Java/J2ME!
    > Thursday, 27 May 2010 22:52 | Written by Michael Paul
    >
    > Although it has read / write and positioning,
    > I don't think it is thread safe. Any other
    > implementations around.
    >
    > Maybe some operations that provide InputStream /
    > OutputStream factory methods.


    Use java.nio.ByteBuffer and handle concurrency in your code.

    Arne
     
    Arne Vajhøj, Nov 25, 2011
    #10
  11. On 25.11.2011 22:36, Jan Burse wrote:
    > Jan Burse schrieb:
    >> The memory stream should be sharable.

    >
    > Well to some extend ByteArrayOutputStream and
    > ByteArrayInputStream are shareble. They are
    > at least thread safe, I find:
    >
    > public synchronized void write(int b) {
    >
    > So the same output object can be used by multiple
    > threads. And the same input object can be used
    > by multiple threads.
    >
    > But I would like to be able to do the same as with
    > a temporary file. Some threads open it for read,
    > and some threads open it for write.


    Why do you want to do that with binary data? What is your use case?

    Kind regards

    robert


    --
    remember.guy do |as, often| as.you_can - without end
    http://blog.rubybestpractices.com/
     
    Robert Klemme, Nov 25, 2011
    #11
  12. Jan Burse

    Roedy Green Guest

    On Fri, 25 Nov 2011 18:39:23 +0100, Jan Burse <>
    wrote, quoted or indirectly quoted someone who said :

    >Would be interested in a good memory stream
    >implementation. Requirement will be multiple
    >readers / writers and positioning.


    What do you use it for? What does it do for you?
    Is it just an ram-resident InputStream?
    --
    Roedy Green Canadian Mind Products
    http://mindprod.com
    Premature optimization is the root of all evil.
    ~ Donald Ervin Knuth 1938-01-10
    Unfortunately, some have misread this quotation as optimisation is
    in itself evil, or even that is it wicked to consider speed when
    choosing an algorithm. I counter, it is less work to do it right
    the first time. Knuth was referring to clarity-destroying
    micro-optimisation, now authomatically handled by optimising
    static compilers and HotSpot runtimes.
     
    Roedy Green, Nov 26, 2011
    #12
  13. Jan Burse

    Jan Burse Guest

    Roedy Green schrieb:
    > On Fri, 25 Nov 2011 18:39:23 +0100, Jan Burse<>
    > wrote, quoted or indirectly quoted someone who said :
    >
    >> Would be interested in a good memory stream
    >> implementation. Requirement will be multiple
    >> readers / writers and positioning.

    >
    > What do you use it for? What does it do for you?
    > Is it just an ram-resident InputStream?


    From my post 25.11.2011 21:23:

    > But I guess the .NET memory streams are not cyclic. I am
    > looking for something like the .NET memory streams, so
    > that I can get rid of temporary file names in my
    > application.
     
    Jan Burse, Nov 26, 2011
    #13
  14. Jan Burse

    Jan Burse Guest

    Arne Vajhøj schrieb:
    > Use java.nio.ByteBuffer and handle concurrency in your code.


    ByteBuffer does not extend automatically. If I do
    put and reach the size of the byte buffer, I will
    get an exception BufferOverflowException.

    That is at least how I interpret the javadoc.

    I am not looking for a bounded memory stream, it
    should be unbounded, like (temporary) files are.
    You just write and write and write, thats it.

    Size control of the memory stream can be done
    via close truncate for example.

    The reference to the memory stream should be
    still valid when an input stream or output stream
    sitting on it does a close. With the same memory
    stream reference I should be able to create new
    input or output streams over it, that will work
    on the previously written content.

    Bye
     
    Jan Burse, Nov 26, 2011
    #14
  15. Jan Burse

    Lew Guest

    On Saturday, November 26, 2011 3:05:27 AM UTC-8, Jan Burse wrote:
    > Roedy Green schrieb:
    > > On Fri, 25 Nov 2011 18:39:23 +0100, Jan Burse<>
    > > wrote, quoted or indirectly quoted someone who said :
    > >
    > >> Would be interested in a good memory stream
    > >> implementation. Requirement will be multiple
    > >> readers / writers and positioning.

    > >
    > > What do you use it for? What does it do for you?
    > > Is it just an ram-resident InputStream?

    >
    > From my post 25.11.2011 21:23:
    >
    >> But I guess the .NET memory streams are not cyclic. I am
    >> looking for something like the .NET memory streams, so
    >> that I can get rid of temporary file names in my
    >> application.


    GREAT! Now we have to do homework just to understand your question! That's a *real* inducement to help you.

    How about you answer the question here, hm?

    What about these "memory streams" do you want to recapitulate? What is the purpose in your project?

    --
    Lew
     
    Lew, Nov 26, 2011
    #15
  16. Jan Burse

    Jan Burse Guest

    Lew schrieb:
    > GREAT! Now we have to do homework just to
    > understand your question! That's a*real*
    > inducement to help you.


    My grandmother did understand it in a blink.

    Bye
     
    Jan Burse, Nov 26, 2011
    #16
  17. Jan Burse

    Lew Guest

    Jan Burse wrote:
    > Lew schrieb:
    >> GREAT! Now we have to do homework just to
    >> understand your question! That's a*real*
    >> inducement to help you.

    >
    > My grandmother did understand it in a blink.


    You're just piling on the inducements, ain't'cha, Dale Carnegie?

    --
    Lew
     
    Lew, Nov 27, 2011
    #17
  18. On 11/26/2011 12:10 PM, Jan Burse wrote:
    > Arne Vajhøj schrieb:
    >> Use java.nio.ByteBuffer and handle concurrency in your code.

    >
    > ByteBuffer does not extend automatically. If I do
    > put and reach the size of the byte buffer, I will
    > get an exception BufferOverflowException.
    >
    > That is at least how I interpret the javadoc.
    >
    > I am not looking for a bounded memory stream, it
    > should be unbounded, like (temporary) files are.
    > You just write and write and write, thats it.
    >
    > Size control of the memory stream can be done
    > via close truncate for example.
    >
    > The reference to the memory stream should be
    > still valid when an input stream or output stream
    > sitting on it does a close. With the same memory
    > stream reference I should be able to create new
    > input or output streams over it, that will work
    > on the previously written content.


    http://docs.oracle.com/javase/6/docs/api/java/io/PipedInputStream.html
    http://docs.oracle.com/javase/6/docs/api/java/io/PipedOutputStream.html

    Still, why do you want to do this? What problem are you trying to
    solve? I'm not convinced that it is a good idea - especially when using
    unbounded buffers.

    Cheers

    robert
     
    Robert Klemme, Nov 27, 2011
    #18
  19. Jan Burse

    Jan Burse Guest

    Robert Klemme schrieb:
    >
    > http://docs.oracle.com/javase/6/docs/api/java/io/PipedInputStream.html
    > http://docs.oracle.com/javase/6/docs/api/java/io/PipedOutputStream.html
    >
    > Still, why do you want to do this? What problem are you trying to
    > solve? I'm not convinced that it is a good idea - especially when using
    > unbounded buffers.


    A pipe is not a memory stream. In a pipe the reader
    does read where the writer does write. But I don't
    have this requirement.

    The only requirement I have, is that I can use
    the memory streams in place of >>TEMPORARY FILES<<.
    Whereas >>TEMPORARY FILES<< have a name and usually
    exist on some media.

    The memory streams should not exist on some media,
    except in the memory. But they should share with
    >>TEMPORARY FILES<< multiple readers / writers and

    positioninng. Since I can use a >>TEMPORARY FILES<<
    name and open a RandomAccessFile with it.

    So basically in my application I would like to have
    some class, call it MemoryStream. And It would like
    to be able to instantiate it. And this instance serves
    than as a kind of a file name.

    So this instance I should be able to use where I normally
    would use the >>TEMPORARY FILES<< name. The only places
    that come to my mind are:
    - creating an input stream from a file name, should
    then work from a memory stream instance.
    - creating an output stream from a file name, should
    then work from a memory stream instance.
    - creating a random acccess file from a file name,
    should then work from a memory stream instance.

    What the memory stream instance will have in common
    with a >>TEMPORARY FILES<< name, is that it will go
    away when the application closes. So they are very
    similar to >>TEMPORARY FILES<< with the option delete
    on close.

    Also I would like to be able to use the same memory stream
    to open input streams, output streams, random access files,
    as many as I want. This is also provided by >>TEMPORARY
    FiLES<<, once a >>TEMPORARY FILES<< name is established, it
    can be used for all kinds of stuff.

    Here is a little example:
    A large clip board: This could be realized by a temporary
    file. When you make the copy operation, you simply write
    the copied content into the temporary file. When you make
    the past operation you reopen the temporary file and
    read the content that should be pasted from it.

    The above could be realized by even passing the >>TEMPORARY
    FILES<< name around in the system clipboard, so that the copy
    paste works across applications. Provided that the applications
    understand the content type.

    But memory streams cannot passed around like this. Here is
    the difference to >>TEMPORARY FILES<<. In their simple
    implementation they are confined to the application that
    created them.

    Of course we could extend them and provide some RMI extension
    of them, or what ever, so that the a memory stream references
    becomes mobile. Or we could implement them in special shared
    memory of the operating system, so that a direct shared access
    from multiple applications is possible.

    This mobility and shared access across multiple application,
    which is a property of >>TEMPORARY FILES<<, is not required
    for the memory streams I am looking for. So I guess if my
    grandmother could program as she can knit pullovers for me,
    she might do a memory stream class for me.

    But unfortunately she knows only COBOL and I am looking for
    a Java library. But she assured me that something like this
    must exist on the web, and that I don't need to go into lengths
    to code it by myself. So she told me:

    It's not difficult, but why reinvent the wheel?

    Bye
     
    Jan Burse, Nov 27, 2011
    #19
  20. On 11/28/2011 12:13 AM, Jan Burse wrote:
    > Robert Klemme schrieb:
    >>
    >> Still, why do you want to do this? What problem are you trying to
    >> solve? I'm not convinced that it is a good idea - especially when using
    >> unbounded buffers.

    >
    > A pipe is not a memory stream. In a pipe the reader
    > does read where the writer does write. But I don't
    > have this requirement.
    >
    > The only requirement I have, is that I can use
    > the memory streams in place of >>TEMPORARY FILES<<.
    > Whereas >>TEMPORARY FILES<< have a name and usually
    > exist on some media.
    >
    > The memory streams should not exist on some media,
    > except in the memory. But they should share with
    >> >TEMPORARY FILES<< multiple readers / writers and

    > positioninng. Since I can use a >>TEMPORARY FILES<<
    > name and open a RandomAccessFile with it.
    >
    > So basically in my application I would like to have
    > some class, call it MemoryStream. And It would like
    > to be able to instantiate it. And this instance serves
    > than as a kind of a file name.
    >
    > So this instance I should be able to use where I normally
    > would use the >>TEMPORARY FILES<< name. The only places
    > that come to my mind are:
    > - creating an input stream from a file name, should
    > then work from a memory stream instance.
    > - creating an output stream from a file name, should
    > then work from a memory stream instance.
    > - creating a random acccess file from a file name,
    > should then work from a memory stream instance.
    >
    > What the memory stream instance will have in common
    > with a >>TEMPORARY FILES<< name, is that it will go
    > away when the application closes. So they are very
    > similar to >>TEMPORARY FILES<< with the option delete
    > on close.
    >
    > Also I would like to be able to use the same memory stream
    > to open input streams, output streams, random access files,
    > as many as I want. This is also provided by >>TEMPORARY
    > FiLES<<, once a >>TEMPORARY FILES<< name is established, it
    > can be used for all kinds of stuff.
    >
    > Here is a little example:
    > A large clip board: This could be realized by a temporary
    > file. When you make the copy operation, you simply write
    > the copied content into the temporary file. When you make
    > the past operation you reopen the temporary file and
    > read the content that should be pasted from it.
    >
    > The above could be realized by even passing the >>TEMPORARY
    > FILES<< name around in the system clipboard, so that the copy
    > paste works across applications. Provided that the applications
    > understand the content type.
    >
    > But memory streams cannot passed around like this. Here is
    > the difference to >>TEMPORARY FILES<<. In their simple
    > implementation they are confined to the application that
    > created them.
    >
    > Of course we could extend them and provide some RMI extension
    > of them, or what ever, so that the a memory stream references
    > becomes mobile. Or we could implement them in special shared
    > memory of the operating system, so that a direct shared access
    > from multiple applications is possible.
    >
    > This mobility and shared access across multiple application,
    > which is a property of >>TEMPORARY FILES<<, is not required
    > for the memory streams I am looking for. So I guess if my
    > grandmother could program as she can knit pullovers for me,
    > she might do a memory stream class for me.
    >
    > But unfortunately she knows only COBOL and I am looking for
    > a Java library. But she assured me that something like this
    > must exist on the web, and that I don't need to go into lengths
    > to code it by myself. So she told me:
    >
    > It's not difficult, but why reinvent the wheel?


    There is a lot of "could" and "should" in there but no description of a
    real problem you are trying to solve. Basically if you want to use
    other libraries which depend on java.io to open files you have little
    chance to smuggle a "memory stream" in there without going through the
    effort to modify byte code of classes. So your "memory stream" could be
    made to work easily with your own code only. But then, you'd probably
    be far better off using a data structure with specific type and not
    sequences of bytes. If you need to store unstructured text, you can use
    a StringBuffer. If you need to store other kinds of data you can use
    any of the classes from java.util.

    Kind regards

    robert


    PS: I hope you are aware that writing to files does not necessarily
    create disk IO. For short lived temporary files chances are that you
    delete the file before it got written to disk. And then there are in
    memory file systems available which you can mount and use like any other
    file system and which hold all the data in memory only.
     
    Robert Klemme, Nov 28, 2011
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. iksrazal
    Replies:
    0
    Views:
    378
    iksrazal
    Aug 21, 2003
  2. vicky

    impl a collection

    vicky, Jun 19, 2004, in forum: Java
    Replies:
    68
    Views:
    1,672
    Dale King
    Apr 15, 2006
  3. Replies:
    0
    Views:
    231
  4. Replies:
    0
    Views:
    93
  5. Replies:
    0
    Views:
    76
Loading...

Share This Page