Fav. Memory Stream Impl.

[Jan Burse]

Dear All,

Would be interested in a good memory stream
implementation. Requirement will be multiple
readers / writers and positioning.

Just found this code:

http://www.mikepaul.ca/index.php?op...ream-for-javaj2me&catid=4:mobile-dev&Itemid=4
..Net style MemoryStream for Java/J2ME!
Thursday, 27 May 2010 22:52 | Written by Michael Paul

Although it has read / write and positioning,
I don't think it is thread safe. Any other
implementations around.

Maybe some operations that provide InputStream /
OutputStream factory methods.

But

 

[markspace]

Dear All,

Would be interested in a good memory stream implementation.

What is a "memory stream?"

Java already has bounded and unbounded queues, in thread safe and
non-thread safe version. For special pruposes Java also has things like
memory buffers for logging.

What's the actual use model for this thing?

(Esp. for those of us who don't know anything about C#/.Net)

 

[Jan Burse]

For special pruposes Java also has things like memory buffers for logging.

Which classes? Part of JDK or some utility?

 

[markspace]

Which classes? Part of JDK or some utility?

It wouldn't kill you to look through the Java API yourself, you know.
Specifically for the logger, since that's what you seem to be interested in:

<http://docs.oracle.com/javase/7/docs/api/java/util/logging/MemoryHandler.html>

 

[Jan Burse]

It wouldn't kill you to look through the Java API yourself, you know.
Specifically for the logger, since that's what you seem to be interested
in:

<http://docs.oracle.com/javase/7/docs/api/java/util/logging/MemoryHandler.html>

No the memory stream need not be cyclic. The Java/J2ME
code reference I gave has maximally 256*chunks of size
16384 each. And there is something with a cicular buffer
in it.

But I guess the .NET memory streams are not cyclic. I am
looking for something like the .NET memory streams, so
that I can get rid of temporary file names in my
application.

Bye

 

[Jan Burse]

But I guess the .NET memory streams are not cyclic. I am
looking for something like the .NET memory streams, so
that I can get rid of temporary file names in my
application.

http://msdn.microsoft.com/en-us/library/system.io.memorystream.aspx

 

[markspace]

http://msdn.microsoft.com/en-us/library/system.io.memorystream.aspx

<http://docs.oracle.com/javase/7/docs/api/java/io/ByteArrayOutputStream.html>

<http://docs.oracle.com/javase/7/docs/api/java/io/ByteArrayInputStream.html>

 

[Jan Burse]

<http://docs.oracle.com/javase/7/docs/api/java/io/ByteArrayOutputStream.html>


<http://docs.oracle.com/javase/7/docs/api/java/io/ByteArrayInputStream.html>

The memory stream should be sharable.

Requirement will be multiple readers / writers

And also seekable.
Thats why I wrote.

and positioning.

To my knowledge ByteArrayOutputStream and ByteArrayInputStream
do not satisfy these requirements.

Bye

 

[Jan Burse]

The memory stream should be sharable.

Well to some extend ByteArrayOutputStream and
ByteArrayInputStream are shareble. They are
at least thread safe, I find:

public synchronized void write(int b) {

So the same output object can be used by multiple
threads. And the same input object can be used
by multiple threads.

But I would like to be able to do the same as with
a temporary file. Some threads open it for read,
and some threads open it for write.

Bye

 

[Arne Vajhøj]

Would be interested in a good memory stream
implementation. Requirement will be multiple
readers / writers and positioning.

Just found this code:

http://www.mikepaul.ca/index.php?op...ream-for-javaj2me&catid=4:mobile-dev&Itemid=4

.Net style MemoryStream for Java/J2ME!
Thursday, 27 May 2010 22:52 | Written by Michael Paul

Although it has read / write and positioning,
I don't think it is thread safe. Any other
implementations around.

Maybe some operations that provide InputStream /
OutputStream factory methods.

Use java.nio.ByteBuffer and handle concurrency in your code.

Arne

 

[Robert Klemme]

Well to some extend ByteArrayOutputStream and
ByteArrayInputStream are shareble. They are
at least thread safe, I find:

public synchronized void write(int b) {

So the same output object can be used by multiple
threads. And the same input object can be used
by multiple threads.

But I would like to be able to do the same as with
a temporary file. Some threads open it for read,
and some threads open it for write.

Why do you want to do that with binary data? What is your use case?

Kind regards

robert

 

[Roedy Green]

Would be interested in a good memory stream
implementation. Requirement will be multiple
readers / writers and positioning.

What do you use it for? What does it do for you?
Is it just an ram-resident InputStream?
--
Roedy Green Canadian Mind Products
http://mindprod.com
Premature optimization is the root of all evil.
~ Donald Ervin Knuth 1938-01-10
Unfortunately, some have misread this quotation as optimisation is
in itself evil, or even that is it wicked to consider speed when
choosing an algorithm. I counter, it is less work to do it right
the first time. Knuth was referring to clarity-destroying
micro-optimisation, now authomatically handled by optimising
static compilers and HotSpot runtimes.

 

[Jan Burse]

What do you use it for? What does it do for you?
Is it just an ram-resident InputStream?

From my post 25.11.2011 21:23:

 

[Jan Burse]

Use java.nio.ByteBuffer and handle concurrency in your code.

ByteBuffer does not extend automatically. If I do
put and reach the size of the byte buffer, I will
get an exception BufferOverflowException.

That is at least how I interpret the javadoc.

I am not looking for a bounded memory stream, it
should be unbounded, like (temporary) files are.
You just write and write and write, thats it.

Size control of the memory stream can be done
via close truncate for example.

The reference to the memory stream should be
still valid when an input stream or output stream
sitting on it does a close. With the same memory
stream reference I should be able to create new
input or output streams over it, that will work
on the previously written content.

Bye

 

[Lew]

From my post 25.11.2011 21:23:

GREAT! Now we have to do homework just to understand your question! That's a *real* inducement to help you.

How about you answer the question here, hm?

What about these "memory streams" do you want to recapitulate? What is the purpose in your project?

 

[Jan Burse]

GREAT! Now we have to do homework just to
understand your question! That's a*real*
inducement to help you.

My grandmother did understand it in a blink.

Bye

 

[Lew]

My grandmother did understand it in a blink.

You're just piling on the inducements, ain't'cha, Dale Carnegie?

 

[Robert Klemme]

ByteBuffer does not extend automatically. If I do
put and reach the size of the byte buffer, I will
get an exception BufferOverflowException.

That is at least how I interpret the javadoc.

I am not looking for a bounded memory stream, it
should be unbounded, like (temporary) files are.
You just write and write and write, thats it.

Size control of the memory stream can be done
via close truncate for example.

The reference to the memory stream should be
still valid when an input stream or output stream
sitting on it does a close. With the same memory
stream reference I should be able to create new
input or output streams over it, that will work
on the previously written content.

http://docs.oracle.com/javase/6/docs/api/java/io/PipedInputStream.html
http://docs.oracle.com/javase/6/docs/api/java/io/PipedOutputStream.html

Still, why do you want to do this? What problem are you trying to
solve? I'm not convinced that it is a good idea - especially when using
unbounded buffers.

Cheers

robert

 

[Jan Burse]

http://docs.oracle.com/javase/6/docs/api/java/io/PipedInputStream.html
http://docs.oracle.com/javase/6/docs/api/java/io/PipedOutputStream.html

Still, why do you want to do this? What problem are you trying to
solve? I'm not convinced that it is a good idea - especially when using
unbounded buffers.

A pipe is not a memory stream. In a pipe the reader
does read where the writer does write. But I don't
have this requirement.

The only requirement I have, is that I can use
the memory streams in place of >>TEMPORARY FILES<<.
Whereas >>TEMPORARY FILES<< have a name and usually
exist on some media.

The memory streams should not exist on some media,
except in the memory. But they should share withpositioninng. Since I can use a >>TEMPORARY FILES<<
name and open a RandomAccessFile with it.

So basically in my application I would like to have
some class, call it MemoryStream. And It would like
to be able to instantiate it. And this instance serves
than as a kind of a file name.

So this instance I should be able to use where I normally
would use the >>TEMPORARY FILES<< name. The only places
that come to my mind are:
- creating an input stream from a file name, should
then work from a memory stream instance.
- creating an output stream from a file name, should
then work from a memory stream instance.
- creating a random acccess file from a file name,
should then work from a memory stream instance.

What the memory stream instance will have in common
with a >>TEMPORARY FILES<< name, is that it will go
away when the application closes. So they are very
similar to >>TEMPORARY FILES<< with the option delete
on close.

Also I would like to be able to use the same memory stream
to open input streams, output streams, random access files,
as many as I want. This is also provided by >>TEMPORARY
FiLES<<, once a >>TEMPORARY FILES<< name is established, it
can be used for all kinds of stuff.

Here is a little example:
A large clip board: This could be realized by a temporary
file. When you make the copy operation, you simply write
the copied content into the temporary file. When you make
the past operation you reopen the temporary file and
read the content that should be pasted from it.

The above could be realized by even passing the >>TEMPORARY
FILES<< name around in the system clipboard, so that the copy
paste works across applications. Provided that the applications
understand the content type.

But memory streams cannot passed around like this. Here is
the difference to >>TEMPORARY FILES<<. In their simple
implementation they are confined to the application that
created them.

Of course we could extend them and provide some RMI extension
of them, or what ever, so that the a memory stream references
becomes mobile. Or we could implement them in special shared
memory of the operating system, so that a direct shared access
from multiple applications is possible.

This mobility and shared access across multiple application,
which is a property of >>TEMPORARY FILES<<, is not required
for the memory streams I am looking for. So I guess if my
grandmother could program as she can knit pullovers for me,
she might do a memory stream class for me.

But unfortunately she knows only COBOL and I am looking for
a Java library. But she assured me that something like this
must exist on the web, and that I don't need to go into lengths
to code it by myself. So she told me:

It's not difficult, but why reinvent the wheel?

Bye

 

[Robert Klemme]

A pipe is not a memory stream. In a pipe the reader
does read where the writer does write. But I don't
have this requirement.

The only requirement I have, is that I can use
the memory streams in place of >>TEMPORARY FILES<<.
Whereas >>TEMPORARY FILES<< have a name and usually
exist on some media.

The memory streams should not exist on some media,
except in the memory. But they should share with
positioninng. Since I can use a >>TEMPORARY FILES<<
name and open a RandomAccessFile with it.

So basically in my application I would like to have
some class, call it MemoryStream. And It would like
to be able to instantiate it. And this instance serves
than as a kind of a file name.

So this instance I should be able to use where I normally
would use the >>TEMPORARY FILES<< name. The only places
that come to my mind are:
- creating an input stream from a file name, should
then work from a memory stream instance.
- creating an output stream from a file name, should
then work from a memory stream instance.
- creating a random acccess file from a file name,
should then work from a memory stream instance.

What the memory stream instance will have in common
with a >>TEMPORARY FILES<< name, is that it will go
away when the application closes. So they are very
similar to >>TEMPORARY FILES<< with the option delete
on close.

Also I would like to be able to use the same memory stream
to open input streams, output streams, random access files,
as many as I want. This is also provided by >>TEMPORARY
FiLES<<, once a >>TEMPORARY FILES<< name is established, it
can be used for all kinds of stuff.

Here is a little example:
A large clip board: This could be realized by a temporary
file. When you make the copy operation, you simply write
the copied content into the temporary file. When you make
the past operation you reopen the temporary file and
read the content that should be pasted from it.

The above could be realized by even passing the >>TEMPORARY
FILES<< name around in the system clipboard, so that the copy
paste works across applications. Provided that the applications
understand the content type.

But memory streams cannot passed around like this. Here is
the difference to >>TEMPORARY FILES<<. In their simple
implementation they are confined to the application that
created them.

Of course we could extend them and provide some RMI extension
of them, or what ever, so that the a memory stream references
becomes mobile. Or we could implement them in special shared
memory of the operating system, so that a direct shared access
from multiple applications is possible.

This mobility and shared access across multiple application,
which is a property of >>TEMPORARY FILES<<, is not required
for the memory streams I am looking for. So I guess if my
grandmother could program as she can knit pullovers for me,
she might do a memory stream class for me.

But unfortunately she knows only COBOL and I am looking for
a Java library. But she assured me that something like this
must exist on the web, and that I don't need to go into lengths
to code it by myself. So she told me:

It's not difficult, but why reinvent the wheel?

There is a lot of "could" and "should" in there but no description of a
real problem you are trying to solve. Basically if you want to use
other libraries which depend on java.io to open files you have little
chance to smuggle a "memory stream" in there without going through the
effort to modify byte code of classes. So your "memory stream" could be
made to work easily with your own code only. But then, you'd probably
be far better off using a data structure with specific type and not
sequences of bytes. If you need to store unstructured text, you can use
a StringBuffer. If you need to store other kinds of data you can use
any of the classes from java.util.

Kind regards

robert


PS: I hope you are aware that writing to files does not necessarily
create disk IO. For short lived temporary files chances are that you
delete the file before it got written to disk. And then there are in
memory file systems available which you can mount and use like any other
file system and which hold all the data in memory only.