Fav. Memory Stream Impl.

[Jan Burse]

Basically if you want to use other libraries which depend on java.io to
open files you have little chance to smuggle a "memory stream" in there
without going through the effort to modify byte code of classes.

No, I don't think that the above claim is right. Look see what
is written for the class InputStream:

This abstract class is the superclass of
all classes representing an input
stream of bytes.
http://docs.oracle.com/javase/1.4.2/docs/api/java/io/InputStream.html

So with proper OO I could implement:

class MemoryStream {

InputStream createInput();

}

class MemoryInput extens InputStream {
...
}

The factory method could return a MemoryInput instance
from a MemoryStream instance. Similarly I could
provide OutputStrem createOutput etc..

The MemoryStream would be not the first stream that
has been created outside of java.lang via proper OO.
Think for example of the request and respons streams
of a web server. They are also made like this.

Bye

 

[Robert Klemme]

No, I don't think that the above claim is right. Look see what
is written for the class InputStream:

        This abstract class is the superclass of
        all classes representing an input
         stream of bytes.
       http://docs.oracle.com/javase/1.4.2/docs/api/java/io/InputStream.html

So with proper OO I could implement:

     class MemoryStream {

          InputStream createInput();

     }

     class MemoryInput extens InputStream {
          ...
     }

The factory method could return a MemoryInput instance
from a MemoryStream instance. Similarly I could
provide OutputStrem createOutput etc..

The MemoryStream would be not the first stream that
has been created outside of java.lang via proper OO.
Think for example of the request and respons streams
of a web server. They are also made like this.

And how do you make library code use your memory stream factory
class? Remember, there will by typically a line like this somewhere

FileInputStream fileIn = new FileInputStream(fileName);

There is no factory. There is just an invocation of the constructor.

From what you write I get the impression that you are hooked into your
idea of using in memory piles of bytes as replacement for temporary
files because it looks like a good idea (and simple to do) on first
sight. But I haven't seen anything so far which would convince me
that it's really a good idea altogether.

Kind regards

robert

 

[Jan Burse]

And how do you make library code use your memory stream factory
class? Remember, there will by typically a line like this somewhere

FileInputStream fileIn = new FileInputStream(fileName);

I will replace these constructors. The input streams /
output streams are factored via methods from the
memory stream instance, and not passed as an argument
to other constructors.

From what you write I get the impression that you are hooked into your
idea of using in memory piles of bytes as replacement for temporary
files because it looks like a good idea (and simple to do) on first sight.

Well it might be a matter of taste what is a
good idea and what not.

In the present case I am expecting more performance,
since these memory streams will be used in high
frequency for small contents of about 10-50 bytes.

Maybe a part of Google Protocol Buffers could do:
http://code.google.com/intl/de-DE/apis/protocolbuffers/docs/reference/java/index.html

They have newInput() and newOutput(). Have to check.

Bye

 

[Robert Klemme]

I will replace these constructors. The input streams /
output streams are factored via methods from the
memory stream instance, and not passed as an argument
to other constructors.

And I said you will have to use bytecode manipulation for this in case
of other libraries (i.e. ones which you do not control or have access to
sources to).

Well it might be a matter of taste what is a
good idea and what not.

In a way, yes. I'd rather say it's a matter of goals which one wants to
achieve. There are means which work well for some ends and not so well
for others. If the goal is clear solutions can often be pretty easy
aligned according to "good" and "not good".

In the present case I am expecting more performance,
since these memory streams will be used in high
frequency for small contents of about 10-50 bytes.

*What* content? Why do you not answer this question? You didn't even
state that you do not want to answer the question (which I could
understand if you do not want to leak out ideas you are working on).
But coming here and asking while at the same time ignoring those who are
willing to help to find a good solution is not a strategy to success.

Regards

robert

 

[Jan Burse]

*What* content? Why do you not answer this question? You didn't even
state that you do not want to answer the question (which I could
understand if you do not want to leak out ideas you are working on). But
coming here and asking while at the same time ignoring those who are
willing to help to find a good solution is not a strategy to success.

Strange approach to make help dependent on disclosure
of application details. Please note this is a public
forum, so when somebody posts a solution here everybody
profits from this, not only eventually me.

Bye

 

[Jan Burse]

Strange approach to make help dependent on disclosure
of application details. Please note this is a public
forum, so when somebody posts a solution here everybody
profits from this, not only eventually me.

Bye

Last but not least, the solution poster might also
profit by getting some feedback etc..

Bye

 

[Gene Wirchenko]

Strange approach to make help dependent on disclosure
of application details. Please note this is a public

Leaving out critical details can lead to suggestions that are not
so good, or even bad.

forum, so when somebody posts a solution here everybody
profits from this, not only eventually me.

That is the idea, yes.

Sincerely,

Gene Wirchenko

 

[Mayeul]

*What* content? Why do you not answer this question? You didn't even
state that you do not want to answer the question (which I could
understand if you do not want to leak out ideas you are working on). But
coming here and asking while at the same time ignoring those who are
willing to help to find a good solution is not a strategy to success.

Long story short, *what* in the world needs some kind of in-memory file
virtualization, that could not be done better with the ultra-simple
concept of ArrayList? Why is there such a need for bytes, what are bytes
any use of?

In the Java World it is often the case that you do not want to work with
bytes, you merely write and read them on the boundaries of your program
to the external world. You'd usually work with structured data, and
therefore not keep bytes lying in-memory.
Still, it rarely makes anything more complex than it needs to be.
Everytime one feels it does, one should just question one's approach to
the task.
I spoke of ArrayList just as an example. Whatever the case is, someone
can probably point out a satisfactory way to address it. It might have
nothing to do with file virtualization, though.

 

[Jan Burse]

I spoke of ArrayList just as an example. Whatever the case is, someone
can probably point out a satisfactory way to address it. It might have
nothing to do with file virtualization, though.

Well idea is not to change the original application which uses
already some byte serialization in a fundamental way. But only
to replace the temporary files by memory streams.

Of course one could opt for a total rewrite of the original
application, but then there is no guarantee that a non-byte
serialization gives a better design.

Also we cannot deduce from the requirement of an byte input /
output stream that the application deals with binary data. It
could be also that on top of the byte input / output stream
character reader / writer are created.

So that basically the application deals with character streams,
and maybe with something like documents or some such. It might
be that ArrayLists come into play when working with documents,
but only peripherically, for example JTextPane & Co.. there is
an array of elements involved which represent paragraphs.

But when you strip all the styles from JTextPane & Co you
get to the characters of the paragraphs. And XML / HTML shows
you for example how you can serialize characters and styles
into character streams.

Now again one could opt for working with a DOM instead of working
with streams. But in XML both options are viable and not alien
to Java. Sometimes stream based approaches are even more efficient
than DOM based approaches.

So be it as it is, the requiremets stands at it is, there is
the request for memory streams, despite possible alternative
designs. Just assume that among the variant designs, we currently
focus on memory streams and not something else.

Bye

 

[Lew]

So be it as it is, the requiremets stands at it is, there is
the request for memory streams, despite possible alternative
designs. Just assume that among the variant designs, we currently
focus on memory streams and not something else.

Doesn't seem like this group has what you're looking for. The thread's gone on for a while with no movement on your rather narrow and close-to-the-vest requirements. I suggest expanding your search beyond Usenet.

GIYF. And it's faster, too.

 

[Martin Gregorie]

Well idea is not to change the original application which uses already
some byte serialization in a fundamental way. But only to replace the
temporary files by memory streams.

As you say the individual file sizes are a few tens of bytes, have you
looked at using ByteInputStream and ByteOutputStream?

If you have considered them, why are they unsuitable?

 

[Martin Gregorie]

They could play a part in the game. But two problems are unsolved:
sharing and positioning.

My bad: I meant ByteArrayInputStream and ByteArrayOutputStream

The names are a little confusing: ByteArrayInputStream reads *from* a
byte array. The ByteArrayOutputStream writes *into* a byte array.

I've used them to move byte arrays to and from CLOB columns in databases,
where they were straight-forward to use with CLOB sizes ranging from a
few bytes to a few megabytes, though I never had more than one or two
instances in use at a time.

 

[Jan Burse]

My bad: I meant ByteArrayInputStream and ByteArrayOutputStream

The names are a little confusing: ByteArrayInputStream reads *from* a
byte array. The ByteArrayOutputStream writes *into* a byte array.

I've used them to move byte arrays to and from CLOB columns in databases,
where they were straight-forward to use with CLOB sizes ranging from a
few bytes to a few megabytes, though I never had more than one or two
instances in use at a time.

I don't see a constructor where I could hand over a
buffer content from the memory stream. So that multiple
reader and/or writers can share the same buffer content.

Remember requirement is something along:

class MemoryStream {

InputStream createInput();
OutputStream createOutput();
...

}

class MemoryInput extens InputStream {
...
}

class MemoryOutput extens OutputStream {
...
}

 

[markspace]

I don't see a constructor where I could hand over a
buffer content from the memory stream. So that multiple
reader and/or writers can share the sa

OK, I'll throw myself on this grenade.

<http://docs.oracle.com/javase/7/doc...InputStream.html#ByteArrayInputStream(byte[])>

 

[Martin Gregorie]

I don't see a constructor where I could hand over a buffer content from
the memory stream. So that multiple reader and/or writers can share the
same buffer content.

The idea is that you write to the ByteArrayOutputStream, close it, grab
the byte array using toByteArray() and store that until it is needed.
Then you construct a ByteArrayInputStream with the byte array as the
constructor argument. This does, of course, assume that you never want to
write and read an instance simultaneously.

I'd think implementing non-simultaneous access should be easy enough to
bury inside a <MemoryStream> class.

Beyond that I'm guessing since you've said little or nothing about
whether you need simultaneous read and write access to a <MemoryStream>
instance, whether an instance gets read more than once and how many
instances can exist at the same time.

 

[John B. Matthews]

I don't see a constructor where I could hand over a
buffer content from the memory stream. So that multiple
reader and/or writers can share the sa

OK, I'll throw myself on this grenade.

<http://docs.oracle.com/javase/7/doc...InputStream.html#ByteArrayInputStream(byte[])>

This related example uses an in-memory byte stream to transport
character encoded data:

<http://groups.google.com/group/comp.lang.java.programmer/msg/4f6eab3230444531>

 

[Jan Burse]

Beyond that I'm guessing since you've said little or nothing about
whether you need simultaneous read and write access to a<MemoryStream>
instance, whether an instance gets read more than once and how many
instances can exist at the same time.

I only said multiple readers and writers. But idea
is concurrently and simultaneous. Idea is not
sequentially one after the other.

What will transpire between these multiple readers
and writers will be the size of the memory stream,
and eventually a delete status. And of course the
content.

No special limit on the number of memory stream instances,
and no special limit on the number of input / output /
random access objects created from one memory stream.

Bye

 

[Jan Burse]

What do you use it for? What does it do for you?
Is it just an ram-resident InputStream?

Dear All,

I found an interesting implementation. On linux
level there is ashmem for Android. And then on Java
level there is an implementation of a class called
MemoryFile. The memory file acts as a factory for
input/output streams:


http://grepcode.com/file/repository....2_r1.1/android/os/MemoryFile.java#MemoryFile

public InputStream getInputStream() {
return new MemoryInputStream();
}

public OutputStream getOutputStream() {
return new MemoryOutputStream();
}

The above two streams account for sequential access,
and via mark/release some pseudo random access.
But there is also an api for random access. Namely
the following two beasts:

public int readBytes(byte[] buffer, int srcOffset,
int destOffset, int count);

public void writeBytes(byte[] buffer, int srcOffset,
int destOffset, int count)

Compare to a normal read/write there is an additional
parameter for the offset inside the memory file. But
the implementation is not really light weight. Namely
the memory file can shared accross processes, which
I do not need:

http://elinux.org/Android_Kernel_Features#ashmem

But at least it gives an idea how to bootstrap a
memory file implementation. Namely the input/output
streams are implemented based on the read/write with
the extra argument. Makes for examples mark/release
quite simple....

Cool!

Bye

 

[Arne Vajhøj]

ByteBuffer does not extend automatically. If I do
put and reach the size of the byte buffer, I will
get an exception BufferOverflowException.

That is at least how I interpret the javadoc.

I am not looking for a bounded memory stream, it
should be unbounded, like (temporary) files are.
You just write and write and write, thats it.

Size control of the memory stream can be done
via close truncate for example.

The reference to the memory stream should be
still valid when an input stream or output stream
sitting on it does a close. With the same memory
stream reference I should be able to create new
input or output streams over it, that will work
on the previously written content.

Given that memory/available address space is
bounded then what you can store in memory will
be bounded no matter what.

You will just need to code to handle it.

Arne

 

[Arne Vajhøj]

Strange approach to make help dependent on disclosure
of application details. Please note this is a public
forum, so when somebody posts a solution here everybody
profits from this, not only eventually me.

It happens all the time that people ask for help
with implementation of solution XYZ while there is another
solution ABC that is a much better solution
for the problem to be solved.

Picking the right solution is the difficult part and
implementing it is the easy part (if it is a good
solution).

The interest in helping implement XYZ when poster
will not reveal what problem is being solved is not
particular attractive, because the chance that XYZ
is optimal is very small.

Arne