How to tie

[Abble]

I'd like to tie a Java array to some code so when the array is
accessed my code gets run.

Is there some simple way to do this in Java?

 

[Christian]

I'd like to tie a Java array to some code so when the array is
accessed my code gets run.

Is there some simple way to do this in Java?

Rather not ..
but when you use Collections instead of an array doing this becomes
rather easy.

Christian

 

[Lew]

Rather not ..
but when you use Collections instead of an array doing this becomes
rather easy.

Give us an example, Christian, or a description of how.

To the OP: You want to wrap access to your protected resource (array,
whatever) in an object that either performs the desired action on access, or
sends a notification to a listener to trigger such action.

JavaBeans have a mechanism available for notifications of property changes and
such events. You can use this to fire an event on each access of interest.

 

[Abble]

Give us an example, Christian, or a description of how.

To the OP: You want to wrap access to your protected resource (array,
whatever) in an object that either performs the desired action on access, or
sends a notification to a listener to trigger such action.

JavaBeans have a mechanism available for notifications of property changes and
such events. You can use this to fire an event on each access of interest.

Thanks, Lew, but I'd like if possible to not have to wrap my array.

Here's another angle to the question: Can I dynamically declare new
array names, for instance, if I have a function that opens up a
database, can that function look up the names of the fields in the
database and make similar variables "appear"?
Something like:

OpenDB( "mydb.xyz" ); //mydb.xyz has in it a definition of arrays
A,B, C
// now at this point arrays A B C spring into existence and
populated.

D = A[ i ] + B + C[ i ]; // use A B c


Thanks,

 

[Lew]

Thanks, Lew, but I'd like if possible to not have to wrap my array.

Why not?

Here's another angle to the question: Can I dynamically declare new
array names, for instance, if I have a function that opens up a
database, can that function look up the names of the fields in the
database and make similar variables "appear"?
Something like:

OpenDB( "mydb.xyz" ); //mydb.xyz has in it a definition of arrays
A,B, C
// now at this point arrays A B C spring into existence and
populated.

D = A[ i ] + B + C[ i ]; // use A B c

What does declaring new names get for you that simply assigning new arrays to
existing variables doesn't?

 

[Abble]

Why not?

Because I have about 60,000 lines of old code from C, C++, and FORTRAN
with array references like:

D = A[ i ] + B + C[ i ]; // use A B c

I think to objectize these the lines would look like:


D.setval( i, A.getval( i ) + B.getval( i ) + C.getval( i ) ); // use
A B c

So the goal is to do whatever it takes so we can keep the natural
array notations and legibility of the expressions( which can get very
long), AND speed would be nice, as the native code C, C++ and FORTRAN
codes are already not as fast as one might like.

 

[Eric Sosman]

Why not?

Because I have about 60,000 lines of old code from C, C++, and FORTRAN
with array references like:

D = A[ i ] + B + C[ i ]; // use A B c

I think to objectize these the lines would look like:


D.setval( i, A.getval( i ) + B.getval( i ) + C.getval( i ) ); // use
A B c

So the goal is to do whatever it takes so we can keep the natural
array notations and legibility of the expressions( which can get very
long), AND speed would be nice, as the native code C, C++ and FORTRAN
codes are already not as fast as one might like.

Odd use of "natural," that ...

If the goal is to interpose your own code on array accesses
in Java, I think you're out of luck: Array accesses use built-in
opcodes of the Java Virtual Machine, and there's no provision for
you to insert your own code into the JVM. (Hard to see how the
security and integrity models could permit such a thing, anyhow.)

Perhaps it's time to take a step back: What kind of code do
you want to interpose, and for what purpose? Perhaps the cat
can be skinned some other way.

 

[Abble]

Abble wrote:

If the goal is to interpose your own code on array accesses
in Java, I think you're out of luck: Array accesses use built-in
opcodes of the Java Virtual Machine, and there's no provision for
you to insert your own code into the JVM. (Hard to see how the
security and integrity models could permit such a thing, anyhow.)

Perhaps it's time to take a step back: What kind of code do
you want to interpose, and for what purpose? Perhaps the cat
can be skinned some other way.

Thanks Eric,
What I'd like to do is tie a Java array to a a sparse array
database of sorts. These arrays might be really sparse, or maybe
dimensioned far in excess of any RAM or even VM. It sure would be
nice to present the appearance of an array[million,million,million].
I can do this in FORTRAN (by trapping segment violations), I can do
this in C (with macros), I can do it in C++, I think, by overloading
operators. I can do it in Delphi Pascal, same as in FORTRAN.

Sure would be nice to do it in Java too.

I understand this is likely to a bit of a stretch in any language
that is security-conscious.

 

[Patricia Shanahan]

Abble wrote:

If the goal is to interpose your own code on array accesses
in Java, I think you're out of luck: Array accesses use built-in
opcodes of the Java Virtual Machine, and there's no provision for
you to insert your own code into the JVM. (Hard to see how the
security and integrity models could permit such a thing, anyhow.)

Perhaps it's time to take a step back: What kind of code do
you want to interpose, and for what purpose? Perhaps the cat
can be skinned some other way.

Thanks Eric,
What I'd like to do is tie a Java array to a a sparse array
database of sorts. These arrays might be really sparse, or maybe
dimensioned far in excess of any RAM or even VM. It sure would be
nice to present the appearance of an array[million,million,million].
I can do this in FORTRAN (by trapping segment violations), I can do
this in C (with macros), I can do it in C++, I think, by overloading
operators. I can do it in Delphi Pascal, same as in FORTRAN.

Sure would be nice to do it in Java too.

I understand this is likely to a bit of a stretch in any language
that is security-conscious.

I think your best approach is going to be to write an interface
representing your sparse arrays. For example, if they are two
dimensional doubles:

public interface SparseArray{
double get(long i, long j);
void set(double val, long i, long j);
}

and then write various implementations of the interface. The interface
may need additional methods for constructing views, such as rows,
columns, subblocks etc.

Most of the code would work in terms of the interface.

Unfortunately, Java lacks the operator overloading that would let you
represent get and set using array notation. That lack gets in the way of
easy translation from e.g. subscripted matrix notation to code.

Patricia

 

[Lew]

I think your best approach is going to be to write an interface
representing your sparse arrays. For example, if they are two
dimensional doubles:

public interface SparseArray{
double get(long i, long j);
void set(double val, long i, long j);
}

and then write various implementations of the interface. The interface
may need additional methods for constructing views, such as rows,
columns, subblocks etc.

Most of the code would work in terms of the interface.

Unfortunately, Java lacks the operator overloading that would let you
represent get and set using array notation. That lack gets in the way of
easy translation from e.g. subscripted matrix notation to code.

One possible implementation of SparseArray is to use a
Map <Pair <Integer, Integer>, T>.

You'll need to write your own Pair <U, U>, or you could use java.awt.Dimension
if you're willing to settle for int indexes instead of long.

====
package example;
public interface SparseArray <X, T>
{
T get( X i, X j );
void set( T val, X i, X j );
}

====
package example;
import java.awt.Dimension;
public class Sparse2D <T> implements SparseArray <Integer, T>
{
private final Map <Dimension, T> internal = new HashMap <Dimension, T> ();

public T get( Integer i, Integer j )
{
return internal.get( new Dimension( j, i ));
}
public void set( T val, Integer i, Integer j )
{
internal.put( new Dimension( j, i ), val );
}
}

====

Substitute 'new Pair <Long, Long> ( i, j )' for the Dimension constructions to
implement a SparseArray<Long, T>.

I reverse i and j in the Dimension constructors because of the documented
interpretation of the 'width' and 'height' members of Dimension. This
wouldn't apply to Pair.

 

[Mark Rafn]

Because I have about 60,000 lines of old code from C, C++, and FORTRAN
with array references like:
D = A[ i ] + B + C[ i ]; // use A B c
I think to objectize these the lines would look like:
D.setval( i, A.getval( i ) + B.getval( i ) + C.getval( i ) ); // use
A B c

Well List has shorter names, so it's:
D.set(i, A.get(i) + B.get(i) + C.get(i)); // use A B c

Which IMO isn't bad. But it's not the same as what you're asking for.

So the goal is to do whatever it takes so we can keep the natural
array notations and legibility of the expressions( which can get very
long), AND speed would be nice, as the native code C, C++ and FORTRAN
codes are already not as fast as one might like.

Maybe you should go a different direction. Write a mini-language that uses
whatever syntax you like, and convert that to java, in a way that lets you do
whatever optimizations you like.

 

[Owen Jacobson]

Why not?

Because I have about 60,000 lines of old code from C, C++, and FORTRAN
with array references like:

    D = A[ i ] + B + C[ i ];  // use A B c

I think to objectize these the lines would look like:

D.setval( i, A.getval( i ) + B.getval( i ) + C.getval( i )  );  // use
A B c

So the goal is to do whatever it takes so we can keep the natural
array notations and legibility of the expressions( which can get very
long), AND speed would be nice, as the native code C, C++ and FORTRAN
codes are already not as fast as one might like.

You could translate array (or std::vector/std::list) access from C++
to array access in Java, of course, which avoids the verbose-code
"problem". But, to answer your question, no; you can't create new
compile-time names at runtime (and that sentence should probably be
sufficient to explain why not... . Since the algorithm you're
describing has implicit knowledge of the structure of the data, why
not something like:

Database db = openDatabase ("foo.xml");
long[] a = db.getElements ("a"); // or getA ();
long[] b = db.getElements ("b"); // ibid
long[] c = db.getElements ("c"); // ibid

// validate a, b, c here, and determine n
long[] d = new long[n];

for (int i = 0; i < n; ++i)
d = a + b + c;

?

If you really need sparseness, then at least D and probably A, B, and
C would need to be objects of some sparse indexed collection type, in
which case you're kind of back to square one, and there's no way
around it. Java's syntax is not great for purely-numeric programs.

You might have a look at writing the number crunchy bits of the
program in Scala, which targets the JVM and can both call into and be
called from Java classes.

 

[Eric Sosman]

Abble wrote:

If the goal is to interpose your own code on array accesses
in Java, I think you're out of luck: Array accesses use built-in
opcodes of the Java Virtual Machine, and there's no provision for
you to insert your own code into the JVM. (Hard to see how the
security and integrity models could permit such a thing, anyhow.)

Perhaps it's time to take a step back: What kind of code do
you want to interpose, and for what purpose? Perhaps the cat
can be skinned some other way.

Thanks Eric,
What I'd like to do is tie a Java array to a a sparse array
database of sorts. These arrays might be really sparse, or maybe
dimensioned far in excess of any RAM or even VM. It sure would be
nice to present the appearance of an array[million,million,million].
I can do this in FORTRAN (by trapping segment violations), I can do
this in C (with macros), I can do it in C++, I think, by overloading
operators. I can do it in Delphi Pascal, same as in FORTRAN.

Sure would be nice to do it in Java too.

I understand this is likely to a bit of a stretch in any language
that is security-conscious.

It's been many years since I used FORTRAN, but the idea
of using a normal array reference and then trapping segment
violations strikes me as both slow and fragile. "Slow" because
practically every reference will produce a trap that needs to
be untangled and handled (didn't you say speed was an issue?),
and "fragile" because a bug somewhere else in the program might
get mis-routed to your trap handler with unhappy consequences.
Also, there's no hope at all of getting it to work for arrays
"far in excess of any [...] VM:" what if two sets of indices
wrap around to the same VM address, or to some unrelated but
valid address in your program?

C's preprocessor macros just get you a notational sugar-
coating, at the cost of using FORTRAN-esque array indexing
A(i,j,k) instead of C's own A[j][k]. You still wind up with
a function call.

I'll leave C++ for someone else; I've avoided it thus far
and would just as soon continue to do so.

Pascal is another language I haven't used in donkey's years,
but when I used it there were no facilities for interjecting one's
own code into array operations nor for trapping address faults.
I never used this Delphi dialect you mention, but if it allows
the same dodge you're contemplating for FORTRAN then I suspect
it has the same drawbacks.

What you seem to be after is the notational convenience of
an A(i,j,k) "array" reference. If you really really really must
have it and cannot stand writing A.get(i,j,k) and A.put(i,j,k,val),
I suggest you write your code in a Java-ish language that gets
transformed into actual Java by running it through a preprocessing
program of some kind:

$A(i,j,k) = $A(i,j,k) + $B(i,j) * $C(j,k);

might become

A.put(i,j,k, A.get(i,j,k) + B.get(i,j) + C.get(j,k));

It all comes down to what price you're willing to pay for the
convenience.

 

[Christian]

Give us an example, Christian, or a description of how.

I meant something like:

List<Object> a = new ArrayList<Object>() {
public Object get(int index) {
//do something..
return super.get(index);
}
};

 

[Roedy Green]

I'd like to tie a Java array to some code so when the array is
accessed my code gets run.

Is there some simple way to do this in Java?

You would write an accessor, similar to ArrayList. then can invoke
methods as side effects.

 

[Lew]

I meant something like:

List<Object> a = new ArrayList<Object>() {
public Object get(int index) {
//do something..
return super.get(index);
}
};

Nice idiom!

 

[Abble]

It's been many years since I used FORTRAN, but the idea
of using a normal array reference and then trapping segment
violations strikes me as both slow and fragile.

You're correct that it would be too slow if it trapped on every array
access.
But it's possible to arrange things so when it traps, you read in
anything from one to "n" pages,
and the page size depending on the CPU can range from 512 bytes to
4MB. So it's only going to trap
every n * pagesize / floatsize . And the trap handler can surmise
from the pattern of traps what the next likely access might be.

Also, there's no hope at all of getting it to work for arrays
"far in excess of any [...] VM:" what if two sets of indices
wrap around to the same VM address, or to some unrelated but
valid address in your program?

hat is a concern, so we better go to a 64-bit CPU in 64-bit mode,
there index and address wrap-around is not a problem.


I think we will go with the suggestion of cooking up our own array
language and map it to Java or similar.

thanks,

grg

 

[Eric Sosman]

You're correct that it would be too slow if it trapped on every array
access.

Perhaps I've misunderstood, but from your description it
appears that it *must* trap on every access. Your example
was an three-dimensional array with a million indices in
each dimension; 10^6^3 = 10^18 ~= 2^60 elements in all. If
the program presents you with [12345][67890][31415] you get
a trap for "no such memory address," which you intercept and
then resolve to some actual in-memory spot. Then the program
turns around and asks for [12345][67890][31416] and you get
another invalid address and another trap. The fact that you
eventually resolve the second trap to an actual memory address
that's next door to the first one doesn't help.

Also, there's no hope at all of getting it to work for arrays
"far in excess of any [...] VM:" what if two sets of indices
wrap around to the same VM address, or to some unrelated but
valid address in your program?

hat is a concern, so we better go to a 64-bit CPU in 64-bit mode,
there index and address wrap-around is not a problem.

Well, you said you planned to exceed the capacity of "any" VM,
not just a VM with 32-bit addresses. Even on a 64-bit machine,
you'd need to check how many address bits are actually used;
plenty of "64-bit" CPUs actually export fewer than 64 address
lines to the bus -- for example, the Itanium literature says it
can address "a full petabyte," which amounts to 50 address lines.
Bolt 1024 Itaniums (Itania?) together and you'll be almost home ;-)

I think we will go with the suggestion of cooking up our own array
language and map it to Java or similar.

Far more practical, IMHO.

 

[Abble]

Then the program

turns around and asks for [12345][67890][31416] and you get
another invalid address and another trap.

Sorry, I was not totally clear-- it's not a regular "invalid address"
trap, but a "address not in page table" trap.
The Array's address is pointing into this process' address space, but
there is no physical memory allocated to those addresses.

On the first trap for Array[12345][67890][31415] we guessed there
might be subsequent sequential accesses, so we ask the OS to map some
memory to the addressed location. Depending on the page size and our
guess as to how far up the app might access, we will ask for P pages
of memory. Then we read the data file into those locations. That
will prevent traps for as long as accesses hit the newly mapped area
(or any previously mapped area).

With careful management of how much RAM you map into the address
space, and perhaps unmapping of unrecently used RAM areas, I think
this scheme will be usable. It's much like what the VM system does,
except we know a lot more about what is likely to be needed and the
locality of references.


We could even ask the CPU to prefetch heavily used arrays into the
CPU cache memory. In one recent case this prefetching sped up the
program by a factor of 8.


Regards,