Doing one last thing to a WeakReference

[Tom Anderson]

Yes it is. In Java, you have very limited ways you can interact with
the garbage collector. If one of the ways, say WeakReferences as is the
case here, doesn't do what you want, you're just plain out of luck.

In C++, however, I could easily code my own reference-counted class and
destructors and get exactly the behavior I want (thanks to, among other
things, stack allocation of objects that permit the "resource acquisition is
initialization" technique).

Okay: any time you can use RAII in C++, you can use an explicit close,
perhaps ensured by a try-finally, in java. Yes, in Java, you have to write
the close explicitly. Boo hoo.

It's a perfectly reasonable (and common) thing to want to:

a) guarantee an object is destroyed as soon as it's no longer used (and not
"some later time").

b) do one last thing to an object before it's destroyed.

Java simply doesn't let you do those.

Yes, it does. It just doesn't give you a way to do it automatically. I
agree that this is a shame - i believe C# has a 'with' construct that lets
you do RAII, and i think that's a good feature. But it's hardly a
showstopper.

tom

 

[Tom Anderson]

That's just it: for my use case, I *can't* do it explicitly because I
have no way of knowing if my resource is either currently in use or
will be used again in the not-too-distant future (because some
code/object somewhere still has a hard reference to it).

For the record, the use-case is as follows. I have some class
(ImageInfo) that reads image files (possibly very large image files).
As part of the class's implementation, it opens a RandomAccessFile on
the image and reads "chunks" of the file into memory. (The "chunks"
themselves are held by SoftReferences -- a SoftChunkyByteBuffer -- and
this part works fine.)

The class has many methods, each of which accesses different parts of
the image, e.g., get the image's metadata (getMetadata()), the image's
color profile (getColorProfile()), the image's thumbnail
(getThumbnail()), and the image itself (getImage()). All said methods
must ultimately read data from the "chunks."

For a given image file F, a new ImageInfo(F) is created. Once created,
the rest of the code will use the various methods of ImageInfo in an
arbitrary order. The first method called (regardless of which method it
is) creates the SoftChunkyByteBuffer and, after creation, it (and the
RandomAccessFile it uses) sticks around.

After the first method is called, I can't just close the
RandomAccessFile because it might be the case that another method is
about to be called for the same image F via the same ImageInfo object.
In fact, I can *never* explicitly close the RandomAccessFile because of
this.

Open the file on-demand then. Keep your chunky buffers as
WeakReferences, so that they stick around a tad bit stronger, and close
the file after, say, a short timeout. By experimenting around, you ought
to find a reasonable trade-off balance for how soon to close. I mean,
there isn't a particularly huge overhead associated with opening a RAF,
is there? One FD. You may even be able to do without the buffering of
significant amounts of data (as opposed to metadata) altogether.

[Apologies if I missed some important parts of the thread -- I haven't
read it all]

You didn't miss anything in Paul's case that would stop this working.
Closing and reopening files is a good solution here, and one i would have
suggested if you hadn't got there first.

In my earlier discussion with Mark, i described an isomorphic situation,
but added a requirement that meant closing and reopening couldn't be used.
In that case, you do need to add refcounting.

tom

 

[Tom Anderson]

Because I'd want the VM to *guarantee* that the dispose() method would
be called *as* *soon* *as* the count reaches 0. This is in contrast to
WeakReferences that may never have their referrents be reclaimed.

Right. But if the objects that are holding your CountedReferences are
themselves managed by the GC, then they'll only decrement their counts
when they get collected, so you're still dependent on the GC to make your
refcounted objects die.

Of course, you could manage your reference-holding objects by refcounting.
But here, obviously, you have an infinite regress: for this to work, you
have to use refcounting all the way back to the root set, ie replace GC
with refcounting altogether. And even then, you're stuffed if you hit
cyclic garbage: you can only free that when the cycle detector runs, which
happens periodically, like GC. Unless you're going to cycle-detect every
time you decrement a pointer, which is going to have some performance
penalties, to put it mildly. On top of the fact that large-scale
refcounting is incredibly slow anyway, because of all the writes to
memory.

Now, having said that, the fragrant David Bacon has come up with an edgy
modern style of refcounting which does actually deliver pretty good
performance:

http://www.research.ibm.com/people/d/dfb/recycler-publications.html

It deals with cyclic garbage, runs only 5-10% slower than a proper GC,
and has really good realtime properties. However, one of the tradeoffs it
makes to achieve this is deferring decrements, which means it doesn't
quite have the properties you want.

That's just it: for my use case, I *can't* do it explicitly because I
have no way of knowing if my resource is either currently in use or will
be used again in the not-too-distant future (because some code/object
somewhere still has a hard reference to it).

For the record, the use-case is as follows. I have some class
(ImageInfo) that reads image files (possibly very large image files).
As part of the class's implementation, it opens a RandomAccessFile on
the image and reads "chunks" of the file into memory. (The "chunks"
themselves are held by SoftReferences -- a SoftChunkyByteBuffer -- and
this part works fine.)

The class has many methods, each of which accesses different parts of
the image, e.g., get the image's metadata (getMetadata()), the image's
color profile (getColorProfile()), the image's thumbnail
(getThumbnail()), and the image itself (getImage()). All said methods
must ultimately read data from the "chunks."

For a given image file F, a new ImageInfo(F) is created. Once created,
the rest of the code will use the various methods of ImageInfo in an
arbitrary order. The first method called (regardless of which method it
is) creates the SoftChunkyByteBuffer and, after creation, it (and the
RandomAccessFile it uses) sticks around.

After the first method is called, I can't just close the
RandomAccessFile because it might be the case that another method is
about to be called for the same image F via the same ImageInfo object.
In fact, I can *never* explicitly close the RandomAccessFile because of
this.

This is indeed a painful situation. But my point stands: since you're
dependent on GC of the reference-holding objects to the decrement the
count, you'd still lose with refcounting.

In C++, you could do this, because you implicitly or explicitly destroy
objects as soon as you're done with them - but there's nothing stopping
you doing this in java! Create an ImageInfoHandle class, through which you
use your ImageInfo. Give it a release() method. Call it whenever a
destructor would be called in C++ - when a locally-held instance goes out
of scope (using try-finally), or when you know you're done with a shared
instance (where you'd use delete in C++).

Or, do what Daniele suggested, and hide the details of the RAF being
opened and closed altogether.

tom

 

[John B. Matthews]

C++ has destructors.

Are there any other (object-oriented) programming
languages with destructors?

In Ada, objects derived from the abstract type
Ada.Finalization.Controlled have a Finalize method that is invoked
before the object's storage is reclaimed. This is sometimes used with
System.Storage_Pools to implement custom garbage collection.

<http://www.adaic.org/whyada/intro3.html>

 

[Paul J. Lucas]

For you [guaranteed dispose()] a flaw because apparently you don't want to write
this particular corner case yourself.

Guaranteeing object destruction is hardly a corner case. Part of the big point
of Java is the garbage collector to free the programmer from the burden of
managing memory. Unfortunately it's very myopic in that it's *only* about
object reclamation and doesn't lift a finger to help you manage other resources
(like files, sockets, etc.).

- Paul

 

[Paul J. Lucas]

Right. But if the objects that are holding your CountedReferences are
themselves managed by the GC, then they'll only decrement their counts
when they get collected, so you're still dependent on the GC to make
your refcounted objects die.

In my ideal implementation, a counted reference would not be an object like a
WeakReference, i.e., a separate object. Imagine a new syntax like:

MyObject# ref = SomeObject.getMyObject();

where the # marks "ref" as a counted reference; or, if you don't like that, then
a new keyword:

counted MyObject ref = SomeObject.getMyObject();

Or, if you don't like that either, mark the method to return only counted
references so the caller can't forget:

class SomeObject {
public counted MyObject getMyObject() { /* ... */ }
}

MyObject ref = SomeObject.getMyObject();

Then the whole problem of having a hard reference to a CountedReference (as a
distinct object) goes away. As soon as ref goes out of scope, its counter gets
decremented.

As far as having a counted reference *inside* another object (to which one has a
hard reference), well, as with everything else, you have to know what you're
doing. But at least this scheme affords you the ability whereas current Java
doesn't.

In C++, you could do this, because you implicitly or explicitly destroy
objects as soon as you're done with them - but there's nothing stopping
you doing this in java! Create an ImageInfoHandle class, through which
you use your ImageInfo. Give it a release() method. Call it whenever a
destructor would be called in C++ - when a locally-held instance goes
out of scope (using try-finally), or when you know you're done with a
shared instance (where you'd use delete in C++).

Yes, sure; but, as a library designer, I want the burden of doing all this to be
mine and not the users of the library who will often forget.

I find it ironic that Java, which has garbage collection to free the programmer
from the burden of memory management, instead burdens the programmer with having
to remember to use try/finally.

- Paul

 

[Paul J. Lucas]

Open the file on-demand then. Keep your chunky buffers as
WeakReferences, so that they stick around a tad bit stronger, and close
the file after, say, a short timeout.

Unfortunately that can't be guaranteed to work. When one wants to
move or delete a file under Windows, the file can *not* be open.
Having a time-out is a race-condition and bug waiting to happen.

- Paul

 

[Tom Anderson]

In my ideal implementation, a counted reference would not be an object like a
WeakReference, i.e., a separate object. Imagine a new syntax like:

MyObject# ref = SomeObject.getMyObject();

where the # marks "ref" as a counted reference; or, if you don't like that,
then a new keyword:

counted MyObject ref = SomeObject.getMyObject();

Or, if you don't like that either, mark the method to return only counted
references so the caller can't forget:

class SomeObject {
public counted MyObject getMyObject() { /* ... */ }
}

MyObject ref = SomeObject.getMyObject();

Then the whole problem of having a hard reference to a CountedReference
(as a distinct object) goes away. As soon as ref goes out of scope, its
counter gets decremented.
Okay.

As far as having a counted reference *inside* another object (to which
one has a hard reference), well, as with everything else, you have to
know what you're doing. But at least this scheme affords you the
ability whereas current Java doesn't.

Hang on, would you ban the use of counted references as instance fields?
Or just let this happen and live with having delayed decrements in those
cases?

Yes, sure; but, as a library designer, I want the burden of doing all
this to be mine and not the users of the library who will often forget.

That's fair enough. I was trying to make the point that this kind of
refcounting *can* be done in java, but of course you're quite right to say
that it's a pain.

I think the best accomodation that could be made without a major and
potentially awkward addition to the language and runtime system would be a
C#-style 'using' construct:

using (ImageInfo info = getImageInfo(fileName)) {
int w = info.getWidth() ;
// etc
}

Which would mean something like:

ImageInfo info ;
try {
info = getImageInfo(fileName) ;
int w = info.getWidth() ;
// etc
}
finally {
info.close() ;
}

This is an addition to the language, but a minor one, and one that's
syntactic sugar, so the compiler can deal with it without needing the
runtime system to change. Like the new for loops. Also like the new for
loops, it would depend on the used object implementing some interface,
like Closeable, Releasable, Usable, whatever.

You might even want to use finalize() as the method called, or make
finalization call close() if an object is Closeable. Not sure about that.

It may not be quite as neat as in C++ (and i never thought i'd say that!),
but all it involves doing is adding a keyword, a pair of parens, and a
pair of braces to your code.

It would mean that the lifetimes of usable objects have to nest - you
can't do:

acquire foo
acquire bar
release foo
release bar

At least, not using this construct alone. You could nest them and
explicitly close foo when you were done.

I find it ironic that Java, which has garbage collection to free the
programmer from the burden of memory management, instead burdens the
programmer with having to remember to use try/finally.

Fair point!

tom

 

[Joshua Cranmer]

Guaranteeing object destruction is hardly a corner case. Part of the
big point of Java is the garbage collector to free the programmer from
the burden of managing memory. Unfortunately it's very myopic in that
it's *only* about object reclamation and doesn't lift a finger to help
you manage other resources (like files, sockets, etc.).

Memory tends to be a special case. Every object uses memory, and writing
a good memory manager is very taxing on the programmer.

Other resources are different. Relatively few objects need access to a
file, stream, or socket, for example. In addition, these resources tend
to have more uniform life spans. If you want to write your own global
resource manager, go ahead.

By the way, there is an RFE to add a using construct to make the
try-finally syntax. With any luck, it will come in Java 7.

 

[Lew]

For you [guaranteed dispose()] a flaw because apparently you don't want to write
this particular corner case yourself.

Guaranteeing object destruction is hardly a corner case.  Part of the big point

That isn't a refutation of what I said, it's a refutation of some
point I did not make. The "corner case" to which I referred is not
object destruction per se, but the unique requirement you have that it
occur at a particular time and release additional resources. That is
a corner case, Mr. Straw-Man Arguer.

of Java is the garbage collector to free the programmer from the burden of
managing memory.  Unfortunately it's very myopic in that it's *only* about
object reclamation and doesn't lift a finger to help you manage other resources
(like files, sockets, etc.).

That may be unfortunate, but it's not a flaw in the language, it's
just something you don't get automatically with every object. You
don't get it with every object because not every object needs it.
Only the objects involved in your corner case need that capability.
To burden every object with relatively rarely-needed features would be
the flaw, so you have it exactly backwards.

 

[Stefan Ram]

I think the best accomodation that could be made without a major and
potentially awkward addition to the language and runtime system would be a
C#-style 'using' construct:
using (ImageInfo info = getImageInfo(fileName)) {
int w = info.getWidth() ;
// etc
}

In [1], it is described how to write a class
»FileInputStreamOpenCloseAndNull« so that

for( final java.io.FileInputStream fileInputStream:
new FileInputStreamOpenCloseAndNull( "input.txt" ))
use( fileInputStream );

will close »fileInputStream« after the use.

This is not much longer than the way it is written
without such a feature, that is:

final java.io.FileInputStream fileInputStream =
new FileInputStreamOpenCloseAndNull( "input.txt" );
use( fileInputStream );

When opening the stream was not possible, it also will not be
closed, and it is possible to set up things to either not call
»use« in this case at all or call it with »null«.

Also, when multiple resources are needed, the for statements
can be written in sequence to get the proper behavior.

For more details, See:

[1]

http://www.purl.org/stefan_ram/pub/dual-band_if

 

[Mike Schilling]

How is this different from C++ (excuse any syntax errors: it's been a while)

{
Object a = new Object();
exit(-1); // a's destructor won't run now
}

That doesn't invalidate my point. And it should be used when appropriate.
That's why Java has shutdownHooks.

There are two sorts of resources that might need cleaning up.

1. Those which disappear on process exit. The shutdownHooks are not
necessary..
2. Those which don't disappear on process exit. The shutdownHooks are
insufficient, since the JVM may not exit cleanly.

 

[Joshua Cranmer]

How is this different from C++ (excuse any syntax errors: it's been a while)

{
Object a = new Object();
exit(-1); // a's destructor won't run now
}

The code should be:
{
Object a;
exit(-1);
}

And no, it isn't any different (at least, in g++-4.3 it isn't).

 

[Tom Anderson]

I think the best accomodation that could be made without a major and
potentially awkward addition to the language and runtime system would be a
C#-style 'using' construct:
using (ImageInfo info = getImageInfo(fileName)) {
int w = info.getWidth() ;
// etc
}

In [1], it is described how to write a class
»FileInputStreamOpenCloseAndNull« so that
[1]

http://www.purl.org/stefan_ram/pub/dual-band_if

That's clever. Gross, but clever!

tom

 

[Daniele Futtorovic]

Unfortunately that can't be guaranteed to work. When one wants to
move or delete a file under Windows, the file can *not* be open.

Yeah, that's how it goes. Work on a temporary copy if you're concerned
about that. By the way, how come you're so much concerned about deleting
a file when your program is actually working on it? Isn't that like...
all right if it can't be deleted?

Also, are you sure you open the file in read-only mode?

Having a time-out is a race-condition and bug waiting to happen.

Not if you do it properly.

But you may also forget about the timeout, load the file/image metadata
once, and remaining data on-demand.

 

[Daniel Pitts]

For you [guaranteed dispose()] a flaw because apparently you don't
want to write
this particular corner case yourself.

Guaranteeing object destruction is hardly a corner case. Part of the
big point of Java is the garbage collector to free the programmer from
the burden of managing memory. Unfortunately it's very myopic in that
it's *only* about object reclamation and doesn't lift a finger to help
you manage other resources (like files, sockets, etc.).

- Paul

Which is exactly what the try{}finally{} is to help you manage. Seems
like more than a finger to me.

 

[Lasse Reichstein Nielsen]

In my ideal implementation, a counted reference would not be an object
like a WeakReference, i.e., a separate object. Imagine a new syntax
like:

MyObject# ref = SomeObject.getMyObject();

where the # marks "ref" as a counted reference; or, if you don't like
that, then a new keyword:

counted MyObject ref = SomeObject.getMyObject();

Is the variable reference-counted or is the object?

Or, if you don't like that either, mark the method to return only
counted references so the caller can't forget:

class SomeObject {
public counted MyObject getMyObject() { /* ... */ }

If the method returns the same reference twice, how is it counted?
If it returns a reference to an existing object with other references
out there, how will it be counted?

I can only make reference counting make sense if it tracks an object
from its creation to its final reference is dead.

/L

 

[Paul J. Lucas]

Hang on, would you ban the use of counted references as instance fields?
Or just let this happen and live with having delayed decrements in those
cases?

I wouldn't ban anything, just put a big, fat warning to developers. I don't
believe in designing language to save programmers from themselves. I'd rather
err on the side of flexibility.

Sure, C++ allows you to shoot yourself in the foot (and blow away your whole
leg); but then it's *my* fault. I'd rather have that situation than my current
one with Java where I *can't* do what I want easily.

- Paul

 

[Paul J. Lucas]

Other resources are different. Relatively few objects need access to a
file, stream, or socket, for example. In addition, these resources tend
to have more uniform life spans. If you want to write your own global
resource manager, go ahead.

Even if I were to attempt that, there's no way I can leverage Java's garbage
collector to help me because, as I already mentioned, one can interact with the
GC in only very limited ways.

By the way, there is an RFE to add a using construct to make the
try-finally syntax. With any luck, it will come in Java 7.

Using is just syntactic sugar that the library *user* must *still* *remember* to
use. As a library implementor, I want to remove that burden from my users entirely.

Unfortunately, I don't think Java will ever get stack-allocation of objects
(which would enable true destructors and RAII) which is what's really needed to
solve the problem at hand.

- Paul

 

[Paul J. Lucas]

How is this different from C++ (excuse any syntax errors: it's been a while)

{
Object a = new Object();
exit(-1); // a's destructor won't run now
}

Small point: you really want stack-based allocation like:

{
Object a; // constructs an Object
exit(-1); // terminates process *now*
} // ~Object() for 'a' would have happened here normally

Anyway, it isn't different; but then *I* never claimed that destructors are
*always* executed. (I was *not* the poster who posted about "finally" always
being run.)

There are two sorts of resources that might need cleaning up.

1. Those which disappear on process exit. The shutdownHooks are not
necessary..
2. Those which don't disappear on process exit. The shutdownHooks are
insufficient, since the JVM may not exit cleanly.

But, for my current application, shutdownHooks are useful in that they let
things like Lucene indexes be closed cleanly. Sure, it's not *guaranteed* to
happen due to a non-clean JVM exit, but then such an exit is usually a bug which
we'll fix.

- Paul