Weak/Soft references?

M

Mike Schilling

Tegiri said:
You do realize that the key is also referenced by the hash map
internals (to be specific, the entry that contains the key is in
some
hash chain.) .Absent some mechanism like weak references, the key
can't be collection unless it's explicitly removed from the map.

Ok, let's be more specific.

class HashMap {
Entry[] table;
...
}

No problem here, I assume if there is no references to HashMap all
the
rederences to Entry would be marked for deletion.

class Entry<K,V> {
K key;
V value;
Entry<K,V> next;
}

Next if the Entry object is marked for deletion, then key, value,
and
the next entry in the chain are maked as well.

So, there is nothing specific about the key at all!

What you're saying is correct, but doesn't describe the situatioin
where WeakHashMaps are used. Suppose the HashMap is the only
remaining reference to a key value, and you don't want the fact that
the key is in the map to keep it and its associated value in memory.
That is, you'd like iterating over the map to return only entries that
are currently referenced by something other than the map.. Normal
HashMaps, which have hard references to their keys and values, keep
all of them in memory regardless of whether they're used in any other
way. WeakHashMaps do not.
 
M

Mike Schilling

Tegiri said:
To quote another guy, who is mathematician, computer buffs like to
complicate everything. Let see what is important in your example.
You
have a parent widget, child widget and an external data object that
child references. No matter how complicated references subgraph for
the system of all the three objects in our picture are, this whole
hairball of references would be disconnected from the outside world
as
soon as parent goes away, no?

No, not if the external object doesn't go away. Thnk about how events
and listeners are implemented. The object that causes the event needs
reference to the one that's listening for it. Reread my explanation
above if you're still not seeing it.
 
T

Tegiri Nenashi

The internals of the HashMap implementation


Read up a bit on how HashMaps are implemented and you'll begin to
understand it.  If you can design an alternate implementation that
allows the sort of collection you just described, I'd be very
interested in seeing it.- Hide quoted text -

Yep, there is a parallel exchange below. How about little test?

HashMap<String, Byte[]> m = new HashMap<String, Byte[]>();
System.gc();
long h = Runtime.getRuntime().totalMemory();
long hf = Runtime.getRuntime().freeMemory();
m.put("1", new Byte[1000000]);
long h1 = Runtime.getRuntime().totalMemory();
long hf1 = Runtime.getRuntime().freeMemory();
System.out.println("mem="+ ((h1-hf1)-(h-hf)) );
m = null;
System.gc();
h1 = Runtime.getRuntime().totalMemory();
hf1 = Runtime.getRuntime().freeMemory();
System.out.println("mem="+ ((h1-hf1)-(h-hf)) );
 
T

Tegiri Nenashi

Ok, let's be more specific.
class HashMap {
   Entry[] table;
   ...
}
No problem here, I assume if there is no references to HashMap all
the
rederences to Entry would be marked for deletion.
class Entry<K,V> {
   K key;
   V value;
   Entry<K,V> next;
}
Next if the Entry object is marked for deletion, then key, value,
and
the next entry in the chain are maked as well.
So, there is nothing specific about the key at all!

What you're saying is correct, but doesn't describe the situatioin
where WeakHashMaps are used.  Suppose the HashMap is the only
remaining reference to a key value, and you don't want the fact that
the key is in the map to keep it and its associated value in memory.
That is, you'd like iterating over the map to return only entries that
are currently referenced by something other than the map..  Normal
HashMaps, which have hard references to their keys and values, keep
all of them in memory regardless of whether they're used in any other
way.  WeakHashMaps do not.- Hide quoted text -

I would like to move to more rigorous ground and to see the evidence
of HashMap not being GCd in the code (similar to I put in the parallel
thread).

If you are talking about caching (this is supposedly where
WeakHashMaps are used?), then the topic has been covered already.
 
T

Tegiri Nenashi

No. The graph looks like this:

Window <-> Widget <- Data Manager

Fair enough. Let's return to Mike's original claim then:
1. G can be careful to unsubscribing when its parent window exits.
2. G can subscribe via a WeakReference to itself.
2 involves less bookkeeping, and so is more robust.

The bookkeeping can be hidden in the widget cleanup procedure, can't
it? Just make widget to be aware of the objects it lisens to. Joshua
diagramm becomes

Window <-> Widget <-> Data Manager

So one needs to enhance two methods:
1. The one that registers the widget as a listener.
2. The widget cleanup.

Both are the GUI library methods, so whatever happens onside them is
transparent to the end user! This can be done high enough in widget
class hierarchy, of course, to avoid repeating it for every widget
type.
 
A

Andreas Leitgeb

Tegiri Nenashi said:
If you are talking about caching (this is supposedly where
WeakHashMaps are used?), then the topic has been covered already.

No, "Caching" is, where SoftReferences are used.

For WeakReferences, the #1-example of use is "canonicalizing HashMaps".

In other words, it is typically used for HashMaps on Objects whose
equal() is not overridden from Object (thus reflecting identity),
and that (the HashMap) is intended to be *never iterated*.

The only way left to access it, is by retrieving the value for some
particular Key-object. If I ever drop my last reference to such a
Key-object, then, because of the identity-based equality, and due to
my sworn oath never to iterate the Map, that Key in the Map would
become unaccessible for me, but not for the jvm (which doesn't know
about my oath :)

By using a WeakHashMap
<http://java.sun.com/javase/6/docs/api/java/util/WeakHashMap.html>
you effectively tell the jvm, that you're never going to iterate
the map, and that nothing else than the originally added key-objects
will ever be able to successfully retrieve a value through get().

Both of these promises can of course be lies, but then it's you,
who suffers from seemingly odd behaviour.
 
T

Tegiri Nenashi

Fair enough. Let's return to Mike's original claim then:


The bookkeeping can be hidden in the widget cleanup procedure, can't
it? Just make widget to be aware of the objects it lisens to. Joshua
diagramm becomes

Window <-> Widget <-> Data Manager

So one needs to enhance two methods:
1. The one that registers the widget as a listener.
2. The widget cleanup.

Both are the GUI library methods, so whatever happens onside them is
transparent to the end user! This can be done high enough in widget
class hierarchy, of course, to avoid repeating it for every widget
type.

Actually, I overengineered it. No change in the reference graph, just
when the data object is about to notify the widget, check if the
widget is not dead. Null the reference if it is, how complicated is
that?
 
T

Tegiri Nenashi

No, "Caching" is, where SoftReferences are used.

Are you sure? The guy at the end of thread
http://forum.java.sun.com/thread.jspa?threadID=173031&messageID=534016
seems to indicate otherwise?
For WeakReferences, the #1-example of use is "canonicalizing HashMaps".

In other words, it is typically used for HashMaps on Objects whose
equal() is not overridden from Object (thus reflecting identity),
and that (the HashMap) is intended to be *never iterated*.

Frankly, I don't see the point of the map where key identity is not
defined (and is, therefore, reduced to pointer comparison). The map of
such key values pairs is no different from

Array<Pair<Object, Object>>

Everywhere in the code, istead of referring to the key object, you
just refer to the Pair object.

Once again, this discussion (and other threads) convince me that fancy
reference is not really a good idea. What, this concept has been 10
years out already, and perhaps 1 of the 10 programmers in the industry
is even aware of it?
 
T

Tegiri Nenashi

No, not if the external object doesn't go away.  Thnk about how events
and listeners are implemented.  The object that causes the event needs
reference to the one that's listening for it.  Reread my explanation
above if you're still not seeing it

Joshua was generous enough to put a diagramm. (As usual, picture is
worth more than words).

To summarize, we have two usage scenarious which arguably warrants
"fancy" references. So, 4 different types of references for two use
cases? Give me a break.

Wouldn't it be more reasonable to add couple of sections about memory
leak potential problems into java/swing tutorial? Explain there that
if you roll up your own observer/listener class then you face the
potential problem. If you just reuse the library classes, then you
don't have to worry.
 
C

conrad

Tegiri said:
Yep, there is a parallel exchange below. How about little test?

                HashMap<String, Byte[]> m = new HashMap<String, Byte[]>();
        System.gc();

What do you expect the gc() call to do? It's only documented to
provide a hint to the JVM that it is all right with you if GC
happens. It does not require the JVM to perform a collection,
especially if there is no memory pressure at the moment.
 
M

Mike Schilling

Tegiri said:
Actually, I overengineered it. No change in the reference graph, just
when the data object is about to notify the widget, check if the
widget is not dead. Null the reference if it is, how complicated is
that?

Deifne "dead", in such a way that the data object doesn't need to know
anything extraneous about its listeners.
 
M

Mike Schilling

Tegiri said:
Joshua was generous enough to put a diagramm. (As usual, picture is
worth more than words).

To summarize, we have two usage scenarious which arguably warrants
"fancy" references. So, 4 different types of references for two use
cases? Give me a break.

In other words, you're not interested in facts that might interfere with
your existing conclusions. Bye-bye.
 
T

Tegiri Nenashi

Deifne "dead", in such a way that the data object doesn't need to know
anything extraneous about its listeners.

Easy: dead() should be a method in listener interface. Any standard
widget should define it appropriately.
 
T

Tegiri Nenashi

Tegiri said:
Yep, there is a parallel exchange below. How about little test?
                HashMap<String, Byte[]> m = new HashMap<String, Byte[]>();
        System.gc();

What do you expect the gc() call to do?  It's only documented to
provide a hint to the JVM that it is all right with you if GC
happens.  It does not require the JVM to perform a collection,
especially if there is no memory pressure at the moment.

Does it matter? My java console output evidences all the memory
reclaimed at the end of the test where I nulled reference to the
HashMap. If the output were different, then we can suspect the
disclaimer you mentioned among the other reasons.
 
C

conrad

Tegiri said:
Yep, there is a parallel exchange below. How about little test?
                HashMap<String, Byte[]> m = new HashMap<String, Byte[]>();
        System.gc();
What do you expect the gc() call to do?  It's only documented to
provide a hint to the JVM that it is all right with you if GC
happens.  It does not require the JVM to perform a collection,
especially if there is no memory pressure at the moment.

Does it matter? My java console output evidences all the memory
reclaimed at the end of the test where I nulled reference to the
HashMap. If the output were different, then we can suspect the
disclaimer you mentioned among the other reasons.

Of course it matters.

The question of Weak- and SoftReferences is all about what happens at
GC, when memory is tight. If memory is never tight, you don't see any
benefit from them.
 
J

Joshua Cranmer

Tegiri said:
Once again, this discussion (and other threads) convince me that fancy
reference is not really a good idea. What, this concept has been 10
years out already, and perhaps 1 of the 10 programmers in the industry
is even aware of it?

Since when does usage make something a good idea? In retrospect, COBOL
was a bad idea, but that didn't stop it from becoming a popular language.
 
T

Tegiri Nenashi

On Jul 10, 10:50 am, (e-mail address removed) wrote:
Tegiri Nenashi wrote:
Yep, there is a parallel exchange below. How about little test?
                HashMap<String, Byte[]> m = new HashMap<String, Byte[]>();
        System.gc();
What do you expect the gc() call to do?  It's only documented to
provide a hint to the JVM that it is all right with you if GC
happens.  It does not require the JVM to perform a collection,
especially if there is no memory pressure at the moment.
Does it matter? My java console output evidences all the memory
reclaimed at the end of the test where I nulled reference to the
HashMap. If the output were different, then we can suspect the
disclaimer you mentioned among the other reasons.

Of course it matters.

The question of Weak- and SoftReferences is all about what happens at
GC, when memory is tight.  If memory is never tight, you don't see any
benefit from them.

Let me spell it out. I want any example of HashMap usage that
manifests memory leak problem beyond object cache. (That has been
discussed already). I don't see the problem in any other usage
scenario, where the HashMap is not global.

My code snippet demonstrates that if one nulls the reference to
HashMap then it would be marked for garbage collection. Granted your
JVM can theoretically show different output (due to disclamer), but if
the test works in at least one JVM it proves that no matter how
complicated the subgraph of HashMap references is it would be marked
for GC as soon as there are no external references to it.

Both cases investigated here -- Cache and Notifier/Observer -- involve
some objects within the global scope. I reiterate that there is
nothing special about HasMap if it is a local object. What is the case
where a user might want a global HashMap (or any other map
implementation for this matter)?
 
T

Tegiri Nenashi

Since when does usage make something a good idea? In retrospect, COBOL
was a bad idea, but that didn't stop it from becoming a popular language.

Analogies would force us off topic really fast. If anybody cares, XML
occupies #1 spot on my hate list.

Lets make some more progress. We have considered 2 cases so far:

1. Cache. This is not convincing ecxample at all. Make Cache a class
that extends your favorite Map and register it to listen to low memory
event. Implement whatever policy you like: blow up all the content, or
90% of LRU, etc. I see no reason why this class is not in the standard
library already, releaving application programmer of even knowing that
such a problem exists.

2. Observer case is little more challenging. The notifier object
implementation would always have a collection of listener references,
and this collection might grow out of control. This collection brings
a little similarity to the provious case. Why don't this collection
subscribe to the low memory event too? When such an event is triggered
the collection can walk its content and check how many references each
object posesses. Therefore, we need another system API call that would
query GC how many references to the object are out there. Similar to
the previous case, this collection class naturally belongs to the
standard library.
 
A

Andreas Leitgeb

Tegiri Nenashi said:
1. Cache. This is not convincing ecxample at all. Make Cache a class
that extends your favorite Map and register it to listen to low memory
event. Implement whatever policy you like: blow up all the content, or
90% of LRU, etc.
... and check how many references each object posesses.
Therefore, we need another system API call that would
query GC how many references to the object are out there.

You may mean that the GC would get the ref-count cheaply as a
byproduct of the normal gc'ing-pass, but I strongly doubt that.

While an unreachable object can not become reachable
concurrently to the GC's mark-phase (that's before
finalizers are run), a refcount of 1 can easly grow
anytime. So even if the gc identified a ref-count
of 1 for some object: by the time the OutOfMemory-
notification is processed by the first Listener, this
ref-count may already be 42 and higher.

That leads to the point which the Java-devs have
reached back at 1.2, namely to leave the task of
on-demand breaking of certain (non-hard) references
to the GC directly. :)
 
H

Hendrik Maryns

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tegiri Nenashi schreef:
| My code snippet demonstrates that if one nulls the reference to
| HashMap then it would be marked for garbage collection. Granted your

You’ve been missing the point for quite a while now. Nobody wants to
null the reference to the HashMap. *Of course* everything is collected
once you throw away the HashMap. The point is, you want to have a hash
map (note no capitals here, speaking generically), which you keep as
long as you want, without worrying about memory. The problem is, the
stuff that is inside the hash map is referenced by it so can’t be
collected, even if it is no longer used anywhere else in the program.
If you use WeakHashMap, the stuff *inside* the map can get collected,
once nothing else points at it; the reference the map has to it doesn’t
hinder collection.

Use case: You have some pool of objects which are used in some part of
your program. Another part of your program uses them, but needs to do
some effort to get references to them. At the same time it wants to
have them within reach. It puts them in a WeakHashMap, so they are
there whenever it needs them. When OTOH, the first part of the program
stops using the object, it will be removed from the hash map as well, so
the latter part cannot get it any more, which is exactly what is needed,
since it would be invalid to use it any longer.

Hm, maybe not an ideal use case, but something like it.

H.
- --
Hendrik Maryns
http://tcl.sfs.uni-tuebingen.de/~hendrik/
==================
http://aouw.org
Ask smart questions, get good answers:
http://www.catb.org/~esr/faqs/smart-questions.html
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iEYEARECAAYFAkh3HPYACgkQe+7xMGD3itTxXACeIzMtZHCb/mjFvQQXUz//z/Rs
F6gAn2h3PMz8Wnt8oEs13cu/4rqIfoHg
=z+yI
-----END PGP SIGNATURE-----
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,479
Members
44,899
Latest member
RodneyMcAu

Latest Threads

Top