HttpUrlConnection caching ip

[Christophe Darville]

Hello,

I got an ip caching problem when using HttpUrlConnection class.

When using the code :

URL url = new URL("http://a.domain");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();

The VM seems to cache the ip address associated with the domain "a.domain".
If the ip associated with this domain change, the VM still connect to the
old ip. The only way to connect to the new ip, is to restart the VM.

I have tried
java.security.Security.setProperty("networkaddress.cache.ttl", "0");
but it does not work with HttpUrlConnection (it works fine with InetAddress
class for instance).

Any idea ?

Thank you,
Christophe

 

[Murray]

Hello,

I got an ip caching problem when using HttpUrlConnection class.

When using the code :

URL url = new URL("http://a.domain");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();

The VM seems to cache the ip address associated with the domain "a.domain".
If the ip associated with this domain change, the VM still connect to the
old ip. The only way to connect to the new ip, is to restart the VM.

I have tried
java.security.Security.setProperty("networkaddress.cache.ttl", "0");
but it does not work with HttpUrlConnection (it works fine with InetAddress
class for instance).

Any idea ?

Thank you,
Christophe

Which JDK are you using? If it's 1.3 then I think the property is
sun.net.inetaddr.ttl

Also try setting the property from the command line when starting the JVM.
It might be too late by the time you set it in your code if the
configuration is done statically (no idea of Sun's implementation ...)

 

[Christophe Darville]

Hi Murray,

I am using JDK 1.4.2

I have also tried the "sun.net.inetaddr.ttl" parameter without success.

What I fear, is that HttpUrlConnection does not use this parameter at all
because the parameter works fine when using the InetAddress class.

I will try passing the parameter when starting the VM

Thank you,
Christophe

 

[Christophe Darville]

Passing the parameter when starting JVM does not solve the problem :-(

 

[Murray]

Passing the parameter when starting JVM does not solve the problem :-(

Bugger

 

[iksrazal]

Hello,

I got an ip caching problem when using HttpUrlConnection class.

I don't know about your problem specifically, but I've generally had
much better luck with org.apache.commons.httpclient.HttpClient as a
drop in replacement for HttpUrlConnection .

HTH
iksrazal

 

[Sam]

Hello,

I got an ip caching problem when using HttpUrlConnection class.

When using the code :

URL url = new URL("http://a.domain");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();

The VM seems to cache the ip address associated with the domain "a.domain".
If the ip associated with this domain change, the VM still connect to the
old ip. The only way to connect to the new ip, is to restart the VM.

I have tried
java.security.Security.setProperty("networkaddress.cache.ttl", "0");
but it does not work with HttpUrlConnection (it works fine with InetAddress
class for instance).

Any idea ?

Thank you,
Christophe

Christope,

There is a chance the is a DNS caching problem. When you look up the
IP address corresponding to a url, DNS looks inside it's local cache,
and if found DNS will use that cached IP address instead of going out
to the internet to look it up on a remote database. DNS records have
some kind of timeout paramter where they eventually expire. But
typically ip addresses for a static domain don't change rapidly, so it
could take a while.

On some windows systems, you can use "nslookup" to see what ip address
is beeing returned. I think it works on unix too. You could play
around with that, maybe clear the cache or set a timeout.

Good luck,
Sam90

 

[Christophe Darville]

"Christophe Darville" <[email protected]> wrote in message

Christope,

There is a chance the is a DNS caching problem. When you look up the
IP address corresponding to a url, DNS looks inside it's local cache,
and if found DNS will use that cached IP address instead of going out
to the internet to look it up on a remote database. DNS records have
some kind of timeout paramter where they eventually expire. But
typically ip addresses for a static domain don't change rapidly, so it
could take a while.

On some windows systems, you can use "nslookup" to see what ip address
is beeing returned. I think it works on unix too. You could play
around with that, maybe clear the cache or set a timeout.

Good luck,
Sam90

Sam,

Unfortunately, this is not DNS caching problem. When I ping (on the same
machine where my java application is running) the domain, the new ip address
is exact (after a few seconds, of course) but it is never exact in the java
application. The only way to use the new ip, is to restart the java
application.
So it is a DNS caching problem, but at the java level

Christophe

 

[Sam]

Sam,

Unfortunately, this is not DNS caching problem. When I ping (on the same
machine where my java application is running) the domain, the new ip address
is exact (after a few seconds, of course) but it is never exact in the java
application. The only way to use the new ip, is to restart the java
application.
So it is a DNS caching problem, but at the java level

Christophe

Uhf. I had high hopes. It's the jvm that's caching it, not the URL nor
the connection class - in other words, you are running this code
before and after the ip address change?
url.openConnection();

And this code reflects the change?
).getHostAddress();

This fact that this code even exists:
"0");

indicates that Java security is caching DNS address.

The more I look at it, the more it appears that for whatever reason,
the URL or HTTPURLConnection is ignoring the java security setting and
using the default, which is to keep the same address it found
"forever". This is presumably some static behavior of the command, and
could be a bug. You could probably find the source code and isolate
the problem if you were motivated, but otherwise you may need to find
an alternative the class in question. Sorry I couldn't figure it out.

Below is a relevant post I found on usenet which you've probably seen,
but I'm filing for prosperity:

InetAddress Caching

The InetAddress class has a cache to store successful as well as
unsuccessful host name resolutions. The positive caching is there to
guard
against DNS spoofing attacks; while the negative caching is used to
improve
performance.
By default, the result of positive host name resolutions are cached
forever, because there is no general rule to decide when it is safe to
remove cache entries. The result of unsuccessful host name resolution
is
cached for a very short period of time (10 seconds) to improve
performance.

Under certain circumstances where it can be determined that DNS
spoofing
attacks are not possible, a Java security property can be set to a
different Time-to-live (TTL) value for positive caching. Likewise, a
system
admin can configure a different negative caching TTL value when
needed.

Two Java security properties control the TTL values used for positive
and
negative host name resolution caching:

* networkaddress.cache.ttl (default: -1)
Indicates the caching policy for successful name lookups from the name
service. The value is specified as as integer to indicate the number
of
seconds to cache the successful lookup.
A value of -1 indicates "cache forever".

* networkaddress.cache.negative.ttl (default: 10)
Indicates the caching policy for un-successful name lookups from the
name s
ervice. The value is specified as as integer to indicate the number of
seconds to cache the failure for un-successful lookups.
A value of 0 indicates "never cache". A value of -1 indicates "cache
forever".


Regards,
Sam90

 

[Christophe Darville]

"Christophe Darville" <[email protected]> wrote in message

Uhf. I had high hopes. It's the jvm that's caching it, not the URL nor
the connection class - in other words, you are running this code
before and after the ip address change?

url.openConnection();

And this code reflects the change?

).getHostAddress();

This fact that this code even exists:

"0");

indicates that Java security is caching DNS address.

The more I look at it, the more it appears that for whatever reason,
the URL or HTTPURLConnection is ignoring the java security setting and
using the default, which is to keep the same address it found
"forever". This is presumably some static behavior of the command, and
could be a bug. You could probably find the source code and isolate
the problem if you were motivated, but otherwise you may need to find
an alternative the class in question. Sorry I couldn't figure it out.

Below is a relevant post I found on usenet which you've probably seen,
but I'm filing for prosperity:

InetAddress Caching

The InetAddress class has a cache to store successful as well as
unsuccessful host name resolutions. The positive caching is there to
guard
against DNS spoofing attacks; while the negative caching is used to
improve
performance.
By default, the result of positive host name resolutions are cached
forever, because there is no general rule to decide when it is safe to
remove cache entries. The result of unsuccessful host name resolution
is
cached for a very short period of time (10 seconds) to improve
performance.

Under certain circumstances where it can be determined that DNS
spoofing
attacks are not possible, a Java security property can be set to a
different Time-to-live (TTL) value for positive caching. Likewise, a
system
admin can configure a different negative caching TTL value when
needed.

Two Java security properties control the TTL values used for positive
and
negative host name resolution caching:

* networkaddress.cache.ttl (default: -1)
Indicates the caching policy for successful name lookups from the name
service. The value is specified as as integer to indicate the number
of
seconds to cache the successful lookup.
A value of -1 indicates "cache forever".

* networkaddress.cache.negative.ttl (default: 10)
Indicates the caching policy for un-successful name lookups from the
name s
ervice. The value is specified as as integer to indicate the number of
seconds to cache the failure for un-successful lookups.
A value of 0 indicates "never cache". A value of -1 indicates "cache
forever".


Regards,
Sam90

Sam,

Thank you very much for all the time you spent on my problem.

I think I have red somewhere that InetAddress is keeping a static variable
with ip caching (not so sure now that I have red a lot of things about that
subject) and that this static variable is managed accordingly the
networkaddress.cache.ttl parameter. At least, caching (or no caching)
facility is working fine for the InetAddress class.
What I suspect, is that the URL or UrlConnection or HttpURLConnection or ...
is using the same mechanism but does not care about the
networkaddress.cache.ttl property.
It's just, of course, a suposition.
If only I was able to unload a class, I could try to unload URL for instance
just to see if I am right.
Do you know a way to "reset" all static attributes for a certain class ?

Anyway, this is really a big problem for me and I have no idea how to tackle
it. I hope I will find a solution ....

Best Regards,
Christophe

 

[Roedy Green]

Anyway, this is really a big problem for me and I have no idea how to tackle
it. I hope I will find a solution ....

you may need to work with IPs inside, and use some separate mechanism
to warn your app when they have to change.

 

[marcus]

didn't you say InetAddress was working? just use URLConnection(new
InetAddress("dot.com").getHostAddress()) or some variation. If you can
get the known good IP address your problem is not insurmountable.

 

[Sam]

Sam,

Thank you very much for all the time you spent on my problem.

I think I have red somewhere that InetAddress is keeping a static variable
with ip caching (not so sure now that I have red a lot of things about that
subject) and that this static variable is managed accordingly the
networkaddress.cache.ttl parameter. At least, caching (or no caching)
facility is working fine for the InetAddress class.
What I suspect, is that the URL or UrlConnection or HttpURLConnection or ...
is using the same mechanism but does not care about the
networkaddress.cache.ttl property.
It's just, of course, a suposition.
If only I was able to unload a class, I could try to unload URL for instance
just to see if I am right.
Do you know a way to "reset" all static attributes for a certain class ?

Anyway, this is really a big problem for me and I have no idea how to tackle
it. I hope I will find a solution ....

Best Regards,
Christophe

Christope,

It may be the URL class that's caching, because that's where you
specify the name, so it must do the lookup.

You still have some options. What happens if you give the IP address
instead of the domain name? Does it do the lookup correctly? If so,
you're problem's solved because you could use InetAddress to
pre-process the domain name and give the correct IP address to the URL
constructor.

If not, I would poke through the source code and the API's for the
classes. You can do this by expanding the java source zip file that
comes with the java download from sun. One possibility is to some how
disable the security manager for the duration of the lookup (ugly).
Another thing might be to subclass and override something. You could
also try disconnect and close methods that are specified in the source
code and API's. There is a URI class that might help.

Good luck,
Sam90

 

[Christophe Darville]

Christope,

It may be the URL class that's caching, because that's where you
specify the name, so it must do the lookup.

You still have some options. What happens if you give the IP address
instead of the domain name? Does it do the lookup correctly? If so,
you're problem's solved because you could use InetAddress to
pre-process the domain name and give the correct IP address to the URL
constructor.

I have not tried this but I see a problem with HTTP 1.1 specs. Just because
a single ip address can be mapped to more than one domain (virtual hosting),
I cannot afford to transform a url like http://a.certain.domain/index.html
into http://1.2.3.4/index.html. I have no guarantee that the index.html page
content will be the one of the "a.certain.domain" site and not the one of
another domain hosted on the same machine :-(

If not, I would poke through the source code and the API's for the
classes. You can do this by expanding the java source zip file that
comes with the java download from sun.

I do not see this zip. Where can I find it ? Not in the standard sdk
download ?

One possibility is to some how
disable the security manager for the duration of the lookup (ugly).
Another thing might be to subclass and override something. You could
also try disconnect and close methods that are specified in the source
code and API's. There is a URI class that might help.

Ok, I will investigate those 2 options

Good luck,
Sam90

A big Thank you,
Christophe

 

[Christophe Darville]

didn't you say InetAddress was working? just use URLConnection(new
InetAddress("dot.com").getHostAddress()) or some variation. If you can
get the known good IP address your problem is not insurmountable.

Hi Marcus,

Unfortunately, I do not think replacing domain name by ip in the URL class
can help.

See my answer to Sam about that proposition.

Thank you,
Christophe

 

[marcus]

Now you are being absurd. I assumed when you knew that the DNS ttl was
extremely short you were affiliated with the server folks -- bad
assumption on my part.

What makes you think your situation is so unique it has never occurred
in the history of the internet? DNS caches. Hell, name servers cache
sometimes for inappropriate lengths of time (i used to do tech support
for @Home cable internet -- I know!!), completely ignoring the DNS
record in question. Winsock itself caches. Why is your cacheing
problem the end of the world?

Either get used to the fact that DNS has caches in about a dozen places
and not everywhere on the internet is accessable from everywhere else at
any given time, or explain why your situation os truly different from
what every one else deals with every day.

-- clh

 

[Christophe Darville]

Now you are being absurd. I assumed when you knew that the DNS ttl was
extremely short you were affiliated with the server folks -- bad
assumption on my part.

What makes you think your situation is so unique it has never occurred
in the history of the internet? DNS caches. Hell, name servers cache
sometimes for inappropriate lengths of time (i used to do tech support
for @Home cable internet -- I know!!), completely ignoring the DNS
record in question. Winsock itself caches. Why is your cacheing
problem the end of the world?

Either get used to the fact that DNS has caches in about a dozen places
and not everywhere on the internet is accessable from everywhere else at
any given time, or explain why your situation os truly different from
what every one else deals with every day.

-- clh

I am sorry to be absurd but my problem is not ! And I never said I thought
it was unique in the history of the internet.

The reason is quite simple, I have java applications running nonstop (by non
stop, I mean NON STOP - at least until they crash). Those applications are
using httpUrlConnection to access websites. So, the fact that JVM is caching
DNS until restarting it is a big problem for me, but I am really conscious
it will not prevent the world from running.

My point of view is that a forum is there to submit problem in order to get
answer or turn around or to be sure the problem is not solvable!

Regards,
Christophe

 

[marcus]

Ok I think I see --
You are saying you might have a JVM working for weeks or months, and the
standard API seems to be cacheing the DNS indefinitely (which if it is
true seems to be an outrageously huge bug), and IP addressing doesn't
work because of virtual servers.

I can see a methodology of comparing the IP the JVM is using against an
IP provided by an external service, either a shell command or on a
machine port, and exiting and restarting the JVM if they mismatch -- ugly!

Consider reducing your connection to a simple socket. Is there anything
about URLConnection you cannot readilly emulate with a socket connected
on port 80? Does a socket have the same caching issues?

-- clh

 

[Daniel Hagen]

Sam,

Unfortunately, this is not DNS caching problem. When I ping (on the same
machine where my java application is running) the domain, the new ip address
is exact (after a few seconds, of course) but it is never exact in the java
application. The only way to use the new ip, is to restart the java
application.
So it is a DNS caching problem, but at the java level

Christophe

Just a thought (not considered very carefully, not tested and a very
ugly workaround):

Couldn't you use the API of your underlying OS via JNI to get the IP and
set the HTTP Host Header "manually" for the URLConnection?

Something like:

String formattedIp = someIpFormatMethod(
someNativeGetHostByNameMethod("a.domain") );

URL url = new URL("http://" + formattedIp);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty( "Host", "a.domain");

[...]

Daniel

 

[Christophe Darville]

Ok I think I see --
You are saying you might have a JVM working for weeks or months, and the
standard API seems to be cacheing the DNS indefinitely (which if it is
true seems to be an outrageously huge bug), and IP addressing doesn't
work because of virtual servers.

exact ! (just adding that it is only in the case of HttpUrlConnection, so
the standard api works fine in some circumstances like InetAddress for
instance).

I can see a methodology of comparing the IP the JVM is using against an
IP provided by an external service, either a shell command or on a
machine port, and exiting and restarting the JVM if they mismatch -- ugly!

Consider reducing your connection to a simple socket. Is there anything
about URLConnection you cannot readilly emulate with a socket connected
on port 80? Does a socket have the same caching issues?

Socket will work I guess (InetAddress can be configurated to avoid caching
and I can use InetAddress in cunjunction with socket). But, I will need to
understand in depth HTTP protocol and manage myself what is really done by
HttpUrlConnection (redirect management for instance,...)

Any way, I think I will try the HttpClient from Apache to see if it manage
better caching.

Thank you,
Christophe