Sockets, writing data and shutdownOutput

H

HK

My client connects to an HTTP server by opening a
socket, starting a separate thread to feed it with
data and reading the response in the main
thread. The feeder thread basically does this:

OutputStream out = socket.getOutputStream();
out.write(...);
out.flush();
socket.shutdownOutput();

Problem: roughly half of the time the server
resets the connection before sending its response?

I use the shutdownOutput to make sure the server
understands that we are done and no more data can
be expected, but it seems that it overinterpretes
it as if my client is not interested in
pending response data?

Anyone had this problem and can shed some more
light on it?

Harald.
 
J

jan V

I use the shutdownOutput to make sure the server
understands that we are done and no more data can
be expected, but it seems that it overinterpretes
it as if my client is not interested in
pending response data?

Why don't you socket.shutdownOutput() in your main thread, after you;ve
fully received the web server's response? Why do you have to call it so
early, before having processed the reply?
 
H

HK

jan said:
Why don't you socket.shutdownOutput() in your main thread, after you;ve
fully received the web server's response? Why do you have to call it so
early, before having processed the reply?

I want to make sure that even the last byte
of data gets through to the server and is
not lingering somewhere in an output
buffer of the client. If the data is not
completely send, the response cannot be
complete and then the client will never
get the chance to call shutdownOutput()
if it waits until after it has received
all the response data.

Harald.
 
J

jan V

Why don't you socket.shutdownOutput() in your main thread, after you;ve
I want to make sure that even the last byte
of data gets through to the server and is
not lingering somewhere in an output
buffer of the client.

That's the job of flush(), not shutdownOutput().
 
H

HK

jan said:
That's the job of flush(), not shutdownOutput().

Ok, my explanation was slightly wrong. It is not
the data hanging in a buffer. It is the EOF.

Without shutdownOutput() the server never gets an
EOF while reading the request data. Consequently
it will never issue a close on its outgoing
stream. This results in my clients reader thread
never seeing EOF and so it will keep waiting
for data on the input stream forever.

Should I have mentioned that I am talking about
streaming data. I am not sending fixed sized
chunks back and forth.

Harald.
 
T

Thomas Hawtin

HK said:
My client connects to an HTTP server by opening a
socket, starting a separate thread to feed it with
data and reading the response in the main
thread. The feeder thread basically does this:

OutputStream out = socket.getOutputStream();
out.write(...);
out.flush();
socket.shutdownOutput();

Problem: roughly half of the time the server
resets the connection before sending its response?

Can you switch to chunked encoding? Presumably anything put toy servers
understand that these days (he says).

Tom Hawtin
 
H

HK

Thomas said:
Can you switch to chunked encoding? Presumably anything put toy servers
understand that these days (he says).

I am using chunked encoding. So when I
dechunk, it actually works. But then, for
debugging I would like to read the raw data.
And in addition, I would be interested in
how HTTP is supposed to handle this. I tried
to find something in the RFC2616 about this
without success.

Harald.
 
J

jan V

That's the job of flush(), not shutdownOutput().
Ok, my explanation was slightly wrong.

Ah ;-) It helps to get the explanation right.... (cfr. getting requirements
spec all f*cked up, usually leads to a lot of fun down the line).
It is not the data hanging in a buffer. It is the EOF.

I'm going to kick you [figuratively ;-) ] (and you're going to kick
yourself) if the problem stems from a lack of close(). Are you calling
close() on the stream???? The server is only going to see EOF if you close()
the OutputStream in the client....
 
H

HK

jan said:
It is not the data hanging in a buffer. It is the EOF.

I'm going to kick you [figuratively ;-) ] (and you're going to kick
yourself) if the problem stems from a lack of close(). Are you calling
close() on the stream???? The server is only going to see EOF if you close()
the OutputStream in the client....

Not true. shutdownOutput() on the client side signals
EOF to the server. However, a close() on the
OutputStream shuts down the whole socket,
which is exactly what I not want.

Harald.
 
J

jan V

close() on the stream???? The server is only going to see EOF if you
close()
Not true. shutdownOutput() on the client side signals
EOF to the server. However, a close() on the
OutputStream shuts down the whole socket,

Well, maybe you're using a different version of Java than the version (1.3)
I used to build a system which totally relied on closing output streams to
send EOFs to the recipient. Under 1.3 this worked 100% of the time... what
version are you using?
 
H

HK

jan said:
Well, maybe you're using a different version of Java than the version (1.3)
I used to build a system which totally relied on closing output streams to
send EOFs to the recipient. Under 1.3 this worked 100% of the time... what
version are you using?

I am using 1.4, and 1.5 has the same bug, see
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4717638

Direct from the sources of SocketOutputStream.close():

if (socket != null) {
if (!socket.isClosed())
socket.close();

This does not look like merely closing the output
side of things.

The bug mentioned above, luckily, is still open
despite the unfavorable comment from SUN.
According to the comment, this was always a
problem, which somehow contradicts your statement.
Maybe you did not expect outstanding response data
after closing the output side of the socket in
the client.

Harald.
 
J

jan V

I am using 1.4, and 1.5 has the same bug, see
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4717638

Direct from the sources of SocketOutputStream.close():

if (socket != null) {
if (!socket.isClosed())
socket.close();

This does not look like merely closing the output
side of things.

The bug mentioned above, luckily, is still open
despite the unfavorable comment from SUN.
According to the comment, this was always a
problem, which somehow contradicts your statement.
Maybe you did not expect outstanding response data
after closing the output side of the socket in
the client.

Yep, OK, the Sun source code proves you're right. But my app worked like a
dream... I'd have to go look at *my* source to see precisely what traffic
scenario I subjected my various Sockets to, but all I can say is that I used
the close() API as (then) advertised, and it worked (clearly I can't have
been relying on any stuff coming back via the input stream...). Obviously I
was just a few nanometers away from stepping on yet another bug in Sun's
APIs.

On a software engineering note, this just goes to highlight yet again that
the vast majority of managers completely fail to invest enough resources to
ensure that QUALITY (incl. low defects) is produced before launching some
library API into the big bad world. The above example is one of so many
where the original authors now face the dillema of "Shall we fix this, and
thus break thousands of applications that relied on this flaw, or shall we
leave the flaw as is to maintain backward compatibility?" Either way, the
situation is very messy. The only solution to this is to invest more time
and resources in PREVENTING such bugs creeping in in the first place...
 
T

Thomas Hawtin

jan said:
On a software engineering note, this just goes to highlight yet again that
the vast majority of managers completely fail to invest enough resources to
ensure that QUALITY (incl. low defects) is produced before launching some
library API into the big bad world. The above example is one of so many
where the original authors now face the dillema of "Shall we fix this, and
thus break thousands of applications that relied on this flaw, or shall we
leave the flaw as is to maintain backward compatibility?" Either way, the
situation is very messy. The only solution to this is to invest more time
and resources in PREVENTING such bugs creeping in in the first place...

You mean to criticise your manager because he failed to make you test
your software sufficiently?

For a small product produced with a tough externally set deadline, I
think it was reasonable not to support half-open streams. Other than for
some operations on early version of HTTP I believe it's a relative
obscure feature of TCP.

Tom Hawtin
 
J

jan V

You mean to criticise your manager because he failed to make you test
your software sufficiently?

I'm assuming you mean "in general".. if so, I say "absolutely yes". Testing
should not be left to developers/programmers/software engineers to do any
way they want or are used to, privately, in the comfort of their little
cubicle. It is up to (competent) management to impose a rigorous testing
procedure, and to enforce this testing procedure.

Sun's Bug Database is public (and chronic) proof that Sun don't have a good
enough grip on their Java core libraries development process. Their testing
procedures (despite JCKs etc,,) are self-evidently not good enough. I can
not imagine Sun using extensive code reviews as a *routine* quality control
mechanism, I can also not imagine Sun having an aggressive test-driven
development approach (TDD, Extreme Programming style). The number of
regressions between every major JDK release are proof of this. You don't
sweat regression bugs like a pig if you continually write JUnit-type tests
for every public method in every class you publish for the whole world to
use. I've been hoping for years that Sun would get its act together on this
front, but it's become clear that's never going to happen.
For a small product produced with a tough externally set deadline, I

What small product are you talking about? You're not calling Java's core
libs a "small product" I hope ;-)
 
T

Thomas Hawtin

jan said:
Sun's Bug Database is public (and chronic) proof that Sun don't have a good
enough grip on their Java core libraries development process. Their testing
procedures (despite JCKs etc,,) are self-evidently not good enough. I can
not imagine Sun using extensive code reviews as a *routine* quality control
mechanism, I can also not imagine Sun having an aggressive test-driven
development approach (TDD, Extreme Programming style). The number of
regressions between every major JDK release are proof of this. You don't
sweat regression bugs like a pig if you continually write JUnit-type tests
for every public method in every class you publish for the whole world to
use. I've been hoping for years that Sun would get its act together on this
front, but it's become clear that's never going to happen.

In my opinion the Bug Parade is great proof of Sun's confidence and
openness. Sure there are plenty of bugs, but for a set of libraries that
size it's pretty good (Swing excepted).

I believe Sun claim to have over 150,000 tests.
What small product are you talking about? You're not calling Java's core
libs a "small product" I hope ;-)

About 300 public classes and interfaces, to be downloaded over a dial-up
connection. I'm willing to call that small.

Tom Hawtin
 
J

jan V

In my opinion the Bug Parade is great proof of Sun's confidence and
openness.

That it is too..
Sure there are plenty of bugs, but for a set of libraries that
size it's pretty good (Swing excepted).

I believe Sun claim to have over 150,000 tests.

Let's say the latest JDK has 2500 public classes, let's say with each (on
average) 50 public methods (incl. constructors).. that gives 125,000 entry
points. Let's say 150,000 to simplify things. This would mean that Sun only
have ONE (1) test, on average, per public API method.

If that were so, I think that would suck big time. For my own library
classes I normally write unit tests for every public method and constructor,
and each unit test typically consists of between 5 to 10 different tests
(see example below). If I can hammer my library with at least 5 times more
tests than Sun does, I would expect an organization with the near-bottomless
resources of Sun's to be able to do a lot better than me, a microscopic
little fish with virtually no resources.

-------
Sample extract of my unit test code:

public void z_generateSelfTestReport(SelfTestPrintStream out, boolean
interactiveTest) {

Object[] argValues;
Class[] argTypes;

//------------------------------------------------------------
// tests for rightString()
// tests for rightStringFrom()
// tests for leftString()
// tests for leftStringTo()

argValues = new Object[] {
new Object[] {"Hello", new Integer(0) },
new Object[] {"Hello", new Integer(1) },
new Object[] {"Hello", new Integer(2) },
new Object[] {"Hello", new Integer(3) },
new Object[] {"Hello", new Integer(4) },
new Object[] {"Hello", new Integer(5) },
new Object[] {"Hello", new Integer(6) },

new Object[] {"Hello", new Integer(-1) },
new Object[] {"Hello", new Integer(-2) },
new Object[] {"Hello", new Integer(-3) },
new Object[] {"Hello", new Integer(-4) },
new Object[] {"Hello", new Integer(-5) },
new Object[] {"Hello", new Integer(-6) },
new Object[] {null , new Integer(0) },
};
argTypes = new Class[] {String.class, int.class};
out.testStaticMethod("rightString", argTypes, argValues);
out.testStaticMethod("rightStringFrom", argTypes, argValues);
out.testStaticMethod("leftString", argTypes, argValues);
out.testStaticMethod("leftStringTo", argTypes, argValues);

//------------------------------------------------------------
// tests for chopHead(String body, String tail)
argValues = new Object[] {
new Object[] {"A long sentence","A"},
new Object[] {"A long sentence","A l"},
new Object[] {"A long sentence","A long sentence"},
new Object[] {"A long sentence","long"},
new Object[] {"A long sentence",""},
};
argTypes = new Class[] {String.class, String.class};
out.testStaticMethod("chopHead", argTypes, argValues);

//------------------------------------------------------------
// tests for chopHead(String body, int tailLength)
argValues = new Object[] {
new Object[] {"A long sentence", new Integer(0)},
new Object[] {"A long sentence", new Integer(1)},
new Object[] {"A long sentence", new Integer(2)},
new Object[] {"A long sentence", new Integer(7)},
new Object[] {"A long sentence", new Integer(14)},
new Object[] {"A long sentence", new Integer(15)},
new Object[] {"A long sentence", new Integer(51)},
new Object[] {"A long sentence", new Integer(-1)},
};
argTypes = new Class[] {String.class, int.class};
out.testStaticMethod("chopHead", argTypes, argValues);

//------------------------------------------------------------
// tests for chopTail(String body, String tail)
argValues = new Object[] {
new Object[] {"A tail sentence","A"},
new Object[] {"A tail sentence","A tail sentence"},
new Object[] {"A tail sentence","sentence"},
new Object[] {"A tail sentence","nce"},
new Object[] {"A tail sentence",""},
};
argTypes = new Class[] {String.class, String.class};
out.testStaticMethod("chopTail", argTypes, argValues);

//------------------------------------------------------------
// tests for chopTail(String body, int tailLength)
argValues = new Object[] {
new Object[] {"A long sentence", new Integer(0)},
new Object[] {"A long sentence", new Integer(1)},
new Object[] {"A long sentence", new Integer(8)},
new Object[] {"A long sentence", new Integer(14)},
new Object[] {"A long sentence", new Integer(15)},
new Object[] {"A long sentence", new Integer(51)},
new Object[] {"A long sentence", new Integer(-1)},
};
argTypes = new Class[] {String.class, int.class};
out.testStaticMethod("chopTail", argTypes, argValues);

:
:
and so on. This is just a fraction of the unit tests for my StringKit
utility class.
 
T

Thomas Weidenfeller

Thomas said:
In my opinion the Bug Parade is great proof of Sun's confidence and
openness. Sure there are plenty of bugs, but for a set of libraries that
size it's pretty good (Swing excepted).

Many people are very dissatisfied with the handling of bug reports via
the bug parade, as it has been discussed here in the past. Many have
therefore given up on reporting bugs. Negative experiences include e.g.
that reports were rejected for no apparent reason, that bugs were marked
as duplicate, which weren't duplicates. That bugs were closed as "not a
bug" which were clearly bugs. That when bugs were marked as duplicates
the corresponding votes were not transfered. The habit of marking the
older of two bugs (the one with more votes) as duplicate. A lack of
action on trivial bugs for years.

The worst is maybe Sun's priorities. Instead of fixing what is broken -
often for years - they use their developers to add questionable new
features. I would for example happily live without the new 1.5 for-loop
if a bunch of old GUI bugs would have been fixed.

And when I see that of course they will once again change the class file
format for 1.6, that they add a smart-card API (how many applications
need that?), that they embed the Rhino JavaScript engine, that there are
discussions about adding XML support to the language in 1.7, etc., then
I think Sun has a serious case of featureritis.

/Thomas
 
T

Thomas Weidenfeller

HK said:
OutputStream out = socket.getOutputStream();
out.write(...);
out.flush();
socket.shutdownOutput();

Problem: roughly half of the time the server
resets the connection before sending its response?

I use the shutdownOutput to make sure the server
understands that we are done and no more data can
be expected, but it seems that it overinterpretes
it as if my client is not interested in
pending response data?

It is not an over-interpretation. From the API documentation;

... data will be sent followed by TCP's normal connection termination
sequence.

I read that as that a FIN is sent, triggering of course the close of the
connection on both ends. If you run a network sniffer (highly
recommended for all development involving networking, you should see the
sequence of FIN > , < ACK, < FIN+ACK, ACK >.
( said:
Anyone had this problem and can shed some more
light on it?

Don't close the connection until you are done, or until HTTP indicates
you can do so. See e.g. section 8.1.2 of the RFC 2616 which you
mentioned elsewhere. You might also have to clarify if you are talking
to a 1.1 or 1.0 HTTP server, since the default connection keep-alive
behavior is different between those two HTTP versions.


BTW: The correct Usenet signature separator is '-- ' (two hyphens,
followed my a mandatory space). Many Usenet clients check for this
separator to identify a signature. Your long line of '-' doesn't qualify.

/Thomas
 
E

Esmond Pitt

jan said:
Well, maybe you're using a different version of Java than the version (1.3)
I used to build a system which totally relied on closing output streams to
send EOFs to the recipient. Under 1.3 this worked 100% of the time... what
version are you using?

To clarify this apparent confusion, *both* shutdownOutput* and
Socket.getOutputStream().close() send a FIN and appear to be EOF to the
target, which is why your code worked. However close() closes the input
side of the socket and shutdownOutput() doesn't, which is what the OP
seems to want. I don't see any bugs here. I did submit a bug report once
asking for Socket.getOutputStream().close() to do a flush() and
shutdownOutput() and for Socket.getInputStream().close() to do a
shutdownInput() but Sun decided against it. In retrospect given their
customer base I think they were right, it was already too late.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads


Members online

No members online now.

Forum statistics

Threads
473,769
Messages
2,569,580
Members
45,054
Latest member
TrimKetoBoost

Latest Threads

Top