Slow throughput problem

[JPractitioner]

Hello all.

There was a client connecting to our server using Tcp protocol. The
connection was established successfully, however when the client were
sending packets, the time it takes to reach the server was very long.
Not only that, the packets reach the server one by one and the time gap
between each packet was very long as well. This happends only with 1
client that connects with no concurrency with other clients. (yes,
there is no concurrent/parallel connection in this case). The
connection was using internet.

I assume that the problem is not with the server since i tried
connecting from my pc to the server and simulating the scenario and the
throughput was fast enough.

I know the scope to answer this is wide, but please throw in some
thoughts. perhaps from your experiences, etc.

Thanks.

 

[tom fredriksen]

Hello all.

There was a client connecting to our server using Tcp protocol. The
connection was established successfully, however when the client were
sending packets, the time it takes to reach the server was very long.
Not only that, the packets reach the server one by one and the time gap
between each packet was very long as well. This happends only with 1
client that connects with no concurrency with other clients. (yes,
there is no concurrent/parallel connection in this case). The
connection was using internet.

I assume that the problem is not with the server since i tried
connecting from my pc to the server and simulating the scenario and the
throughput was fast enough.

I know the scope to answer this is wide, but please throw in some
thoughts. perhaps from your experiences, etc.

The problem is most likely with the computer or the link to it, so this
is probably not the forum for it. PS. you can check the link latency by
using ping or traceroute. Figuring out whats slowing things down on the
computer is a different matter and not always an easy task.

/tom

 

[Alun Harford]

Hello all.

There was a client connecting to our server using Tcp protocol. The
connection was established successfully, however when the client were
sending packets, the time it takes to reach the server was very long.
Not only that, the packets reach the server one by one and the time gap
between each packet was very long as well. This happends only with 1
client that connects with no concurrency with other clients. (yes,
there is no concurrent/parallel connection in this case). The
connection was using internet.

Well there's not a lot you can do about latency (well actually there is, but
I'll conveniently ignore that).
It sounds like you're waiting for a response from the server before sending
more data - this should be avoided.

Unless you post some example code about how you're doing it, I don't think I
can help you much more than that.

Alun Harford

 

[Chris Uppal]

There was a client connecting to our server using Tcp protocol. The
connection was established successfully, however when the client were
sending packets, the time it takes to reach the server was very long.

It's worth checking the way you use buffering in both the clients and server
code. It's /just/ possible that you have something screwy there which is
interacting badly with some other characteristic of the link to that client,
but which you manage to get away with in most cases. It's a long shot, but
since it's about the only thing that you can do anything about...

More likely there's a problem with the routing between that client and your
server which is dropping most of the packets. But you can''t fix that from
code ;-)

-- chris

 

[JPractitioner]

thx for the replies, today was another day for the integration test.
and we seem to deal with the same issue all over again.

Chris Uppal , i ve tried using the max size of integer for buffering,
so there should be enough space already. or i shouldnt have done tht?
anyway, there was no significant improvements.

tom fredriksen, yeah we tried something like what u suggested. we tried
to use telnet and the server can read the message just fine. but from
telnet, its obvious that the replied message will not be handled.. (it
dont have the inputstream.. all tht). but from this test, i knew
already that the throughput was fine except when they use their client.

Alun Harford, actually the client didnt wait before sending more
packet, the server sends the reply via an outputstream .. the following
is the part i put on a Thread instance of the server to handle each
connnection. although in this case, no concurrency involved (only one
client connecting to the server), i do threading for future expansion
possiblities.

public void run(){
for(; {
// the part handling packet received
receive(br.readLine().substring(4));
System.out.println("Packet entered: "+i);

System.out.println("---------------------------------------------------");
Thread.sleep(1000);
i++;

// the part handling packet to be sent
OutputStream os = s.getOutputStream();
PrintWriter pw = new PrintWriter(os);
pw.println(Bridge.stringToReply);
pw.flush();
}
}

looks usual and familiar right? From my side, i used inputstream to
handle message received from server.. ( i made a dummy client for my
internal testing and simulation)
something like this..

is = new InputStreamReader(s.getInputStream());
BufferedReader br = new BufferedReader (is);
System.out.println(br.readLine());

On the real testing environment, i dont know wheter the client are
using something like this. They might not even using java, this is out
of my control. However, i assumed that they should have the right
mechanism to get replies from a Tcp sent packet, right guys? I mean,
this should be standard, right?

thx anyway for all replies.. my conclusion (so far) would be to
investigate the latency issues as per adviced but with narrowed scope
targeting the transaction from the client (which i dont know how it
will receive the server messages).

 

[Chris Uppal]

i ve tried using the max size of integer for buffering,
so there should be enough space already. or i shouldnt have done tht?
anyway, there was no significant improvements.

If you mean you set buffer sizes of Integer.MAX_VALUE and it didn't break then
I'm /very/ surprised. At best it suggests that your buffering code isn't doing
what you think it is.

is = new InputStreamReader(s.getInputStream());
BufferedReader br = new BufferedReader (is);
System.out.println(br.readLine());

If you aren't coding your own buffering by hand (which you aren't in the
examples you posted), then you should use a BufferedInputStream wrapped
immediately around the InputStream returned by Socket.getInputStream().
Similarly for the output stream. It is /not/ sufficient to add buffering
around the InputStreamReader (and if the underlying InputStream is properly
buffered then there's no real point in doing so either).

-- chris

 

[JPractitioner]

Hi chris, thx for your comments,
what do u mean by "coding your own buffering by hand" ?

 

[JPractitioner]

Hi chris, thx for your comments,
what do u mean by "coding your own buffering by hand" ?
can you give some examples?

thanks

 

[tom fredriksen]

tom fredriksen, yeah we tried something like what u suggested. we tried
to use telnet and the server can read the message just fine. but from
telnet, its obvious that the replied message will not be handled.. (it
dont have the inputstream.. all tht). but from this test, i knew
already that the throughput was fine except when they use their client.

You need to troubleshoot this by using a divide and conquer technique,
you need to be sure whether it is the software or the surrounding
systems that are at fault. So start in the middle, the network. Also
make sure you use the correct tools otherwise its of little value.

The point of using ping is to measure the round trip time of the network
and the OS latency of network communication. If that number is
consistently low over a period of time then you know its not the
network. but the client or the server. Telnet is not going to help you
with that. Beware that ping uses UDP, while your app is probably using
TCP, but for network latency testing ping is fine.

To test the server, do it from a different machine but from the same
network. if the server is fine then its the client you need to test.
Dont forget to test over a small period, so that you get an average

To test the client, try running the client against another server if
that it possible. Or you create a new and simple client to run on the
same client machine. If the new client software also has problems then
its the OS or the old programs runtime system which has a problem, if
its the new client is fine the the its the old program alone which is
the cause of your problems

Just keep on dividing the component of the system until you find where
the problem is.

I hope this helps.

/tom

 

[JPractitioner]

Hi tom, thx for your advices (which i found are very useful).
We are currently having connection with successful transaction when i
installed a dummy client on the sending site, just to test this .. and
it worked. Thanks buddy, now we know for sure, network latency and the
server was not at fault.

 

[JPractitioner]

tom, just a quick one?
i've heard that ping is on icmp
and that ping is also used on tcp networks.
however, i also kind of agree that ping is on udp.. due to its nature.
hmm... care to discuss?

thx in advance.

 

[Chris Uppal]

what do u mean by "coding your own buffering by hand" ?

If you don't use a BufferedInputStream, but instead read data from the socket
(or file or whatever) straight into a large bye[] array (which you later
process in smaller chunks) then that's what I mean by doing the buffering by
hand. The point of (any) buffering is to minimise the number of calls out to
the underlying OS (or network) since they are expensive, so you want to read
(or write) as much data in one call as possible. Typical code has no real need
for doing that kind of messing around (since the BufferedInputStream provides
the functionality for you), but there are times when it's better to take over
and do it yourself -- normally because you need more precise control over
buffering rather than to gain efficiency.

Note, by the way, that if you are doing buffering yourself then you should be
using an InputStream (or OutputStream) rather than a Reader (or Writer), since
the process of converting bytes to chars may generate many small reads()[*] --
which is precisely what you want to avoid.

Another BTW: you'll often see the advice to do read()s in large chunks anyway
(i.e. even if you are using a BufferedInputStream or BufferedReader) for
performance reasons. I don't think that's very good advice. If your
application is naturally such that it works in large chunks of data then all
well and good -- you may as well read it in large chunks. But if not, and it
buggers up the structure of the application, then it comes under the general
heading of premature optimisation -- especially since the performance gains are
usually very small if you're using the buffered streams correctly.

-- chris

[*] depending on the encoding and how it's implemented.

 

[tom fredriksen]

tom, just a quick one?
i've heard that ping is on icmp
and that ping is also used on tcp networks.
however, i also kind of agree that ping is on udp.. due to its nature.
hmm... care to discuss?

Sorry a slight mistake there... I confused ip datagrams and UDP.

Ping is an ip protocol tool, so the connection is ip datagram based. Ip
protocols are in the network layer while TCP and UDP are in the
Transport layer, i.e. the above layer. Meaning ping can not send TCP
(nor UDP). But what it does use, is the ICMP protocol, by sending
ECHO_REQUEST datagrams.

So, the ip tools are independent of TCP and UDP, they are only bound to
the ip protocol. Basically, the entire internet is connected by the ip
protocol at the network layer (hence the name Internet Protocol), you
can use any transport protocol on top of that as long as it support ip.

/tom

 

[JPractitioner]

thanks a lot tom, that was really enlightening,
thanks again.