Problems with read_eager and Telnet

[Robi]

Hi everybody,
I'm totally new to Python but well motivated

I'm fooling around with Python in order to interface with FlightGear
using a telnet connection.

I can do what I had in mind (send some commands and read output from
Flightgear using the telnetlib) with a read_until() object to catch
every output line I need, but it proved to be very slow (it takes 1/10
of a sec for every read_until().

I tried using the read_eager() object and it's waaaayyyy faster (it
does the job in 1/100 of a sec, maybe more, I didn't tested) but it
gives me problems, it gets back strange strings, repeated ones,
partially broken ones, well ... I don't know what's going on with it.

You see, I don't know telnet (the protocol) very good, I'm very new to
Python and Python's docs are not very specific about that read_eager(9
stuff.

Could someone point me to some more documentation about that? or at
least help me in getting a correct idea of what's going on with
read_eager()?

I'm going on investigating but a help from here would be very
appreciated

Thanks in advance,
Roberto

 

[Jack Diederich]

Hi everybody,
 I'm totally new to Python but well motivated

I'm fooling around with Python in order to interface with FlightGear
using a telnet connection.

I can do what I had in mind (send some commands and read output from
Flightgear using the telnetlib) with a read_until() object to catch
every output line I need, but it proved to be very slow (it takes 1/10
of a sec for every read_until().

I tried using the read_eager() object and it's waaaayyyy faster (it
does the job in 1/100 of a sec, maybe more, I didn't tested) but it
gives me problems, it gets back strange strings, repeated ones,
partially broken ones, well ... I don't know what's going on with it.

You see, I don't know telnet (the protocol) very good, I'm very new to
Python and Python's docs are not very specific about that read_eager(9
stuff.

Could someone point me to some more documentation about that? or at
least help me in getting a correct idea of what's going on with
read_eager()?

Telnet sends two kinds of data over the same channel (a simple TCP
stream). It sends the bytes you actually see in your terminal and it
sends control commands that do things like turn echo on/off and
negotiate what terminal type to use. Each time telnetlib reads from
the socket it puts the control stuff in one bucket and stores the
plain text in a buffer to return from all the read_* commands.

read_eager() returns the plain text that has already been read from
the socket. That might be a partial line. It won't try to read from
the socket to get a full line. That's why it is fast, because it
never does I/O.

-Jack

 

[Robi]

Telnet sends two kinds of data over the same channel (a simple TCP

stream).  It sends the bytes you actually see in your terminal and it
sends control commands that do things like turn echo on/off and
negotiate what terminal type to use.  Each time telnetlib reads from
the socket it puts the control stuff in one bucket and stores the
plain text in a buffer to return from all the read_* commands.

read_eager() returns the plain text that has already been read from
the socket.  That might be a partial line.  It won't try to read from
the socket to get a full line.  That's why it is fast, because it
never does I/O.

-Jack

Ok, that's a start (I'm reading RFC 854 in the meanwhile). Still that
doesn't help me much (sorry, I know it's me, not you).

You mean read_eager() doesn't wait until it gets a complete reading of
a line, instead it reads what's on the socket (even if it's to quick
and there's till nothing) and let's the python script running anyway,
right?
Then with the subsequent read_eager() it will read (if there's
something more on the socket in the meanwhile) the previous data bits
and maybe the new ones too (a new line of data) into a single data
chunk. Is that why I get sometimes repeated empty lines followed by
many consequent lines all together out of a single read_eager() call?

 

[Jack Diederich]

Ok, that's a start (I'm reading RFC 854 in the meanwhile). Still that
doesn't help me much (sorry, I know it's me, not you).

You mean read_eager() doesn't wait until it gets a complete reading of
a line, instead it reads what's on the socket (even if it's to quick
and there's till nothing) and let's the python script running anyway,
right?
Then with the subsequent read_eager() it will read (if there's
something more on the socket in the meanwhile) the previous data bits
and maybe the new ones too (a new line of data) into a single data
chunk. Is that why I get sometimes repeated empty lines followed by
many consequent lines all together out of a single read_eager() call?

Yes. read_eager() will never actually read from the socket, if it has
any data it has already read & processed it will return those. If you
call it enough times it will just start returning empty strings
because it never asks the socket to read & wait for new data.

-Jack

 

[Robi]

Can you point me to a pratical usage example of read_eager()? Maybe
that will help me in making all this clear. I'm still very fuzzy about
the socket and the processing stuff.

I'm still convinced I cannot use read_until() in my project and I'm
determined in looking into the read_eager(), maybe that will be of any
help if carefully used.

 

[Jack Diederich]

Yes. read_eager() will never actually read from the socket, if it has

Can you point me to a pratical usage example of read_eager()? Maybe that
will help me in making all this clear. I'm still very fuzzy about the socket
and the processing stuff.

I'm still convinced I cannot use read_until() in my project and I'm
determined in looking into the read_eager(), maybe that will be of any help
if carefully used.

If you always want to see the response you have to use read_until().
All the other read_* methods don't guarantee you will get the text you
want because they don't wait for the other end of the connection to
have actually sent the data. It's a bummer, but if the server hasn't
returned the data there is nothing magic that telnetlib can do to make
it true.

 

[Robi]

If you always want to see the response you have to use read_until().
All the other read_* methods don't guarantee you will get the text you
want because they don't wait for the other end of the connection to
have actually sent the data.  It's a bummer, but if the server hasn't
returned the data there is nothing magic that telnetlib can do to make
it true.

Well, you see, I could prefetch some usefull data in a spare cycle,
maybe this way I could "fake" almost-realtime ... I don't know, I'm
just guessing here.

Fact is, in my case, when I do a 100 write/read cycles in a second
with telnet.write() followed by telnet.read-eager(), I see I get all
those 100 things done properly, that means, both client and server are
doing it right, there's no lag on the processing and network sides.
But when I use telnet.read_until() I get those feedbacks in the right
order (good) but it's way too slow (it's at least 10 times slower,
toooo bad!!! and, in my case, unusable).

I'm trying to find a way out, and I know in these cases knowledge is
the only way out.
I've seen a huge speed improvement using telnet.read_eager(), I want
to investigate more. Who knows, maybe I can stitch my code around it's
appearant unusability.

 

[Terry Reedy]

Hi everybody,
I'm totally new to Python but well motivated

I'm fooling around with Python in order to interface with FlightGear
using a telnet connection.

Given that FlightGear is a graphical flight simulator
http://www.flightgear.org/
https://secure.wikimedia.org/wikipedia/en/wiki/FlightGear
using a text terminal connection seems a bit odd,
unless using it just to get/set configuration,
in which case, speed should hardly seem an issue.

I can do what I had in mind (send some commands and read output from
Flightgear using the telnetlib) with a read_until() object to catch
every output line I need, but it proved to be very slow (it takes 1/10
of a sec for every read_until().

I presume you are using read_until(b'\n').
read_until cannot return any faster than the server sends complete lines

I tried using the read_eager() object and it's waaaayyyy faster (it
does the job in 1/100 of a sec, maybe more, I didn't tested) but it
gives me problems, it gets back strange strings, repeated ones,
partially broken ones, well ... I don't know what's going on with it.

read_eager is for when you want to parse the bytes yourself, such as
when they come in a continuous stream without newline or other obvious
separators. It will not return b'\n' any faster than read_until. It will
simply give you your output lines in little bits that you would have to
reassemble yourself.

You see, I don't know telnet (the protocol) very good, I'm very new to
Python and Python's docs are not very specific about that read_eager(9
stuff.

Could someone point me to some more documentation about that? or at
least help me in getting a correct idea of what's going on with
read_eager()?

Read the source Lib/telnetlib.py (as I just did). It is pretty
straightforwad code.

 

[Robi]

Given that FlightGear is a graphical flight simulatorhttp://www.flightgear.org/https://secure.wikimedia.org/wikipedia/en/wiki/FlightGear

using a text terminal connection seems a bit odd,
unless using it just to get/set configuration,
in which case, speed should hardly seem an issue.

Right, I'm using it that way, I get/set properties changing them in
real time (I whish!).

I presume you are using read_until(b'\n').
read_until cannot return any faster than the server sends complete lines

Actually it's read_until('\n') (because it's python 2.6, anyway I know
what you meant with b'\n').
And yes, after today I've come to the conclusion that even read_eager
is of no use to me. I have to settle down for that poor telnet speed,
I can't get more than that :-( In fact, the telnet connection speed is
not the issue, it's the time FGFS spends in reading its internals and
answearing back to the telnet client.

I've made some tests, it looks like it doesn't matter how many fgfs's
properties I try and read with telnet at the same time with python, I
always have the same lag, I can even run multiple python instances of
the same script, in order to get/set those same amount of properties
in parallel, well ... result is I always get the same lag, it doesn't
even become slower.

In my test I run a simple loop, in each loop I do a telnet.write() and
a telnet.read_until('\n'). I get back results at 20Hz (that means I
get 120 write/read cycles in 6 seconds).
I repeated the same test doing three telnet.write() followed by three
telnet.read_until('\n') in each loop. Result is always 20Hz (but now
that means I get 120 loop cycles in 6 seconds, which is now 120x3
write/read cycles in 6 seconds).
I even run three instances of the same script at the same time (trying
to saturate fgfs's capabilities and find some limits). No luck (or
yes?!?), I always get a 20hz frequency.

My conclusion being, fgfs cannot answer back quicker than this: 20Hz.

This is quite poor for a quasi-realtime hardware interface (which is
in my intention).
Maybe it's enough for something ... I'll give it a try anyway and find
out what can be a usable scenario for such values.

I guees even telnet is not the best apporach to use for my project,
I'll keep experiencing.

Thank you all anyway, it's always a pleasure to share some
considerations and get back usefull hints.

 

[Terry Reedy]

Right, I'm using it that way, I get/set properties changing them in
real time (I whish!). ....
My conclusion being, fgfs cannot answer back quicker than this: 20Hz.

I suspect that is by design, so as to not interfere with the simulation
itself.

This is quite poor for a quasi-realtime hardware interface (which is
in my intention).

I suggest you discuss your use case on a FlightGear mailing list
(mirrored on gmane as newsgroups) or forum.

 

[Robi]

My conclusion being, fgfs cannot answer back quicker than this: 20Hz.

I suspect that is by design, so as to not interfere with the simulation
itself.

Actually it's not quite like that.

I talked about it in flightgear-devel mailing list; I was told FGFS
default telnet polling frequency is 5Hz (but it's nowhere documented
than in the source code). I had extremely low interaction at first,
slower than what I get now. Increasing this frequency to 120Hz gave me
those results I was talking about in the previous posts; I got
increasing speed but with my (quite crude) benchmark it never goes
over a real 20Hz as stated, it doesn't reach those 120Hz set at hand
by me. But there is no cap at 20Hz set by design in FlightGear, not
that I'm aware of.

I'm going to dig into some other communication media to talk to
FlightGear and leave Telnet for those cases where such a lag is
acceptable.