Problems with read_eager and Telnet

Discussion in 'Python' started by Robi, Feb 28, 2011.

  1. Robi

    Robi Guest

    Hi everybody,
    I'm totally new to Python but well motivated :)

    I'm fooling around with Python in order to interface with FlightGear
    using a telnet connection.

    I can do what I had in mind (send some commands and read output from
    Flightgear using the telnetlib) with a read_until() object to catch
    every output line I need, but it proved to be very slow (it takes 1/10
    of a sec for every read_until().

    I tried using the read_eager() object and it's waaaayyyy faster (it
    does the job in 1/100 of a sec, maybe more, I didn't tested) but it
    gives me problems, it gets back strange strings, repeated ones,
    partially broken ones, well ... I don't know what's going on with it.

    You see, I don't know telnet (the protocol) very good, I'm very new to
    Python and Python's docs are not very specific about that read_eager(9
    stuff.

    Could someone point me to some more documentation about that? or at
    least help me in getting a correct idea of what's going on with
    read_eager()?

    I'm going on investigating but a help from here would be very
    appreciated :)

    Thanks in advance,
    Roberto
    Robi, Feb 28, 2011
    #1
    1. Advertising

  2. On Mon, Feb 28, 2011 at 10:54 AM, Robi <> wrote:
    > Hi everybody,
    >  I'm totally new to Python but well motivated :)
    >
    > I'm fooling around with Python in order to interface with FlightGear
    > using a telnet connection.
    >
    > I can do what I had in mind (send some commands and read output from
    > Flightgear using the telnetlib) with a read_until() object to catch
    > every output line I need, but it proved to be very slow (it takes 1/10
    > of a sec for every read_until().
    >
    > I tried using the read_eager() object and it's waaaayyyy faster (it
    > does the job in 1/100 of a sec, maybe more, I didn't tested) but it
    > gives me problems, it gets back strange strings, repeated ones,
    > partially broken ones, well ... I don't know what's going on with it.
    >
    > You see, I don't know telnet (the protocol) very good, I'm very new to
    > Python and Python's docs are not very specific about that read_eager(9
    > stuff.
    >
    > Could someone point me to some more documentation about that? or at
    > least help me in getting a correct idea of what's going on with
    > read_eager()?


    Telnet sends two kinds of data over the same channel (a simple TCP
    stream). It sends the bytes you actually see in your terminal and it
    sends control commands that do things like turn echo on/off and
    negotiate what terminal type to use. Each time telnetlib reads from
    the socket it puts the control stuff in one bucket and stores the
    plain text in a buffer to return from all the read_* commands.

    read_eager() returns the plain text that has already been read from
    the socket. That might be a partial line. It won't try to read from
    the socket to get a full line. That's why it is fast, because it
    never does I/O.

    -Jack
    Jack Diederich, Feb 28, 2011
    #2
    1. Advertising

  3. Robi

    Robi Guest

    > Telnet sends two kinds of data over the same channel (a simple TCP
    > stream).  It sends the bytes you actually see in your terminal and it
    > sends control commands that do things like turn echo on/off and
    > negotiate what terminal type to use.  Each time telnetlib reads from
    > the socket it puts the control stuff in one bucket and stores the
    > plain text in a buffer to return from all the read_* commands.
    >
    > read_eager() returns the plain text that has already been read from
    > the socket.  That might be a partial line.  It won't try to read from
    > the socket to get a full line.  That's why it is fast, because it
    > never does I/O.
    >
    > -Jack


    Ok, that's a start (I'm reading RFC 854 in the meanwhile). Still that
    doesn't help me much (sorry, I know it's me, not you).

    You mean read_eager() doesn't wait until it gets a complete reading of
    a line, instead it reads what's on the socket (even if it's to quick
    and there's till nothing) and let's the python script running anyway,
    right?
    Then with the subsequent read_eager() it will read (if there's
    something more on the socket in the meanwhile) the previous data bits
    and maybe the new ones too (a new line of data) into a single data
    chunk. Is that why I get sometimes repeated empty lines followed by
    many consequent lines all together out of a single read_eager() call?
    Robi, Feb 28, 2011
    #3
  4. On Mon, Feb 28, 2011 at 11:50 AM, Robi <> wrote:
    >> Telnet sends two kinds of data over the same channel (a simple TCP
    >> stream).  It sends the bytes you actually see in your terminal and it
    >> sends control commands that do things like turn echo on/off and
    >> negotiate what terminal type to use.  Each time telnetlib reads from
    >> the socket it puts the control stuff in one bucket and stores the
    >> plain text in a buffer to return from all the read_* commands.
    >>
    >> read_eager() returns the plain text that has already been read from
    >> the socket.  That might be a partial line.  It won't try to read from
    >> the socket to get a full line.  That's why it is fast, because it
    >> never does I/O.

    >
    > Ok, that's a start (I'm reading RFC 854 in the meanwhile). Still that
    > doesn't help me much (sorry, I know it's me, not you).
    >
    > You mean read_eager() doesn't wait until it gets a complete reading of
    > a line, instead it reads what's on the socket (even if it's to quick
    > and there's till nothing) and let's the python script running anyway,
    > right?
    > Then with the subsequent read_eager() it will read (if there's
    > something more on the socket in the meanwhile) the previous data bits
    > and maybe the new ones too (a new line of data) into a single data
    > chunk. Is that why I get sometimes repeated empty lines followed by
    > many consequent lines all together out of a single read_eager() call?


    Yes. read_eager() will never actually read from the socket, if it has
    any data it has already read & processed it will return those. If you
    call it enough times it will just start returning empty strings
    because it never asks the socket to read & wait for new data.

    -Jack
    Jack Diederich, Feb 28, 2011
    #4
  5. Robi

    Robi Guest

    Can you point me to a pratical usage example of read_eager()? Maybe
    that will help me in making all this clear. I'm still very fuzzy about
    the socket and the processing stuff.

    I'm still convinced I cannot use read_until() in my project and I'm
    determined in looking into the read_eager(), maybe that will be of any
    help if carefully used.
    Robi, Feb 28, 2011
    #5
  6. On Mon, Feb 28, 2011 at 12:13 PM, Roberto Inzerillo
    <> wrote:
    > Yes. read_eager() will never actually read from the socket, if it has
    >>
    >> any data it has already read & processed it will return those.  If you
    >> call it enough times it will just start returning empty strings
    >> because it never asks the socket to read & wait for new data.

    >
    > Can you point me to a pratical usage example of read_eager()? Maybe that
    > will help me in making all this clear. I'm still very fuzzy about the socket
    > and the processing stuff.
    >
    > I'm still convinced I cannot use read_until() in my project and I'm
    > determined in looking into the read_eager(), maybe that will be of any help
    > if carefully used.


    If you always want to see the response you have to use read_until().
    All the other read_* methods don't guarantee you will get the text you
    want because they don't wait for the other end of the connection to
    have actually sent the data. It's a bummer, but if the server hasn't
    returned the data there is nothing magic that telnetlib can do to make
    it true.
    Jack Diederich, Feb 28, 2011
    #6
  7. Robi

    Robi Guest

    On 28 Feb, 18:35, Jack Diederich <> wrote:
    > On Mon, Feb 28, 2011 at 12:13 PM, Roberto Inzerillo
    >
    > <> wrote:
    > > Yes. read_eager() will never actually read from the socket, if it has

    >
    > >> any data it has already read & processed it will return those.  If you
    > >> call it enough times it will just start returning empty strings
    > >> because it never asks the socket to read & wait for new data.

    >
    > > Can you point me to a pratical usage example of read_eager()? Maybe that
    > > will help me in making all this clear. I'm still very fuzzy about the socket
    > > and the processing stuff.

    >
    > > I'm still convinced I cannot use read_until() in my project and I'm
    > > determined in looking into the read_eager(), maybe that will be of any help
    > > if carefully used.

    >
    > If you always want to see the response you have to use read_until().
    > All the other read_* methods don't guarantee you will get the text you
    > want because they don't wait for the other end of the connection to
    > have actually sent the data.  It's a bummer, but if the server hasn't
    > returned the data there is nothing magic that telnetlib can do to make
    > it true.


    Well, you see, I could prefetch some usefull data in a spare cycle,
    maybe this way I could "fake" almost-realtime ... I don't know, I'm
    just guessing here.

    Fact is, in my case, when I do a 100 write/read cycles in a second
    with telnet.write() followed by telnet.read-eager(), I see I get all
    those 100 things done properly, that means, both client and server are
    doing it right, there's no lag on the processing and network sides.
    But when I use telnet.read_until() I get those feedbacks in the right
    order (good) but it's way too slow (it's at least 10 times slower,
    toooo bad!!! and, in my case, unusable).

    I'm trying to find a way out, and I know in these cases knowledge is
    the only way out.
    I've seen a huge speed improvement using telnet.read_eager(), I want
    to investigate more. Who knows, maybe I can stitch my code around it's
    appearant unusability.
    Robi, Feb 28, 2011
    #7
  8. Robi

    Terry Reedy Guest

    On 2/28/2011 10:54 AM, Robi wrote:
    > Hi everybody,
    > I'm totally new to Python but well motivated :)
    >
    > I'm fooling around with Python in order to interface with FlightGear
    > using a telnet connection.


    Given that FlightGear is a graphical flight simulator
    http://www.flightgear.org/
    https://secure.wikimedia.org/wikipedia/en/wiki/FlightGear
    using a text terminal connection seems a bit odd,
    unless using it just to get/set configuration,
    in which case, speed should hardly seem an issue.
    >
    > I can do what I had in mind (send some commands and read output from
    > Flightgear using the telnetlib) with a read_until() object to catch
    > every output line I need, but it proved to be very slow (it takes 1/10
    > of a sec for every read_until().


    I presume you are using read_until(b'\n').
    read_until cannot return any faster than the server sends complete lines

    > I tried using the read_eager() object and it's waaaayyyy faster (it
    > does the job in 1/100 of a sec, maybe more, I didn't tested) but it
    > gives me problems, it gets back strange strings, repeated ones,
    > partially broken ones, well ... I don't know what's going on with it.


    read_eager is for when you want to parse the bytes yourself, such as
    when they come in a continuous stream without newline or other obvious
    separators. It will not return b'\n' any faster than read_until. It will
    simply give you your output lines in little bits that you would have to
    reassemble yourself.

    > You see, I don't know telnet (the protocol) very good, I'm very new to
    > Python and Python's docs are not very specific about that read_eager(9
    > stuff.
    >
    > Could someone point me to some more documentation about that? or at
    > least help me in getting a correct idea of what's going on with
    > read_eager()?


    Read the source Lib/telnetlib.py (as I just did). It is pretty
    straightforwad code.

    --
    Terry Jan Reedy
    Terry Reedy, Feb 28, 2011
    #8
  9. Robi

    Robi Guest

    > Given that FlightGear is a graphical flight simulatorhttp://www.flightgear.org/https://secure.wikimedia.org/wikipedia/en/wiki/FlightGear
    > using a text terminal connection seems a bit odd,
    > unless using it just to get/set configuration,
    > in which case, speed should hardly seem an issue.


    Right, I'm using it that way, I get/set properties changing them in
    real time (I whish!).


    > I presume you are using read_until(b'\n').
    > read_until cannot return any faster than the server sends complete lines


    Actually it's read_until('\n') (because it's python 2.6, anyway I know
    what you meant with b'\n').
    And yes, after today I've come to the conclusion that even read_eager
    is of no use to me. I have to settle down for that poor telnet speed,
    I can't get more than that :-( In fact, the telnet connection speed is
    not the issue, it's the time FGFS spends in reading its internals and
    answearing back to the telnet client.

    I've made some tests, it looks like it doesn't matter how many fgfs's
    properties I try and read with telnet at the same time with python, I
    always have the same lag, I can even run multiple python instances of
    the same script, in order to get/set those same amount of properties
    in parallel, well ... result is I always get the same lag, it doesn't
    even become slower.

    In my test I run a simple loop, in each loop I do a telnet.write() and
    a telnet.read_until('\n'). I get back results at 20Hz (that means I
    get 120 write/read cycles in 6 seconds).
    I repeated the same test doing three telnet.write() followed by three
    telnet.read_until('\n') in each loop. Result is always 20Hz (but now
    that means I get 120 loop cycles in 6 seconds, which is now 120x3
    write/read cycles in 6 seconds).
    I even run three instances of the same script at the same time (trying
    to saturate fgfs's capabilities and find some limits). No luck (or
    yes?!?), I always get a 20hz frequency.

    My conclusion being, fgfs cannot answer back quicker than this: 20Hz.

    This is quite poor for a quasi-realtime hardware interface (which is
    in my intention).
    Maybe it's enough for something ... I'll give it a try anyway and find
    out what can be a usable scenario for such values.

    I guees even telnet is not the best apporach to use for my project,
    I'll keep experiencing.

    Thank you all anyway, it's always a pleasure to share some
    considerations and get back usefull hints.
    Robi, Feb 28, 2011
    #9
  10. Robi

    Terry Reedy Guest

    On 2/28/2011 3:46 PM, Robi wrote:

    >> unless using it just to get/set configuration,
    >> in which case, speed should hardly seem an issue.

    >
    > Right, I'm using it that way, I get/set properties changing them in
    > real time (I whish!).

    ....
    > My conclusion being, fgfs cannot answer back quicker than this: 20Hz.


    I suspect that is by design, so as to not interfere with the simulation
    itself.

    > This is quite poor for a quasi-realtime hardware interface (which is
    > in my intention).


    I suggest you discuss your use case on a FlightGear mailing list
    (mirrored on gmane as newsgroups) or forum.

    --
    Terry Jan Reedy
    Terry Reedy, Feb 28, 2011
    #10
  11. Robi

    Robi Guest

    > > My conclusion being, fgfs cannot answer back quicker than this: 20Hz.
    >
    > I suspect that is by design, so as to not interfere with the simulation
    > itself.


    Actually it's not quite like that.

    I talked about it in flightgear-devel mailing list; I was told FGFS
    default telnet polling frequency is 5Hz (but it's nowhere documented
    than in the source code). I had extremely low interaction at first,
    slower than what I get now. Increasing this frequency to 120Hz gave me
    those results I was talking about in the previous posts; I got
    increasing speed but with my (quite crude) benchmark it never goes
    over a real 20Hz as stated, it doesn't reach those 120Hz set at hand
    by me. But there is no cap at 20Hz set by design in FlightGear, not
    that I'm aware of.

    I'm going to dig into some other communication media to talk to
    FlightGear and leave Telnet for those cases where such a lag is
    acceptable.
    Robi, Mar 1, 2011
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jim Isaacson
    Replies:
    5
    Views:
    593
    Default User
    Nov 5, 2004
  2. Carcarius
    Replies:
    0
    Views:
    275
    Carcarius
    Dec 6, 2007
  3. Sean Warburton

    Problems with Telnet

    Sean Warburton, Jan 23, 2010, in forum: Ruby
    Replies:
    5
    Views:
    161
    Brian Candler
    Jan 27, 2010
  4. Guest

    TELNET and waitfor problems

    Guest, Jul 9, 2003, in forum: Perl Misc
    Replies:
    1
    Views:
    127
    Jay Rogers
    Jul 16, 2003
  5. Mike

    Problems with MS Telnet Server

    Mike, Apr 19, 2004, in forum: Perl Misc
    Replies:
    2
    Views:
    88
    James Willmore
    Apr 20, 2004
Loading...

Share This Page