Getting Net::SFTP handle takes FOR-FLIPPIN-EVER

Discussion in 'Perl Misc' started by usenet@DavidFilmer.com, Jul 7, 2006.

  1. Guest

    Kindly consider the following script which I am having a problem with:

    #!/usr/bin/perl
    use strict; use warnings;
    use Net::SFTP;

    my %args = qw{ user myuserid password mypassword debug 1 };
    $args{'ssh_args'} = [protocol => 2];

    my $sftp = Net::SFTP->new('123.456.78.90', %args) or die $!;

    __END__

    The script works fine, but on one particular machine it takes about a
    minute to get the SFTP handle. The identical script runs instantly on
    another (inferior) machine.

    So, naturally, I would suspect a problem with that particular machine
    that would have nothing to do with my script (maybe a routing issue,
    etc), EXCEPT two things make me think the problem somehow involves the
    Net::SFTP module itself:

    1. "sftp" from a shell (same parameters) connects instantly.
    2. I can get a Net::SSH::perl handle (same %args) instantly.

    With debug mode turned on, I observe that there are long pauses at
    three points. I've posted the debug output (with pauses indicated)
    below. I see no obvious reason for any performance problems (I'm
    running on a new and lightly loaded p520 (POWER5) AIX box). As I said,
    I can get a Net::SSH::perl handle instantly (and Net::SFTP uses N:S:p).

    If anyone has any suggestions on what the problem might be with using
    Net::SFTP from this particular machine, please let me know. Here's my
    Net::SFTP debug output (with pauses indicated):

    server: Reading configuration data /myuserid/.ssh/config
    server: Reading configuration data /etc/ssh_config
    server: Connecting to server123.mycompany.com, port 22.
    server: Remote protocol ver 1.99, remote ver OpenSSH_3.6.1p2
    server: Net::SSH::perl Version 1.29, protocol version 2.0.
    server: No compat match: OpenSSH_3.6.1p2.
    server: Connection established.
    server: Sent key-exchange init (KEXINIT), wait response.
    server: Algorithms, c->s: 3des-cbc hmac-sha1 none
    server: Algorithms, s->c: 3des-cbc hmac-sha1 none

    >>long pause........................................................


    server: Entering Diffie-Hellman Group 1 key exchange.
    server: Sent DH public key, waiting for reply.
    server: Received host key, type 'ssh-dss'.
    server: Host 'server123.mycompany.com' is known and matches host key
    server: Computing shared secret key.

    >>long pause........................................................


    server: Verifying server signature.

    >>long pause........................................................


    server: Waiting for NEWKEYS message.
    server: Enabling incoming encryption/MAC/compression.
    server: Send NEWKEYS, enable outgoing encryption/MAC/compression.
    server: Sending request for user-authentication service.
    server: Service accepted: ssh-userauth.
    server: Trying empty user-authentication request.
    server: Authentication methods that can continue:
    publickey,password,keyboard-interactive.
    server: Next method to try is publickey.
    server: Next method to try is password.
    server: Trying password authentication.
    server: Login completed, opening dummy shell channel.
    server: channel 0: new [client-session]
    server: Requesting channel_open for channel 0.
    server: channel 0: open confirm rwindow 0 rmax 32768
    Establishing SFTP connection with server server123.mycompany.com:
    channel 1: new [client-session]
    server: Requesting channel_open for channel 1.
    server: Sending subsystem: sftp
    server: Requesting service subsystem on channel 1.
    server: channel 1: open confirm rwindow 0 rmax 32768
    server: sftp: Sending SSH2_FXP_INIT
    server: sftp: Remote version: 3
    SFTP connected

    --
    David Filmer (http://DavidFilmer.com)
     
    , Jul 7, 2006
    #1
    1. Advertising

  2. Sisyphus Guest

    <> wrote in message
    ..
    ..
    > server: Algorithms, s->c: 3des-cbc hmac-sha1 none
    >
    > >>long pause........................................................

    >
    > server: Entering Diffie-Hellman Group 1 key exchange.
    > server: Sent DH public key, waiting for reply.
    > server: Received host key, type 'ssh-dss'.
    > server: Host 'server123.mycompany.com' is known and matches host key
    > server: Computing shared secret key.
    >
    > >>long pause........................................................

    >
    > server: Verifying server signature.
    >
    > >>long pause........................................................

    >
    > server: Waiting for NEWKEYS message.


    Looks to me that the 2nd, 3rd, and possibly also the 1st, long pauses occur
    when some mathematical computations take place. Could it be that on the fast
    machine those computations are being performed by compiled (XS) code, but
    are being performed by comparatively slow pure perl code on the slow machine
    ?

    Cheers,
    Rob
     
    Sisyphus, Jul 7, 2006
    #2
    1. Advertising

  3. Guest

    Sisyphus wrote:
    > Could it be that on the fast machine those computations are being performed
    > by compiled (XS) code, but are being performed by comparatively slow pure
    > perl code on the slow machine


    I would agree with your idea if Net::SSH::perl also had these delays on
    the slow machine. That's a pure Perl implementation of ssh. And,
    according to the docs, Net::SFTP uses Net::SSH::perl to broker its
    connection.

    This problem would make a LOT more sense if Net::SSH::perl showed
    similar delays. But I get instant connections with that module, even
    though that very same module is used to establish the Net::SFTP
    connection (and it really is - I looked at the source).

    I'm baffled! It makes NO sense! If anyone has any 'try this and see
    what happens' suggestions then I'm open to testing anything! If anyone
    has an IBM p620 (not a 520 as I typo'd earlier) or something similar
    with AIX 5.3 then I'd appreciate it if you would try my script and tell
    me your results.

    --
    David Filmer (http://DavidFilmer.com)
     
    , Jul 7, 2006
    #3
  4. rahed Guest

    writes:

    > I'm baffled! It makes NO sense! If anyone has any 'try this and see
    > what happens' suggestions then I'm open to testing anything! If anyone
    > has an IBM p620 (not a 520 as I typo'd earlier) or something similar
    > with AIX 5.3 then I'd appreciate it if you would try my script and tell
    > me your results.


    Try to install Math::BigInt::GMP if it's not already there. That helped
    when I met weird behaviour with long ssh authentication times.

    --
    Radek
     
    rahed, Jul 7, 2006
    #4
  5. Guest

    wrote:
    > Sisyphus wrote:
    > > Could it be that on the fast machine those computations are being
    > > performed by compiled (XS) code, but are being performed by
    > > comparatively slow pure perl code on the slow machine

    >
    > I would agree with your idea if Net::SSH::perl also had these delays on
    > the slow machine. That's a pure Perl implementation of ssh. And,
    > according to the docs, Net::SFTP uses Net::SSH::perl to broker its
    > connection.
    >
    > This problem would make a LOT more sense if Net::SSH::perl showed
    > similar delays. But I get instant connections with that module, even
    > though that very same module is used to establish the Net::SFTP
    > connection (and it really is - I looked at the source).


    Look at the %INC to make sure it is the same path to the same module.

    >
    > I'm baffled! It makes NO sense! If anyone has any 'try this and see
    > what happens' suggestions then I'm open to testing anything!


    What is the CPU doing during these long pauses? Idle or working hard?

    Another thing I might do is strace it (or whatever the AIX version of that
    is) and see what system calls are going on during or just before and just
    after the pauses.

    I looked in the Net::SFTP source put couldn't find the lines that trigger
    the debug messages you showed (the ones before and after the pauses). So
    presumably they are coming from one of the modules used inside it. It
    might help if you track those message sources down.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Jul 7, 2006
    #5
  6. Guest

    rahed wrote:
    > Try to install Math::BigInt::GMP if it's not already there. That helped
    > when I met weird behaviour with long ssh authentication times.


    THAT WAS THE PROBLEM!!!!! I owe you a beer!

    FWIW, for the benefit of anyone with this same problem who may stumble
    upon this thread... installing Math::BigInt::GMP on AIX 5.3 requires a
    hack to Makefile.PL - namely that (for some bizarre reason) you must
    change

    'LIBS' => ['-lgmp'],
    to
    'LIBS' => ['-L/usr/local/lib -lgmp'],

    For more info, see my post in comp.unix.aix (http://tinyurl.com/rtrkz)
    where I described various hacks necessary to install several libraries
    and Perl modules (including GPG and Pari stuff) on AIX. This particular
    hack for Math::GMP (which, apparently, is also needed for
    Math::BigInt::GMP) was described by Sisyphus in a thread found here:
    http://tinyurl.com/cb299.

    --
    David Filmer (http://DavidFilmer.com)
     
    , Jul 7, 2006
    #6
  7. apersaud Guest

    Install Math::BigInt::GMP and that should speed things up -- thats how
    I fixed my slowness.


    wrote:
    > Kindly consider the following script which I am having a problem with:
    >
    > #!/usr/bin/perl
    > use strict; use warnings;
    > use Net::SFTP;
    >
    > my %args = qw{ user myuserid password mypassword debug 1 };
    > $args{'ssh_args'} = [protocol => 2];
    >
    > my $sftp = Net::SFTP->new('123.456.78.90', %args) or die $!;
    >
    > __END__
    >
    > The script works fine, but on one particular machine it takes about a
    > minute to get the SFTP handle. The identical script runs instantly on
    > another (inferior) machine.
    >
    > So, naturally, I would suspect a problem with that particular machine
    > that would have nothing to do with my script (maybe a routing issue,
    > etc), EXCEPT two things make me think the problem somehow involves the
    > Net::SFTP module itself:
    >
    > 1. "sftp" from a shell (same parameters) connects instantly.
    > 2. I can get a Net::SSH::perl handle (same %args) instantly.
    >
    > With debug mode turned on, I observe that there are long pauses at
    > three points. I've posted the debug output (with pauses indicated)
    > below. I see no obvious reason for any performance problems (I'm
    > running on a new and lightly loaded p520 (POWER5) AIX box). As I said,
    > I can get a Net::SSH::perl handle instantly (and Net::SFTP uses N:S:p).
    >
    > If anyone has any suggestions on what the problem might be with using
    > Net::SFTP from this particular machine, please let me know. Here's my
    > Net::SFTP debug output (with pauses indicated):
    >
    > server: Reading configuration data /myuserid/.ssh/config
    > server: Reading configuration data /etc/ssh_config
    > server: Connecting to server123.mycompany.com, port 22.
    > server: Remote protocol ver 1.99, remote ver OpenSSH_3.6.1p2
    > server: Net::SSH::perl Version 1.29, protocol version 2.0.
    > server: No compat match: OpenSSH_3.6.1p2.
    > server: Connection established.
    > server: Sent key-exchange init (KEXINIT), wait response.
    > server: Algorithms, c->s: 3des-cbc hmac-sha1 none
    > server: Algorithms, s->c: 3des-cbc hmac-sha1 none
    >
    > >>long pause........................................................

    >
    > server: Entering Diffie-Hellman Group 1 key exchange.
    > server: Sent DH public key, waiting for reply.
    > server: Received host key, type 'ssh-dss'.
    > server: Host 'server123.mycompany.com' is known and matches host key
    > server: Computing shared secret key.
    >
    > >>long pause........................................................

    >
    > server: Verifying server signature.
    >
    > >>long pause........................................................

    >
    > server: Waiting for NEWKEYS message.
    > server: Enabling incoming encryption/MAC/compression.
    > server: Send NEWKEYS, enable outgoing encryption/MAC/compression.
    > server: Sending request for user-authentication service.
    > server: Service accepted: ssh-userauth.
    > server: Trying empty user-authentication request.
    > server: Authentication methods that can continue:
    > publickey,password,keyboard-interactive.
    > server: Next method to try is publickey.
    > server: Next method to try is password.
    > server: Trying password authentication.
    > server: Login completed, opening dummy shell channel.
    > server: channel 0: new [client-session]
    > server: Requesting channel_open for channel 0.
    > server: channel 0: open confirm rwindow 0 rmax 32768
    > Establishing SFTP connection with server server123.mycompany.com:
    > channel 1: new [client-session]
    > server: Requesting channel_open for channel 1.
    > server: Sending subsystem: sftp
    > server: Requesting service subsystem on channel 1.
    > server: channel 1: open confirm rwindow 0 rmax 32768
    > server: sftp: Sending SSH2_FXP_INIT
    > server: sftp: Remote version: 3
    > SFTP connected
    >
    > --
    > David Filmer (http://DavidFilmer.com)
     
    apersaud, Jul 25, 2006
    #7
  8. Guest

    apersaud wrote:
    > Install Math::BigInt::GMP and that should speed things up -- thats how
    > I fixed my slowness.


    Thanks, apersaud. Another poster, Radek (rahed), kindly posed this
    solution in a previous reply to this thread. And you are both correct
    - installing the Math::BigInt::GMP module did solve the Net::SFTP
    problems I was having.

    I do wonder, though... this module is not mentioned in the Net::SSH or
    Net::SFTP perldocs, and a -MCPAN install does not try to install it as
    a prereq. Math::BigInt::GMP is nowhere near being a builtin module.
    Absence of the module does not produce "use" or "require" errors or
    warnings when using Net::SFTP. Dependency on this module is not
    anything I would consider intuitive or easily discerned (and, oddly
    enough, it's not REALLY a dependency, although the horrible performance
    without the module would qualify it as 'dependent' AFAIAC).

    So..... Why doesn't EVERYBODY have this problem with Net::SFTP?

    Hmmmm.

    --
    David Filmer (http://DavidFilmer.com)
     
    , Jul 25, 2006
    #8
  9. Sisyphus Guest

    <> wrote in message
    ..
    ..
    > Absence of the module does not produce "use" or "require" errors or
    > warnings when using Net::SFTP.


    Somewhere in the Net::SFTP source there is presumably something like:

    use Math::BigInt lib => 'GMP';

    That line of code ensures that, if Math::BigInt::GMP is installed, then the
    (very fast) Math::BigInt::GMP module handles the computations. If, however,
    Math::BigInt::GMP is not installed, then the (very slow) Math::BigInt module
    handles the computations. (I have a vague notion that a message is written
    to stderr if Math::BigInt::GMP is not found, telling you that the program is
    reverting to pure perl routines - but I'm not entirely sure about that.)

    That accounts for the difference in the performance that you got - pure perl
    Math::BigInt versus the GMP C library.

    Cheers,
    Rob
     
    Sisyphus, Jul 25, 2006
    #9
  10. Guest

    Sisyphus wrote:
    > Somewhere in the Net::SFTP source there is presumably something like:
    > use Math::BigInt lib => 'GMP';


    There is no such line anywhere within my ../site_perl/5.8.6/Net
    directory structure, including Net::SFTP and Net::SSH::perl. Which
    further compounds my confusion........

    > (I have a vague notion that a message is written
    > to stderr if Math::BigInt::GMP is not found, telling you that the program is
    > reverting to pure perl routines - but I'm not entirely sure about that.)


    That would have been nice in helping me solve my problem, but no such
    message came on STDERR before I installed the GMP module.

    --
    David Filmer (http://DavidFilmer.com)
     
    , Jul 26, 2006
    #10
  11. Sisyphus Guest

    <> wrote in message
    news:...
    > Sisyphus wrote:
    > > Somewhere in the Net::SFTP source there is presumably something like:
    > > use Math::BigInt lib => 'GMP';

    >
    > There is no such line anywhere within my ../site_perl/5.8.6/Net
    > directory structure, including Net::SFTP and Net::SSH::perl. Which
    > further compounds my confusion........
    >


    Sorry .... sloppiness on my part. More likely it's in one of the modules
    loaded by Net::SFTP/Net::SSH::perl.

    After a quick investigation, I think it's Crypt::DH:

    use Math::BigInt lib => "GMP,Pari";

    That will look for Math::BigInt::GMP. If it can't find that, it will look
    for Math::BigInt::pari. And if that can't be found, it reverts to pure perl
    Math::BigInt. That's with Crypt::DH-0.06. With Crypt::DH-0.03 the module
    simply read:

    use Math::pari qw( PARI floor pari2num Mod lift pari2pv );

    Hence, we can see how, with earlier versions of Crypt::DH, the specific
    issue that confronted you would not arise.

    Cheers,
    Rob
     
    Sisyphus, Jul 26, 2006
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jason
    Replies:
    0
    Views:
    214
    Jason
    Jul 6, 2004
  2. Jamis Buck
    Replies:
    0
    Views:
    102
    Jamis Buck
    Jun 18, 2005
  3. Une Bévue

    net sftp getting file mtime ?

    Une Bévue, Jun 29, 2011, in forum: Ruby
    Replies:
    1
    Views:
    1,046
    Une Bévue
    Jun 30, 2011
  4. MaggotChild

    Net::SFTP Fails, sftp OK

    MaggotChild, Oct 28, 2009, in forum: Perl Misc
    Replies:
    2
    Views:
    1,002
    C.DeRykus
    Oct 29, 2009
  5. Kevin13
    Replies:
    3
    Views:
    945
    Kevin13
    Dec 1, 2009
Loading...

Share This Page