Getting Net::SFTP handle takes FOR-FLIPPIN-EVER

[usenet]

Kindly consider the following script which I am having a problem with:

#!/usr/bin/perl
use strict; use warnings;
use Net::SFTP;

my %args = qw{ user myuserid password mypassword debug 1 };
$args{'ssh_args'} = [protocol => 2];

my $sftp = Net::SFTP->new('123.456.78.90', %args) or die $!;

__END__

The script works fine, but on one particular machine it takes about a
minute to get the SFTP handle. The identical script runs instantly on
another (inferior) machine.

So, naturally, I would suspect a problem with that particular machine
that would have nothing to do with my script (maybe a routing issue,
etc), EXCEPT two things make me think the problem somehow involves the
Net::SFTP module itself:

1. "sftp" from a shell (same parameters) connects instantly.
2. I can get a Net::SSH:erl handle (same %args) instantly.

With debug mode turned on, I observe that there are long pauses at
three points. I've posted the debug output (with pauses indicated)
below. I see no obvious reason for any performance problems (I'm
running on a new and lightly loaded p520 (POWER5) AIX box). As I said,
I can get a Net::SSH:erl handle instantly (and Net::SFTP uses N:S).

If anyone has any suggestions on what the problem might be with using
Net::SFTP from this particular machine, please let me know. Here's my
Net::SFTP debug output (with pauses indicated):

server: Reading configuration data /myuserid/.ssh/config
server: Reading configuration data /etc/ssh_config
server: Connecting to server123.mycompany.com, port 22.
server: Remote protocol ver 1.99, remote ver OpenSSH_3.6.1p2
server: Net::SSH:erl Version 1.29, protocol version 2.0.
server: No compat match: OpenSSH_3.6.1p2.
server: Connection established.
server: Sent key-exchange init (KEXINIT), wait response.
server: Algorithms, c->s: 3des-cbc hmac-sha1 none
server: Algorithms, s->c: 3des-cbc hmac-sha1 none

server: Entering Diffie-Hellman Group 1 key exchange.
server: Sent DH public key, waiting for reply.
server: Received host key, type 'ssh-dss'.
server: Host 'server123.mycompany.com' is known and matches host key
server: Computing shared secret key.

server: Verifying server signature.

server: Waiting for NEWKEYS message.
server: Enabling incoming encryption/MAC/compression.
server: Send NEWKEYS, enable outgoing encryption/MAC/compression.
server: Sending request for user-authentication service.
server: Service accepted: ssh-userauth.
server: Trying empty user-authentication request.
server: Authentication methods that can continue:
publickey,password,keyboard-interactive.
server: Next method to try is publickey.
server: Next method to try is password.
server: Trying password authentication.
server: Login completed, opening dummy shell channel.
server: channel 0: new [client-session]
server: Requesting channel_open for channel 0.
server: channel 0: open confirm rwindow 0 rmax 32768
Establishing SFTP connection with server server123.mycompany.com:
channel 1: new [client-session]
server: Requesting channel_open for channel 1.
server: Sending subsystem: sftp
server: Requesting service subsystem on channel 1.
server: channel 1: open confirm rwindow 0 rmax 32768
server: sftp: Sending SSH2_FXP_INIT
server: sftp: Remote version: 3
SFTP connected

 

[Sisyphus]

..
..

server: Algorithms, s->c: 3des-cbc hmac-sha1 none


server: Entering Diffie-Hellman Group 1 key exchange.
server: Sent DH public key, waiting for reply.
server: Received host key, type 'ssh-dss'.
server: Host 'server123.mycompany.com' is known and matches host key
server: Computing shared secret key.


server: Verifying server signature.


server: Waiting for NEWKEYS message.

Looks to me that the 2nd, 3rd, and possibly also the 1st, long pauses occur
when some mathematical computations take place. Could it be that on the fast
machine those computations are being performed by compiled (XS) code, but
are being performed by comparatively slow pure perl code on the slow machine
?

Cheers,
Rob

 

[usenet]

Could it be that on the fast machine those computations are being performed
by compiled (XS) code, but are being performed by comparatively slow pure
perl code on the slow machine

I would agree with your idea if Net::SSH:erl also had these delays on
the slow machine. That's a pure Perl implementation of ssh. And,
according to the docs, Net::SFTP uses Net::SSH:erl to broker its
connection.

This problem would make a LOT more sense if Net::SSH:erl showed
similar delays. But I get instant connections with that module, even
though that very same module is used to establish the Net::SFTP
connection (and it really is - I looked at the source).

I'm baffled! It makes NO sense! If anyone has any 'try this and see
what happens' suggestions then I'm open to testing anything! If anyone
has an IBM p620 (not a 520 as I typo'd earlier) or something similar
with AIX 5.3 then I'd appreciate it if you would try my script and tell
me your results.

 

[rahed]

I'm baffled! It makes NO sense! If anyone has any 'try this and see
what happens' suggestions then I'm open to testing anything! If anyone
has an IBM p620 (not a 520 as I typo'd earlier) or something similar
with AIX 5.3 then I'd appreciate it if you would try my script and tell
me your results.

Try to install Math::BigInt::GMP if it's not already there. That helped
when I met weird behaviour with long ssh authentication times.

 

[xhoster]

I would agree with your idea if Net::SSH:erl also had these delays on
the slow machine. That's a pure Perl implementation of ssh. And,
according to the docs, Net::SFTP uses Net::SSH:erl to broker its
connection.

This problem would make a LOT more sense if Net::SSH:erl showed
similar delays. But I get instant connections with that module, even
though that very same module is used to establish the Net::SFTP
connection (and it really is - I looked at the source).

Look at the %INC to make sure it is the same path to the same module.

I'm baffled! It makes NO sense! If anyone has any 'try this and see
what happens' suggestions then I'm open to testing anything!

What is the CPU doing during these long pauses? Idle or working hard?

Another thing I might do is strace it (or whatever the AIX version of that
is) and see what system calls are going on during or just before and just
after the pauses.

I looked in the Net::SFTP source put couldn't find the lines that trigger
the debug messages you showed (the ones before and after the pauses). So
presumably they are coming from one of the modules used inside it. It
might help if you track those message sources down.

Xho

 

[usenet]

Try to install Math::BigInt::GMP if it's not already there. That helped
when I met weird behaviour with long ssh authentication times.

THAT WAS THE PROBLEM!!!!! I owe you a beer!

FWIW, for the benefit of anyone with this same problem who may stumble
upon this thread... installing Math::BigInt::GMP on AIX 5.3 requires a
hack to Makefile.PL - namely that (for some bizarre reason) you must
change

'LIBS' => ['-lgmp'],
to
'LIBS' => ['-L/usr/local/lib -lgmp'],

For more info, see my post in comp.unix.aix (http://tinyurl.com/rtrkz)
where I described various hacks necessary to install several libraries
and Perl modules (including GPG and Pari stuff) on AIX. This particular
hack for Math::GMP (which, apparently, is also needed for
Math::BigInt::GMP) was described by Sisyphus in a thread found here:
http://tinyurl.com/cb299.

 

[apersaud]

Install Math::BigInt::GMP and that should speed things up -- thats how
I fixed my slowness.

Kindly consider the following script which I am having a problem with:

#!/usr/bin/perl
use strict; use warnings;
use Net::SFTP;

my %args = qw{ user myuserid password mypassword debug 1 };
$args{'ssh_args'} = [protocol => 2];

my $sftp = Net::SFTP->new('123.456.78.90', %args) or die $!;

__END__

The script works fine, but on one particular machine it takes about a
minute to get the SFTP handle. The identical script runs instantly on
another (inferior) machine.

So, naturally, I would suspect a problem with that particular machine
that would have nothing to do with my script (maybe a routing issue,
etc), EXCEPT two things make me think the problem somehow involves the
Net::SFTP module itself:

1. "sftp" from a shell (same parameters) connects instantly.
2. I can get a Net::SSH:erl handle (same %args) instantly.

With debug mode turned on, I observe that there are long pauses at
three points. I've posted the debug output (with pauses indicated)
below. I see no obvious reason for any performance problems (I'm
running on a new and lightly loaded p520 (POWER5) AIX box). As I said,
I can get a Net::SSH:erl handle instantly (and Net::SFTP uses N:S).

If anyone has any suggestions on what the problem might be with using
Net::SFTP from this particular machine, please let me know. Here's my
Net::SFTP debug output (with pauses indicated):

server: Reading configuration data /myuserid/.ssh/config
server: Reading configuration data /etc/ssh_config
server: Connecting to server123.mycompany.com, port 22.
server: Remote protocol ver 1.99, remote ver OpenSSH_3.6.1p2
server: Net::SSH:erl Version 1.29, protocol version 2.0.
server: No compat match: OpenSSH_3.6.1p2.
server: Connection established.
server: Sent key-exchange init (KEXINIT), wait response.
server: Algorithms, c->s: 3des-cbc hmac-sha1 none
server: Algorithms, s->c: 3des-cbc hmac-sha1 none

server: Entering Diffie-Hellman Group 1 key exchange.
server: Sent DH public key, waiting for reply.
server: Received host key, type 'ssh-dss'.
server: Host 'server123.mycompany.com' is known and matches host key
server: Computing shared secret key.

server: Verifying server signature.

server: Waiting for NEWKEYS message.
server: Enabling incoming encryption/MAC/compression.
server: Send NEWKEYS, enable outgoing encryption/MAC/compression.
server: Sending request for user-authentication service.
server: Service accepted: ssh-userauth.
server: Trying empty user-authentication request.
server: Authentication methods that can continue:
publickey,password,keyboard-interactive.
server: Next method to try is publickey.
server: Next method to try is password.
server: Trying password authentication.
server: Login completed, opening dummy shell channel.
server: channel 0: new [client-session]
server: Requesting channel_open for channel 0.
server: channel 0: open confirm rwindow 0 rmax 32768
Establishing SFTP connection with server server123.mycompany.com:
channel 1: new [client-session]
server: Requesting channel_open for channel 1.
server: Sending subsystem: sftp
server: Requesting service subsystem on channel 1.
server: channel 1: open confirm rwindow 0 rmax 32768
server: sftp: Sending SSH2_FXP_INIT
server: sftp: Remote version: 3
SFTP connected

 

[usenet]

Install Math::BigInt::GMP and that should speed things up -- thats how
I fixed my slowness.

Thanks, apersaud. Another poster, Radek (rahed), kindly posed this
solution in a previous reply to this thread. And you are both correct
- installing the Math::BigInt::GMP module did solve the Net::SFTP
problems I was having.

I do wonder, though... this module is not mentioned in the Net::SSH or
Net::SFTP perldocs, and a -MCPAN install does not try to install it as
a prereq. Math::BigInt::GMP is nowhere near being a builtin module.
Absence of the module does not produce "use" or "require" errors or
warnings when using Net::SFTP. Dependency on this module is not
anything I would consider intuitive or easily discerned (and, oddly
enough, it's not REALLY a dependency, although the horrible performance
without the module would qualify it as 'dependent' AFAIAC).

So..... Why doesn't EVERYBODY have this problem with Net::SFTP?

Hmmmm.

 

[Sisyphus]

..
..

Absence of the module does not produce "use" or "require" errors or
warnings when using Net::SFTP.

Somewhere in the Net::SFTP source there is presumably something like:

use Math::BigInt lib => 'GMP';

That line of code ensures that, if Math::BigInt::GMP is installed, then the
(very fast) Math::BigInt::GMP module handles the computations. If, however,
Math::BigInt::GMP is not installed, then the (very slow) Math::BigInt module
handles the computations. (I have a vague notion that a message is written
to stderr if Math::BigInt::GMP is not found, telling you that the program is
reverting to pure perl routines - but I'm not entirely sure about that.)

That accounts for the difference in the performance that you got - pure perl
Math::BigInt versus the GMP C library.

Cheers,
Rob

 

[usenet]

Somewhere in the Net::SFTP source there is presumably something like:
use Math::BigInt lib => 'GMP';

There is no such line anywhere within my ../site_perl/5.8.6/Net
directory structure, including Net::SFTP and Net::SSH:erl. Which
further compounds my confusion........

(I have a vague notion that a message is written
to stderr if Math::BigInt::GMP is not found, telling you that the program is
reverting to pure perl routines - but I'm not entirely sure about that.)

That would have been nice in helping me solve my problem, but no such
message came on STDERR before I installed the GMP module.

 

[Sisyphus]

There is no such line anywhere within my ../site_perl/5.8.6/Net
directory structure, including Net::SFTP and Net::SSH:erl. Which
further compounds my confusion........

Sorry .... sloppiness on my part. More likely it's in one of the modules
loaded by Net::SFTP/Net::SSH:erl.

After a quick investigation, I think it's Crypt:H:

use Math::BigInt lib => "GMP,Pari";

That will look for Math::BigInt::GMP. If it can't find that, it will look
for Math::BigInt:ari. And if that can't be found, it reverts to pure perl
Math::BigInt. That's with Crypt:H-0.06. With Crypt:H-0.03 the module
simply read:

use Math:ari qw( PARI floor pari2num Mod lift pari2pv );

Hence, we can see how, with earlier versions of Crypt:H, the specific
issue that confronted you would not arise.

Cheers,
Rob