Net::SSH Data Stream Problem

[Andrew Stewart]

Hello,

I am trying to copy a file from my local machine to a remote machine
using Net::SSH. The copy fails part way leaving the file partly
written on the remote machine. The size of the remote file portion
is always 131072 bytes (128 kB). My local file is ~1.2MB. This
leads me to suspect that the data are being fed in chunks and
something is going wrong after the first chunk -- though that's a guess.

Here's the output:

Copying /path/to/my/file.zip to /path/to/remote/directory/
file.zip...done.
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.0.9/lib/net/ssh/transport/
session.rb:256:in `wait_for_message': disconnected: Received data for
nonexistent channel 0. (2) (Net::SSH::Transport:isconnect)
from /usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.0.9/lib/net/
ssh/transport/session.rb:240:in `wait_for_message'
from /usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.0.9/lib/net/
ssh/connection/driver.rb:148:in `process'
from /usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.0.9/lib/net/
ssh/connection/driver.rb:138:in `loop'
from /usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.0.9/lib/net/
ssh/service/process/popen3.rb:66:in `popen3'
[snipped rest of trace]

Interestingly the 'puts "...done."' line is executed before the error
is thrown. Since session.process.popen3 is synchronous, does that
not imply that the copy has finished?

And here's the code (based on Ruby Cookbook recipe 14.11):

require 'rubygems'
require 'net/ssh'

host = 'xyz.com'
user = 'me'
upload_dir = '/path/to/remote/directory/'
my_file = '/path/to/my/file.zip'

def copy_file(session, source_path, destination_path=nil)
destination_path ||= source_path
cmd = %{cat > "#{destination_path.gsub('"', '\"')}"}
session.process.popen3 cmd do |stdin, stdout, stderr|
print "Copying #{source_path} to #{destination_path}..."
open(source_path) { |f| stdin.write f.read }
puts "done."
end
end

Net::SSH.start(host, :username => user) do |session|
copy_file session, my_file, my_file.sub(/^.*\//, "#{upload_dir}")
end

I would greatly appreciate any help.

Thanks and regards,
Andy Stewart

 

[Hugh Sasse]

Hello,

I am trying to copy a file from my local machine to a remote machine using

what platform(s)?

Net::SSH. The copy fails part way leaving the file partly written on the
remote machine. The size of the remote file portion is always 131072 bytes
(128 kB). My local file is ~1.2MB. This leads me to suspect that the data [...]
And here's the code (based on Ruby Cookbook recipe 14.11):

require 'rubygems'
require 'net/ssh'

host = 'xyz.com'
user = 'me'
upload_dir = '/path/to/remote/directory/'
my_file = '/path/to/my/file.zip'

def copy_file(session, source_path, destination_path=nil)
destination_path ||= source_path
cmd = %{cat > "#{destination_path.gsub('"', '\"')}"}
session.process.popen3 cmd do |stdin, stdout, stderr|
print "Copying #{source_path} to #{destination_path}..."
open(source_path) { |f| stdin.write f.read }

# I suspect this should be:
open(source_path, 'rb') { |f| stdin.write f.read }
# so that that ctrl-Z isn't treated as EOF, or something ghastly
# of that sort.

puts "done."
end
end

Net::SSH.start(host, :username => user) do |session|
copy_file session, my_file, my_file.sub(/^.*\//, "#{upload_dir}")
end

Hugh

 

[Andrew Stewart]

what platform(s)?

Local: OS X
Remote: Linux

# I suspect this should be:
open(source_path, 'rb') { |f| stdin.write f.read }
# so that that ctrl-Z isn't treated as EOF, or something ghastly
# of that sort.

I tried that just now but sadly it didn't change the result.

Andy

 

[Hugh Sasse]

Local: OS X
Remote: Linux


I tried that just now but sadly it didn't change the result.

OK, given they're both unices, that figures... I'd then read and
write in smaller chunks. I don't do enough deep networking to know
what packet sizes are sensible, but...

--------------------------------------------------------------- IO#write
ios.write(string) => integer
------------------------------------------------------------------------
Writes the given string to _ios_. The stream must be opened for
writing. If the argument is not a string, it will be converted to a
string using +to_s+. Returns the number of bytes written.
[...]

...probably a good idea to check the return value. And for f.read.

Andy

Hugh

 

[Andrew Stewart]

OK, given they're both unices, that figures... I'd then read and
write in smaller chunks. I don't do enough deep networking to know
what packet sizes are sensible, but...

---------------------------------------------------------------
IO#write
ios.write(string) => integer
----------------------------------------------------------------------
--
Writes the given string to _ios_. The stream must be opened for
writing. If the argument is not a string, it will be converted
to a
string using +to_s+. Returns the number of bytes written.
[...]

...probably a good idea to check the return value. And for f.read.

I changed this:

open(source_path) { |f| stdin.write f.read }

To this:

open(source_path) { |f|
x = f.read
puts "x: #{x.length}"
result = stdin.write x
puts "result: #{result}"
}

And it told me that x is the size of my file, so f.read is reading in
the entire file.

The stdin object is an instance of SSHStdinPipe. The documentation
for its write method [1] says: "Write the given data as channel data
to the underlying channel." I.e. it doesn't mention anything about
size limits.

But clearly there is a size limit somewhere and the 'stdin.write
data' method must be the line which hits it. I dug into the source
but got a bit confused.

Any more ideas?

Thanks,
Andy

[1] http://net-ssh.rubyforge.org/api/classes/Net/SSH/Service/Process/
POpen3Manager/SSHStdinPipe.html

 

[ara.t.howard]

Interestingly the 'puts "...done."' line is executed before the error is
thrown. Since session.process.popen3 is synchronous, does that not imply
that the copy has finished?

And here's the code (based on Ruby Cookbook recipe 14.11):

require 'rubygems'
require 'net/ssh'

host = 'xyz.com'
user = 'me'
upload_dir = '/path/to/remote/directory/'
my_file = '/path/to/my/file.zip'

does this help?

def copy_file(session, source_path, destination_path=nil)
destination_path ||= source_path
cmd = %{cat > "#{destination_path.gsub('"', '\"')}"}
session.process.popen3 cmd do |stdin, stdout, stderr|
print "Copying #{source_path} to #{destination_path}..."
open(source_path) { |f| stdin.write f.read } stdin.flush
stdin.close_write
stdout.read
stderr.read

puts "done."
end
end

-a

 

[Hugh Sasse]

OK, given they're both unices, that figures... I'd then read and
write in smaller chunks. I don't do enough deep networking to know
what packet sizes are sensible, but... [ri output]
...probably a good idea to check the return value. And for f.read.

I changed this: [...]
To this:

open(source_path) { |f|
x = f.read
puts "x: #{x.length}"
result = stdin.write x
puts "result: #{result}"
}

And it told me that x is the size of my file, so f.read is reading in the
entire file.

what was #{result}? Oh ...

The stdin object is an instance of SSHStdinPipe. The documentation for its
write method [1] says: "Write the given data as channel data to the underlying
channel." I.e. it doesn't mention anything about size limits.

... and write doesn't return how many things it wrote breaking the Duck Type
that says "treat me like IO". Rats.

But clearly there is a size limit somewhere and the 'stdin.write data' method
must be the line which hits it. I dug into the source but got a bit confused.

Any more ideas?

I'd read then write q bytes at a time until the whole file is read
I think the value for q is the length of the file you ended up with
at the far end. That would make a good start for a search, anyway.

Thanks,
Andy

[1]
http://net-ssh.rubyforge.org/api/classes/Net/SSH/Service/Process/POpen3Manager/SSHStdinPipe.html

Nice docs, except for not being able to see and navigate to the parent
class easily.

Hugh

 

[Andrew Stewart]

does this help?

It gives me an undefined method error for flush:

undefined method `flush' for
#<Net::SSH::Service:rocess:Open3Manager::SSHStdinPipe:0x6f387c>
(NoMethodError)
from /usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.0.9/lib/net/
ssh/service/process/popen3.rb:52:in `popen3'

Regards,
Andy

 

[Pau Garcia i Quiles]

--nextPart2646694.82fQx2m1pP
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

On Wednesday 08 November 2006 18:28, Andrew Stewart wrote:

What about this:
http://www.elpauer.org/index.php?p=3D213

require 'net/ssh'
require 'net/sftp'

class SSHAgent
def initialize
@agent_env =3D Hash.new
agenthandle =3D IO.popen("/usr/bin/ssh-agent -s", "r")
agenthandle.each_line do |line|
if line.index("echo") =3D=3D nil
line =3D line.slice(0..(line.index(';')-1))
key, value =3D line.chomp.split(/=3D/)
puts "Key =3D #{key}, Value =3D #{value}"
@agent_env[key] =3D value
end
end
end
def [](key)
return @agent_env[key]
end
end

agent =3D SSHAgent.new
ENV["SSH_AUTH_SOCK"] =3D agent["SSH_AUTH_SOCK"]
ENV["SSH_AGENT_PID"] =3D agent["SSH_AGENT_PID"]
system("/usr/bin/ssh-add")

Net::SSH.start( '192.168.1.12',
:username=3D>'pgquiles',
:compression_level=3D>0,
:compression=3D>'none'
) do |session|
session.sftp.connect do |sftp|
sftp.put_file("bigvideo.avi", "bigvideo.avi")
end
end

That code is using public-key, password-less cryptography, but with slight=
=20
modifications it will work with public-key+password or only password. There=
=20
is some more info about Net::SFTP in the blog post.

Hello,
=20
I am trying to copy a file from my local machine to a remote machine =20
using Net::SSH. The copy fails part way leaving the file partly =20
written on the remote machine. The size of the remote file portion =20
is always 131072 bytes (128 kB). My local file is ~1.2MB. This =20
leads me to suspect that the data are being fed in chunks and =20
something is going wrong after the first chunk -- though that's a guess.
=20
Here's the output:
=20
Copying /path/to/my/file.zip to /path/to/remote/directory/=20
file.zip...done.
/usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.0.9/lib/net/ssh/transport/=20
session.rb:256:in `wait_for_message': disconnected: Received data for =20
nonexistent channel 0. (2) (Net::SSH::Transport:isconnect)
from /usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.0.9/lib/net/=20
ssh/transport/session.rb:240:in `wait_for_message'
from /usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.0.9/lib/net/=20
ssh/connection/driver.rb:148:in `process'
from /usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.0.9/lib/net/=20
ssh/connection/driver.rb:138:in `loop'
from /usr/local/lib/ruby/gems/1.8/gems/net-ssh-1.0.9/lib/net/=20
ssh/service/process/popen3.rb:66:in `popen3'
[snipped rest of trace]
=20
Interestingly the 'puts "...done."' line is executed before the error =20
is thrown. Since session.process.popen3 is synchronous, does that =20
not imply that the copy has finished?
=20
And here's the code (based on Ruby Cookbook recipe 14.11):
=20
require 'rubygems'
require 'net/ssh'
=20
host =3D 'xyz.com'
user =3D 'me'
upload_dir =3D '/path/to/remote/directory/'
my_file =3D '/path/to/my/file.zip'
=20
def copy_file(session, source_path, destination_path=3Dnil)
destination_path ||=3D source_path
cmd =3D %{cat > "#{destination_path.gsub('"', '\"')}"}
session.process.popen3 cmd do |stdin, stdout, stderr|
print "Copying #{source_path} to #{destination_path}..."
open(source_path) { |f| stdin.write f.read }
puts "done."
end
end
=20
Net::SSH.start(host, :username =3D> user) do |session|
copy_file session, my_file, my_file.sub(/^.*\//, "#{upload_dir}")
end
=20
I would greatly appreciate any help.
=20
Thanks and regards,
Andy Stewart
=20
=20
=20
=20

=2D-=20
Pau Garcia i Quiles
http://www.elpauer.org
(Due to the amount of work, I usually need 10 days to answer)

--nextPart2646694.82fQx2m1pP
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQBFUiVk/DzYv9iGJzsRAoU2AJ9HKVMGPCKxsX9t1PLpuSsiBcq/OgCg+EXI
qlkwBlhrEApFtHm7yfGf7bM=
=ezSK
-----END PGP SIGNATURE-----

--nextPart2646694.82fQx2m1pP--

 

[Andrew Stewart]

I'd read then write q bytes at a time until the whole file is read
I think the value for q is the length of the file you ended up with
at the far end. That would make a good start for a search, anyway.

Thanks for the suggestion -- it worked. Here are the code changes
(based on Ruby Cookbook recipe 6.6):

I added:

class File
def each_chunk(chunk_size=1024)
yield read(chunk_size) until eof?
end
end

And I changed:

open(source_path) { |f| stdin.write f.read }

To:

open(source_path) { |f|
f.each_chunk { |chunk| stdin.write chunk }
}

I tried various chunk sizes descending from 128kB, the size which
made it over the wire previously. The failures remained until the
chunk size was 1024 bytes.

The overall speed was comparable with scp from the shell.

Thanks for your help,
Andy

 

[Andrew Stewart]

What about this:
http://www.elpauer.org/index.php?p=213
[snipped]

That code is using public-key, password-less cryptography, but with
slight
modifications it will work with public-key+password or only
password. There
is some more info about Net::SFTP in the blog post.

Thanks very much for this. I'll check it out (though I've just got
my code working after several helpful suggestions from others)
because it's using a different approach.

Thanks again,
Andy

 

[greg]

I had the same problem where my ssh connection closing early. I solved
it by opening a shell- don't know if that can help with sftp though. I
am having no problems copying files with the ftp library.

Net::SSH.start(SSH_SERVER, :username => u, assword => p) do |session|
shell = session.shell.sync
shell.exec( cmd ).stdout
end