calling scp from ruby

[Seid Rudy]

I have a problem using "scp" from a ruby scrip. It copies my sql file to
the destination but it cuts the file the middle of line 290.
When i run scp manually i the copied file is normal.

I am using `` to call the unix command

Dir.chdir(sql_folder)
`scp -r #{sql_folder} #{remotelogin}:#{remotesqlfolder}/#{document_id}`

Do you have any idea what i am doing wrong?

 

[Jan Svitok]

What I'm using when scping is: scp -r dirToCopy destination ( -r for
recursive copy folder)

but, I believe that when dirToCopy or destination is remote, you should write:

USER@IP:/FOLDER/FILE

in your lines I can't see the @ sign

Could it be like:
`scp -r #{sql_folder} #{remotelogin}:#{remotesqlfolder}/#{document_id}`
`scp -r #{sql_folder} #{remoteUSER}@#{remotesIP}:/#{document_id}`

(The OP seems to know the proper syntax, as he is able to do it by hand.)

(I have no clue what the cause is, these are just hint what to try)
- try another file (to see whether it is dependent on some pattern in
the file, or whether it transfers the same size all the time)
- try removing few line around that line
- try using system instead of ``
- look at the logs on the remote side

 

[Seid Rudy]

Try the -v switch to scp. Perhaps it will break even more but perhaps
it would do the same and show useful information.

Thanks

Michal

hi i tried -v option and found out that scp is getting chopped @ byte
40960. Any clue how i can change the byte limit?

debug1: Sending command: scp -v -r -t
/var/local/rudyserv/wilcox/hr/_sql/
Entering directory: D0755 0 069999
Sending file modes: C0644 40960 069999.sql
Sending file modes: C0644 0 069999.err
Sending file modes: C0644 0 069999.tst
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.1 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status 0

rudy@localhost $ ls -al ../sql/069999/069999.sql
-rw-r--r-- 1 seid users 41455 Sep 9 03:18 ../sql/069999/069999.sql

 

[ara.t.howard]

hi i tried -v option and found out that scp is getting chopped @ byte
40960. Any clue how i can change the byte limit?

debug1: Sending command: scp -v -r -t
/var/local/rudyserv/wilcox/hr/_sql/
Entering directory: D0755 0 069999
Sending file modes: C0644 40960 069999.sql
Sending file modes: C0644 0 069999.err
Sending file modes: C0644 0 069999.tst
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.1 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status 0

rudy@localhost $ ls -al ../sql/069999/069999.sql
-rw-r--r-- 1 seid users 41455 Sep 9 03:18 ../sql/069999/069999.sql

i spawn scp all the time from ruby, with gigantic files of tens of gigs and
have never seen this. i think this maybe either an os issue or your disk is
failing (though didn't you mention it worked by hand?).

one other possiblity is some pipe filling issue. by using backticks you are
relying on ruby to read stdout as fast as scp produces it. i can't imagine
why there would be an issue there, but i'd try your command using something
like

system "scp #{ src } #{ dst } >#{ log } 2>&1"

are you on windows by any chance?

-a

 

[Seid Rudy]

On Sat, 9 Sep 2006, Seid Rudy wrote:

system "scp #{ src } #{ dst } >#{ log } 2>&1"

are you on windows by any chance?

no, unix

I did that and i got an empty log.

I also rerun the script without deleting the sql file so it appended to
the file like it should. When i looked at the destination file i noticed
the file stopped at the same spot. But when i look up further to the top
of the destinatio file i noticed it had the a complete content of the
sql file followed by an incomplete content of the sql file. Ofcourse the
source has content of two complete sql files appended in one.

 

[Seid Rudy]

no, unix

I did that and i got an empty log.

I also rerun the script without deleting the sql file so it appended to
the file like it should. When i looked at the destination file i noticed
the file stopped at the same spot. But when i look up further to the top
of the destinatio file i noticed it had the a complete content of the
sql file followed by an incomplete content of the sql file. Ofcourse the
source has content of two complete sql files appended in one.

I DON'T THINK IT'S scp ISSUE.
i get the same problem when i do this

system("cat #{sqloutput}/#{date_folder}.sql
#{sqloutput}/#{date_folder}.jnk >#{sqloutput}/#{date_folder}.out")

 

[ara.t.howard]

I DON'T THINK IT'S scp ISSUE.
i get the same problem when i do this

system("cat #{sqloutput}/#{date_folder}.sql
#{sqloutput}/#{date_folder}.jnk >#{sqloutput}/#{date_folder}.out")

hmmm. are you sure you don't have disk issues?

-a