net::ssh popen3 issue?

[ritchie]

hi

I'm using popen3 to query a database. I'm returning 20 records but I'm
only getting 5 back. The command executed is a simple psql query to a
pg database so there's not much room for error there. Is there some
kind of timeout or fixed buffer size when using popen3? Thanks for any
help.

Here's some relevant code:

records = []
headings = []
Commands.session.process.popen3( "sudo bin/get_account" ) { |i,o,e|

x = o.read
puts x
lines = x.split("\n")

lines.each_with_index { |l,i|
fields = l.split("|")
if i == 0
headings = fields
else
if i != 1 # got a line
records << fields
end
end
#puts "line #{i}, fields #{records}"
}
puts "number of lines is #{lines.length}"
}

tw = headings.length
th = records.length

 

[Jamis Buck]

hi

I'm using popen3 to query a database. I'm returning 20 records but I'm
only getting 5 back. The command executed is a simple psql query to a
pg database so there's not much room for error there. Is there some
kind of timeout or fixed buffer size when using popen3? Thanks for any
help.

Here's some relevant code:

records = []
headings = []
Commands.session.process.popen3( "sudo bin/get_account" ) { |i,o,e|

x = o.read
puts x
lines = x.split("\n")

lines.each_with_index { |l,i|
fields = l.split("|")
if i == 0
headings = fields
else
if i != 1 # got a line
records << fields
end
end
#puts "line #{i}, fields #{records}"
}
puts "number of lines is #{lines.length}"
}

tw = headings.length
th = records.length

It sounds like it might be a timing issue... there are a lot of those
when using the Net::SSH stuff, unfortunately.

Perhaps try using the shell.sync service to do it:

shell = session.shell.sync

result = shell.send_command "sudo bin/get_account"
lines = result.stdout
...

The asynchronous nature of Net::SSH really wreaks havoc with these
kinds of things. The shell.sync service is probably the most robust
way I've seen for executing remote commands via ssh, but even it still
suffers from some weak areas. If anyone has come up with a clever way
to execute remote commands using Net::SSH, I'd love to hear about
them.

- Jamis

 

[ritchie]

Jamis thanks for your answer, it worked.

These timing issues are related only to the blocking methods? If I use
channels and state machine stuff all should be well for my production
code?

Cheers

R

 

[Daniel Berger]

hi

I'm using popen3 to query a database. I'm returning 20 records but I'm
only getting 5 back. The command executed is a simple psql query to a
pg database so there's not much room for error there.

What a peculiar way to query a database. Is there a reason you're not
using DBI with the pg driver, or the pg driver directly?

You've got me curious now.

Regards,

Dan

 

[ritchie]

I'm writing an admin system to a remote server, which I'd like to be
able to extend easily. Therefore, I can create remote sql queries or
any other command on the server trivially, and then massage the data as
I need on the client based on return type, table, list etc. I have a
yaml spec file outlining the commands,descriptions and params that are
available on the server and then have my fxruby client generate
appropriate param boxes based on that specification. Further, this is
db agnostic,command type agnositc and secure from the get go. So If I
add a trivial query on the server I document it in the yaml file and
the client is able to use it instantly. Peculiar yet super functional;
I'm done screwing around with sql embedded in php or whatnot for site
admin.

R

 

[Daniel Berger]

I'm writing an admin system to a remote server, which I'd like to be
able to extend easily. Therefore, I can create remote sql queries or
any other command on the server trivially, and then massage the data as
I need on the client based on return type, table, list etc.

I was going to suggest DBD:roxy, but it's not secure.

However, now that Ruby has an openssl package as part of the standard
library, perhaps an encrypted version of DBD:roxy is in order.

Michael?

Regards,

Dan

 

[Jamis Buck]

Jamis thanks for your answer, it worked.

These timing issues are related only to the blocking methods? If I use
channels and state machine stuff all should be well for my production
code?

Yes, on both counts. The timing issues occur because the output from
your command may or may not be sent all in one packet to the client.
Thus, calling "pipe.read" once may only return part of the
output...but the questions is, how many times should you keep trying
to call it? If you call it too many times, your program will block,
which is not usually a good thing.

In general, the channels and state machines will provide a more
general-purpose solution, but they are (unfortunately) rather unwieldy
to implement and maintain.

As I said, I'm still looking for a better option. In the meantime, the
shell.sync service seems to do okay.

- Jamis