piping input when shelling out

[Caio Chassot]

Hi all,

I want to use Kernel#system or one of its friends to manipulate some
data I have in a ruby string. Ideally, I'd want the executed command
to read the string from stdin, as if ruby had piped the data to it.
The data may be reasonably large. Any tips?

thanks

 

[Caio Chassot]

Could you be a bit more clear? Do you want to write a program that
will
cooperate with a stream? As in:

data_source | ruby-program | data_dest

Yes?

No, from within a ruby program I want to call some shell command to
act on data stored in a ruby string, so, in the ruby program,
something like, in pseudoruby:

output = system('shell_cmd --params etc', :input => a_string)

 

[Caio Chassot]

http://www.ruby-doc.org/core/classes/IO.html#M001247

Hope this is what your after.

Looks like it. Thanks a lot.

Do you know if it places the exit code in $? ?

I wonder what happens if the subprocess is killed by the system?

I guess I'll run some tests, but if you happen to know the answers,
I'd be glad to hear them nonetheless.

 

[Caio Chassot]

output = `echo #{s} | perl -pe "s/a/x/g"`

I want to emphasize there are a dozen ways to do this. This one
happens to
be simple to understand. Another way is File.popen(), but that
approach
makes bidirectional communications difficult.

I thought about doing it with echo but then I'd have to escape the
string, and what if I was dealing with binary data, and all that...
so popen looks like a more robust solution.

 

[Caio Chassot]

The problem with File.popen() is that you want bidirectional
communication
with the child process, and AFAIK that doesn't offer it -- you can
either
read, or write, to the pipe, but not both.

Hmm, that's not cool. I'll run some tests, but looks like it's back
to using tempfiles then.

 

[Caio Chassot]

Again, I havn't used it before, but the IO#pipe command seems to
allow for
bi-directional control.

Yea, I figured out how to open a pipe for both read and write. You
use a numeric mode. Here's what I got:

#!/usr/bin/env ruby
io = File.popen('perl -pe s/a/b/g', 0666)
io.puts('abc')
io.close_write
puts io.read

outputs:
bbc

So it's fine for my needs. I just need to wrap it into something more
idiomatic.

Thanks Paul and Daniel

 

[ara.t.howard]

No, from within a ruby program I want to call some shell command to act on
data stored in a ruby string, so, in the ruby program, something like, in
pseudoruby:

output = system('shell_cmd --params etc', :input => a_string)

harp:~ > cat a.rb
require 'open4' # gem install open4
require 'yaml'

stdin = 'foobar'
stdout = ''
stderr = ''

Open4.spawn 'cat', :stdin => stdin, :stdout => stdout, :stderr => stderr

y 'stdout' => stdout
y 'stderr' => stderr



harp:~ > ruby a.rb
---
stdout: foobar
---
stderr: ''



-a

 

[ara.t.howard]

Yea, I figured out how to open a pipe for both read and write. You use a
numeric mode. Here's what I got:

#!/usr/bin/env ruby
io = File.popen('perl -pe s/a/b/g', 0666)
io.puts('abc')
io.close_write
puts io.read

outputs:
bbc

So it's fine for my needs. I just need to wrap it into something more
idiomatic.

Thanks Paul and Daniel

you're making it pretty hard on yourself:

harp:~ > cat a.rb
cat = IO.popen 'cat', 'r+'
cat.sync = true

cat.puts 'foobar'
puts cat.gets

cat.puts 'barfoo'
puts cat.gets



harp:~ > ruby a.rb
foobar
barfoo

my session and open4 lib both abstract this largely, including a good deal of
error handling. there is also the built-in open3 module. check out the docs.

regards.

-a

 

[Caio Chassot]

you're making it pretty hard on yourself:

I guess I was

my session and open4 lib both abstract this largely, including a
good deal of
error handling. there is also the built-in open3 module. check
out the docs.

Thanks.

Having a separate stdout and stderr is great, in fact, just the next
thing I was going to ask about. So both open3 and open4 seem like
great candidates.

Could you take some time to sell open4 over open3?

The main advantage I see is the greater abstraction, i.e. the spawn
method.

 

[ara.t.howard]

I guess I was


Thanks.

Having a separate stdout and stderr is great, in fact, just the next thing I
was going to ask about. So both open3 and open4 seem like great candidates.

Could you take some time to sell open4 over open3?

The main advantage I see is the greater abstraction, i.e. the spawn method.

- you get the child pid returned also with open4

- a command that fails to exec raises and error (note this is sent back from
the child fork)

- when using the spawn method a great deal of abstraction is provided, not
least of which that stdin, stdout, stderr are each handled via a thread so
you may provide objects that can handle async output

Open4.spawn cmd, 1=>stdout_handler 2=>stderr_handler

and it will be passed to the hanlder in an async way


the main reason, however, is having the pid - it's hard to kill a proces
without it! ;-)

that said - if all you require is stdout/stderr stay with the stdlib.

regards.


-a

 

[x1]

Is Open4 not win32 compatable? With all win32ole hooks, I'm suprised
no one has tackled this yet.

Ara, you helped me a while back with spawning processes in win32,
catching pids and error levels.

--just assumed if I could install it as a gem, it would work.



Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\d0t1q>gem install open4
Attempting local installation of 'open4'
Local gem file not found: open4*.gem
Attempting remote installation of 'open4'
Updating Gem source index for: http://gems.rubyforge.org
Successfully installed open4-0.7.0

C:\Documents and Settings\d0t1q>irb
irb(main):001:0> require 'open4'
=> false
irb(main):002:0> Open4.spawn "echo", :stdin => "hello"
NameError: uninitialized constant Fcntl::F_SETFD
from c:/ruby/lib/ruby/gems/1.8/gems/open4-0.7.0/lib/open4.rb:19:in `pope
n4'
from c:/ruby/lib/ruby/gems/1.8/gems/open4-0.7.0/lib/open4.rb:263:in `spa
wn'
from c:/ruby/lib/ruby/1.8/timeout.rb:48:in `timeout'
from c:/ruby/lib/ruby/gems/1.8/gems/open4-0.7.0/lib/open4.rb:262:in `spa
wn'
from (irb):2
irb(main):003:0>

 

[ara.t.howard]

Is Open4 not win32 compatable? With all win32ole hooks, I'm suprised
no one has tackled this yet.

Ara, you helped me a while back with spawning processes in win32,
catching pids and error levels.

--just assumed if I could install it as a gem, it would work.

ah. no - it is not.

popen, or something out of the win32 project, it is then.

there is also this

http://popen4.rubyforge.org/

which i haven't tried.

regards.


-a

 

[Rick DeNatale]

you're making it pretty hard on yourself:

harp:~ > cat a.rb
cat = IO.popen 'cat', 'r+'

Interestingly, I just found a use for popen this morning. I was
playing around with letting open-uri use an 'scp' url.

This is a complete hack based on the URI::HTTP class, and it's pretty
rudimentary. Even though it's an 'scp' uri, it uses ssh internally.

As far as I can tell, without looking too hard, open-uri only supports
opening a URI for reading.

rick@frodo:/public/rubyscripts$ cat uris/scp.rb
#
# = uri/scp.rb
#
# Author:: Rick DeNatale
# License:: You can redistribute it and/or modify it under the same
term as Ruby.
#

require 'uri/generic'

module URI


# scp_URI = "scp://" [ userinfo "@" ] host [ ":" port ]
# [ ; parameter = value ] [ abs_path ]


class SCP < Generic
DEFAULT_PORT = 22

COMPONENT = [
:scheme,
:userinfo, :host, ort,
ath
].freeze

#
# == Description
#
# Create a new URI::SCP object from components of URI::SCP with
# check. It is scheme, userinfo, host, port, and path. It
provided by an Array of a Hash.
#
def self.build(args)
tmp = Util::make_components_hash(self, args)
return super(tmp)
end

#
# == Description
#
# Create a new URI::SCP object from ``generic'' components with no
# check.
#
def initialize(*arg)
super(*arg)
end

def ssh_command
result = ["ssh "]
result << "-p" + port.to_s + " " unless port == 22
result << user + "@" if user
result << host + " cat ~"
result << path
result.join('')
end

def open(&b)
IO.popen(ssh_command, &b)
end
end

@@schemes['SCP'] = SCP
end