System calls with ` in parameters

[Gerad S.]

Hi All,

I am trying to add functionality to an interesting script. The part
that is tripping me up is that I have to make a system call to echo, and
feed several parameters. One of the parameters occasionally includes
the ` character.

Because of this, the syntax of:

`echo "#{someVar}" | ./some-script.py > "#{outputFile}"`

is breaking when #{someVar} contains the tick `. This causes the
command to fail because it executes up to the tick ` in someVar and not
to the end as intended.

I have also tried using the system() command as follows:

system("echo \"#{someVar}\" | ./some-script.py > \"#{outputFile}\"")

But this fails when there is a tick ` in #{someVar} as well.

Any help would be appreciated.

Thanks,
Gerad

 

[Eric Hodel]

Hi All,

I am trying to add functionality to an interesting script. The part
that is tripping me up is that I have to make a system call to echo, and
feed several parameters. One of the parameters occasionally includes
the ` character.

Because of this, the syntax of:

`echo "#{someVar}" | ./some-script.py > "#{outputFile}"`

is breaking when #{someVar} contains the tick `. This causes the
command to fail because it executes up to the tick ` in someVar and not
to the end as intended.

I have also tried using the system() command as follows:

system("echo \"#{someVar}\" | ./some-script.py > \"#{outputFile}\"")

But this fails when there is a tick ` in #{someVar} as well.

Any help would be appreciated.

Shellwords should do it:

ruby -rshellwords -e 'p Shellwords.escape "`"'

 

[Robert Klemme]

Hi All,

I am trying to add functionality to an interesting script. =A0The part
that is tripping me up is that I have to make a system call to echo, and
feed several parameters. =A0One of the parameters occasionally includes
the ` character.

Because of this, the syntax of:

`echo "#{someVar}" | ./some-script.py > "#{outputFile}"`

is breaking when #{someVar} contains the tick `. =A0This causes the
command to fail because it executes up to the tick ` in someVar and not
to the end as intended.

I have also tried using the system() command as follows:

system("echo \"#{someVar}\" | ./some-script.py > \"#{outputFile}\"")

Why do you escape the string interpolation? Don't you want those file
names inserted there? It seems you rather want

system("echo '#{someVar}' | ./some-script.py >'#{outputFile}'")

But this fails when there is a tick ` in #{someVar} as well.

Here's a pure Ruby solution:

File.open outputFile, "w" do |out|
IO.popen "./some-script.py", "rw" do |io|
t =3D Thread.new do
IO.copy_stream(io, out, 1024)
# pre 1.9:
# while ( b =3D io.read(1024) )
# out.write(b)
# end
end

io.puts(someVar)
io.close_write

t.join
end
end

Of course, it would be even better if you rewrote the Python script in Ruby=
;-)

Cheers

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

 

[Tanaka Akira]

`echo "#{someVar}" | ./some-script.py > "#{outputFile}"`

If you use Ruby 1.9.2, Open3.pipline() provides a way to run a pipline
without shell.

% ruby -ropen3 -e 'Open3.pipeline(["echo", "foo`bar`baz"], ["head",
"-6c", ut => "/tmp/qux"])'
% od -c /tmp/qux
0000000 f o o ` b a
0000006

 

[brabuhr]

Because of this, the syntax of:

`echo "#{someVar}" | ./some-script.py > "#{outputFile}"`

is breaking when #{someVar} contains the tick `. =A0This causes the
command to fail because it executes up to the tick ` in someVar and not
to the end as intended.

I have also tried using the system() command as follows:

system("echo \"#{someVar}\" | ./some-script.py > \"#{outputFile}\"")

But this fails when there is a tick ` in #{someVar} as well.

Could you show us a working example of the problem?
=3D> "foo `date` baz"
foo Thu Feb 10 07:18:23 EST 2011 baz
=3D> true
foo Thu Feb 10 07:18:32 EST 2011 baz
=3D> true
foo `date` baz
=3D> true
=3D> "foo `date` baz\n"

 

[Brian Candler]

Gerad S. wrote in post #980690:

Hi All,

I am trying to add functionality to an interesting script. The part
that is tripping me up is that I have to make a system call to echo, and
feed several parameters. One of the parameters occasionally includes
the ` character.

Because of this, the syntax of:

`echo "#{someVar}" | ./some-script.py > "#{outputFile}"`

is breaking when #{someVar} contains the tick `.

Does this test case represent your problem?
sh: Syntax error: EOF in backquote substitution
=> ""

Quick workaround: use single quotes instead (but see below).
=> "foo`bar\n"

The same occurs with system():
sh: Syntax error: EOF in backquote substitution
=> falsefoo`bar
=> true

It's an issue with the shell, not with ruby; the same happens at the
shell prompt too.

$ echo "foo`bar"

bash: unexpected EOF while looking for matching ``'

bash: syntax error: unexpected end of file

The issue is that in the shell, double-quoted strings can *contain*
backtick expansions, for example

$ echo "foo`hostname`bar"
foox100bar

Backticks inside single quotes are not treated specially. However,
single quotes inside single quotes are still a problem:
sh: Syntax error: Unterminated quoted string
=> ""

So using the Shellwords escaping mechanism, as already shown, is
probably what you want.

 

[Robert Klemme]

`echo "#{someVar}" | ./some-script.py > "#{outputFile}"`

If you use Ruby 1.9.2, Open3.pipline() provides a way to run a pipline
without shell.

% ruby -ropen3 -e 'Open3.pipeline(["echo", "foo`bar`baz"], ["head",
"-6c", ut =3D> "/tmp/qux"])'
% od -c /tmp/qux
0000000 =A0 f =A0 o =A0 o =A0 ` =A0 b =A0 a
0000006

Cool! Thanks for sharing.

Is there also a version which would return an IO or accept a block
with IO parameter so we can feed something into the head of the
pipeline and read from the tail?

Kind regards

robert

--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

 

[Gerad S.]

Robert Klemme wrote in post #980746:

Why do you escape the string interpolation? Don't you want those file
names inserted there? It seems you rather want

system("echo '#{someVar}' | ./some-script.py >'#{outputFile}'")

Hi Robert,

Thanks for the help. I used your suggestion of single quotes and now my
script is working like a charm.

Cheers,
Gerad

 

[Gerad S.]

Brian Candler wrote in post #980806:

Does this test case represent your problem?

sh: Syntax error: EOF in backquote substitution
=> ""

This indeed is a good representation of the problem.

I will have to research if my #{someVar} could possibly contain a '
character. Perhaps Shellwords would be a safer bet.


In case you are still interested, an example of the command with actual
data is below (notice how the sequence @@@@@@S` in the string before " |
collab-table.py" is causing the issue):

`echo "Facs Rep|21|@@@@@@Ir||FACS Version 2.1.36.0|2010-09-27
10:57:04.7108-07|21|2|Under Review|||||f||||||f|2010-09-27
11:14:42.686225-07|09/27/2010||||message|t|Facs
Rep|09/27/2010|closed|f||18| ||||
Facs Rep|21|@@@@@@Lz||FACS Version 3.1.0.0|2010-10-27
10:02:43.778138-07|21|2|Under Review|||||f||||||f|2010-10-27
10:51:17.216686-07|10/27/2010||||message|t|Facs
Rep|10/27/2010|closed|f||19| ||||
Facs Rep|21|@@@@@@N]||FACS Version 3.2.0.0|2010-11-09
16:56:17.214108-08|21|1|Under Review|||||f||||||f|2010-11-09
16:56:17.214108-08|11/09/2010||||message|t|Facs
Rep|11/09/2010|open|f||20| ||||
Facs Rep|21|@@@@@@QT||FACS Version 3.3.0.0|2010-12-09
10:19:06.322005-08|21|1|Under Review|||||f||||||f|2010-12-09
10:19:06.322005-08|12/09/2010||||message|t|Facs
Rep|12/09/2010|open|f||21| ||||
Facs Rep|21|@@@@@@S`||FACS Version 3.4.0.0|2011-01-07
10:17:25.142238-08|21|1|Under Review|||||f||||||f|2011-01-07
10:17:25.142238-08|01/07/2011||||message|t|Facs
Rep|01/07/2011|open|f||22| ||||" | ./collab-table.py "message" "D260888"
"hub" "ps_nysdot" "New York State Department of Transportation"
"Alexander Hamilton Bridge & Highbridge Interchange Ramps Rehab" "Facs
Rep" "21" "2011-02-10 11:58:12-0800" "20" "4" "4">
"cor_6/2011-02-10/D260888/correspondence/CollabmessageList_4.html"`

 

[Tanaka Akira]

Is there also a version which would return an IO or accept a block
with IO parameter so we can feed something into the head of the
pipeline and read from the tail?

% ruby -ropen3 -e '
Open3.pipeline_w(%w[head -6c], %w[cat -n]) {|w|
w.write "ab\ncd\nef\n"
}'
1 ab
2 cd

% ruby -ropen3 -e '
Open3.pipeline_r(["echo", "foo`bar`baz"], ["head","-6c"]) {|r|
p r.read
}'
"foo`ba"

 

[Robert Klemme]

Is there also a version which would return an IO or accept a block
with IO parameter so we can feed something into the head of the
pipeline and read from the tail?

% ruby -ropen3 -e '
Open3.pipeline_w(%w[head -6c], %w[cat -n]) {|w|
=A0w.write "ab\ncd\nef\n"
}'
=A0 =A0 1 =A0ab
=A0 =A0 2 =A0cd

% ruby -ropen3 -e '
Open3.pipeline_r(["echo", "foo`bar`baz"], ["head","-6c"]) {|r|
=A0p r.read
}'
"foo`ba"

Very nice! I tried

13:09:25 Temp$ ./pl.rb
< 1 000>
< 2 001>
< 3 002>
< 4 003>
< 5 004>
13:09:31 Temp$ cat pl.rb
#!/bin/env ruby19

require 'open3'

Open3.pipeline_rw(["cat", "-n"], ["sed", "-e", 's/^.*$/<&>/']) do
|sin, sout, threads|
th =3D Thread.new(sout) do |io|
io.each_line do |l|
puts l
end
end

5.times do |i|
sin.printf "%03d\n", i
end

sin.close
th.join
end
13:09:58 Temp$

Cheers

robert


--=20
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

 

[Avdi Grimm]

=A0 =A0 =A0IO.copy_stream(io, out, 1024)

I keep finding out about nifty 1.9 features like this piecemeal. Is
there an up-to-date guide to 1.8-1.9 library changes somewhere?