Sigh! I'm depressed. Debian vs Ruby and the backtick and subshells

J

John Carter

I have just strace'd weird bug to it's root cause.

For decades bash was the "sh" of choice for linux systems.

Then somebody threw some money at the Debian foundation to make Debian
boot faster.

The solution was to tweak the init scripts to be perfectly POSIX
Lowest Common Denominator "sh" compatible, and then make dash the
default "sh".

To allow for legacy scripts that expect /bin/sh === /bin/bash you can
sudo dpkg-reconfigure dash
and switch back to sh === bash

Thus on Debian (and Debian derivatives like Ubuntu) Rubies backtick
(iff there is a glob or a redirection present) is HARDCODED to /bin/sh
but will (depending on the users whim) give you dash or bash.

For example the &> redirection is a bashism...
ls -l /bin/sh
lrwxrwxrwx 1 root root 4 2008-08-12 16:54 /bin/sh -> dash
build@ws1419:~/AutomatedBuild/test$ ruby -e 'system("ls NOFILEWITHTHISNAME&>foo")'
ls: cannot access NOFILEWITHTHISNAME: No such file or directory
build@ws1419:~/AutomatedBuild/test$ cat foo
build@ws1419:~/AutomatedBuild/test$ sudo dpkg-reconfigure dash
Removing `diversion of /bin/sh to /bin/sh.distrib by dash'
Removing `diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash'
build@ws1419:~/AutomatedBuild/test$ ruby -e 'system("ls NOFILEWITHTHISNAME&>foo")'
build@ws1419:~/AutomatedBuild/test$ cat foo
ls: cannot access NOFILEWITHTHISNAME: No such file or directory

AAARGH!

As I always say... if a bug is hard to find... you won't know what to
do with it once you find it.



John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : (e-mail address removed)
New Zealand
 
R

Robert Klemme

I have just strace'd weird bug to it's root cause.

For decades bash was the "sh" of choice for linux systems.

Then somebody threw some money at the Debian foundation to make Debian
boot faster.

The solution was to tweak the init scripts to be perfectly POSIX
Lowest Common Denominator "sh" compatible, and then make dash the
default "sh".

To allow for legacy scripts that expect /bin/sh === /bin/bash you can
sudo dpkg-reconfigure dash
and switch back to sh === bash

Thus on Debian (and Debian derivatives like Ubuntu) Rubies backtick
(iff there is a glob or a redirection present) is HARDCODED to /bin/sh
but will (depending on the users whim) give you dash or bash.

For example the &> redirection is a bashism...
ls -l /bin/sh
lrwxrwxrwx 1 root root 4 2008-08-12 16:54 /bin/sh -> dash
build@ws1419:~/AutomatedBuild/test$ ruby -e 'system("ls NOFILEWITHTHISNAME&>foo")'
ls: cannot access NOFILEWITHTHISNAME: No such file or directory
build@ws1419:~/AutomatedBuild/test$ cat foo
build@ws1419:~/AutomatedBuild/test$ sudo dpkg-reconfigure dash
Removing `diversion of /bin/sh to /bin/sh.distrib by dash'
Removing `diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash'
build@ws1419:~/AutomatedBuild/test$ ruby -e 'system("ls NOFILEWITHTHISNAME&>foo")'
build@ws1419:~/AutomatedBuild/test$ cat foo
ls: cannot access NOFILEWITHTHISNAME: No such file or directory

AAARGH!

As I always say... if a bug is hard to find... you won't know what to
do with it once you find it.

You should probably mention that the bug is actually in _your_ code: if
you rely on bash features present in /bin/sh then that's the error.

:)

Kind regards

robert
 
M

Magicloud Magiclouds

I am using Debian sid, which sh is bash.
And, in fact, I hate ruby's way of system call. Ugly and hard to use and
stupid hardcoded.
 
R

Robert Klemme

I am using Debian sid, which sh is bash.
And, in fact, I hate ruby's way of system call. Ugly and hard to use and
stupid hardcoded.

/bin/sh is the least common denominator for all Unix like systems. So
in terms of portability it is a good choice to make this the default
shell. And if you do not like that shell, you can use another one
anytime:

09:06:03 ~$ ruby -e 'system "/bin/bash", "-c", "echo \"$
{BASH_VERSINFO[*]}\""'
3 2 39 19 release i686-pc-cygwin
09:06:06 ~$ ruby -e 'system ENV["SHELL"], "-c", "echo \"$
{BASH_VERSINFO[*]}\""'
3 2 39 19 release i686-pc-cygwin
09:06:07 ~$

Put that into a method and you can use your bash whenever you want.
No need to blame Ruby here when in fact it behaves reasonable.

Kind regards

robert
 
F

F. Senault

Le 12 août 2008 à 09:07, Robert Klemme a écrit :
Put that into a method and you can use your bash whenever you want.
No need to blame Ruby here when in fact it behaves reasonable.

Not that you can overload the method, too...

module Kernel
def `(v)
...
end
end

(Maybe it's a bit eeeeevil...)

Fred
 
P

Phlip

build@ws1419:~/AutomatedBuild/test$ ruby -e 'system("ls
Magicloud said:
I am using Debian sid, which sh is bash.
And, in fact, I hate ruby's way of system call. Ugly and hard to use and
stupid hardcoded.

The choice of `` is deliberate. It's to inspire you to get off your lazy butt
and learn the Ruby-internal way to do something, such as Pathname.glob. There's
always a way to avoid a shell command...
 
M

Michael Guterl

The choice of `` is deliberate. It's to inspire you to get off your lazy
butt and learn the Ruby-internal way to do something, such as Pathname.glob.
There's always a way to avoid a shell command...
This is a ridiculous claim. I'd spend more time writing a response,
but I'm hoping you're just being sarcastic.

Michael Guterl
 
M

M. Edward (Ed) Borasky

I have just strace'd weird bug to it's root cause.

For decades bash was the "sh" of choice for linux systems.

Then somebody threw some money at the Debian foundation to make Debian
boot faster.

The solution was to tweak the init scripts to be perfectly POSIX
Lowest Common Denominator "sh" compatible, and then make dash the
default "sh".

To allow for legacy scripts that expect /bin/sh === /bin/bash you can
sudo dpkg-reconfigure dash
and switch back to sh === bash

Thus on Debian (and Debian derivatives like Ubuntu) Rubies backtick
(iff there is a glob or a redirection present) is HARDCODED to /bin/sh
but will (depending on the users whim) give you dash or bash.

For example the &> redirection is a bashism...
ls -l /bin/sh
lrwxrwxrwx 1 root root 4 2008-08-12 16:54 /bin/sh -> dash
build@ws1419:~/AutomatedBuild/test$ ruby -e 'system("ls NOFILEWITHTHISNAME&>foo")'
ls: cannot access NOFILEWITHTHISNAME: No such file or directory
build@ws1419:~/AutomatedBuild/test$ cat foo
build@ws1419:~/AutomatedBuild/test$ sudo dpkg-reconfigure dash
Removing `diversion of /bin/sh to /bin/sh.distrib by dash'
Removing `diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash'
build@ws1419:~/AutomatedBuild/test$ ruby -e 'system("ls NOFILEWITHTHISNAME&>foo")'
build@ws1419:~/AutomatedBuild/test$ cat foo
ls: cannot access NOFILEWITHTHISNAME: No such file or directory

AAARGH!

As I always say... if a bug is hard to find... you won't know what to
do with it once you find it.

Why does Debian use dash? I've never even *heard* of dash until I saw
this thread! Is there some religious war against bash?
--
M. Edward (Ed) Borasky
ruby-perspectives.blogspot.com

"A mathematician is a machine for turning coffee into theorems." --
Alfréd Rényi via Paul Erdős
 
B

brabuhr

Why does Debian use dash? I've never even *heard* of dash until I saw
this thread! Is there some religious war against bash?

http://en.wikipedia.org/wiki/Debian_Almquist_shell
"Dash, like ash, executes scripts faster than bash and depends on
fewer libraries. It is believed to be more reliable in case of upgrade
problems or disk failures."

https://wiki.ubuntu.com/DashAsBinSh
"The major reason to switch the default shell was efficiency. bash is
an excellent full-featured shell appropriate for interactive use;
indeed, it is still the default login shell. However, it is rather
large and slow to start up and operate by comparison with dash. A
large number of shell instances are started as part of the Ubuntu boot
process. Rather than change each of them individually to run
explicitly under /bin/dash, a change which would require significant
ongoing maintenance and which would be liable to regress if not paid
close attention, the Ubuntu core development team felt that it was
best simply to change the default shell."
 
A

Avdi Grimm

Thus on Debian (and Debian derivatives like Ubuntu) Rubies backtick
(iff there is a glob or a redirection present) is HARDCODED to /bin/sh
but will (depending on the users whim) give you dash or bash.

For as long as I've been working with *nix systems the rule was
always: either *explicitly* invoke Bash, or understand and use only
the lowest common denominator bourne-compatible shell features. In
fact, the best practice is to only use features which are common to
both Bourne and CSH-compatible shells. It may come as a shock, but
there are actually more *NIXen than Linux, and many of them don't use
Bash as the default shell.

If you expect portability, you shouldn't be be doing anything that
depends on bash-specific features in your #system()/backtick calls.
If you need to invoke a specific shell, the popen() family of methods
are your friends.

--
Avdi

Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com
 
A

ara.t.howard

Thus on Debian (and Debian derivatives like Ubuntu) Rubies backtick
(iff there is a glob or a redirection present) is HARDCODED to /bin/sh
but will (depending on the users whim) give you dash or bash.


i could easily be wrong, but i have a dim recollection that the
location of sh is a POSIX standard itself.

backticks seem to respect the SHELL var here

cfp:~ > SHELL=tcsh ruby -e' puts `echo $SHELL 2>out` '
tcsh


a @ http://codeforpeople.com/
 
A

Avdi Grimm

I am using Debian sid, which sh is bash.
The choice of `` is deliberate. It's to inspire you to get off your lazy
butt and learn the Ruby-internal way to do something, such as Pathname.glob.
There's always a way to avoid a shell command...

The backtick syntax is a convenience inherited from Bash and Perl. It
makes one very common simple case of shell interaction easy and
concise. I typically only use backticks in quick one-off scripts. If
I need to interact with the shell in a larger program I avoid it and
use either system() (for clarity) or popen*() (for greater control).

--
Avdi

Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com
 
M

Martin DeMello

T24gVHVlLCBBdWcgMTIsIDIwMDggYXQgMTozMyBQTSwgSm9obiBDYXJ0ZXIgPGpvaG4uY2FydGVy
QHRhaXQuY28ubno+IHdyb3RlOgo+IE9uIFR1ZSwgMTIgQXVnIDIwMDgsIE0uIEVkd2FyZCAoRWQp
IEJvcmFza3kgd3JvdGU6Cj4KPj4gIkEgbWF0aGVtYXRpY2lhbiBpcyBhIG1hY2hpbmUgZm9yIHR1
cm5pbmcgY29mZmVlIGludG8gdGhlb3JlbXMuIiAtLQo+PiBBbGZy6WQgUulueWkgdmlhIFBhdWwg
RXJk9XMKPgo+IEkgdHVybiBjb2ZmZWUgaW50byBSdWJ5LgoKSG93IGlzIHJ1YmJ5IGZvcm1lZD8g
SG93IGNvZGUgZ2V0IHByb3JnYW0/CgptYXJ0aW4K
 
J

John Carter

The choice of `` is deliberate. It's to inspire you to get off your lazy butt
and learn the Ruby-internal way to do something, such as Pathname.glob.
There's always a way to avoid a shell command...

There are two reasons I use ``.

1) Quick and easy. Much quicker to type ` ` than system " "

2) When I want the output of a program as a string, much easier to type
`blah` than IO.read("|blah")

Whilst shell globbing has some shorthandedness about it, the main
reason is are file redirects. > &> >2&1

I'm quite happy to override `` with that trick someone mentioned above
if it grants me deterministic behaviour between client boxes. (I don't
care if its the wrong behaviour, so long as it's the _same_
behaviour!)

I often use Dir["bla*"] and glob with glee.

Anyone have a short and easy Ruby way of doing file redirects?




John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : (e-mail address removed)
New Zealand
 
J

John Carter

i could easily be wrong, but i have a dim recollection that the location of
sh is a POSIX standard itself.

backticks seem to respect the SHELL var here

cfp:~ > SHELL=tcsh ruby -e' puts `echo $SHELL 2>out` '
tcsh

At risk of being a little rude to one who doesn't deserve it, I
couldn't resist (purely for humorous reasons) framing my illustration
of the problem with that line like so....

I_AM_EASILY_WRONG='yes you are :)' ruby -e' puts `echo $I_AM_EASILY_WRONG 2>out`'
yes you are :)


The location at /bin/sh is standard and fixed. SHELL is what you set
that users default shell to in your /etc/password.

In unix Ruby backtick ignores ENV["SHELL"] and just runs whatever is at /bin/sh


John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : (e-mail address removed)
New Zealand
 
A

ara.t.howard

i could easily be wrong, but i have a dim recollection that the
location of sh is a POSIX standard itself.

backticks seem to respect the SHELL var here

cfp:~ > SHELL=tcsh ruby -e' puts `echo $SHELL 2>out` '
tcsh

At risk of being a little rude to one who doesn't deserve it, I
couldn't resist (purely for humorous reasons) framing my illustration
of the problem with that line like so....

I_AM_EASILY_WRONG='yes you are :)' ruby -e' puts `echo
$I_AM_EASILY_WRONG 2>out`'
yes you are :)


The location at /bin/sh is standard and fixed. SHELL is what you set
that users default shell to in your /etc/password.

In unix Ruby backtick ignores ENV["SHELL"] and just runs whatever is
at /bin/sh


John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : (e-mail address removed)
New Zealand


heh - right you are!

a @ http://codeforpeople.com/
 
M

Marc Heiler

The location at /bin/sh is standard and fixed.

Could you point out where it is defined? I could understand if the FHS
wants to impose such locations, but someone claimed that POSIX in itself
makes that claim.

I would be delighted to hear more, because as of today I only thought
the only thing that imposes something is FHS (and glibc which relies on
a hardcoded path. For more info about this read NixOS design choices, i
think /bin/bash or something else is hardcoded because the glibc guys
mandated that)
 
A

ara.t.howard

Could you point out where it is defined? I could understand if the FHS
wants to impose such locations, but someone claimed that POSIX in =20
itself
makes that claim.

[ahoward@localhost ~]$ PAGER=3Dcat man system
SYSTEM=20
(3) =20
Linux Programmer=92s =20
Manual=20
=20
SYSTEM(3)



NAME
system - execute a shell command

SYNOPSIS
#include <stdlib.h>

int system(const char *string);

DESCRIPTION
system() executes a command specified in string by =20
calling /bin/sh -c string, and returns after the command has been =20
completed. During execution of the command,
SIGCHLD will be blocked, and SIGINT and SIGQUIT will be ignored.

RETURN VALUE
The value returned is -1 on error (e.g. fork failed), and the =20=

return status of the command otherwise. This latter return status is =20=

in the format specified in wait(2).
Thus, the exit code of the command will be =20
WEXITSTATUS(status). In case /bin/sh could not be executed, the exit =20=

status will be that of a command that does exit(127).

If the value of string is NULL, system() returns nonzero if =20
the shell is available, and zero if not.

system() does not affect the wait status of any other children.

CONFORMING TO
ANSI C, POSIX.2, BSD 4.3

NOTES
As mentioned, system() ignores SIGINT and SIGQUIT. This may =20=

make programs that call it from a loop uninterruptable, unless they =20
take care themselves to check the exit
status of the child. E.g.

while(something) {
int ret =3D system("foo");

if (WIFSIGNALED(ret) &&
(WTERMSIG(ret) =3D=3D SIGINT || WTERMSIG(ret) =3D=3D =
=20
SIGQUIT))
break;
}

Do not use system() from a program with suid or sgid =20
privileges, because strange values for some environment variables =20
might be used to subvert system integrity. Use
the exec(3) family of functions instead, but not execlp(3) =20
or execvp(3). system() will not, in fact, work properly from programs =20=

with suid or sgid privileges on sys-
tems on which /bin/sh is bash version 2, since bash 2 drops =20
privileges on startup. (Debian uses a modified bash which does not do =20=

this when invoked as sh.)

The check for the availability of /bin/sh is not actually =20
performed; it is always assumed to be available. ISO C specifies the =20=

check, but POSIX.2 specifies that the
return shall always be non-zero, since a system without the =20
shell is not conforming, and it is this that is implemented.

It is possible for the shell command to return 127, so that =20
code is not a sure indication that the execve() call failed.

SEE ALSO
sh(1), signal(2), wait(2), exec(3)



http://en.wikipedia.org/wiki/System_(C_Standard_Library)



a @ http://codeforpeople.com/
 
R

Robert Klemme

Anyone have a short and easy Ruby way of doing file redirects?

Well, define it once and include it in all scripts where you need it.
For example:

robert@fussel /cygdrive/c/Temp
$ ./redir.rb
Done #<Process::Status: pid=5384,exited(0)>
Done #<Process::Status: pid=5464,exited(2)>

robert@fussel /cygdrive/c/Temp
$ cat output
drwx------+ 5 robert Kein 0 Aug 13 08:42 .

robert@fussel /cygdrive/c/Temp
$ cat output2
ls: cannot access does_not_exis: No such file or directory
drwx------+ 5 robert Kein 0 Aug 13 08:42 .

robert@fussel /cygdrive/c/Temp
$ cat redir.rb
#!/bin/env ruby

module Kernel
private
def system_redir(args, fout = ::STDOUT, ferr = ::STDERR, fin = ::STDIN)
child = fork do
redir_fd ::STDIN, fin, "r"
redir_fd ::STDOUT, fout, "w"
redir_fd ::STDERR, ferr, "w"
exec *args
end
Process.wait2(child).last
end

def redir_fd(io, target, mode = nil)
case target
when String
io.reopen(File.open(target, mode))
when IO
io.reopen target
when io
# do nothing
when nil
io.close
else
raise ArgumentError, "Invalid target argument #{target.inspect}"
end
end
end

res = system_redir %w{ls -ld}, "output"
printf "Done %p\n", res

File.open "output2", "w" do |io|
res = system_redir %w{ls -ld . does_not_exis}, io, io
printf "Done %p\n", res
end

robert@fussel /cygdrive/c/Temp


Kind regards

robert
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,906
Latest member
SkinfixSkintag

Latest Threads

Top