zombie invasion - six ways to invoke shell processes

[Giles Bowkett]

I've got code somebody else wrote. The code uses exec() to invoke
shell processes. It spawns zombies by the thousands.

I need to redo this over the long term to stop using shell processes at all.

I need it to stop spawning zombies immediately.

This blog post gives six ways to invoke a shell process:

http://pasadenarb.com/2007/03/ruby-shell-commands.html

How do I choose which of these six methods to use?

All I remember off the top of my head is that backticks is the most
terse and system() is the one I use the most. I had forgotten exec()
even existed. "The Ruby Way" says that exec() doesn't use a subshell
and system() does. I think that's *why* I forgot exec() existed - I
think I blacklisted the method from the moment I discovered it.

I'm switching the legacy code to use system(). Way too much
superstition and guessing in this process for my taste though. How do
you choose the best (least zombie-prone) method to invoke a shell
process?

 

[Giles Bowkett]

It sounds like your use cases are rather simple and really only need

to be focusing on `` and system().
In this case use `` when you need to capture the output and system()
otherwise.

Of course I'm assuming that your need is simple here.

The need is simple, I think. But the only difference between using
exec() and system() in this case is that the zombies now have a
different listing for CMD in ps.

On examining the code it looks as if these external processes require
something to be configured which is not configured. So it appears that
every one of these zombies is failing to even do anything.

 

[Gregory Brown]

All I remember off the top of my head is that backticks is the most
terse and system() is the one I use the most. I had forgotten exec()
even existed. "The Ruby Way" says that exec() doesn't use a subshell
and system() does. I think that's *why* I forgot exec() existed - I
think I blacklisted the method from the moment I discovered it.

I wonder if exec() might be useful in forking. Just playing around:

fork do
exec 'ls'
end
puts "I can has ls?"
Process.wait

 

[Daniel DeLorme]

I've got code somebody else wrote. The code uses exec() to invoke
shell processes. It spawns zombies by the thousands.

I need to redo this over the long term to stop using shell processes at
all.

why? it's one of ruby's strengths that it can act as glue between
external components rather than having to code everything yourself.

I need it to stop spawning zombies immediately.

Look for the cause. Zombie processes are child processes whose return
response has not yet been collected by the parent. Since you say that
the code uses exec() extensively (when normally it can only be used once
and then the program exits), it leads me to think you have something
like this:
Process.fork{ exec(cmd) }
which of course would spawn a lot of zombies. You need to detach the
child process from the parent:
pid = Process.fork{ exec(cmd) }
Process.detach(pid)

Daniel

 

[Giles Bowkett]

On examining the code it looks as if these external processes require

In that case, it sounds like you could just comment out the code that's
trying to call the external processes, and you would immediately eliminate
the zombies without losing any functionality that you have at the
moment. I'd probably start there.

In fact, that's what I did. The code didn't do anything, so I deleted it.

Next figure out whether you really need those external processes to
run. If not then you're done; maybe the reason to run them has gone away
since this was first set up. If you do need them, then configure them so
they run properly in 'standalone' mode, then uncomment the original code and
chances are you won't have the same zombie problem anymore.

The external processes actually called the app itself, via curl.

The external processes are now considered evil in my book. Anything
which spawns zombies is evil; anything which calls itself recursively
via HTTP is also evil. Therefore, evil.

QED.

(Actually, one day we'll probably have apps that call themselves via
HTTP, and it'll be higher-order HTTP, and that'll be great, but for
now, evil.)

As for exec() vs. system(), the biggest difference is whether you want any
code after that call to get executed. Since exec replaces the current
process with the new one, any code after the exec() call will never get run.
If it's the last line of the program anyway, then this won't matter to you.

Well, it wasn't the last line of code, but it wasn't really being
called properly. Ordinarily I use system() rather than exec(), on the
assumption that in most cases I want the program to continue its end.

 

[Giles Bowkett]

I've got code somebody else wrote. The code uses exec() to invoke

why? it's one of ruby's strengths that it can act as glue between
external components rather than having to code everything yourself.

Yeah, but given the userbase etc. it should be using BackgrounDRb
rather than external shell processes, if it needs external processes.
In fact, it didn't have such a need; the shell process was calling the
app itself via HTTP over curl, and the code was already within the
same app.

Yay rapid prototyping.