Slave not removing slave_proc_* files

J

Jos Backus

Hi. In order to be able to run swiftiply_mongrel_rails under daemontools I
patched swiftiply_mongrel_rails to use Ara's Slave library. The patch does
essentially this:

slaves = []
3.times do |i|
require 'slave'
require "#{File.dirname(__FILE__)}/mongrel_rails"
slaves << Slave.object:)async => true) {
Mongrel::Runner.new.run(args) # args == similar to mongrel_rails
# command line args
}
end
slaves.each {|t| t.join}

(See http://rubyforge.org/pipermail/swiftiply-users/2007-August/000054.html
for the actual patch).

Note that Mongrel::Runner.new.run never returns, hence the use of the :async
option. It is just a wrapper around mongrel_rails.

When a SIGTERM is sent to the swiftiply_mongrel_rails process, the following
output is seen:

http://pastie.caboo.se/88665

This shows swiftiply_mongrel_rails as well as its slaves exiting, and
daemontools subsequently restarting swiftiply_mongrel_rails.

Now for the problem: after this, /tmp holds 3 slave_proc_* UNIX domain sockets
that have not been unlinked. Subsequent restarts yield more slave_proc_*
sockets, and so on.

So my question is: what could cause these sockets not to be removed? am I
using Slave incorrectly? I instrumented slave.rb to make sure the
Kernel.at_exit method is called and it is - it's just that the associated
block isn't executed in the SIGTERM case. It works fine when a SIGUSR2 signal
is sent - the sockets are cleaned up as expected. A simple test script
suggests that Kernel.at_exit works okay on the platform (ruby 1.8.5 from the
CentOS testing repo on CentOS 4).

Any help is appreciated...
 
A

ara.t.howard

Hi. In order to be able to run swiftiply_mongrel_rails under
daemontools I
patched swiftiply_mongrel_rails to use Ara's Slave library. The
patch does
essentially this:

slaves = []
3.times do |i|
require 'slave'
require "#{File.dirname(__FILE__)}/mongrel_rails"
slaves << Slave.object:)async => true) {
Mongrel::Runner.new.run(args) # args == similar to mongrel_rails
# command line args
}
end
slaves.each {|t| t.join}

what is this code supposed to do exactly? slave.rb puts an object in
another process which cannot outlive it's parent, but this object is
meant to be used: a handle to it is expected. iff
Mongrel::Runner.new.run never returns then all those slaves are
essentially half baked: they will not have a reference to their
objects - in particular the lifelines (sockets) have not been setup
completely. so essentially all this code is close to simply

fork { Mongrel .... }

and never setting up collection of the child. however i'm not 100%
clear on how mongrel implements run nor what you are really trying to
do here?
(See http://rubyforge.org/pipermail/swiftiply-users/2007-August/
000054.html
for the actual patch).

Note that Mongrel::Runner.new.run never returns, hence the use of
the :async
option. It is just a wrapper around mongrel_rails.

all async does is wrap your code with a thread.
When a SIGTERM is sent to the swiftiply_mongrel_rails process, the
following
output is seen:


yeah - that makes sense: slave.rb is trying to exit because the
parent has died - the question is why SystemExit is being rescued.
perhaps a blanket 'resuce Exception' in swiftiply?

f
This shows swiftiply_mongrel_rails as well as its slaves exiting, and
daemontools subsequently restarting swiftiply_mongrel_rails.

Now for the problem: after this, /tmp holds 3 slave_proc_* UNIX
domain sockets
that have not been unlinked. Subsequent restarts yield more
slave_proc_*
sockets, and so on.

So my question is: what could cause these sockets not to be
removed? am I
using Slave incorrectly? I instrumented slave.rb to make sure the
Kernel.at_exit method is called and it is - it's just that the
associated
block isn't executed in the SIGTERM case. It works fine when a
SIGUSR2 signal
is sent - the sockets are cleaned up as expected. A simple test script
suggests that Kernel.at_exit works okay on the platform (ruby 1.8.5
from the
CentOS testing repo on CentOS 4).

Any help is appreciated...

for SIGTERM the socket should be cleaned up by the parent process -
however it seems like the normal exit chain is being interupted by
swiftiply since SystemExit is being rescued. can you find/show the
code that is rescuing the SystemExit call and see what's happening
there: are the normal exit handlers getting called in the case of
SystemExit being thrown? in otherwords the code should look vaguely
like this

cfp:~ > cat a.rb
begin
exit 42
rescue Exception => e
p e.class
p e.status
exit e.status if SystemExit === e
p :somethig_else
end


cfp:~ > ruby a.rb
SystemExit
42

regards.


a @ http://drawohara.com/
 
J

Jos Backus

Hi Ara,

Hi. In order to be able to run swiftiply_mongrel_rails under daemontools I
patched swiftiply_mongrel_rails to use Ara's Slave library. The patch does
essentially this:

slaves = []
3.times do |i|
require 'slave'
require "#{File.dirname(__FILE__)}/mongrel_rails"
slaves << Slave.object:)async => true) {
Mongrel::Runner.new.run(args) # args == similar to mongrel_rails
# command line args
}
end
slaves.each {|t| t.join}

what is this code supposed to do exactly?

Mongrel::Runner.new.run(args) will start running relative to the parent,
meaning that (since we are talking about the swiftiplied Mongrel here) it will
connect to the swiftiply proxy and handle any requests sent by it. The parent
isn't interested in making any method calls to the slaves; the slaves are
meant to run forever.
slave.rb puts an object in
another process which cannot outlive it's parent, but this object is meant
to be used: a handle to it is expected. iff Mongrel::Runner.new.run never
returns then all those slaves are essentially half baked: they will not
have a reference to their objects - in particular the lifelines (sockets)
have not been setup completely. so essentially all this code is close to
simply

fork { Mongrel .... }

and never setting up collection of the child. however i'm not 100% clear
on how mongrel implements run nor what you are really trying to do here?

So my guess that I am using Slave wrong appears correct. The above explains
why the sockets remain. Since Slave.object returns a thread I figured I could
just join them to block the parent.

My idea is to have a simple way to host a bunch of Rails apps/multiple
instances of the same Rails app. So I am trying to start N mongrel_rails
processes managed by swiftiply_mongrel_rails which in turn will be managed by
daemontools. To accomplish this, I added the change in the post below
mentioned to mongrel_rails.rb so that rather than using backticks I can just
call this Mongrel::Runner.run method (maybe that is the wrong approach and I
should just use backticks/Kernel.system()).

Slave seemed like an easy way to manage the children and avoid zombies. Is
there a correct way of doing this with Slave? Or is it the wrong tool for the
job? In that case I'll use fork/exec, Process.waitall and a SIGTERM handler in
the parent which kills the children, etc. to get the job done.
all async does is wrap your code with a thread.




yeah - that makes sense: slave.rb is trying to exit because the parent has
died - the question is why SystemExit is being rescued. perhaps a blanket
'resuce Exception' in swiftiply?

Honestly, I don't know. Maybe Kirk can comment. The only relevant `rescue
Exception' in swiftiply (in src/swiftcore/{evented,swiftiplied,}_mongrel.rb)
doesn't seem involved (to check, I added a warn which isn't seen upon sending
SIGTERM).
for SIGTERM the socket should be cleaned up by the parent process - however
it seems like the normal exit chain is being interupted by swiftiply since
SystemExit is being rescued. can you find/show the code that is rescuing
the SystemExit call and see what's happening there: are the normal exit
handlers getting called in the case of SystemExit being thrown? in
otherwords the code should look vaguely like this

cfp:~ > cat a.rb
begin
exit 42
rescue Exception => e
p e.class
p e.status
exit e.status if SystemExit === e
p :somethig_else
end


cfp:~ > ruby a.rb
SystemExit
42

There are no references to SystemExit in swiftiply-0.6.1:

# pwd
/usr/lib/ruby/gems/1.8/gems/swiftiply-0.6.1
# grep -r SystemExit .
#

Thanks for your help, Ara.
 
A

ara.t.howard


how do.
Mongrel::Runner.new.run(args) will start running relative to the
parent,
meaning that (since we are talking about the swiftiplied Mongrel
here) it will
connect to the swiftiply proxy and handle any requests sent by it.
The parent
isn't interested in making any method calls to the slaves; the
slaves are
meant to run forever.

does mongrel fork to accomplish that? if not why not simply

threads << Thread.new{ Mongrel::Runner.new.run(args) }
So my guess that I am using Slave wrong appears correct. The above
explains
why the sockets remain. Since Slave.object returns a thread I
figured I could
just join them to block the parent.

you could *if* mongrel returned an object. the fact that i never
returns leaves slave half baked...

My idea is to have a simple way to host a bunch of Rails apps/
multiple
instances of the same Rails app. So I am trying to start N
mongrel_rails
processes managed by swiftiply_mongrel_rails which in turn will be
managed by
daemontools. To accomplish this, I added the change in the post below
mentioned to mongrel_rails.rb so that rather than using backticks I
can just
call this Mongrel::Runner.run method (maybe that is the wrong
approach and I
should just use backticks/Kernel.system()).

Slave seemed like an easy way to manage the children and avoid
zombies. Is
there a correct way of doing this with Slave? Or is it the wrong
tool for the
job? In that case I'll use fork/exec, Process.waitall and a SIGTERM
handler in
the parent which kills the children, etc. to get the job done.

i think the LifeLine class of slave is exactly what you need and that
you can re-use it. basically the idea is this

socket = Socket.pair


cid = fork

if cid
stuff
else
Thread.new{ socket.read rescue exit }
stuff
end

in summary, the child reads from a pipe to the parent. if the read
ever returns the parent has died - so exit. this is the 'zombie'
preventer of slave.rb. if you look at slave.rb and grep for LifeLine
and @lifeline you'll see the usage. summary is:

lifeline = LifeLine.new

cid = fork

unless cid # child
@lifeline.catch
@lifeline.cling
else
@lifeline.throw
end


and the poor man's version:



cfp:~ > cat a.rb
r, w = IO.pipe
cid = fork

unless cid ### child
w.close

Thread.new{
begin
r.read
ensure
STDERR.puts 'parent died... exiting!'
Kernel.exit
end
}

sleep and 'pretend some processing is going on...'
else
r.close
sleep 2 and 'pretend the parent exited'
end



cfp:~ > ruby a.rb
parent died... exiting!

There are no references to SystemExit in swiftiply-0.6.1:

# pwd
/usr/lib/ruby/gems/1.8/gems/swiftiply-0.6.1
# grep -r SystemExit .
#

right. but there *should* be *if* blanket Exceptions are rescued.
Thanks for your help, Ara.

always good to have testers. er, i mean users ;-)

a @ http://drawohara.com/
 
J

Jos Backus

On Aug 18, 2007, at 12:54 PM, Jos Backus wrote: [snip]
does mongrel fork to accomplish that? if not why not simply

threads << Thread.new{ Mongrel::Runner.new.run(args) }

AfaIk, Mongrel doesn't fork so this won't work. (Hey, that rhymes!)
you could *if* mongrel returned an object. the fact that i never returns
leaves slave half baked...

Yeah, I see that now. Otoh, if it returned it would mean that it would have
stopped running, which is, uh, undesirable. ;-)
i think the LifeLine class of slave is exactly what you need and that you
can re-use it. basically the idea is this

socket = Socket.pair


cid = fork

if cid
stuff
else
Thread.new{ socket.read rescue exit }
stuff
end

in summary, the child reads from a pipe to the parent. if the read ever
returns the parent has died - so exit. this is the 'zombie' preventer of
slave.rb. if you look at slave.rb and grep for LifeLine and @lifeline
you'll see the usage. summary is:

lifeline = LifeLine.new

cid = fork

unless cid # child
@lifeline.catch
@lifeline.cling
else
@lifeline.throw
end

I tried this and couldn't get it to work. For some reason the Mongrel children
don't progress to connect to the swiftiply proxy. Not sure why.
and the poor man's version:



cfp:~ > cat a.rb
r, w = IO.pipe
cid = fork

unless cid ### child
w.close

Thread.new{
begin
r.read
ensure
STDERR.puts 'parent died... exiting!'
Kernel.exit
end
}

sleep and 'pretend some processing is going on...'
else
r.close
sleep 2 and 'pretend the parent exited'
end



cfp:~ > ruby a.rb
parent died... exiting!

The poor man's version appears to work great. Thanks! I'll post a new patch to
the swiftiply list.
right. but there *should* be *if* blanket Exceptions are rescued.


always good to have testers. er, i mean users ;-)

Surely one day soon there'll be an opportunity for me to use Slave, but this
doesn't seem to be the right fit, contrary to my earlier impression.

Thanks again for the neat blocking-pipe-read-in-thread trick, Ara. I can see
myself using it frequently in the future.

Cheers,
 
J

Jos Backus

no that's good - if it doesn't fork this might work great - if it did it
could not. might want to give it whirl...

I'm not sure I understand. Are you suggesting that swiftiply_mongrel_rails run
all these Mongrels (who themselves use threads) in threads instead of forking?

If Mongrel doesn't fork, how does this give me multiple independent Mongrel
instances? If this Mongrel executes a blocking db call, won't this block the
whole process and prevent other threads from executing? I thought the idea was
to fork multiple Mongrels so that each can execute blocking calls without
affecting other requests being served by other Mongrels.

Can you elaborate please?

Cheers,
 
A

ara.t.howard

I'm not sure I understand. Are you suggesting that
swiftiply_mongrel_rails run
all these Mongrels (who themselves use threads) in threads instead
of forking?

If Mongrel doesn't fork, how does this give me multiple independent
Mongrel
instances? If this Mongrel executes a blocking db call, won't this
block the
whole process and prevent other threads from executing? I thought
the idea was
to fork multiple Mongrels so that each can execute blocking calls
without
affecting other requests being served by other Mongrels.

Can you elaborate please?

i was just wondering if it might work - i've been suprised both by
the the things that work, and also the things that do not work,
within ruby threads - i'd guess that it would not work but was
brainstorming out loud.

back to your regularly scheduled programming...

;-)

a @ http://drawohara.com/
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads


Members online

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top