Python signal delivery under BSD 4.4

Discussion in 'Python' started by Elf M. Sternberg, Aug 12, 2003.

  1. I'm having a devil of a time with signal handling under BSD 4.4 with
    Python and I was hoping that this might ring a bell with someone. Using
    the Webware Application Server (a multi-threaded Python application), I
    fork off a process to run independently and do some heavy-duty multi-
    threaded I/O churning. The process is launched with os.spawnvp().

    The user may, at any time, elect to terminate this process
    before it reaches its end. I do that by having the Application Server
    send a SIGUSR1 signal to the process ID returned by os.spawnvp(), which
    so far has been reliably returned and stored in the appserver's session
    mananger. So far, so good.

    This worked well under Python 2.1, where I used os.fork() instead
    of os.spawnvp().

    We have upgraded to Python 2.2.3 and it has mysteriously stopped
    working. Processes launched off the appserver, whether using os.fork()
    or os.spawnvp(), are now completely deaf to signals. Nothing gets its
    attention except, of course, SIGKILL. Not when sent from the appserver,
    and not when sent from the shell, not even by root.

    If the process is launched from the command line, however, it
    is well-behaved and responds to signal reliably.

    What gives?
     
    Elf M. Sternberg, Aug 12, 2003
    #1
    1. Advertising

  2. Elf M. Sternberg

    Donn Cave Guest

    In article <>,
    "Elf M. Sternberg" <> wrote:
    > I'm having a devil of a time with signal handling under BSD 4.4 with
    > Python and I was hoping that this might ring a bell with someone. Using
    > the Webware Application Server (a multi-threaded Python application), I
    > fork off a process to run independently and do some heavy-duty multi-
    > threaded I/O churning. The process is launched with os.spawnvp().
    >
    > The user may, at any time, elect to terminate this process
    > before it reaches its end. I do that by having the Application Server
    > send a SIGUSR1 signal to the process ID returned by os.spawnvp(), which
    > so far has been reliably returned and stored in the appserver's session
    > mananger. So far, so good.
    >
    > This worked well under Python 2.1, where I used os.fork() instead
    > of os.spawnvp().
    >
    > We have upgraded to Python 2.2.3 and it has mysteriously stopped
    > working. Processes launched off the appserver, whether using os.fork()
    > or os.spawnvp(), are now completely deaf to signals. Nothing gets its
    > attention except, of course, SIGKILL. Not when sent from the appserver,
    > and not when sent from the shell, not even by root.
    >
    > If the process is launched from the command line, however, it
    > is well-behaved and responds to signal reliably.
    >
    > What gives?


    Don't know. Can you duplicate it in a simpler environment?
    I'd like to run the following C program from spawnv, from a
    Python program like

    import os
    import time
    import signal

    p = os.spawnv(os.P_NOWAIT, 'prsig', ['prsig'])
    time.sleep(1.0)
    os.kill(p, signal.SIGTERM)
    print 'process exit', os.waitpid(p, 0)

    if you think that represents what you're doing. This seems
    to work as expected with Python 2.2 on a BSD platform (if
    your "BSD 4.4" is actually NetBSD 1.6 or FreeBSD 5.1, I could
    try that.)

    The reasons I can think of in principle are
    1. handler set to SIG_IGN (if the child process is supposed
    to exit due to the default signal handler.)
    2. blocked via signal mask
    3. wrong process

    Donn Cave, /

    include <signal.h>
    #include <stdio.h>
    #include <string.h>

    static void
    handler(int sig)
    {
    switch (sig) {
    case SIGALRM:
    fprintf(stderr, "caught SIGALRM\n");
    break;
    case SIGUSR1:
    fprintf(stderr, "caught SIGUSR1\n");
    break;
    default:
    fprintf(stderr, "caught %d\n", sig);
    }
    exit(0);
    }

    static void
    prsig(int sig)
    {
    struct sigaction sa;
    int status;
    memset(&sa, 0, sizeof(sa));
    status = sigaction(sig, 0, &sa);
    if (status == 0) {
    printf("sigaction {\n");
    printf(" sa_handler 0x%08x,\n", sa.sa_handler);
    printf(" sa_flags 0x%08x\n", sa.sa_flags);
    /* printf(" sa_mask 0x%08x\n", sa.sa_mask); */
    printf("}\n");
    } else {
    perror("sigaction query");
    exit(2);
    }
    }

    static void
    prmask()
    {
    sigset_t mask;
    int status;
    status = sigprocmask(SIG_SETMASK, 0, &mask);
    if (status == 0)
    fprintf(stdout, "mask 0x%lx\n", mask);
    else {
    perror("sigprocmask");
    exit(2);
    }
    }

    static void
    sethandler(int sig, void (*hf)(int))
    {
    struct sigaction sa;
    int status;
    memset(&sa, 0, sizeof(sa));
    sa.sa_handler = hf;
    status = sigaction(sig, &sa, 0);
    if (status == 0) {
    } else {
    perror("sigaction");
    exit(2);
    }
    }

    static void
    pause()
    {
    char buffer[4];
    read(0, buffer, 4);
    }

    int
    main(int argc, char **argv)
    {
    prsig(SIGUSR1);
    prmask();
    sethandler(SIGUSR1, handler);
    sethandler(SIGALRM, handler);
    alarm(3);
    pause();
    return 0;
    }
     
    Donn Cave, Aug 13, 2003
    #2
    1. Advertising

  3. Donn Cave <> writes:

    > In article <>,
    > "Elf M. Sternberg" <> wrote:


    >> This worked well under Python 2.1, where I used os.fork() instead
    >> of os.spawnvp().


    >> We have upgraded to Python 2.2.3 and it has mysteriously stopped
    >> working. Processes launched off the appserver, whether using os.fork()
    >> or os.spawnvp(), are now completely deaf to signals. Nothing gets its
    >> attention except, of course, SIGKILL. Not when sent from the appserver,
    >> and not when sent from the shell, not even by root.


    I'm sorry about the confusion. FreeBSD 4.4 is what we're
    running where I work.

    > Can you duplicate it in a simpler environment?
    > I'd like to run the following C program from spawnv, from a
    > Python program like


    > import os
    > import time
    > import signal


    > p = os.spawnv(os.P_NOWAIT, 'prsig', ['prsig'])
    > time.sleep(1.0)
    > os.kill(p, signal.SIGTERM)
    > print 'process exit', os.waitpid(p, 0)


    I'm going to assume you meant 'SIGUSR1' here since that's what
    you're catching in the C code you included.

    In any event, your code worked as expected.

    However, this variant worked as I've reported:

    import os
    import sys
    import time
    import signal
    import thread


    def do_it():
    p = os.spawnv(os.P_NOWAIT, 'prsig', ['prsig'])
    print >>sys.stderr, "Sleeping"
    time.sleep(1.0)
    os.kill(p, signal.SIGUSR1)
    print 'process exit', os.waitpid(p, 0)
    sys.exit()

    thread.start_new_thread(do_it, ())
    time.sleep(4.0)

    Not only did it behave as I reported, but it slept the full four
    seconds even though the sys.exit() call was made. This leads me to the
    conclusion that something rather unusual is going on within Python's
    threading module and the libc_r.a within FreeBSD4.4.

    Right now, I just want a way to make signal delivery work.

    Thanks for your help so far, though.

    Elf
     
    Elf M. Sternberg, Aug 13, 2003
    #3
  4. Elf M. Sternberg

    Donn Cave Guest

    In article <>,
    "Elf M. Sternberg" <> wrote:
    ....
    > I'm going to assume you meant 'SIGUSR1' here since that's what
    > you're catching in the C code you included.


    Right, sorry.

    > However, this variant worked as I've reported:

    [... forks from thread ]

    I tried it from FreeBSD 5.1, using 2.2.2 built from ports,
    and didn't see this problem. I believe 5.1 has a significantly
    different thread implementation from 4.x, though.

    Note that there is pthread_sigmask(3) with the same arguments as
    sigprocmask(2), and Python's thread_pthread.h does appear to block
    all signals that way when starting a new thread. That's a clue,
    perhaps. If sigprocmask() didn't show any signals blocked, maybe
    pthread_sigmask() would, or maybe at any rate you can set the
    mask to 0 using one or the other of those functions.

    Donn Cave,
     
    Donn Cave, Aug 13, 2003
    #4
  5. On Wed, 13 Aug 2003, Donn Cave wrote:

    > In article <>,
    > "Elf M. Sternberg" <> wrote:
    >
    > > However, this variant worked as I've reported:

    > [... forks from thread ]
    >
    > I tried it from FreeBSD 5.1, using 2.2.2 built from ports,
    > and didn't see this problem. I believe 5.1 has a significantly
    > different thread implementation from 4.x, though.


    And later releases of FreeBSD 4.x have a number of bugfixes to the pthread
    support, including some signal related changes. The 4.x pthread support
    still has some issues though.

    Elf refers to this working with Python 2.1 - I vaguely recall some changes
    to Python's pthreads support to try and "harmonise" the behaviour across a
    number of platforms, and signal delivery may have been affected such that
    signals are only delivered to the primary thread. As this is only a vague
    recollection, treat with caution...

    Does building Python with the linuxthreads port, instead of FreeBSD's
    native pthreads, behave as expected?

    > Note that there is pthread_sigmask(3) with the same arguments as
    > sigprocmask(2), and Python's thread_pthread.h does appear to block
    > all signals that way when starting a new thread. That's a clue,
    > perhaps. If sigprocmask() didn't show any signals blocked, maybe
    > pthread_sigmask() would, or maybe at any rate you can set the
    > mask to 0 using one or the other of those functions.


    I also recall Michael Hudson trying to get signal mask support working in
    the presence of threads, but he could only get it to work on Linux. We
    did eventually get a required fix to FreeBSD's libc_r committed (in the
    4.7 timeframe I think), but Michael had given up by then and canned his
    code.

    --
    Andrew I MacIntyre "These thoughts are mine alone..."
    E-mail: (pref) | Snail: PO Box 370
    (alt) | Belconnen ACT 2616
    Web: http://www.andymac.org/ | Australia
     
    Andrew MacIntyre, Aug 14, 2003
    #5
  6. Andrew MacIntyre <> writes:

    >> I tried it from FreeBSD 5.1, using 2.2.2 built from ports,
    >> and didn't see this problem. I believe 5.1 has a significantly
    >> different thread implementation from 4.x, though.


    > And later releases of FreeBSD 4.x have a number of bugfixes to the pthread
    > support, including some signal related changes. The 4.x pthread support
    > still has some issues though.


    I went and looked again at the evidence.

    When calling the function with apply(), the child process spawned off
    reports:

    mask 0x0

    When calling the function with thread.start_new_thread(), the child
    process reports:

    mask 0xfffefeff

    So I think the two of you have more or less hit it squarely on the
    head. Now, I just have to find a way around it... *Sigh*

    Elf
     
    Elf M. Sternberg, Aug 14, 2003
    #6
  7. Michael Hudson wrote:

    > "Elf M. Sternberg" <> writes:
    >
    >> Right now, I just want a way to make signal delivery work.

    >
    > At some point, the threads implementation got changed so that new
    > threads get their signal mask set to "block everything and it's cat"
    > (there was a reason for this, but I'm not really sure what it was).
    > As I see it, you have two options: 1) write a little C to unset the
    > procmask after fork() but before execve() 2) only launch apps from the
    > main thread.


    Incidentally, these aren't far from the advice in "Pthreads Programming"
    (by Nichols, Buttlar and Proulx Farrell -- O'Reilly) in Chapter 5, under
    "Threads and Process Management". The authors are dealing with C issues,
    not Python by any means, but still they conclude that "forking from a
    multithreaded program is no picnic" and suggest 1) fork before you've
    created any threads, and 2) consider the surrogate parent model (fork a
    child process at init time, IPC with the child -- which remains single
    threaded -- to delegate forking and execing on our behalf).


    Alex
     
    Alex Martelli, Aug 15, 2003
    #7
  8. Alex Martelli <> writes:

    > Incidentally, these aren't far from the advice in "Pthreads Programming"
    > (by Nichols, Buttlar and Proulx Farrell -- O'Reilly) in Chapter 5, under
    > "Threads and Process Management". The authors are dealing with C issues,
    > not Python by any means, but still they conclude that "forking from a
    > multithreaded program is no picnic" and suggest 1) fork before you've
    > created any threads, and 2) consider the surrogate parent model (fork a
    > child process at init time, IPC with the child -- which remains single
    > threaded -- to delegate forking and execing on our behalf).


    (1) wasn't really viable since the parent process is ruled by an
    application server (Webware, in this case) and I have no desire to go
    trolling through the guts of that excellent program to make it "work"
    the way I want. I almost went with (2). But, fortunately, we're
    developing a single-OS appliance and don't really care about
    cross-platform compatibility, so I worked with the FreeBSD 4.4 "model"
    of forking from multithreaded applications. It's fortunate that the
    next call is an exec(); I'd hate to have to manage a forked copy of the
    process with that one thread...

    Elf
     
    Elf M. Sternberg, Aug 15, 2003
    #8
  9. Elf M. Sternberg

    Donn Cave Guest

    Quoth "Elf M. Sternberg" <>:
    | ... But, fortunately, we're
    | developing a single-OS appliance and don't really care about
    | cross-platform compatibility, so I worked with the FreeBSD 4.4 "model"
    | of forking from multithreaded applications. It's fortunate that the
    | next call is an exec(); I'd hate to have to manage a forked copy of the
    | process with that one thread...

    So, you resolved the problem and the spawnv child process can now
    receive signals? Did you use sigprocmask() to clear the signal
    mask? What is the FreeBSD 4.4 model of forking from multithreaded
    applications? (And is it the same as the FreeBSD 4.7 model, 5.1, etc.?)

    Donn Cave,
     
    Donn Cave, Aug 16, 2003
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. timanfaya/
    Replies:
    0
    Views:
    458
    timanfaya/
    May 18, 2004
  2. lpw
    Replies:
    2
    Views:
    346
    werasm
    Nov 3, 2005
  3. PurpleServerMonkey

    Distributing Python Apps on Linux\BSD

    PurpleServerMonkey, Mar 21, 2008, in forum: Python
    Replies:
    3
    Views:
    440
    Diez B. Roggisch
    Mar 22, 2008
  4. Replies:
    9
    Views:
    1,038
  5. Robert Nicholson

    Why is this dup failing under BSD?

    Robert Nicholson, Dec 30, 2006, in forum: Perl Misc
    Replies:
    1
    Views:
    102
    John W. Krahn
    Dec 30, 2006
Loading...

Share This Page