Event Driven programming - Doubts

Discussion in 'Python' started by Kottiyath, Dec 22, 2008.

  1. Kottiyath

    Kottiyath Guest

    Hi,
    I have been looking at Twisted and lately Circuits as examples for
    event driven programming in Python.
    Even though I understood how to implement the code in these and
    what is deferred etc, I have not yet understood the implementation of
    deferred. I went through a lot of tutorials, but I guess most places
    they expect that the user already understands how events are
    generated. The tutorials mention that there is no more threads once
    twisted is used.

    My question is as follows:
    I have not understood how the callbacks are hit without (a)
    blocking the code or (b) having new threads.

    The usual example given is that of a program waiting for data coming
    through a socket. In the tutorials, it is mentioned that -in an event
    driven program, we schedule the code to hit when the remote server
    gets back to us - .
    Now, my question is - somebody has to still wait on that socket and
    check whether the data is received, and once all the data is received,
    call the appropriate callbacks.

    Is twisted creating a micro-thread which just waits on the socket and
    once the data is received, calls callFromThread for it to run on the
    main loop?

    If so, Even though data locking etc is not a problem, are we not still
    having threads? Will it not still cause scalability problems in high
    traffic?
    If not, could somebody let me know how it is done?
    Kottiyath, Dec 22, 2008
    #1
    1. Advertising

  2. Kottiyath

    David Stanek Guest

    On Mon, Dec 22, 2008 at 9:57 AM, Kottiyath <> wrote:
    >
    > If so, Even though data locking etc is not a problem, are we not still
    > having threads? Will it not still cause scalability problems in high
    > traffic?
    > If not, could somebody let me know how it is done?


    This somewhat depends on the application. Is it IO bound or CPU bound?
    >From what I understand about twisted you will only have one process

    and one thread which means you will only be using one CPU.

    --
    David
    http://www.traceback.org
    David Stanek, Dec 22, 2008
    #2
    1. Advertising

  3. Kottiyath

    James Mills Guest

    On Tue, Dec 23, 2008 at 12:57 AM, Kottiyath <> wrote:
    > Hi,
    > I have been looking at Twisted and lately Circuits as examples for
    > event driven programming in Python.


    Wonderful! :) "circuits" that is :)

    > Even though I understood how to implement the code in these and
    > what is deferred etc, I have not yet understood the implementation of
    > deferred. I went through a lot of tutorials, but I guess most places
    > they expect that the user already understands how events are
    > generated. The tutorials mention that there is no more threads once
    > twisted is used.


    deferred to me is nothing more than
    scheduling "things" to happen "later".

    ie: Timers, or Timed Events.

    I clearly biased here as I'm the developer
    of circuits, so I can't comment on Twisted's
    design or architecture, but the way to
    achieve Twisted's so-called deferred (events?)
    is to use the Timer component, Timed Events.

    > My question is as follows:
    > I have not understood how the callbacks are hit without (a)
    > blocking the code or (b) having new threads.


    Again speaking in terms of circuits,
    but also in the event-driven paradigm:

    a) Event Handlers _can_ block - but ideally shouldn't (ihmo).
    b) Forking new threads/processes per event is mostly not necessary.
    (There are exceptions).

    > The usual example given is that of a program waiting for data coming
    > through a socket. In the tutorials, it is mentioned that -in an event
    > driven program, we schedule the code to hit when the remote server
    > gets back to us - .
    > Now, my question is - somebody has to still wait on that socket and
    > check whether the data is received, and once all the data is received,
    > call the appropriate callbacks.


    In the case of circuits' TCPServer component
    you simply poll it in your main event-loop.
    The pattern looks like this:

    from circuits.lib.sockets import TCPServer

    server = TCPServer(8000)
    while True:
    server.poll()
    server.flush()

    The TCPServer component has various builtin
    event handlers that do all the work for you
    and expose other more useful events to
    the application, such as:
    * connect
    * disconnect
    * read
    * error

    > Is twisted creating a micro-thread which just waits on the socket and
    > once the data is received, calls callFromThread for it to run on the
    > main loop?


    I can't comment - I tend to avoid threads
    where ever possible.

    > If so, Even though data locking etc is not a problem, are we not still
    > having threads? Will it not still cause scalability problems in high
    > traffic?


    Regarding scalability btw ... (in case you're interested)
    circuits comes with circuits.lib.web and several
    sets of Web Components. A lot of code was
    borrowed from the BaseHTTPServer from the
    python standard library and bits and pieces
    from CherryPy and the cgi module.... In terms
    of performance, it has a "raw" performance
    ~3k req/s on good hardware.

    Do have fun in your endeavours :)

    cheers
    James

    Circuits: http://trac.softcircuit.com.au/circuits/
    James Mills, Dec 22, 2008
    #3
  4. Kottiyath

    Bryan Olson Guest

    Kottiyath wrote:
    > [...] I have not yet understood the implementation of
    > deferred. I went through a lot of tutorials, but I guess most places
    > they expect that the user already understands how events are
    > generated. The tutorials mention that there is no more threads once
    > twisted is used.
    >
    > My question is as follows:
    > I have not understood how the callbacks are hit without (a)
    > blocking the code or (b) having new threads.
    >
    > The usual example given is that of a program waiting for data coming
    > through a socket. In the tutorials, it is mentioned that -in an event
    > driven program, we schedule the code to hit when the remote server
    > gets back to us - .
    > Now, my question is - somebody has to still wait on that socket and
    > check whether the data is received, and once all the data is received,
    > call the appropriate callbacks.
    >
    > Is twisted creating a micro-thread which just waits on the socket and
    > once the data is received, calls callFromThread for it to run on the
    > main loop?


    No. Event-driven frameworks are built upon some system call that waits
    for events from multiple sources at once. The Python standard library
    exposes several such calls, the most portable of which is select(), in
    the module also named "select".

    http://docs.python.org/library/select.html

    So how is something like deferred implemented? When you schedule action
    on your socket, you are telling the framework to include your socket in
    in subsequent calls to select(), until the socket selects as ready.

    The framework keeps a single global list of sockets on which clients are
    waiting, and from this list it builds the select() parameters. A single
    call to select() waits on behalf of all the callbacks. The select() call
    returns a list of sockets ready for I/0; the framework iterates over the
    ready list, invoking the corresponding callbacks one by one. After the
    last callback returns, the framework loops back to select() again.

    select() is not the only call to do multi-source I/O, and I'm not an
    expert on these frameworks, so take the above as a simplified general
    description.


    --
    --Bryan
    Bryan Olson, Dec 23, 2008
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. christopher diggins
    Replies:
    4
    Views:
    393
    christopher diggins
    Dec 21, 2004
  2. Steve
    Replies:
    2
    Views:
    543
    Carl Cerecke
    Jan 23, 2006
  3. sebek

    Event-driven programming

    sebek, May 27, 2007, in forum: Java
    Replies:
    10
    Views:
    705
    =?ISO-8859-1?Q?Arne_Vajh=F8j?=
    Jun 3, 2007
  4. Bruce Cook

    Re: event driven programming

    Bruce Cook, Dec 25, 2009, in forum: C Programming
    Replies:
    0
    Views:
    410
    Bruce Cook
    Dec 25, 2009
  5. Herbert Rosenau

    Re: event driven programming

    Herbert Rosenau, Dec 27, 2009, in forum: C Programming
    Replies:
    1
    Views:
    500
    Nobody
    Dec 29, 2009
Loading...

Share This Page