Best architecture for proxy?

Discussion in 'Python' started by Andrew Warkentin, Jul 10, 2007.

  1. I am going to write a general-purpose modular proxy in Python. It will
    consist of a simple core and several modules for things like filtering
    and caching. I am not sure whether it is better to use multithreading,
    or to use an event-driven networking library like Twisted or Medusa/
    Asyncore. Which would be the better architecture to use?
     
    Andrew Warkentin, Jul 10, 2007
    #1
    1. Advertising

  2. Andrew Warkentin wrote:

    > I am going to write a general-purpose modular proxy in Python. It
    > will consist of a simple core and several modules for things like
    > filtering and caching. I am not sure whether it is better to use
    > multithreading, or to use an event-driven networking library like
    > Twisted or Medusa/ Asyncore. Which would be the better
    > architecture to use?


    I'd definitely use an event-driven approach with Twisted.

    Generally, multithreading is less performant than multiplexing. High
    performance servers mostly use a combination of both, though.

    Regards,


    Björn

    --
    BOFH excuse #306:

    CPU-angle has to be adjusted because of vibrations coming from the
    nearby road
     
    Bjoern Schliessmann, Jul 10, 2007
    #2
    1. Advertising

  3. Andrew Warkentin

    Steve Holden Guest

    Bjoern Schliessmann wrote:
    > Andrew Warkentin wrote:
    >
    >> I am going to write a general-purpose modular proxy in Python. It
    >> will consist of a simple core and several modules for things like
    >> filtering and caching. I am not sure whether it is better to use
    >> multithreading, or to use an event-driven networking library like
    >> Twisted or Medusa/ Asyncore. Which would be the better
    >> architecture to use?

    >
    > I'd definitely use an event-driven approach with Twisted.
    >
    > Generally, multithreading is less performant than multiplexing. High
    > performance servers mostly use a combination of both, though.
    >

    Converselt I'd recommend Medusa - not necessarily because it's "better",
    but becuase I know it better. There's also a nice general-purpose proxy
    program (though I'd be surprised if Twisted didn't also have one).

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC/Ltd http://www.holdenweb.com
    Skype: holdenweb http://del.icio.us/steve.holden
    --------------- Asciimercial ------------------
    Get on the web: Blog, lens and tag the Internet
    Many services currently offer free registration
    ----------- Thank You for Reading -------------
     
    Steve Holden, Jul 11, 2007
    #3
  4. On Jul 10, 8:19 pm, Steve Holden <> wrote:
    > Bjoern Schliessmann wrote:
    > > Andrew Warkentin wrote:

    >
    > >> I am going to write a general-purpose modular proxy in Python. It
    > >> will consist of a simple core and several modules for things like
    > >> filtering and caching. I am not sure whether it is better to use
    > >> multithreading, or to use an event-driven networking library like
    > >> Twisted or Medusa/ Asyncore. Which would be the better
    > >> architecture to use?

    >
    > > I'd definitely use an event-driven approach with Twisted.

    >
    > > Generally, multithreading is less performant than multiplexing. High
    > > performance servers mostly use a combination of both, though.

    >
    > Converselt I'd recommend Medusa - not necessarily because it's "better",
    > but becuase I know it better. There's also a nice general-purpose proxy
    > program (though I'd be surprised if Twisted didn't also have one).
    >
    >

    Would an event-driven proxy be able to handle multiple connections
    with large numbers of possibly CPU-bound filters? I use The
    Proxomitron (and would like to write my own proxy that can use the
    same filter sets, but follows the Unix philosophy) and some of the
    filters appear to be CPU-bound, because they cause The Proxomitron to
    hog the CPU (although that might just be a Proxomitron design flaw or
    something). Wouldn't CPU-bound filters only allow one connection to be
    filtered at a time? On the Medusa site, it said that an event-driven
    architecture only works for I/O-bound programs.
     
    Andrew Warkentin, Jul 11, 2007
    #4
  5. Andrew Warkentin

    Steve Holden Guest

    Andrew Warkentin wrote:
    > On Jul 10, 8:19 pm, Steve Holden <> wrote:
    >> Bjoern Schliessmann wrote:
    >>> Andrew Warkentin wrote:
    >>>> I am going to write a general-purpose modular proxy in Python. It
    >>>> will consist of a simple core and several modules for things like
    >>>> filtering and caching. I am not sure whether it is better to use
    >>>> multithreading, or to use an event-driven networking library like
    >>>> Twisted or Medusa/ Asyncore. Which would be the better
    >>>> architecture to use?
    >>> I'd definitely use an event-driven approach with Twisted.
    >>> Generally, multithreading is less performant than multiplexing. High
    >>> performance servers mostly use a combination of both, though.

    >> Converselt I'd recommend Medusa - not necessarily because it's "better",
    >> but becuase I know it better. There's also a nice general-purpose proxy
    >> program (though I'd be surprised if Twisted didn't also have one).
    >>
    >>

    > Would an event-driven proxy be able to handle multiple connections
    > with large numbers of possibly CPU-bound filters? I use The
    > Proxomitron (and would like to write my own proxy that can use the
    > same filter sets, but follows the Unix philosophy) and some of the
    > filters appear to be CPU-bound, because they cause The Proxomitron to
    > hog the CPU (although that might just be a Proxomitron design flaw or
    > something). Wouldn't CPU-bound filters only allow one connection to be
    > filtered at a time? On the Medusa site, it said that an event-driven
    > architecture only works for I/O-bound programs.
    >

    Well since Medusa doesn't thread or fork you are correct, it's likely
    that connection requests could be rejected if the response processors
    take a significant amount of real and CPU time.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC/Ltd http://www.holdenweb.com
    Skype: holdenweb http://del.icio.us/steve.holden
    --------------- Asciimercial ------------------
    Get on the web: Blog, lens and tag the Internet
    Many services currently offer free registration
    ----------- Thank You for Reading -------------
     
    Steve Holden, Jul 12, 2007
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Muhammad Khan
    Replies:
    4
    Views:
    1,255
    Mike Treseler
    Jul 10, 2003
  2. Replies:
    3
    Views:
    529
    Malcolm
    Sep 29, 2005
  3. rashmi
    Replies:
    2
    Views:
    485
    Grumble
    Jul 5, 2005
  4. Replies:
    3
    Views:
    441
    Malcolm
    Sep 29, 2005
  5. Replies:
    4
    Views:
    686
    Malcolm
    Sep 29, 2005
Loading...

Share This Page