High-performance Python websites

Discussion in 'Python' started by Nick Mellor, Nov 26, 2009.

  1. Nick Mellor

    Nick Mellor Guest

    Hi all,

    I'm contemplating setting up a Python-powered website for the tourist
    industry, which will involve a web service, a good deal of XML
    processing, and a Django-powered front-end. If the project works, it
    could get a lot of traffic. I'm sure it can be done, but I'm looking
    to find out more about how existing high-volume Python sites have
    managed their workload. Can anyone give me examples of high-volume
    Python-powered websites, if possible with some idea of their
    architecture?

    Many thanks,

    Nick
     
    Nick Mellor, Nov 26, 2009
    #1
    1. Advertising

  2. Nick Mellor

    ShoqulKutlu Guest

    Hi,

    Managing load of high volume of visitors is a common issue for all
    kind of web technologies. I mean this is not the python issue. This
    issue is mostly about server level designs. You need to supply load
    balancing for both web servers and databases to make your web site
    able to respond to several concurrent visitors. Of course a good
    programmed website is a key performance issue but for your mention I
    would also suggest considering how many hardwares, how many
    webservers, how many database cluster and which database server should
    be used or will be used in the future..

    Regards,
    Kutlu

    On Nov 26, 2:24 am, Nick Mellor <> wrote:
    > Hi all,
    >
    > I'm contemplating setting up a Python-powered website for the tourist
    > industry, which will involve a web service, a good deal of XML
    > processing, and a Django-powered front-end. If the project works, it
    > could get a lot of traffic. I'm sure it can be done, but I'm looking
    > to find out more about how existing high-volume Python sites have
    > managed their workload. Can anyone give me examples of high-volume
    > Python-powered websites, if possible with some idea of their
    > architecture?
    >
    > Many thanks,
    >
    > Nick
     
    ShoqulKutlu, Nov 26, 2009
    #2
    1. Advertising

  3. Nick Mellor

    ShoqulKutlu Guest

    If you need an example one is in front of you and you are using it
    now. Google uses python at their thousands of servers. But as you see
    Google gains this performance from the power of hundreds of thousands
    servers. The main idea behind the Google search engine was indexing
    the whole web through hundreds of servers. So you can imagine where
    Google's power comes from..

    Regards


    On Nov 26, 2:33 am, ShoqulKutlu <> wrote:
    > Hi,
    >
    > Managing load of high volume of visitors is a common issue for all
    > kind of web technologies. I mean this is not the python issue. This
    > issue is mostly about server level designs. You need to supply load
    > balancing for both web servers and databases to make your web site
    > able to respond to several concurrent visitors. Of course a good
    > programmed website is a key performance issue but for your mention I
    > would also suggest considering how many hardwares, how many
    > webservers, how many database cluster and which database server should
    > be used or will be used in the future..
    >
    > Regards,
    > Kutlu
    >
    > On Nov 26, 2:24 am, Nick Mellor <> wrote:
    >
    >
    >
    > > Hi all,

    >
    > > I'm contemplating setting up a Python-powered website for the tourist
    > > industry, which will involve a web service, a good deal of XML
    > > processing, and a Django-powered front-end. If the project works, it
    > > could get a lot of traffic. I'm sure it can be done, but I'm looking
    > > to find out more about how existing high-volume Python sites have
    > > managed their workload. Can anyone give me examples of high-volume
    > > Python-powered websites, if possible with some idea of their
    > > architecture?

    >
    > > Many thanks,

    >
    > > Nick- Hide quoted text -

    >
    > - Show quoted text -
     
    ShoqulKutlu, Nov 26, 2009
    #3
  4. Nick Mellor

    Nick Mellor Guest

    Thanks Kutlu,

    I wasn't aware that Google used Python for running their Google groups
    servers. Can you confirm that? The only place
    I've seen Google explicitly use Python on their web front end is in
    the Google Ads tests.

    I am impressed by the responsiveness of lawrence.com, ljworld.com and
    others on the Django home page (http://www.djangoproject.com/)

    They seem to do a great job of loading large, complex pages using
    Django (stacked on Python, stacked on bytecode, stacked on C.)
    Shows it can be done.

    Nick
     
    Nick Mellor, Nov 26, 2009
    #4
  5. Nick Mellor

    ShoqulKutlu Guest

    Hi Nick,

    Sorry about my concern on Google. It caused a confusion about Google
    groups. I didn't mean explicitly where Google uses python, I mentioned
    just Google uses Python. A Google officer told that they run Python on
    thousands of their servers at an interview. Due to this claim I wanted
    to say it for you.
    Actualy of course it can be done and even it will not be worse than
    any other frameworks, and I bet can be better than Java and ASP.NET if
    configured and programmed well. I really encourage you to use
    mod_python for any project. Mod_python and mod_wsgi made it very
    powerful at web side. As I said in my previous message a web
    application's responsiveness is dependent to several issues. A good
    web framework, server speed, database design etc.. In this case you
    want to use django, which as I know build for mod_python and can be
    configured to run on a mod_wsgi web server. Consider that you will
    have one million members on your site. That traffic simply needs
    several clustered web servers and clustered databases. This means you
    supply a load balancing. So concurrent user sessions will be shared on
    different web servers. You can do your best with such a clustered
    system with such a powerful language. Really don't worry about that.

    Regards,
    Kutlu

    On Nov 26, 7:21 am, Nick Mellor <> wrote:
    > Thanks Kutlu,
    >
    > I wasn't aware that Google used Python for running their Google groups
    > servers. Can you confirm that? The only place
    > I've seen Google explicitly use Python on their web front end is in
    > the Google Ads tests.
    >
    > I am impressed by the responsiveness of lawrence.com, ljworld.com and
    > others on the Django home page (http://www.djangoproject.com/)
    >
    > They seem to do a great job of loading large, complex pages using
    > Django (stacked on Python, stacked on bytecode, stacked on C.)
    > Shows it can be done.
    >
    > Nick
     
    ShoqulKutlu, Nov 26, 2009
    #5
  6. Nick Mellor

    ShoqulKutlu Guest

    Hi again,

    I also want to say something about creating a better web application.
    If I were you I wouldn't use any web framework like django or other.
    If you want to create a commercial project and want to manage all
    modules of the application yourself I suggest you to create your own
    framework. I don't mean create a Django like framework. Determine your
    needs, modules, services and build your own modules. Use other
    independent open source modules for any specific issues. To make a
    successful project requires your own business logic. And you can do
    this with your own algorithm. Don't avoid collecting other modules for
    templating, xml parsing, DB connectivity issues. But build your own
    framework with these and your own modules..

    Regards,
    Kutlu
     
    ShoqulKutlu, Nov 26, 2009
    #6
  7. Nick Mellor a écrit :
    > Hi all,
    >
    > I'm contemplating setting up a Python-powered website for the tourist
    > industry, which will involve a web service, a good deal of XML
    > processing, and a Django-powered front-end. If the project works, it
    > could get a lot of traffic. I'm sure it can be done, but I'm looking
    > to find out more about how existing high-volume Python sites have
    > managed their workload. Can anyone give me examples of high-volume
    > Python-powered websites, if possible with some idea of their
    > architecture?


    youtube once used quite a lot of Python IIRC. You may be able to find
    relevant infos on the net.

    While I may disagree with Kutlu on some points[1], it's clear that the
    key to handling huge traffic is the ability to scale up. So better to
    avoid solutions that make it hard - or impossible - to setup load
    balancing, replication etc. Now that doesn't mean than decent
    performance and reasonnable memory usage are not a concern - even a
    simple website with moderate traffic can become a PITA if you choose the
    wrong tools / architecture (Plone perfs problems anyone ?).

    Anyway : just make sure your solution is both simple enough to avoid
    becoming a resource-eater yet serious enough to allow for fine-grained
    caching, load-balancing and the like.


    [1] like reinventing your own framework - whatever architecture
    (including non-blocking IO/event-based server like Twisted) you settle
    on, chances are most of the grunt work has already been done, and
    probably better than what you could come with in a reasonable amount of
    time - unless you have a really BIG budget of course.
     
    Bruno Desthuilliers, Nov 26, 2009
    #7
  8. Nick Mellor

    Nick Mellor Guest

    On Nov 27, 3:26 am, Bruno Desthuilliers <bruno.
    > wrote:
    > Nick Mellor a écrit :
    >
    > > Hi all,

    >
    > > I'm contemplating setting up a Python-powered website for the tourist
    > > industry, which will involve a web service, a good deal of XML
    > > processing, and a Django-powered front-end. If the project works, it
    > > could get a lot of traffic. I'm sure it can be done, but I'm looking
    > > to find out more about how existing high-volume Python sites have
    > > managed their workload. Can anyone give me examples of high-volume
    > > Python-powered websites, if possible with some idea of their
    > > architecture?

    >
    > youtube once used quite a lot of Python IIRC. You may be able to find
    > relevant infos on the net.
    >
    > While I may disagree with Kutlu on some points[1], it's clear that the
    > key to handling huge traffic is the ability to scale up. So better to
    > avoid solutions that make it hard - or impossible - to setup load
    > balancing, replication etc. Now that doesn't mean than decent
    > performance and reasonnable memory usage are not a concern - even a
    > simple website with moderate traffic can become a PITA if you choose the
    > wrong tools / architecture (Plone perfs problems anyone ?).
    >
    > Anyway : just make sure your solution is both simple enough to avoid
    > becoming a resource-eater yet serious enough to allow for fine-grained
    > caching, load-balancing and the like.
    >
    > [1] like reinventing your own framework - whatever architecture
    > (including non-blocking IO/event-based server like Twisted) you settle
    > on, chances are most of the grunt work has already been done, and
    > probably better than what you could come with in a reasonable amount of
    > time - unless you have a really BIG budget of course.


    Bruno and Kutlu,

    It's a small start-up project.

    By mentioning lawrence and other Django-powered websites, I was
    pointing out that the problem of creating a high-performance
    web solution using Python has already been solved in several places,
    and the lessons learned have been given back to us
    in the form of products like Django. I tend to agree with Bruno that
    I'm unlikely to do a
    better job than Django.

    Thanks for your responses,

    Nick
     
    Nick Mellor, Nov 27, 2009
    #8
  9. On Nov 25, 4:24 pm, Nick Mellor <> wrote:
    > Hi all,
    >
    > I'm contemplating setting up a Python-powered website for the tourist
    > industry, which will involve a web service, a good deal of XML
    > processing, and a Django-powered front-end. If the project works, it
    > could get a lot of traffic. I'm sure it can be done, but I'm looking
    > to find out more about how existing high-volume Python sites have
    > managed their workload. Can anyone give me examples of high-volume
    > Python-powered websites, if possible with some idea of their
    > architecture?
    >
    > Many thanks,
    >
    > Nick


    You should now that there's a Google service called Google App Engine
    that lets you host yur website in google's own infraestructure (this
    is known nowadays as "cloud computing").
    It's free to start (as long as you don't exceed the minimum quotas of
    space and traffic, which are quite generous).
    The good thing is that you don't have to think about scallig issues or
    about your overall site's arquitecture or hardware. It's te whole
    google infraestructure at your disposal, which can scale from one user
    to tens of thousands without having to change anyting from your part.
    Simply code correctly your site in python or java, using Django or any
    other wsgi compliant framework, and you are set to go.

    Check it out: http://code.google.com/appengine/docs/whatisgoogleappengine.html

    Luis
     
    Luis M. González, Nov 27, 2009
    #9
  10. Nick Mellor

    inhahe Guest

    On Wed, Nov 25, 2009 at 7:33 PM, ShoqulKutlu <> wrote:
    > Hi,
    >
    > Managing load of high volume of visitors is a common issue for all
    > kind of web technologies. I mean this is not the python issue. This
    > issue is mostly about server level designs. You need to supply load
    > balancing for both web servers and databases to make your web site
    > able to respond to several concurrent visitors. Of course a good
    > programmed website is a key performance issue but for your mention I
    > would also suggest considering how many hardwares, how many
    > webservers, how many database cluster and which database server should
    > be used or will be used in the future..
    >


    I don't know a lot about this issue, but take apache + php. every
    time a page is loaded a new instance of php is loaded to run the
    page, so i imagine load balancing can easiry be done on the page
    request level by distributing instances of php processes.
    whereas if you use python, you don't really want to load the python
    interpreter for every page request. as far as i can tell, the
    canonical way is to have one app for the whole website that's
    constantly running and communicates with the server via WSGI. or is
    that wrong? and wouldn't that make load balancing a little bit more
    tricky, or at least different? not sure..
     
    inhahe, Nov 30, 2009
    #10
  11. On Monday 30 November 2009 10:55:55 inhahe wrote:
    > On Wed, Nov 25, 2009 at 7:33 PM, ShoqulKutlu <>

    wrote:
    > > Hi,
    > >
    > > Managing load of high volume of visitors is a common issue for all
    > > kind of web technologies. I mean this is not the python issue. This
    > > issue is mostly about server level designs. You need to supply load
    > > balancing for both web servers and databases to make your web site
    > > able to respond to several concurrent visitors. Of course a good
    > > programmed website is a key performance issue but for your mention
    > > I would also suggest considering how many hardwares, how many
    > > webservers, how many database cluster and which database server
    > > should be used or will be used in the future..

    >
    > I don't know a lot about this issue, but take apache + php. every
    > time a page is loaded a new instance of php is loaded to run the
    > page,


    AFAIK that's only the case for PHP-CGI, and Python as a CGI scripting
    language is used the same way. Apache is very often run with mod_php,
    though, which embeds the PHP interpreter; mod_python does something
    similar for Python.


    ----
    Rami Chowdhury
    "As an online discussion grows longer, the probability of a comparison
    involving Nazis or Hitler approaches one." -- Godwin's Law
    408-597-7068 (US) / 07875-841-046 (UK) / 0189-245544 (BD)
     
    Rami Chowdhury, Dec 2, 2009
    #11
  12. Rami Chowdhury a écrit :
    > On Monday 30 November 2009 10:55:55 inhahe wrote:
    >> On Wed, Nov 25, 2009 at 7:33 PM, ShoqulKutlu <>

    > wrote:
    >>> Hi,
    >>>
    >>> Managing load of high volume of visitors is a common issue for all
    >>> kind of web technologies. I mean this is not the python issue. This
    >>> issue is mostly about server level designs. You need to supply load
    >>> balancing for both web servers and databases to make your web site
    >>> able to respond to several concurrent visitors. Of course a good
    >>> programmed website is a key performance issue but for your mention
    >>> I would also suggest considering how many hardwares, how many
    >>> webservers, how many database cluster and which database server
    >>> should be used or will be used in the future..

    >> I don't know a lot about this issue, but take apache + php. every
    >> time a page is loaded a new instance of php is loaded to run the
    >> page,

    >
    > AFAIK that's only the case for PHP-CGI, and Python as a CGI scripting
    > language is used the same way.


    Yeps.

    > Apache is very often run with mod_php,
    > though, which embeds the PHP interpreter; mod_python does something
    > similar for Python.


    Indeed.

    And FWIW, this has very few impact wrt/ load balancing issues.
     
    Bruno Desthuilliers, Dec 2, 2009
    #12
  13. Nick Mellor

    Dr. Marco Guest

    On Wed, 25 Nov 2009 21:21:25 -0800 (PST),
    Nick Mellor <> wrote :
    Hi,

    > I wasn't aware that Google used Python for running their Google groups
    > servers. Can you confirm that? The only place
    > I've seen Google explicitly use Python on their web front end is in
    > the Google Ads tests.
    >
    > I am impressed by the responsiveness of lawrence.com, ljworld.com and
    > others on the Django home page (http://www.djangoproject.com/)


    I'm running two sites using Django which are certainly infinitely more
    modest and much less visited than the ones you quote but which
    nevertheless are extremely responsive compared to other frameworks
    tested, using a single machine (http://nonlineaire.univ-lille1.fr/SNL/
    and http://nonlineaire.univ-lille1.fr/GDR3070/).

    --
    Dr. Marco
     
    Dr. Marco, Dec 6, 2009
    #13
  14. The key to scaling a web site is to make
    sure you can load-balance to as many front
    ends as needed and then use a common database
    backend that is fast enough or possibly a
    common file system that is fast enough.

    I can't speak to Django specifically but
    you can certainly get essentially unlimited
    scalability on the front-end side of the
    equation using a Python based web app.

    The google app engine will set such a
    configuration up for you automatically, but
    they are still working some bugs out
    with regard to performance, I think,
    based on my experience here
    http://whiffdoc.appspot.com/
    and here
    http://listtree.appspot.com/

    I hope that helps.
    -- Aaron Watters

    ===
    an apple every 8 hours
    will keep 3 doctors away. -kliban
     
    Aaron Watters, Dec 7, 2009
    #14
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jimmy

    High traffic websites

    Jimmy, Apr 28, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    719
    Juan T. Llibre
    Apr 28, 2005
  2. David McNab
    Replies:
    0
    Views:
    677
    David McNab
    Apr 23, 2004
  3. Irmen de Jong
    Replies:
    7
    Views:
    565
    Josiah Carlson
    Oct 27, 2004
  4. Michael Sparks
    Replies:
    1
    Views:
    474
    Elbert Lev
    Oct 28, 2004
  5. Jack
    Replies:
    6
    Views:
    397
Loading...

Share This Page