Caching in memory for Apache

Discussion in 'Python' started by Simon Johnson, Mar 24, 2006.

  1. Dear All,

    I have decided to take the big plunge and drop the Microsoft platform
    and use Mod_Python and Apache in it's place.

    I've never used Linux before this project so it's a really big learning
    curve, but so far it's going well. I've managed to create some simple
    pages using the basic of Python.

    One of the questions I have is: How do you cache stuff in memory? Say
    I've got a bunch of stuff I need to create from a set of database rows.
    Obviously, I do not want to consult the database each time I have a
    request for my page. Is there anyway in Mod_python for me to store
    values inside memory so I can look it up on the next GET request?

    Simon.
    Simon Johnson, Mar 24, 2006
    #1
    1. Advertising

  2. Pickle (http://www.network-theory.co.uk/docs/pytut/tut_58.html) comes
    to mind when anyone mentions wanting to save a value, method, object,
    etc without writing to storage (file or db). Is it possible for you to
    post some code? There may be an alternative algorithm so you can avoid
    writing to "memory."

    Harlin Seritt
    Harlin Seritt, Mar 24, 2006
    #2
    1. Advertising

  3. Simon  Johnson

    Paul Boddie Guest

    Simon Johnson wrote:
    >
    > I have decided to take the big plunge and drop the Microsoft platform
    > and use Mod_Python and Apache in it's place.


    Congratulations! ;-)

    [...]

    > One of the questions I have is: How do you cache stuff in memory? Say
    > I've got a bunch of stuff I need to create from a set of database rows.
    > Obviously, I do not want to consult the database each time I have a
    > request for my page. Is there anyway in Mod_python for me to store
    > values inside memory so I can look it up on the next GET request?


    I suppose that you probably need to use sessions in your application in
    order to achieve what you're aiming for. If you're not familiar with
    the concept of a session in the context of Web applications, sessions
    are just things which associate a particular user or client with
    persistent information stored on the server: when sending responses
    (ie. Web pages) to users/clients, you include a token which those
    users/clients then submit back to you with each request; you use these
    tokens to look up any information that you've stored for quick
    retrieval; the method of transmitting tokens is most often a "cookie"
    which is automatically sent back by browsers unless configured
    otherwise.

    I know that mod_python does support sessions, including in-memory
    sessions, as do a number of different frameworks. It should be fairly
    convenient to take database rows and store them in a session, and in
    most cases the framework (eg. mod_python) will manage the low-level
    details (such as the sending of cookies, and so on). Sometimes, there
    are restrictions on what the persistent information in sessions can
    actually look like: some implementations use things like pickle, and
    you can run into "unpickleable" objects; some implementations recommend
    that you only store things like strings or text.

    Whether sessions are ultimately the "right" answer rests upon a number
    of technical considerations, and some people have stated that they
    believe them to be generally inappropriate. However, they're probably
    the most immediately appropriate thing for you to use in this case.

    Paul
    Paul Boddie, Mar 24, 2006
    #3
  4. On Fri, 2006-03-24 at 06:56 -0800, Simon Johnson wrote:
    > Dear All,
    >
    > I have decided to take the big plunge and drop the Microsoft platform
    > and use Mod_Python and Apache in it's place.
    >
    > I've never used Linux before this project so it's a really big learning
    > curve, but so far it's going well. I've managed to create some simple
    > pages using the basic of Python.
    >
    > One of the questions I have is: How do you cache stuff in memory? Say
    > I've got a bunch of stuff I need to create from a set of database rows.
    > Obviously, I do not want to consult the database each time I have a


    Why not hit the database each time? That's what it is there for. What
    you are trying to do is called premature optimization.

    You are facing a huge learning curve. My recommendation is to build it
    first with close adherence to the KISS principal. After its built, take
    it for a spin. If it is too slow, then tune it. Think towards
    implementation simplicity, not speed. You will probably find that
    whatever you build will be fast enough when you are finished.


    > request for my page. Is there anyway in Mod_python for me to store
    > values inside memory so I can look it up on the next GET request?


    Your database really is the best place for that.

    Good luck - Adam DePrince
    Adam DePrince, Mar 25, 2006
    #4
  5. Simon  Johnson

    Guest

    Simon Johnson wrote:
    > Dear All,
    >
    > I have decided to take the big plunge and drop the Microsoft platform
    > and use Mod_Python and Apache in it's place.
    >
    > I've never used Linux before this project so it's a really big learning
    > curve, but so far it's going well. I've managed to create some simple
    > pages using the basic of Python.
    >
    > One of the questions I have is: How do you cache stuff in memory? Say
    > I've got a bunch of stuff I need to create from a set of database rows.
    > Obviously, I do not want to consult the database each time I have a
    > request for my page. Is there anyway in Mod_python for me to store
    > values inside memory so I can look it up on the next GET request?


    It all depends a bit on what you want to cache and whether the data is
    specific to one users interaction with a page, or whether the data is
    able to be used across requests from multiple users. This will dictate
    whether you can use a global data cached within modules used to
    generate
    the page, whether you can use sessions, shared memory or whether you
    have no choice but to always go back to the database.

    As others have pointed out, don't get trapped in doing premature
    optimisation. More often than not what you may think will be a bottle
    neck will not and any problems if they even exist will be elsewhere.

    In respect of mod_python, it is also vitally important to understand
    the
    process/interpreter module under which Python runs within the context
    of Apache. A starting point for getting information about this is:

    http://www.dscpl.com.au/articles/modpython-004.html

    If you do decide that sessions may be useful to what you are doing, as
    pointed out by others, there can be limitations as to what data can be
    stored. For limitations in respect of sessions in mod_python have a
    look
    at:

    http://www.dscpl.com.au/articles/modpython-005.html

    Finally, make sure you use the latest version of mod_python available.
    Avoid using mod_python 2.7.X or 3.1.X as there are lots of known issues
    and problems. Not all problems are solved in 3.2.X version, but good
    progress has been made on getting rid of many issues. The main area
    which can still cause problems, especially when trying to cache data in
    the Apache process, is that of module loading and reloading. For a run
    down on issues in that area see:

    http://www.dscpl.com.au/articles/modpython-003.html

    A lot of work has already been done on solving these issues for 3.3.X
    version of mod_python.

    BTW, the best place to ask questions about mod_python is the mod_python
    mailing list. Thus, get yourself subscribed to it. Details of the
    mailing list
    are on the mod_python web site.

    Graham
    , Mar 25, 2006
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hypo
    Replies:
    6
    Views:
    392
  2. Troy Simpson

    Fragment Caching inside page caching?

    Troy Simpson, Jan 19, 2004, in forum: ASP .Net
    Replies:
    0
    Views:
    768
    Troy Simpson
    Jan 19, 2004
  3. Replies:
    1
    Views:
    282
  4. JimLad
    Replies:
    3
    Views:
    901
    JimLad
    Jan 21, 2010
  5. howa

    Memory caching in Apache/mod_perl

    howa, Nov 17, 2007, in forum: Perl Misc
    Replies:
    2
    Views:
    339
    Mark Clements
    Nov 17, 2007
Loading...

Share This Page