How to stop DBM/Perl crashing servers?

Discussion in 'Perl Misc' started by OttawaTrade, Sep 13, 2005.

  1. OttawaTrade

    OttawaTrade Guest

    Why a DBM made by Perl can crash a server? It is reported by many people an
    unprivileged perl script can load the memory and crash the server. How do I
    fix this, since many working on the project can not fix. I hope I can find
    more perl experts here.

    http://www.absolutefreebies.com/phpBB2/viewtopic.php?t=7960371

    Does anyone know why turning on Cache will cause the crash every too often?
    Yep - the process of causing the crash is pretty well understood. The cache
    uses a 'technology' called DBM within Perl, which uses a single file as a
    database. The DBM 'technology' uses a rather basic file locking mechanism
    which is susceptible to corruption when multiple instances of the script try
    writing to the DB at the same time.


    http://www.absolutefreebies.com/phpBB2/viewtopic.php?t=7959851&highlight=cache
    OttawaTrade, Sep 13, 2005
    #1
    1. Advertising

  2. OttawaTrade

    Guest

    "OttawaTrade" <> wrote:
    > Why a DBM made by Perl can crash a server?


    A search for "DBM" on search.cpan.org yields 374 hits. Perhaps you could
    be so kind as to tell us exactly what DBM module you are talking about.

    There are thousands of different kinds of servers. Please tell us what
    server you are talking about.


    > It is reported by many people
    > an unprivileged perl script can load the memory and crash the server. How
    > do I fix this, since many working on the project can not fix. I hope I
    > can find more perl experts here.
    >
    > http://www.absolutefreebies.com/phpBB2/viewtopic.php?t=7960371
    >
    > Does anyone know why turning on Cache will cause the crash every too
    > often?


    A search on "Cache" in search.cpan.org yields 2228 hits. What precisely
    are you talking about?


    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
    , Sep 13, 2005
    #2
    1. Advertising

  3. wrote in news:20050913133810.808$:

    > "OttawaTrade" <> wrote:
    >> Why a DBM made by Perl can crash a server?

    >
    > A search for "DBM" on search.cpan.org yields 374 hits. Perhaps you
    > could be so kind as to tell us exactly what DBM module you are talking
    > about.
    >
    > There are thousands of different kinds of servers. Please tell us
    > what server you are talking about.
    >
    >
    >> It is reported by many people
    >> an unprivileged perl script can load the memory and crash the server.
    >> How do I fix this, since many working on the project can not fix. I
    >> hope I can find more perl experts here.
    >>


    <URL snipped>

    >> Does anyone know why turning on Cache will cause the crash every too
    >> often?

    >
    > A search on "Cache" in search.cpan.org yields 2228 hits. What
    > precisely are you talking about?


    My (admittedly less charitable) guess is that this guy is trying to drive
    traffic to the absolutefreebies site. Going to the URL posted above:

    asu1@PAM-JY03P01 ~
    $ links http://.../viewtopic.php?t=7960371


    showed no discussion, but a telling sign:

    Cursing is forbidden.
    Posting of affiliate codes by Newbies is forbidden.
    Nudity is forbidden (not by me, by hosting company, i rather

    Hmmmm ...

    Sinan
    A. Sinan Unur, Sep 13, 2005
    #3
  4. OttawaTrade

    OttawaTrade Guest

    Another copy from the link:
    http://www.absolutefreebies.com/phpBB2/viewtopic.php?t=7959851&highlight=cache

    Support this web site by purchasing from one of our Amazon affiliate stores:
    Canada | France | Germany | Japan | United Kingdom | United States


    Absolute Freebies Forums
    Cursing is forbidden.
    Posting of affiliate codes by Newbies is forbidden.
    Nudity is forbidden (not by me, by hosting company, i rather like nudity).
    Body modification of housepets is forbidden (except for Bonsai Kittens).
    FAQ Search Memberlist Chat (0) Usergroups
    Profile You have no new messages Log out [ qllonceagain ]


    MrRat's APF - Weaknesses - your thoughts please.


    Absolute Freebies Forums Forum Index -> MrRat's Script Support
    View previous topic :: View next topic
    Author Message
    Dean
    Patron of all things Cheesy



    Joined: 26 Jul 2004
    Posts: 1588
    Location: Lancaster, Lancashire, United Kingdom
    Posted: Tue Apr 05, 2005 8:43 pm Post subject: MrRat's APF -
    Weaknesses - your thoughts please.

    ----------------------------------------------------------------------------
    ----

    THE PREAMBLE

    As promised a few days ago here are my thoughts about weaknesses in MrRat's
    Amazon Products Feed (APF). As before - this is NOT an attack on MrR, or on
    his work - merely my thoughts on what is not perfect. Some of these 'issues'
    can be improved, some are limitations imposed from elsewhere - but which I
    believe should still be borne in mind, for example so that feature requests
    can be made to Amazon - or alternative workarounds sought.

    See this earlier thread, for a run down of APF's strengths.
    http://www.absolutefreebies.com/phpBB2/viewtopic.php?t=7959821

    In a day or two I will publish my ideas for specific tweaks and
    improvements, based upon my own ideas and building upon your thoughts and
    feedback. I hope that everyone who uses the script will join in this effort
    to highlight the strengths, weaknesses and perhaps future direction of
    MrRat's APF script.


    WEAK POINTS


    One Man Show - this is MrRat's baby.
    I listed this (him) as the script's greatest strength - well the corollary
    of this is that the future of the script and any future development is
    dependent upon one man. It is inherently in our own interests to keep MrRat
    interested in this project - You will all shortly be receiving your
    commission payments - I will say no more on the matter.


    Documentation is perhaps a weakpoint.
    In particular installation instructions. Newcommers to the script do seem to
    struggle - and the same few issues do seem to come up time after time:
    users NOT uploading all of the required files.
    File and folder permissions.
    How to debug server 500 errors.
    How to use the templates to produce VALID HTML (avoiding the multiple
    <html><head><body> tags)
    Creating and using a second (basic) template set for integrating into
    existing pages (which already have <html> and <body> tags) using Server Side
    Includes or php Includes.

    Some of the difficulties experienced can be put down to users wanting to run
    websites but who have no HTML, CSS, FTP or basic scripting knowledge and who
    either miss the docs or decide to just ask questions first and do their
    reading later - or never. I know that most who experience difficulties do
    NOT fit this description.

    The documentation DOES jump straight from an install guide (probably
    misleadingly named the beginners guide), straight to the 'Advanced guide'
    with template variables, apf_ loop logic and Server Side Includes / php
    includes.

    BTW: to improve trouble-shooting, the script could be distributed with two
    copies of the script, one named normally the other with the _test.cgi string
    in the name (enabling additional debugging output). Additionally MrRat,
    those of us who use .pl file extensions still have to edit the script to
    allow debugging with .pl file suffix.


    AMAZON

    Most of the major issues with the script are in my humble opinion down to
    Amazon. These may not be directly related to this script but bear with me
    for a moment anyway. If enough people raise these issues with Amazon in a
    persuasive enough manner we may make some headway.


    Co-branded Shopping Cart
    The lack of co-branding on the landing pages / shopping cart pages at Amazon
    shows a certain lack of value in affiliates. I wonder whether AWS / ECS has
    all been about driving up Amazon's own rankings in Google. I wonder about
    the long term future of AWS / ECS.

    Returning 'OUR' visitors to us AFTER the purchase process is completed.
    This would be a sign that associates were valued - rather than just a means
    of inflating Amazon's Page Rank, and link count. After all once the order
    process is complete why would shoppers want to be pointed back to Amazon's
    front page? This is even more relevent now that opening new windows is
    becoming less polite / more difficult for the avarage surfer. If we can't
    open Amazon in new windows then we lose our open page in the browser.

    The 'Price too low to display'
    (the price isn't too low to display - typically these are high value items -
    it is just a cynical marketing move they don't want to tell the shopper the
    price before they try and buy it.) This is disgraceful practise and it is on
    OUR sites. Imagine a supermarket not putting the price on the shelf but
    instead telling you that you can find out the price at the checkout:

    Availability: This item is currently not available by this merchant
    I am not sure where the responsibility for this one lies - I suspect
    Amazon - but I don't have the in-depth understanding about AWS/ECS needed to
    'prove the point'. It could be that MrRat's Why are product that are not
    available for purchase returned in the listings.
    Purple elephant eaters from the planet Tharg aren't available at the
    moment - that is why I don't have any listed on my site. Imagine if we all
    went around shouting about what is not available.

    Searching and Sorting
    Searching and Sorting are just plain awful - no two ways about it. I am sure
    the problems here lie firmly at Amazon's door - They sort (badly) by one
    variable whilst displaying another. For example - I hit upon the idea of
    integrating a product search with the encyclopaedia on the Professional
    Researcher web site, but after a month or so I removed the feature, the
    products were unrelated wherever the article title (Which we used as teh
    search string) in question was more than one word. This is currently
    recognised by Amazon as an issue - but this is the second or third release
    of AWS where it is listed as an issue - so it is not a high priority then.



    THE APF SCRIPT ITSELF

    Error Message Handling
    Error messages come either from Amazon, or in the case of the Unable to
    process in a timely manner - are hardcoded into the script. I would like to
    see:
    1. Better Error Messages - possibly from the language files, or from a
    template file. Perhaps not showing the error message from Amazon, but a user
    configurable message based upon the Amazon Error Code.
    2. Link to Amazon - Easy way to offer the old 'Please try amazon directly'
    type messages.
    3. A hook - for logging / recording / handling error.

    The above refers to errors returned by Amazon in the XML. For example when a
    search returns no matches. I also have concerns about Internal Server Errors
    and catching those by placing the main 'script control block' inside an eval
    block (You know the bit I mean MrR - the seven or eight calls to
    sub-routines with the START_PROCESSING_LABEL: in amongst them). A friendly
    error message could then be shown to the browser, and a hook could enable
    logging / e-mailing to the webmaster in the event of code that would have
    produced Server 500s.


    The Cache File
    There have been multiple reports of issues with the cache file -
    particularly from sites that are hit hard with high traffic requests - such
    as a concerted burst from Google (although traffic throttling server modules
    may help here).

    Anyone who gets into difficulty with the script - particularly problems with
    the script working through the menus but not displaying product listings, or
    high CPU warnings from their host should try deleting the cachefile and the
    related lock file.

    I know that when the cache file becomes corrupt it can cause problems - but
    I disagree with the stance that some take that this means it is useless. In
    fact for high traffic sites I would have thought that the cache can be very
    useful. I know my (medium traffic) sites benefit greatly from cacheing -
    there is a recognisable repetition in products viewed when items become
    'hot' or when a page gets a good ranking in the Search Engine Results Pages.

    I think that this is an issue though and that at some point it would benefit
    from some attention. I think MrRat has indicated an intent to review the
    cache system (although that was some time ago - I don't know if he changed
    his mind, or just got sidetracked).

    I wonder what versions of Perl those who experience frequent cache problems
    are running, whether they have the most up-to-date versions of Fcntl or
    DM_File (if these are external modules???)
    Code:
    eval 'use Fcntl'; $dbm_error = $@;
    eval 'use DB_File'; $dbm_error .= $@;

    I don't know if these particular modules have known performance / concurrent
    user limitations, but there may be better performing modules that carry out
    similar functions.



    Navigation
    I think that Navigation is still an issue - if I find an APF site whilst
    searching the web, I often have a little poke around - I often find myself
    on pages with NOWHERE to go - almost literally. Especially after errors.
    MrRat's 'showcase' store - with no nav menu is a prime example, and whilst I
    understand that he wants to show the basic script in action I don't think it
    is a good 'shop front' for his masterpiece.

    Two solutions spring to mind:
    1. Improve navigation in the core script / default template.
    2. Alternatively, show an 'out of the box' installation and a 'look at what
    you could have' installation, which includes a few select mod files to
    improve the navigation.


    Template Difficulties
    Template design can be a little tricky sometimes - trying to position
    template variables when you are not certain whether the variable is going to
    produce any output. For example, placing a <hr />above and below a product
    description only to find that there is no product description but two <hr/>
    for no apparent reason. For those of you who struggle with this sort of
    thing there is a hack for this - available in the mods section - it is a
    little technical, but it works.

    I would like to see this hack, or something like it built into the core
    program - but then I am biased. (MrR - it is yours if you want it) I know it
    adds an extra layer of complexity - but only for those who want it.

    Another issue is the exclamation mark / exclamation point - might be known
    by different names in other parts of the world ( bang? ). Whatever you call
    it - it looks like this --> !

    Because of the parsing of apf!!! commands used in templates using
    exclamation marks can cause things to behave unpredictably. This includes
    <!-- html --> comments or exclamation marks that you might enter into the
    language files. Other than changing the parse routines - and invalidating
    every template in use - I see no solution to this, other than awareness -
    and a mention in any FAQ / help documentation.


    The 'Header' or breadcrumb trail or path
    Bestsellers > Books > Topic here. When you go to the sub topic the
    breadcrumb becomes:
    Bestsellers > Books > SubTopic rather than Bestsellers > Books > Topic >
    SubTopic
    This inability of the script to maintain a more informative trail is a
    weakness - it is not MrRat's fault, but it is inefficient, ineffective and
    at times confusing.
    I can only think of one solution to this at present - and that is to build a
    static set of browsenode xml files of our own that incorporate the extra
    data required, similar to my workaround for the on/off UK browsenode lookup
    issue. On balance this would probably be a lot of work to maintain, would be
    error prone and only available for the UK and US locales. Then again just
    because we haven't got a solution doesn't mean it is not an issue - at lease
    if we recognise it as such we can look for solutions.


    Too many query string variables
    I know that these variables are how the script knows what to do - but there
    a lot of them at times. I think some search engines will still balk at 5 or
    more key=value pairs after the question mark. I have identified the
    following situations where these could be dropped.

    Following the use of: link_templates=default
    all urls show this (forever more) templates=default - no need to propogate
    these - just ugly and don't achieve anything. Similarly the number of
    results to retrieve max_results / link_max_results


    Basenodes - The starting point within each SearchIndex
    Choices have been made as to the 'basenode' for each SearchIndex (product
    category such as books DVD software). This is necessary because there is no
    true ZERO NODE or start node within SearchIndexes. By which I mean there is
    no one books node from which ALL book nodes spring. So potentially books are
    available listed by Topic, by publisher, bargain books, used books. By
    choosing 'by topic' above the others, many books and book categories are not
    easily available - they are available - but perhaps not as readily, you
    would have to build some other link to these alternate schema.

    I quite like the idea of constructing, perhaps as an option - some form of
    zero node - perhaps distributing a zero node xml file for each SearchIndex.
    For example, when books is selected in the browse menus - the script could
    load our home grown xml file which when processed as normal would offer the
    user the options organised by topic, organised by publisher, etc. But there
    I go proffering solutions rather than detailing issues.


    Title and Meta Tags / Search Engine Optimisation
    NO script should be designed from the ground up just to appeal to Search
    Engines, and I believe MrRat has expressed his intention NOT to incorporate
    SEO 'techniques' into APF (could be wrong on that one - I have a vague
    memory of this but can't find it with a quick search) I think that there are
    some rough edges. The subject tag after a search is 'sub optimal' and
    because of the way that Store and Subject tags are defined (and possibly
    redefined) throughout the script these are difficult issues to fix.


    Look ahead processing - Processor / memory usage, and bandwidth usage.
    This one is purely personal opinion (what do you mean most of it has been?)
    and is built on deduction rather than 'absolute' knowledge of the script's
    workings. I wonder whether the script isn't a little inefficient in its
    logic. Data is requested and processed and variables are calculated /
    concatenated even though they may not be needed or used in the final 'page
    build'. As an example, I know of one fool (alright it was me) who when
    switching from the script's own shopping cart ( %%shopping_cart_button%% )
    to using Amazon's cart (%%buy_button%%) left the variations_html variable
    lying around. I suspect the script requested variation data, created the
    variations_html template variable (as well as the buy_button variable), and
    for what. Without the %%shopping_cart_button%% present in the template what
    was the point? In fact look at the harm - variations requested, variables
    built, etc.

    I wonder how many situations would benefit from a little look ahead logic -
    to predetermine what is needed and only request data / build variables that
    are needed.


    Showing standard text / html pages within APF templates.
    (I know this one has cropped up a few times recently MrR, but for others who
    didn't catch it)
    I think many people would benefit from the ability to show standard text /
    html 'pages' within the apf script - embedded with the template structure.
    Trying to link to a static privacy policy external to the script is easy,
    but assume the shopper is not in your default locale. They click to your
    privacy policy, when they come to click to re-enter your store, they will
    end up in the default locale. What are you going to do - educate them to use
    their brower back button? Make them select from a list of countries to click
    back to where they were? Both of these could work, but both are sub-optimal.

    Postal / shipping rates, privacy policy, index pages, special offers, even
    the apf_config.cgi setup script could ALL be incorporated directly into the
    main script.


    These are the issues that I feel are worthy of attention - what do you
    think? Am I being harsh, are there other issues you think are a higher
    priority. If you think I am plain wrong then please step up and say so.

    Once again, just so everyone is clear of my intentions I still recommend
    APF, and I commend MrRat for all of his hard work.

    So tell me - and MrRat, what do you think are the script's weaknesses.


    Dean Marshall

    PS - as I said earlier, please hold fire for a few days and I will summarise
    everyone else's thoughts and suggestions and together with my own ideas we
    will see if we can't add a few items to MrRat's TODO list.

    PPS - I have a minor 'analysis' of APF's mod file system - I will probably
    post that next, before I wrap up the process with the aforementioned
    summary.
    _________________
    DeanMarshall.co.uk - Professional Researcher - UK Computer Store

    Back to top


    Nintendo
    Patron of all things Cheesy



    Joined: 30 Nov 2002
    Posts: 1583
    Location: Winston, OR, USA
    Posted: Wed Apr 06, 2005 9:37 am Post subject:

    ----------------------------------------------------------------------------
    ----

    Ouch!!!!!!

    1. You can't block main categories using the config file. ie, keep the full
    video game section out, unless I think if you edit one of the files.
    2. Special characters in URLs!!!!! ie those music on CD links. (Not sure how
    they are in the newest version.)
    3. Cache files. I think by default it's on. Blarg!!!! (I know, I know, I'm
    the one who always says it's worthless!!)
    4. Too EASY to set up!!!! Upload files, change permission on two files, go
    to config file, and bang, you are done!!!! There for you get a ton of
    competition!!!
    5. It's free!!!! Start charging every one!!!!!!!!
    6. Amazon.com has a crummy AWS system!!!!! Rumor has it that there allready
    working on AWS 4, even with AWS 3 covered with bugs!!!
    _________________
    Amazon/Webmaster Services (AWS)
    Public Domain Content
    News Rant

    Back to top


    Dean
    Patron of all things Cheesy



    Joined: 26 Jul 2004
    Posts: 1588
    Location: Lancaster, Lancashire, United Kingdom
    Posted: Thu Apr 07, 2005 7:51 pm Post subject:

    ----------------------------------------------------------------------------
    ----

    Nintendo wrote:
    Ouch!!!!!!
    Do you think I was harsh - I hope I haven't offended MrRat. I expect if
    someone delivered such an analysis of something I had put so much time and
    effort into I would probably not go back to the project. Usually when he
    goes this quiet he comes back with some amazing new features. I hope the
    latter is the case rather than the former.

    Nintendo wrote:

    1. You can't block main categories using the config file. ie, keep the full
    video game section out, unless I think if you edit one of the files.
    Not quite with you here. SearchIndexes can be excluded individually in
    config.cgi on a per country basis.
    I expect that you mean something else and that I am being dim.

    Nintendo wrote:
    2. Special characters in URLs!!!!! ie those music on CD links. (Not sure how
    they are in the newest version.)
    Not sure what the current status is on this one - I am no longer on the
    B?le[ea]ding edge - MrRat's work rate is too high for me.

    Nintendo wrote:
    3. Cache files. I think by default it's on. Blarg!!!! (I know, I know, I'm
    the one who always says it's worthless!!)
    I know you are the one who calls it worthless - I wrote a rant telling you
    not to be so rude the last time (but held off from posting while I counted
    to 10) in the meantime MrR posted with an emoticon sticking his tongue out
    or similar. So I figured he had had his say. I know you find the cache file
    error prone in your 'unique' situation. By the way - have you tried any of
    the bandwidth throttling server modules to calm Google down when it hits you
    hard? I don't think G penalises based on server load, and the lower
    performance demands might make the cache more stable.

    Nintendo wrote:
    4. Too EASY to set up!!!! Upload files, change permission on two files, go
    to config file, and bang, you are done!!!! There for you get a ton of
    competition!!!
    5. It's free!!!! Start charging every one!!!!!!!!
    I have some sympathy here. I think it should be distributed in a crippled
    format that only those with a modicum of programming knowledge can resolve.

    Still there are some who still struggle with setup - usually total scripting
    newbies or people with unusual server configuarations.

    Nintendo wrote:
    6. Amazon.com has a crummy AWS system!!!!! Rumor has it that there allready
    working on AWS 4, even with AWS 3 covered with bugs!!!
    I am not sure if there is some subtle humour at work here (too subtle for me
    if so) or whether your numbering is deliberately off by one for some other
    reason. As I understand it APF4 is based on AWS/ECS4. Then again I am
    probably missing something.

    Your point about Amazon is valid though - I posted elsewhere today that
    having read "expendable's" posts on the Amazon boards (although he does seem
    argumentative in the extreme at times) I am warming to his overall
    conclusions. I think the developement of AWS/ECS is both a means of gaining
    favourable press coverage and a potential future revenue stream that we are
    just the guinea pigs for. In addition our sites gain Amazon backlinks in
    Google which doesn't do Amazon any harm.

    Most computer developments approach some sort of respectable stability by
    version 3 and maturity by version 4. I am not convinced that Amazon's ECS is
    close to either of these definitions.


    Having said all that, fair play to MrRat for all of his efforts. I have a
    fair understanding of what is involved in providing this project (although I
    admit I don't have the discipline or commitment - or frankly the ability -
    to do anything similar) I respect and thank him for all of his efforts, even
    moreso because of my own failings.

    Dean.
    _________________
    DeanMarshall.co.uk - Professional Researcher - UK Computer Store

    Back to top


    Nintendo
    Patron of all things Cheesy



    Joined: 30 Nov 2002
    Posts: 1583
    Location: Winston, OR, USA
    Posted: Fri Apr 08, 2005 1:20 am Post subject:

    ----------------------------------------------------------------------------
    ----

    :::: Do you think I was harsh - I hope I haven't offended MrRat.

    Ouch as in your post here was much bigger than the other one!

    :::::::::Not quite with you here. SearchIndexes can be excluded individually
    in config.cgi on a per country basis. I expect that you mean something else
    and that I am being dim.

    Now I think I can see.....

    Limit US SearchIndexes to browse (comma separated and case sensitive)

    (The 'COUNTRY SearchIndexes' links there are dead links now, so I'm not
    sure.)

    Ie, don't want a books section, or don't want the DVD section, or video game
    section. I'm guessing that is where to take it out.

    :::I know you are the one who calls it worthless - I wrote a rant telling
    you not to be so rude the last time (but held off from posting while I
    counted to 10)

    Ouch!!! You should of posted it!!!! I probably deserverd it since I flame
    the cache all the time!

    :::I know you find the cache file error prone in your 'unique' situation.

    At some where around 20 stores, at the rate of each cache crashing an
    average of about every six months, that would be a crash every 9 days, and
    the server wasn't at all happy when that happened! I had to go to technical
    support to figure it out when it happened the first time! So yah, I
    understand why the web hosting companies ban the script when this
    occures!!!! If I was a web hosting company I would of done the same thing,
    unless I knew how simple it was to fix it!!

    ::::By the way - have you tried any of the bandwidth throttling server
    modules to calm Google down when it hits you hard?

    Nope. I've never had to worry about bandwidth, I got a terabyte of that a
    month and don't even use close to that, not even 1/3rd of that a month!

    :::I am not sure if there is some subtle humour at work here (too subtle for
    me if so) or whether your numbering is deliberately off by one for some
    other reason

    er my brain is off by one year!!! Rumor has it that there allready working
    on AWS 5, even with AWS 4 being covered with bugs!!!

    :::having read "expendable's" posts on the Amazon boards (although he does
    seem argumentative in the extreme at times)

    He HATES me, BIG time!!!!!

    ::In addition our sites gain Amazon backlinks in Google which doesn't do
    Amazon any harm.

    Unless you use MrRats script, which uses a redirect link.
    _________________
    Amazon/Webmaster Services (AWS)
    Public Domain Content
    News Rant

    Back to top


    MrRat
    The Big Cheese



    Joined: 28 Nov 2000
    Posts: 3764
    Location: Alabama
    Posted: Mon Apr 11, 2005 10:33 am Post subject:

    ----------------------------------------------------------------------------
    ----

    Nintendo wrote:
    :::: Do you think I was harsh - I hope I haven't offended MrRat.

    Ouch as in your post here was much bigger than the other one!



    rotflmao


    Quote:
    :::I know you are the one who calls it worthless - I wrote a rant telling
    you not to be so rude the last time (but held off from posting while I
    counted to 10)

    Ouch!!! You should of posted it!!!! I probably deserverd it since I flame
    the cache all the time!


    i think the idea of leaving the cache off by default is worth considering.

    i don't think Amazon would like that thought. they really want my script to
    cache.

    Back to top


    Nintendo
    Patron of all things Cheesy



    Joined: 30 Nov 2002
    Posts: 1583
    Location: Winston, OR, USA
    Posted: Mon Apr 11, 2005 7:24 pm Post subject:

    ----------------------------------------------------------------------------
    ----

    Or just place a <H1> warning next to the cache option.

    <H1>Turn this off if you don't want your site crashing every three or four
    months generating a mad E-Mail from your web host!!! Don't use with
    mod_rewrite. Don't use if the Googlebot likes your store. Dont use if your
    store doesn't get over 10,000 pageviews a day. Don't use if....... </H1>
    _________________
    Amazon/Webmaster Services (AWS)
    Public Domain Content
    News Rant

    Back to top


    Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6
    Months1 Year Oldest FirstNewest First


    Absolute Freebies Forums Forum Index -> MrRat's Script Support All
    times are GMT - 6 Hours

    Page 1 of 1
    Watch this topic for replies

    Jump to: Select a forum AbsoluteFreebies.com----------------General
    DiscussionFreebies MrRat.com----------------MrRat's Script SupportModsBeta
    TestingGeneralGraphics

    You can post new topics in this forum
    You can reply to topics in this forum
    You can edit your posts in this forum
    You can delete your posts in this forum
    You can vote in polls in this forum




    Powered by phpBB ? 2001, 2005 phpBB Group



    ----------------------------------------------------------------------------
    ----

    Home | Freebies | Freeware | Gaming | Forum | CGI Scripts | Site Search
    ----------------------------------------------------------------------------
    ----
    OttawaTrade, Sep 13, 2005
    #4
  5. OttawaTrade

    OttawaTrade Guest

    Copy from http://www.absolutefreebies.com/phpBB2/viewtopic.php?t=7960371

    Support this web site by purchasing from one of our Amazon affiliate stores:
    Canada | France | Germany | Japan | United Kingdom | United States


    Absolute Freebies Forums
    Cursing is forbidden.
    Posting of affiliate codes by Newbies is forbidden.
    Nudity is forbidden (not by me, by hosting company, i rather like nudity).
    Body modification of housepets is forbidden (except for Bonsai Kittens).
    FAQ Search Memberlist Chat (0) Usergroups
    Profile You have no new messages Log out [ qllonceagain ]


    APF causes server down?
    Goto page 1, 2 Next

    Absolute Freebies Forums Forum Index -> MrRat's Script Support
    View previous topic :: View next topic
    Author Message
    qllonceagain
    Cheese Cube



    Joined: 13 Apr 2004
    Posts: 59

    Posted: Sun Sep 11, 2005 6:52 pm Post subject: APF causes server down?

    ----------------------------------------------------------------------------
    ----

    I added some mod to APF. But load is never heavy. APF is the only script and
    the only thing running on my server.

    It kept getting crashed for no reason. Can not find a thing in log file or
    breakin etc.

    I tried with 2 providers, both using AMD chips. I evern replaced RAMs, but
    still keep happening.

    Does anyone's APF with MOD cause crash like that?

    What is your load average?

    Back to top


    Dean
    Patron of all things Cheesy



    Joined: 26 Jul 2004
    Posts: 1588
    Location: Lancaster, Lancashire, United Kingdom
    Posted: Sun Sep 11, 2005 7:06 pm Post subject:

    ----------------------------------------------------------------------------
    ----

    What mod was it? Hope it is not one of mine.

    I have about 20 mods running on my APF installations - on shared hosting
    accounts, and never a problem. Well hardly ever. About twice per year I get
    a corrupted cache file that results high memory and CPU usage temporarily.
    It also results in a perl core dump - I suspect this is due to some CPU
    usage limit on the account. The core dump kills the APF process freeing the
    server's memory (I think).

    Corrupt cache is number one cause of high CPU and memory usage.

    If the problem is rapid calls from Search Engine Bots, and you have your own
    server, then you can use bandwidth throttling technologies to slow down the
    bots.

    Dean.
    _________________
    DeanMarshall.co.uk - Professional Researcher - UK Computer Store

    Back to top


    qllonceagain
    Cheese Cube



    Joined: 13 Apr 2004
    Posts: 59

    Posted: Sun Sep 11, 2005 7:13 pm Post subject:

    ----------------------------------------------------------------------------
    ----

    i use mod_rewrite.
    the script is crashing servers for the last 3 months.
    will cache crash feed.cgi or the whole server?

    when the server is crashed, the load/cpu/memory is never high.
    load like 0.4
    cpu ike 25%
    memory is like 40M out of 512M


    by the way, which shared hosting allow you to run this banned script?

    Back to top


    qllonceagain
    Cheese Cube



    Joined: 13 Apr 2004
    Posts: 59

    Posted: Sun Sep 11, 2005 11:09 pm Post subject:

    ----------------------------------------------------------------------------
    ----

    Does anyone know why turning on Cache will cause the crash every too often?

    Can we stop the crash maybe adding some error catch in the perl program?

    Back to top


    Dean
    Patron of all things Cheesy



    Joined: 26 Jul 2004
    Posts: 1588
    Location: Lancaster, Lancashire, United Kingdom
    Posted: Mon Sep 12, 2005 8:41 am Post subject:

    ----------------------------------------------------------------------------
    ----

    qllonceagain wrote:
    Does anyone know why turning on Cache will cause the crash every too often?
    Yep - the process of causing the crash is pretty well understood. The cache
    uses a 'technology' called DBM within Perl, which uses a single file as a
    database. The DBM 'technology' uses a rather basic file locking mechanism
    which is susceptible to corruption when multiple instances of the script try
    writing to the DB at the same time.

    qllonceagain wrote:
    Can we stop the crash maybe adding some error catch in the perl program?

    That won't work - once the problem occurs the script 'spins' trying to read
    the cache which is corrupted, I think internal links or instructions within
    the DB file point to each other causing an endless loop while being read.
    Imagine a file of infinite length being read into a finite amount of memory.
    Perl never gets a chance to evaluate anything at this point.

    I have suggested an alternative approach, and will soon begin testing of a
    basic implementation which stores the xml from each call to Amazon in its
    own file - thus reducing the risk of corruption. This will almost certainly
    be less efficient at retrieval from disk, but hopefully the improved
    'reliability' will be worth it.

    Dean.
    _________________
    DeanMarshall.co.uk - Professional Researcher - UK Computer Store

    Back to top


    qllonceagain
    Cheese Cube



    Joined: 13 Apr 2004
    Posts: 59

    Posted: Mon Sep 12, 2005 3:47 pm Post subject:

    ----------------------------------------------------------------------------
    ----

    Will MySQL be helpful here? ...

    I see there is a cache.lock. Unix based system should also have a file write
    lock. Why won't they prevent it?

    Back to top


    mishmash
    Grade AA Cheese



    Joined: 09 Oct 2004
    Posts: 224

    Posted: Mon Sep 12, 2005 5:27 pm Post subject:

    ----------------------------------------------------------------------------
    ----

    A loop or some other error in your .htaccess could be causing these
    symptoms...

    Though it should give a server "500" error not crash the server.

    Look at your error logs - do you have lots of server errors as well as the
    crashes?
    _________________
    http://www.online-buy.co.uk http://knives.buy-kitchen.co.uk
    http://www.reviewsreviews.co.uk http://www.gadgets-buy.co.uk
    http://web4.000025.net/~shop1co/cgi-bin/reviews.cgi

    Back to top


    Dean
    Patron of all things Cheesy



    Joined: 26 Jul 2004
    Posts: 1588
    Location: Lancaster, Lancashire, United Kingdom
    Posted: Mon Sep 12, 2005 5:29 pm Post subject:

    ----------------------------------------------------------------------------
    ----

    qllonceagain wrote:
    Will MySQL be helpful here? ...

    It could be an option if someone wants to implement a solution based on a
    real database - but it would create an extra dependency that MrRat wants to
    avoid. The script is entirely self contained, relying only on modules
    distributed with the script or part of the standard Perl distribution.

    qllonceagain wrote:
    I see there is a cache.lock. Unix based system should also have a file write
    lock. Why won't they prevent it?
    Not sure - Unix is a foreign beast to me. I am not sure how strict OS
    filesystem locks are, I believe they are 'advisory'. Additionally there is
    probably the risk of a 'race condition' where a file is opened before the
    lock is applied - in between times another instance of the script could get
    access. Additionally file system corruption from outside the script could
    occur.

    I think a DBI implementation is regarded as being a more rigorous solution
    but again I am out of my depth - this is just something I read somewhere -
    the DBI FAQ I think!

    Dean.
    _________________
    DeanMarshall.co.uk - Professional Researcher - UK Computer Store

    Back to top


    Dean
    Patron of all things Cheesy



    Joined: 26 Jul 2004
    Posts: 1588
    Location: Lancaster, Lancashire, United Kingdom
    Posted: Tue Sep 13, 2005 11:05 am Post subject:

    ----------------------------------------------------------------------------
    ----

    There was an early version of APF4 that could go into an infinite loop when
    no browsenodes were returned for sub menus. I don't know which version it
    was but I wonder whether that could be the cause of your problems -
    especially when UK browsenodes aren't working.

    Dean.
    _________________
    DeanMarshall.co.uk - Professional Researcher - UK Computer Store

    Back to top


    Frank71
    Patron of all things Cheesy



    Joined: 15 Sep 2003
    Posts: 565
    Location: Germany
    Posted: Tue Sep 13, 2005 11:16 am Post subject: hm

    ----------------------------------------------------------------------------
    ----

    Hm, I'm not the server expert, but I have a very good server and I had
    tousands of crashs last months / years.
    Its a question of visitors ANF all these hobby spiders which will call your
    scripts in seconds.

    Every one who is using mod_rewrite and is suxessful will get this day, most
    times if he is really earning money.
    I was kicked in past from 4 hosts before I rent my own server and also than
    I changed it two times, because all are to small for the script

    By the way, mysql will crash normally at the first before http is losing
    control.

    Regards
    Frank
    _________________
    Sport 25
    Online Verkauf
    With live Ebay XML results !

    Back to top


    Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6
    Months1 Year Oldest FirstNewest First


    Absolute Freebies Forums Forum Index -> MrRat's Script Support All
    times are GMT - 6 Hours
    Goto page 1, 2 Next
    Page 1 of 2
    Watch this topic for replies

    Jump to: Select a forum AbsoluteFreebies.com----------------General
    DiscussionFreebies MrRat.com----------------MrRat's Script SupportModsBeta
    TestingGeneralGraphics

    You can post new topics in this forum
    You can reply to topics in this forum
    You can edit your posts in this forum
    You can delete your posts in this forum
    You can vote in polls in this forum




    Powered by phpBB ? 2001, 2005 phpBB Group



    ----------------------------------------------------------------------------
    ----

    Home | Freebies | Freeware | Gaming | Forum | CGI Scripts | Site Search
    ----------------------------------------------------------------------------
    ----
    OttawaTrade, Sep 13, 2005
    #5
  6. OttawaTrade

    OttawaTrade Guest

    OttawaTrade, Sep 13, 2005
    #6
  7. OttawaTrade

    OttawaTrade Guest


    > A search for "DBM" on search.cpan.org yields 374 hits. Perhaps you could
    > be so kind as to tell us exactly what DBM module you are talking about.
    >
    > There are thousands of different kinds of servers. Please tell us what
    > server you are talking about.
    >
    >
    > > It is reported by many people
    > > an unprivileged perl script can load the memory and crash the server.

    How
    > > do I fix this, since many working on the project can not fix. I hope I
    > > can find more perl experts here.
    > >
    > > http://www.absolutefreebies.com/phpBB2/viewtopic.php?t=7960371
    > >
    > > Does anyone know why turning on Cache will cause the crash every too
    > > often?

    >
    > A search on "Cache" in search.cpan.org yields 2228 hits. What precisely
    > are you talking about?
    >
    >
    > Xho


    Well documenated here on the link.
    http://www.absolutefreebies.com/phpBB2/viewtopic.php?t=7960371

    Do you hav any idea why it happens?
    OttawaTrade, Sep 13, 2005
    #7
  8. OttawaTrade

    Guest

    "OttawaTrade" <> wrote:
    > > A search for "DBM" on search.cpan.org yields 374 hits. Perhaps you
    > > could be so kind as to tell us exactly what DBM module you are talking
    > > about.
    > >
    > > There are thousands of different kinds of servers. Please tell us what
    > > server you are talking about.
    > >

    ....
    > >
    > > A search on "Cache" in search.cpan.org yields 2228 hits. What
    > > precisely are you talking about?
    > >
    > >
    > > Xho

    >
    > Well documenated here on the link.

    ....
    > Do you hav any idea why it happens?


    No, in fact it is not well documented at that link. That link clarifies
    none of the things which I requested that you clarify. Maybe if I clicked
    around the various hierarchies there and googled all the terms from those
    pages, I could figure out what the Hell you are talking about. But I am
    not going to do that. This is a Perl group, not an "AFP" group or "Mr Rat"
    group.

    As this is the sixth post you've made here with absolutely no useful
    information, but rather with links to irrelevant pages, I will have to
    assume that Sinan was right.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
    , Sep 13, 2005
    #8
  9. OttawaTrade

    OttawaTrade Guest


    > > > A search for "DBM" on search.cpan.org yields 374 hits. Perhaps you
    > > > could be so kind as to tell us exactly what DBM module you are talking
    > > > about.
    > > >
    > > > There are thousands of different kinds of servers. Please tell us

    what
    > > > server you are talking about.
    > > >

    > ...
    > > >
    > > > A search on "Cache" in search.cpan.org yields 2228 hits. What
    > > > precisely are you talking about?
    > > >
    > > >
    > > > Xho

    > >
    > > Well documenated here on the link.

    > ...
    > > Do you hav any idea why it happens?

    >
    > No, in fact it is not well documented at that link. That link clarifies
    > none of the things which I requested that you clarify. Maybe if I clicked
    > around the various hierarchies there and googled all the terms from those
    > pages, I could figure out what the Hell you are talking about. But I am
    > not going to do that. This is a Perl group, not an "AFP" group or "Mr

    Rat"
    > group.
    >
    > As this is the sixth post you've made here with absolutely no useful
    > information, but rather with links to irrelevant pages, I will have to
    > assume that Sinan was right.
    >
    > Xho
    >


    It has nothing to do with search.cpan.org. It is a general perl programming
    question.
    OttawaTrade, Sep 13, 2005
    #9
  10. OttawaTrade

    Keith Keller Guest

    On 2005-09-13, OttawaTrade <> wrote:
    >
    > It has nothing to do with search.cpan.org. It is a general perl programming
    > question.


    If it is a general perl programming question, then post the problem
    following the Posting Guidelines (short but complete script that
    replicates the problem). You probably shouldn't expect an answer to
    your question otherwise.

    --keith

    --
    -francisco.ca.us
    (try just my userid to email me)
    AOLSFAQ=http://wombat.san-francisco.ca.us/cgi-bin/fom
    see X- headers for PGP signature information
    Keith Keller, Sep 13, 2005
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Matt Johnson
    Replies:
    0
    Views:
    1,267
    Matt Johnson
    Aug 8, 2003
  2. Meron Lavie

    How To Stop Caching By ISP Proxy Servers

    Meron Lavie, Apr 27, 2004, in forum: ASP .Net
    Replies:
    4
    Views:
    8,776
    anilakhwaja
    Jul 16, 2007
  3. Peter Richards

    Perl script crashing at lockfile ?

    Peter Richards, Oct 1, 2003, in forum: Perl Misc
    Replies:
    3
    Views:
    114
    Malcolm Dew-Jones
    Oct 2, 2003
  4. Topher

    How to create a DBM database in Perl

    Topher, Nov 27, 2003, in forum: Perl Misc
    Replies:
    6
    Views:
    95
    Chris Mattern
    Nov 28, 2003
  5. Replies:
    4
    Views:
    152
    Steve C
    Feb 26, 2010
Loading...

Share This Page