Problems flushing my buffer! (perl)

Discussion in 'Perl Misc' started by Nigel, Oct 3, 2008.

  1. Nigel

    Nigel Guest

    Hi there,

    I hope you can help - I'm writing a perl program and I want it report
    back to the browser
    on its progress. I read the perl faq and saw that by setting $| true
    the buffer would be
    flushed each time I printed instead of waiting until the program
    terminates. But I can't
    get it to work.

    I'm expecting the web page to say that it's on the job, then every
    second (for 10 seconds) to say Please Wait and then at the end to say
    it's finished. But what happens is I wait ten seconds for a response
    and the whole lot appears at once. Any advice would be VERY welcome.

    Here is my VERY basic program...

    #!/usr/bin/perl -w
    use strict;
    use CGI::Carp qw(fatalsToBrowser);
    use CGI qw:)standard escape escapeHTML);

    # Force the buffer to flush
    $|++;

    my $i;

    print << "EOF";

    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://
    www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
    <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
    <head>
    <title>Test Buffer</title>
    </head>
    <body>
    Working on it!<br />
    EOF

    for ($i=0;$i<10;$i++) {
    print "Please wait!<br />\n";
    sleep 1;
    };

    print << "EOF";
    All Done!
    </body>
    </html>

    EOF

    exit (0);
    Nigel, Oct 3, 2008
    #1
    1. Advertising

  2. Nigel <> wrote:
    >Hi there,
    >
    >I hope you can help - I'm writing a perl program and I want it report
    >back to the browser
    >on its progress. I read the perl faq and saw that by setting $| true
    >the buffer would be
    >flushed each time I printed instead of waiting until the program
    >terminates. But I can't
    >get it to work.
    >
    >I'm expecting the web page to say that it's on the job, then every
    >second (for 10 seconds) to say Please Wait and then at the end to say
    >it's finished. But what happens is I wait ten seconds for a response
    >and the whole lot appears at once. Any advice would be VERY welcome.


    The whole lot appears at once _WHERE_?

    Run your sample program from the command line and you will notice,that
    there is the desired one second pause between each "Please wait!<br
    />".

    Of course your web server typically will not transmit the HTTP response
    in such small chuncks but wait until it recieves EOF. And a web browser
    may not display any HTML-page until it has been downloaded completely.
    But neither of these has _ANYTHING_ to do with Perl but would happen
    exactly the same way no matter in which programming language the CGI
    program was written in.

    jue
    Jürgen Exner, Oct 3, 2008
    #2
    1. Advertising

  3. Nigel

    Nigel Guest

    On Oct 3, 1:30 pm, Jürgen Exner <> wrote:
    > Nigel <> wrote:
    > >Hi there,

    >
    > >I hope you can help - I'm writing a perl program and I want it report
    > >back to the browser
    > >on its progress. I read the perl faq and saw that by setting $| true
    > >the buffer would be
    > >flushed each time I printed instead of waiting until the program
    > >terminates. But I can't
    > >get it to work.

    >
    > >I'm expecting the web page to say that it's on the job, then every
    > >second (for 10 seconds) to say Please Wait and then at the end to say
    > >it's finished. But what happens is I wait ten seconds for a response
    > >and the whole lot appears at once. Any advice would be VERY welcome.

    >
    > The whole lot appears at once _WHERE_?
    >
    > Run your sample program from the command line and you will notice,that
    > there is the desired one second pause between each  "Please wait!<br
    > />".
    >
    > Of course your web server typically will not transmit the HTTP response
    > in such small chuncks but wait until it recieves EOF. And a web browser
    > may not display any HTML-page until it has been downloaded completely.
    > But neither of these has _ANYTHING_ to do with Perl but would happen
    > exactly the same way no matter in which programming language the CGI
    > program was written in.
    >
    > jue


    To clarify - it all appears at once in my web browser.

    So it seems I must rephrase my question and perhaps address it to a
    different Group as apparently the problem isn't with my Perl as
    such...but I am writing this in Perl so I guess I still have a Perl
    related question: Is it possible to achieve what I want i.e. to
    request my perl program from my browser and have the program tell me
    the progress it is making from time to time and have that appear in my
    browser in real time? My impression from what I read in the faq (and
    elsewhere) was that such a thing IS possible, but perhaps I've
    misunderstood. In the worked examples I saw, they talked about
    'Forking' - which I understand to mean that one part of the program
    responds to the browser, whilst another part carries on with the
    processing. I have to say I couldn't really get my head around the
    examples I saw which is why I was trying to start off simple! Thanks
    for the response anyway.
    Nigel, Oct 3, 2008
    #3
  4. Nigel

    Larry Guest

    In article
    <>,
    Nigel <> wrote:

    > To clarify - it all appears at once in my web browser.


    I would go something like that:

    #!/perl

    use strict;
    use warnings;

    select( (select ( STDOUT ), $|=1)[0] );

    syswrite STDOUT "Content-type: text/plain\n\n";

    syswrite STDOUT, "hello world!\n";

    sleep 10;
    syswrite STDOUT, "hello world!\n";

    sleep 10;
    syswrite STDOUT, "hello world!\n";

    exit;

    __END__;

    hopefully your web server will support "chunked" content-lenght as you
    do not know how long the response size is going to be...
    Larry, Oct 3, 2008
    #4
  5. Nigel

    Guest

    Nigel <> wrote:
    ....
    > I'm expecting the web page to say that it's on the job, then every
    > second (for 10 seconds) to say Please Wait and then at the end to say
    > it's finished. But what happens is I wait ten seconds for a response
    > and the whole lot appears at once. Any advice would be VERY welcome.
    >
    > Here is my VERY basic program...
    >
    > #!/usr/bin/perl -w
    > use strict;
    > use CGI::Carp qw(fatalsToBrowser);
    > use CGI qw:)standard escape escapeHTML);
    >
    > # Force the buffer to flush
    > $|++;
    >
    > my $i;
    >
    > print << "EOF";
    >
    > <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://
    > www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
    > <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">


    Where is the content type header?

    Without a header, my browser downloads the results as a file and only lets
    me open that file once it is done. Once I add a proper header, the page
    is rendered in my browser incrementally, as you expected.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
    , Oct 3, 2008
    #5
  6. Nigel

    Jim Gibson Guest

    In article
    <>,
    Nigel <> wrote:


    > To clarify - it all appears at once in my web browser.
    >
    > So it seems I must rephrase my question and perhaps address it to a
    > different Group as apparently the problem isn't with my Perl as
    > such...but I am writing this in Perl so I guess I still have a Perl
    > related question: Is it possible to achieve what I want i.e. to
    > request my perl program from my browser and have the program tell me
    > the progress it is making from time to time and have that appear in my
    > browser in real time? My impression from what I read in the faq (and
    > elsewhere) was that such a thing IS possible, but perhaps I've
    > misunderstood. In the worked examples I saw, they talked about
    > 'Forking' - which I understand to mean that one part of the program
    > responds to the browser, whilst another part carries on with the
    > processing. I have to say I couldn't really get my head around the
    > examples I saw which is why I was trying to start off simple! Thanks
    > for the response anyway.


    You need the "client-pull" refresh feature of HTTP that causes your
    browser to periodically request a page (or CGI):

    <http://www.stonehenge.com/merlyn/LinuxMag/col39.html>

    --
    Jim Gibson
    Jim Gibson, Oct 3, 2008
    #6
  7. On 2008-10-03 11:30, Jürgen Exner <> wrote:
    > Nigel <> wrote:
    >>Hi there,
    >>
    >>I hope you can help - I'm writing a perl program and I want it report
    >>back to the browser on its progress. I read the perl faq and saw that
    >>by setting $| true the buffer would be flushed each time I printed
    >>instead of waiting until the program terminates. But I can't get it to
    >>work.
    >>
    >>I'm expecting the web page to say that it's on the job, then every
    >>second (for 10 seconds) to say Please Wait and then at the end to say
    >>it's finished. But what happens is I wait ten seconds for a response
    >>and the whole lot appears at once. Any advice would be VERY welcome.

    >
    > The whole lot appears at once _WHERE_?
    >
    > Run your sample program from the command line and you will notice,that
    > there is the desired one second pause between each "Please wait!<br
    > />".
    >
    > Of course your web server typically will not transmit the HTTP response
    > in such small chuncks but wait until it recieves EOF.


    Why "of course"? The most common web server (Apache) DOES transmit
    responses from CGI scripts as it receives them - it doesn't gather them
    into bigger chunks. Other web servers I've worked with behave the same.
    So which "typical" web server are you thinking about?

    > And a web browser may not display any HTML-page until it has been
    > downloaded completely.


    Right. Although that's extremely rare these days. All the major
    graphical HTML rendering engines (IE, Gecko, KHTML, ...) are able to
    render a page incrementally. However, many text based browsers (e.g.,
    lynx, w3m) aren't - if Nigel is using one of them that might be the
    problem.

    hp
    Peter J. Holzer, Oct 4, 2008
    #7
  8. On 2008-10-03 13:34, Larry <> wrote:
    > In article
    ><>,
    > Nigel <> wrote:
    >> To clarify - it all appears at once in my web browser.

    >
    > I would go something like that:

    [basically the same as the OP's code]

    > hopefully your web server will support "chunked" content-lenght as you
    > do not know how long the response size is going to be...


    Neither the server nor the browser needs to know that for HTTP/1.0. And
    for HTTP/1.1, chunked is mandatory.

    hp
    Peter J. Holzer, Oct 4, 2008
    #8
  9. [A complimentary Cc of this posting was NOT [per weedlist] sent to
    Peter J. Holzer
    <>], who wrote in article <>:
    > Right. Although that's extremely rare these days. All the major
    > graphical HTML rendering engines (IE, Gecko, KHTML, ...) are able to
    > render a page incrementally. However, many text based browsers (e.g.,
    > lynx, w3m) aren't - if Nigel is using one of them that might be the
    > problem.


    lynx is incremental as well...

    Yours,
    Ilya
    Ilya Zakharevich, Oct 4, 2008
    #9
  10. Ilya Zakharevich <> wrote:
    > [A complimentary Cc of this posting was NOT [per weedlist] sent to
    > Peter J. Holzer <>], who wrote in article
    > <>:
    >> Right. Although that's extremely rare these days. All the major
    >> graphical HTML rendering engines (IE, Gecko, KHTML, ...) are able to
    >> render a page incrementally. However, many text based browsers (e.g.,
    >> lynx, w3m) aren't - if Nigel is using one of them that might be the
    >> problem.

    > lynx is incremental as well...


    elinks too...

    --
    Torvalds' goal for Linux is very simple: World Domination
    Eric Pozharski, Oct 5, 2008
    #10
  11. On Fri, 03 Oct 2008 02:57:24 -0700, Nigel wrote:

    > Hi there,
    >
    > I hope you can help - I'm writing a perl program and I want it report
    > back to the browser
    > on its progress. I read the perl faq and saw that by setting $| true the
    > buffer would be
    > flushed each time I printed instead of waiting until the program
    > terminates. But I can't
    > get it to work.
    >
    > I'm expecting the web page to say that it's on the job, then every
    > second (for 10 seconds) to say Please Wait and then at the end to say
    > it's finished. But what happens is I wait ten seconds for a response and
    > the whole lot appears at once. Any advice would be VERY welcome.


    Some versions (all?) of IE needs 256 characters before they will behave
    correctly. Try to print 256 spaces somewhere. I wrote scripts that did
    behave like what you told until I added the spaces.

    HTH,
    M4
    Martijn Lievaart, Oct 5, 2008
    #11
  12. On 2008-10-05 09:21, Eric Pozharski <> wrote:
    > Ilya Zakharevich <> wrote:
    >> <>:
    >>> Right. Although that's extremely rare these days. All the major
    >>> graphical HTML rendering engines (IE, Gecko, KHTML, ...) are able to
    >>> render a page incrementally. However, many text based browsers (e.g.,
    >>> lynx, w3m) aren't - if Nigel is using one of them that might be the
    >>> problem.

    >> lynx is incremental as well...


    Not the version I have here (2.8.7dev9) which appears to be almost the
    newest developer version (there's a 2.8.7dev10 on lynx.isc.org). So I
    wouldn't expect the stable release (2.8.6) to be incremental, either.

    > elinks too...


    Yup. That's the exception I am aware of.

    hp
    Peter J. Holzer, Oct 6, 2008
    #12
  13. Nigel

    Nigel Guest

    On Oct 5, 2:40 pm, Martijn Lievaart <> wrote:
    > On Fri, 03 Oct 2008 02:57:24 -0700, Nigel wrote:
    > > Hi there,

    >
    > > I hope you can help - I'm writing a perl program and I want it report
    > > back to the browser
    > > on its progress. I read the perl faq and saw that by setting $| true the
    > >bufferwould be
    > > flushed each time I printed instead of waiting until the program
    > > terminates. But I can't
    > > get it to work.

    >
    > > I'm expecting the web page to say that it's on the job, then every
    > > second (for 10 seconds) to say Please Wait and then at the end to say
    > > it's finished. But what happens is I wait ten seconds for a response and
    > > the whole lot appears at once. Any advice would be VERY welcome.

    >
    > Some versions (all?) of IE needs 256 characters before they will behave
    > correctly. Try to print 256 spaces somewhere. I wrote scripts that did
    > behave like what you told until I added the spaces.
    >
    > HTH,
    > M4


    On Oct 5, 2:40 pm, Martijn Lievaart <> wrote:
    > On Fri, 03 Oct 2008 02:57:24 -0700, Nigel wrote:
    > > Hi there,

    >
    > > I hope you can help - I'm writing a perl program and I want it report
    > > back to the browser
    > > on its progress. I read the perl faq and saw that by setting $| true the
    > >bufferwould be
    > > flushed each time I printed instead of waiting until the program
    > > terminates. But I can't
    > > get it to work.

    >
    > > I'm expecting the web page to say that it's on the job, then every
    > > second (for 10 seconds) to say Please Wait and then at the end to say
    > > it's finished. But what happens is I wait ten seconds for a response and
    > > the whole lot appears at once. Any advice would be VERY welcome.

    >
    > Some versions (all?) of IE needs 256 characters before they will behave
    > correctly. Try to print 256 spaces somewhere. I wrote scripts that did
    > behave like what you told until I added the spaces.
    >
    > HTH,
    > M4


    Thank you all for your input. In the end I decided to adapt the
    solution suggested by Jim Gibson:

    >You need the "client-pull" refresh feature of HTTP that causes your
    >browser to periodically request a page (or CGI):


    <http://www.stonehenge.com/merlyn/LinuxMag/col39.html>

    My needs are rather specific as I'm developing a web front-ended
    database for a company not a website for use by the public. Anyway my
    approach may interest some others: I have a program that collects the
    parameters with which the User wishes to run the batch process in
    question. Once their input is validated I open a new window with the
    URL of my batch program + the parameters. The batch program in turns
    runs some intitialisation processing before sending a page back to the
    browser. In that page I have an automatic redirect back to my batch
    program with slightly amend parameters so that it now starts to
    process the required records. After a predetermined number of records
    or a period of time (as yet undecided about that) it again sends a
    page to the browser - again with an automatic redirect. This cycle
    continues until the processing is complete at which point the Batch
    Program sends a last page to the browser (without the automatic
    redirect!) to tell the user the result of the process. I use some
    parameters to store information between iterations of the batch
    process to avoid it having to repeat too much processing each time it
    is invoked. I also store some information in cookies so the function
    that 'submits' the batch process can keep track of its progress. If
    the batch process fails or the window is closed, the cookies expire
    and the next time the user runs the submission function it tidies up
    any submitted process(es) that no longer have a cookie i.e. have
    failed in one way or another.

    So in conclusion - I am using the Client-Pull method, but I'm not
    keeping the batch process active all the time with the use of a fork
    as described in the article above. Why? Purely pragmatic reasons - I'm
    a completely self taught perl programer and the article gets into
    aspects of perl and server side prcessing that are well outside my
    level of understanding. The processing time is not critical and I
    don't think it will be huge anyway. In any case the architecture of
    the program is pretty similar to that outlined in the article so if it
    becomes necessary I could change it in the future.

    One particular point about the article that I couldn't get my head
    around is how, when the batch program is re-invoked by the client-
    pull, one avoids that simply starting up another instance of the
    program instead of interupting the existing instance to get an update
    on its progress. Can anyone enlighten me?

    Thanks again for the input!

    Nigel
    Nigel, Oct 6, 2008
    #13
  14. [A complimentary Cc of this posting was NOT [per weedlist] sent to
    Peter J. Holzer
    <>], who wrote in article <>:
    > >>> Right. Although that's extremely rare these days. All the major
    > >>> graphical HTML rendering engines (IE, Gecko, KHTML, ...) are able to
    > >>> render a page incrementally. However, many text based browsers (e.g.,
    > >>> lynx, w3m) aren't - if Nigel is using one of them that might be the
    > >>> problem.
    > >> lynx is incremental as well...

    >
    > Not the version I have here (2.8.7dev9) which appears to be almost the
    > newest developer version (there's a 2.8.7dev10 on lynx.isc.org). So I
    > wouldn't expect the stable release (2.8.6) to be incremental, either.


    Mine is lynx2-8-5-pre4. Maybe you are not aware that lynx will behave
    as you ask it? My config file is about 3K large... E.g., I have

    # Set the threshold # of lines Lynx must render before it
    # redraws the screen in PARTIAL mode. Anything < 0 implies
    # use of the screen size.
    PARTIAL_THRES:6

    Hope this helps,
    Ilya
    Ilya Zakharevich, Oct 6, 2008
    #14
  15. Nigel

    Jim Gibson Guest

    In article
    <>,
    Nigel <> wrote:

    > On Oct 5, 2:40 pm, Martijn Lievaart <> wrote:
    > > On Fri, 03 Oct 2008 02:57:24 -0700, Nigel wrote:
    > > > Hi there,

    > >
    > > > I hope you can help - I'm writing a perl program and I want it report
    > > > back to the browser
    > > > on its progress. I read the perl faq and saw that by setting $| true the
    > > >bufferwould be
    > > > flushed each time I printed instead of waiting until the program
    > > > terminates. But I can't
    > > > get it to work.


    >
    > Thank you all for your input. In the end I decided to adapt the
    > solution suggested by Jim Gibson:
    >
    > >You need the "client-pull" refresh feature of HTTP that causes your
    > >browser to periodically request a page (or CGI):

    >
    > <http://www.stonehenge.com/merlyn/LinuxMag/col39.html>
    >



    > One particular point about the article that I couldn't get my head
    > around is how, when the batch program is re-invoked by the client-
    > pull, one avoids that simply starting up another instance of the
    > program instead of interupting the existing instance to get an update
    > on its progress. Can anyone enlighten me?


    The program uses the session id to indicate that a request is a
    follow-up for a previous new request. The first request does not have a
    session id, so param('session') in line 9 of the sample program returns
    false (undef). Each subsequent refresh request contains the 'session'
    parameter, and the value of that parameter is used to fetch the results
    so far, as in line 11.

    Good luck!

    --
    Jim Gibson
    Jim Gibson, Oct 7, 2008
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Eric

    Flushing the Buffer in .Net

    Eric, Apr 25, 2004, in forum: ASP .Net
    Replies:
    3
    Views:
    602
  2. arnuld
    Replies:
    5
    Views:
    789
    =?ISO-8859-1?Q?Erik_Wikstr=F6m?=
    Mar 18, 2007
  3. Ethan Metsger

    Decorators and buffer flushing

    Ethan Metsger, Feb 28, 2008, in forum: Python
    Replies:
    0
    Views:
    274
    Ethan Metsger
    Feb 28, 2008
  4. Andyza

    Flushing servers buffer

    Andyza, Jul 6, 2005, in forum: ASP General
    Replies:
    3
    Views:
    123
    Andyza
    Jul 7, 2005
  5. tj
    Replies:
    2
    Views:
    143
    Walter Roberson
    May 4, 2004
Loading...

Share This Page