XMLHTTPRequest streaming data

Discussion in 'Javascript' started by googlegroups@scottsavarese.com, Jun 22, 2005.

  1. Guest

    I am playing with the XMLHTTPRequest method to perform client/server
    transactions. I have it set up right now so that when readyState is 4,
    it takes the XML and processes it. This works great until there is alot
    of data. In that case, the user will have to wait for the data to come
    back which may take a minute or so.

    I don't want the user to have to wait. Is it possible for javascript to
    periodically (while still receiving more data) stop and display what it
    has received thus far? I guess this would be considered a type of
    streaming.

    In mozilla/firefox, I have read that I can use readyState 3 to run my
    callback function every 4096 bytes. I can then take those 4K, parse
    them, and then continue on. However I have also read that IE cannot do
    this. Since I need this to work in IE, is there a workaround?

    Thanks,
    Scott
    , Jun 22, 2005
    #1
    1. Advertising

  2. Matt Kruse Guest

    wrote:
    > I don't want the user to have to wait. Is it possible for javascript
    > to periodically (while still receiving more data) stop and display
    > what it has received thus far?


    Not that I know of.
    One suggestion - use JSON instead of XML. Or use your own compact data
    structure.

    XML is verbose, and you may be able to cut down your transmission time if
    you use a data format that is more compact.

    --
    Matt Kruse
    http://www.JavascriptToolbox.com
    http://www.AjaxToolbox.com
    Matt Kruse, Jun 22, 2005
    #2
    1. Advertising

  3. wrote:


    > In mozilla/firefox, I have read that I can use readyState 3 to run my
    > callback function every 4096 bytes. I can then take those 4K, parse
    > them, and then continue on. However I have also read that IE cannot do
    > this. Since I need this to work in IE, is there a workaround?


    You should be able to access responseText once readyState gives you the
    value 3 for interactive.
    From the MSXML docs for XMLHTTP:

    (3) INTERACTIVE Some data has been received. You can call the
    responseBody and responseText properties to get the current partial
    results.

    --

    Martin Honnen
    http://JavaScript.FAQTs.com/
    Martin Honnen, Jun 22, 2005
    #3
  4. Martin Honnen wrote:


    > You should be able to access responseText once readyState gives you the
    > value 3 for interactive.
    > From the MSXML docs for XMLHTTP:
    >
    > (3) INTERACTIVE Some data has been received. You can call the
    > responseBody and responseText properties to get the current partial
    > results.


    I have done some tests here trying to access responseText when
    readyState is signalled as 3 but unfortunately MSXML then always gives
    an exception that the data necessary is not yet available so in this
    case it looks like the docs are promising more than is possible.

    --

    Martin Honnen
    http://JavaScript.FAQTs.com/
    Martin Honnen, Jun 22, 2005
    #4
  5. Matt Kruse Guest

    Martin Honnen wrote:
    > (3) INTERACTIVE Some data has been received. You can call the
    > responseBody and responseText properties to get the current partial
    > results.


    However, with XML it needs to be well-formed and parsed before being
    available, afaik.
    You can get to the responseText, but it won't be XML that can be
    manipulated.

    --
    Matt Kruse
    http://www.JavascriptToolbox.com
    http://www.AjaxToolbox.com
    Matt Kruse, Jun 22, 2005
    #5
  6. Guest

    Matt Kruse wrote:
    > Not that I know of.
    > One suggestion - use JSON instead of XML. Or use your own compact data
    > structure.


    I'll look into JSON. Right now our XML is pretty compact. The structure
    is very little overhead as compared to the data itself. I am more
    worried by the shear number of data rows we may return than anything
    else.

    I was playing around with Mozilla. It doesn't seem to hard to keep
    track of how much data I've read so far and use substring to extract
    the latest. Then parse the latest information and present it to the
    user.

    However with IE, I can't use the readyState variable. Plus I tried
    using a timer (every second) to read the responseText and IE comes back
    with an error. Is there anyway to get the partial response from IE or
    do I have to wait until it completes? Is there any hack I can do to IE
    to get it to work?

    Thanks,
    Scott
    , Jun 22, 2005
    #6
  7. Jim Ley Guest

    On 22 Jun 2005 12:40:41 -0700, wrote:

    >However with IE, I can't use the readyState variable. Plus I tried
    >using a timer (every second) to read the responseText and IE comes back
    >with an error. Is there anyway to get the partial response from IE or
    >do I have to wait until it completes? Is there any hack I can do to IE
    >to get it to work?


    No, just use the IFRAME remote scripting method and JSON, it's much
    more reliable and cross-platform for achieving what you want, although
    it's the server that has to do the breaking up.

    Jim.
    Jim Ley, Jun 22, 2005
    #7
  8. Guest

    Martin Honnen wrote:
    > I have done some tests here trying to access responseText when
    > readyState is signalled as 3 but unfortunately MSXML then always gives
    > an exception that the data necessary is not yet available so in this
    > case it looks like the docs are promising more than is possible.


    I get the same thing... That is exactly what I am trying to get around.

    Scott
    , Jun 22, 2005
    #8
  9. Guest

    Jim Ley wrote:
    > No, just use the IFRAME remote scripting method and JSON, it's much
    > more reliable and cross-platform for achieving what you want, although
    > it's the server that has to do the breaking up.


    I really like the idea of JSON. Unfortunately in this case, I need my
    protocol to be XML. The code can't be language specific. If I was just
    using a web browser to obtain the information from the server, then
    JSON can be used, and may even be better, however I also have a perl
    client side module, and I have a feeling my users are going to want a
    Java one as well. How is JSON with security... I run the returned value
    in a eval block. Of the examples I've seen everyone seems to trust that
    the data returned is valid and not harmful. Can you craft a JSON reply
    that does something malicious?

    As for the IFRAME piece, I did some research on it... Is there an
    example on line of people doing the type of streaming that I am looking
    for? Also, it looks like the IFRAME seems to imply a need to use GET as
    opposed to POST (as it updates the history). Is that true?

    Scott
    , Jun 22, 2005
    #9
  10. Jim Ley Guest

    On 22 Jun 2005 13:37:20 -0700, wrote:

    >Jim Ley wrote:
    >> No, just use the IFRAME remote scripting method and JSON, it's much
    >> more reliable and cross-platform for achieving what you want, although
    >> it's the server that has to do the breaking up.

    >
    >I really like the idea of JSON. Unfortunately in this case, I need my
    >protocol to be XML.


    No, you need to provide an XML protocol, that doesn't mean the one you
    provide to your website is XML, a server XML->json is entirely
    appropriate solution - especially as IFRAME's (which require JSON to
    do this) are the only way you're going to get the incremental approach
    in IE.

    As for a link, no idea, you just need to write out an HTML document
    with

    <script type="text/javascript">
    .... block 1
    </script>

    <script type="text/javascript">
    .... block 2
    </script>

    <script type="text/javascript">
    .... block 3
    </script>

    etc.

    Jim.
    Jim Ley, Jun 22, 2005
    #10
  11. writes:

    [...]

    > I really like the idea of JSON. Unfortunately in this case, I need
    > my protocol to be XML. The code can't be language specific. If I was
    > just using a web browser to obtain the information from the server,
    > then JSON can be used, and may even be better, however I also have a
    > perl client side module, and I have a feeling my users are going to
    > want a Java one as well.


    Why is that a problem? JSON has a simple syntax, which ought to be
    easy to parse (into any language which has associative arrays, anyway)
    by hand even if you can't find an existing implementation.

    > How is JSON with security... I run the returned value in a eval
    > block. Of the examples I've seen everyone seems to trust that the
    > data returned is valid and not harmful. Can you craft a JSON reply
    > that does something malicious?


    Think of the overall solution: the JSON reply can do anything that the
    web browser lets it do. But we can get the browser to do all that
    anyway, since the Javascript that's evaling the JSON came from the
    server in the first place.

    How about splitting up the message, so rather than one big message
    that you try and read in a stream, you read a number of messages which
    you make well-formed in some way? Otherwise whatever solution you
    pick, you're going to end up with the client getting a part of a
    message, and it's not clear that you can usefully do anything with
    that.
    Bruce Stephens, Jun 23, 2005
    #11
  12. wrote:
    > Jim Ley wrote:
    > > No, just use the IFRAME remote scripting method and JSON, it's much
    > > more reliable and cross-platform for achieving what you want, although
    > > it's the server that has to do the breaking up.

    >
    > I really like the idea of JSON. Unfortunately in this case, I need my
    > protocol to be XML. The code can't be language specific. If I was just
    > using a web browser to obtain the information from the server, then
    > JSON can be used, and may even be better, however I also have a perl
    > client side module, and I have a feeling my users are going to want a
    > Java one as well. How is JSON with security... I run the returned value
    > in a eval block. Of the examples I've seen everyone seems to trust that
    > the data returned is valid and not harmful. Can you craft a JSON reply
    > that does something malicious?


    Just brainstorming here, because this looks interesting.

    1. If your audience is so open to many and various clients, how hard
    could it be to get them to install another scripting language? If they
    already have Perl, maybe ActiveState's PerlScript could do what you're
    looking for, here. Theoretically that should be an easy port from the
    stand-alone client you already have, to boot.

    2. You could also just dump back fully-formed HTML, which is loaded in
    a visible frame or iframe. Depending on how you do it (avoid tables),
    most browsers will "stream" this to the screen in the way you're
    thinking.

    3. If you don't mind some ugly scripting on both client- and
    server-side, you can have your server-side script break the message
    apart so that the client has to request it in (well-formed) segments.
    This means that your client-side scripting has to know what to request,
    and your server-side obviously has to be able to fulfill those requests
    with exactly the expected response. This could also increase your load
    time quite a bit, as each segment is a separate HTTP request with all
    the overhead that goes with.

    4. Return a multipart (MIME) response, with each part being text/html
    enclosing segments of your data as JavaScript. Each section of JS code
    will execute independently (but asynchronously) of the others, and
    because the execution completes before the next HTML body loads, it may
    cause the browser to render the changes. I've never done anything like
    that before, but it might work. You may also be able to do something
    similar with your current XML approach, but I don't know how well (or
    if at all) XMLHTTPRequest would handle a multipart response, even if
    all the message parts are well-formed XML with the right content-type.
    Christopher J. Hahn, Jun 23, 2005
    #12
  13. Guest

    Christopher J. Hahn wrote:
    > Just brainstorming here, because this looks interesting.
    >
    > 1. If your audience is so open to many and various clients, how hard
    > could it be to get them to install another scripting language? If they
    > already have Perl, maybe ActiveState's PerlScript could do what you're
    > looking for, here. Theoretically that should be an easy port from the
    > stand-alone client you already have, to boot.
    >

    Practically impossible. There are hundreds of developers that would
    want to use the tool we are creating (both the web side and the client
    API piece). Those developers don't have time to learn another language,
    and have the tendency to exert pressure on my management to get things
    done their way. The API has to be language independent. I don't want to
    write it twice, so I would prefer one API for both the web piece and
    whatever client side modules get written.

    > 2. You could also just dump back fully-formed HTML, which is loaded in
    > a visible frame or iframe. Depending on how you do it (avoid tables),
    > most browsers will "stream" this to the screen in the way you're
    > thinking.
    >

    I will have to think about this. I currently send back some data that
    doesn't initially get displayed so i need to figure out a way to embed
    it in the data 'tree' in such a way as to make it hidden (css?). It
    would really add a lot to the structure of what is returned back though
    as I now how to put a lot of stuff in div blocks to create a structure
    of sorts. It would also make it harder to manipulate the data (re-sort
    on different columns, etc.)

    > 4. Return a multipart (MIME) response, with each part being text/html
    > enclosing segments of your data as JavaScript. Each section of JS code
    > will execute independently (but asynchronously) of the others, and
    > because the execution completes before the next HTML body loads, it may
    > cause the browser to render the changes. I've never done anything like
    > that before, but it might work. You may also be able to do something
    > similar with your current XML approach, but I don't know how well (or
    > if at all) XMLHTTPRequest would handle a multipart response, even if
    > all the message parts are well-formed XML with the right content-type.


    This URL http://www.xulplanet.com/tutorials/mozsdk/serverpush.php (look
    at the section on "multipart") suggests that a solution like this might
    be possible (in theory I have not tried it out). I could hypothetically
    split the data into its rows and send each row as its own part. Using
    the onload handler each row would then get handled seperately. There is
    however no mention on how IE will handle this.

    Thanks,
    Scott
    , Jun 23, 2005
    #13
  14. Guest

    wrote:
    > This URL http://www.xulplanet.com/tutorials/mozsdk/serverpush.php (look
    > at the section on "multipart") suggests that a solution like this might
    > be possible (in theory I have not tried it out). I could hypothetically
    > split the data into its rows and send each row as its own part. Using
    > the onload handler each row would then get handled seperately. There is
    > however no mention on how IE will handle this.


    Just tested this... It doesn't work for IE. XMLHTTPRequest in IE does
    not support multipart.

    Scott
    , Jun 23, 2005
    #14
  15. wrote:
    > wrote:
    > > This URL http://www.xulplanet.com/tutorials/mozsdk/serverpush.php (look
    > > at the section on "multipart") suggests that a solution like this might
    > > be possible (in theory I have not tried it out). I could hypothetically
    > > split the data into its rows and send each row as its own part. Using
    > > the onload handler each row would then get handled seperately. There is
    > > however no mention on how IE will handle this.

    >
    > Just tested this... It doesn't work for IE. XMLHTTPRequest in IE does
    > not support multipart.
    >
    > Scott


    Shame about the multipart-- that would've been a cool solution, I
    thought.

    But as regards extending the browser's language support, I wasn't
    thinking of making the API output PerlScript (that would be practically
    Satanic for other languages to deal with), but rather using client-side
    browser-hosted PerlScript as your parser/DOM manipulator. Perl is an
    excellent language for any parsing need, and it may operate
    asynchronously enough that you can cause rendering to occur
    simultaneously. But I wouldn't know-- I've never touched PerlScript,
    just PERL. I know it has support on Windows, but other platforms I am
    unsure of. See http://www.activestate.com

    If it works out, you could still use XML over HTTP, but using
    PerlScript as your parser.

    The other option, which I would prefer to that, is this:
    Use a hidden iframe to load the data from the server.
    The server's response would still be multipart, but XMLHTTPRequest
    support for it would no longer be an issue.
    The responses should be fully-formed HTML documents, but with nothing
    in the body. Instead, it would primarily be outputing JSON (easy to
    parse) structures.
    A second script tag would also include certain function definitions.
    The <body> tag would include an onload to call a loader function in the
    parent document (the one that contains the iframe) that actually clones
    the JSON structures.

    For people who want it to be language-independent:
    The HTML is predictable and easy to parse, and the tags should be
    considered comments outside of a browser context.
    The JSON is simple and should be analogous to data structures available
    in most modern languages. Parsing JSON is incredibly simple-- far
    simpler in my opinion than, say, XML. It also has the advantage of
    extreme ease of transferability to JavaScript (ahem) and Java, and
    other languages with similar syntax for declaring literals.

    Just strip out the html ( s/<.*?>//g ) and parse the JSON and you've
    got your data.

    The only issue is the complexity of the multipart.
    Christopher J. Hahn, Jun 23, 2005
    #15
  16. Guest

    Christopher J. Hahn wrote:
    > The other option, which I would prefer to that, is this:
    > Use a hidden iframe to load the data from the server.
    > The server's response would still be multipart, but XMLHTTPRequest
    > support for it would no longer be an issue.
    > The responses should be fully-formed HTML documents, but with nothing
    > in the body. Instead, it would primarily be outputing JSON (easy to
    > parse) structures.
    > A second script tag would also include certain function definitions.
    > The <body> tag would include an onload to call a loader function in the
    > parent document (the one that contains the iframe) that actually clones
    > the JSON structures.
    >


    Can the iframes do POST requests? If I only get to manipulate the URL
    it calls, it seems that only GET requests would be possible. Also, how
    would I do the streaming? Multipart and onload handlers?

    Thanks,
    Scott
    , Jun 24, 2005
    #16
  17. Jim Ley Guest

    On 24 Jun 2005 07:23:21 -0700, wrote:

    >Can the iframes do POST requests?


    Yes, just submit a form in it.

    > Also, how
    >would I do the streaming? Multipart and onload handlers?


    No, you'd just but script blocks in and have them call, no event.

    Jim.
    Jim Ley, Jun 24, 2005
    #17
  18. wrote:
    > Christopher J. Hahn wrote:
    > > The other option, which I would prefer to that, is this:
    > > Use a hidden iframe to load the data from the server.
    > > The server's response would still be multipart, but XMLHTTPRequest
    > > support for it would no longer be an issue.
    > > The responses should be fully-formed HTML documents, but with nothing
    > > in the body. Instead, it would primarily be outputing JSON (easy to
    > > parse) structures.
    > > A second script tag would also include certain function definitions.
    > > The <body> tag would include an onload to call a loader function in the
    > > parent document (the one that contains the iframe) that actually clones
    > > the JSON structures.
    > >

    >
    > Can the iframes do POST requests? If I only get to manipulate the URL
    > it calls, it seems that only GET requests would be possible. Also, how
    > would I do the streaming? Multipart and onload handlers?
    >
    > Thanks,
    > Scott


    Just to clarify, in the responses from the server, there should only be
    one script "block". The other script tag (technically a block, but
    leave it empty) should reference an external .js file.

    This way, you can put your JSON data in the one block, and in the empty
    one define functions for the parent document to handle the data. Call
    the onload handler from the body tag.

    The reason you'd do it this way is to satisfy the language independence
    requirement. Because all JS unrelated to the JSON structures is defined
    or referenced entirely in tag attributes, when you strip the HTML with
    that simple s/<.*?>//g regex, you're left with nothing but JSON and
    whitespace. No pesky function definitions cluttering up the JSON.

    Or you might find it easier to make the server-side logic accept an
    optional query-string variable requesting an HTML response, but
    defaulting to XML.

    Whatever works. I like the multipart idea, though.
    Christopher J. Hahn, Jun 25, 2005
    #18
  19. Jim Ley Guest

    On 24 Jun 2005 21:25:34 -0700, "Christopher J. Hahn"
    <> wrote:

    > wrote:
    >> Can the iframes do POST requests? If I only get to manipulate the URL
    >> it calls, it seems that only GET requests would be possible. Also, how
    >> would I do the streaming? Multipart and onload handlers?
    >>

    >Just to clarify, in the responses from the server, there should only be
    >one script "block". The other script tag (technically a block, but
    >leave it empty) should reference an external .js file.
    >
    >This way, you can put your JSON data in the one block, and in the empty
    >one define functions for the parent document to handle the data. Call
    >the onload handler from the body tag.


    This will not however be "streaming", waiting for onload is a
    pointless reduced quality degradation of the page.

    Jim.
    Jim Ley, Jun 25, 2005
    #19
  20. Jim Ley wrote:
    > On 24 Jun 2005 21:25:34 -0700, "Christopher J. Hahn"
    > <> wrote:
    >
    > > wrote:
    > >> Can the iframes do POST requests? If I only get to manipulate the URL
    > >> it calls, it seems that only GET requests would be possible. Also, how
    > >> would I do the streaming? Multipart and onload handlers?
    > >>

    > >Just to clarify, in the responses from the server, there should only be
    > >one script "block". The other script tag (technically a block, but
    > >leave it empty) should reference an external .js file.
    > >
    > >This way, you can put your JSON data in the one block, and in the empty
    > >one define functions for the parent document to handle the data. Call
    > >the onload handler from the body tag.

    >
    > This will not however be "streaming", waiting for onload is a
    > pointless reduced quality degradation of the page.
    >
    > Jim.


    Of course it wouldn't be streaming. Lacking a real and worthwhile
    cross-browser way to implement streaming given the requirements of the
    project, we're left with emulation.

    Whether waiting for onload is pointless is another matter entirely.
    Since you've neither asked for an explanation nor given one yourself,
    there's little more to say than that.
    Christopher J. Hahn, Jun 26, 2005
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    1
    Views:
    373
    Dag Sunde
    May 22, 2007
  2. Noozer
    Replies:
    7
    Views:
    156
    Noozer
    Apr 15, 2006
  3. Jarson
    Replies:
    5
    Views:
    468
    Lasse Reichstein Nielsen
    May 10, 2005
  4. Vijay
    Replies:
    3
    Views:
    123
    Jim Ley
    Apr 3, 2006
  5. libsfan01
    Replies:
    5
    Views:
    229
    Jeff North
    Dec 20, 2006
Loading...

Share This Page