Probelm to post XML data in a loop. First time XML is posted, second time data is getting truncated.

Discussion in 'Perl Misc' started by vamsi.aluru@gmail.com, Feb 9, 2006.

  1. Guest

    Hi,
    I am posting an XML document to an asp applicaiton. This XML
    document, i am sending in a loop. That is, if the XML document is big,
    I am sending the first 50 records first time, then next 50 records and
    so on.

    The problem is, I am able to send first 50, but next 50 is
    causing problem and the asp application is receiving the truncated XML.
    When I send 20 after 20, first two times the xml goes fine, but the
    third time the XML is truncating.

    Any idea what is the problem? I am using post, so the size
    should not be a problem. And more over the XML size is not more than
    50k each time.

    I am using the below code. I am psing only part of the code
    where I am posting. if i post the full code, it will be very big.


    for($x=0;$x<=$count-1;$x++){

    # getting the XML content.
    $XML_OUT=$xmlcnt[$x];


    print 'xml==========='.$XML_OUT;
    $ua = LWP::UserAgent->new;
    $ua->timeout(36000);
    $ua->max_size(1000000000);
    # post the XML to the below url.
    $req = HTTP::Request->new(POST =>
    'http://viz-ams-001/facility/mapping.aspx');
    # set the conetnt type of the post.
    $req->content_type('application/x-www-form-urlencoded');
    $req->content_length(1000000000);

    #while posting, assign the XML to Doc variable. This variable acts
    a form variable.
    $req->content('Doc='.$XML_OUT);
    # Pass request to the user agent and get a response back
    $res = $ua->request($req);

    # if response is available do the below.
    if ($res->is_success) {

    print 'success';
    }
    else{

    print 'error';
    }
     
    , Feb 9, 2006
    #1
    1. Advertising

  2. Guest

    I forgot to add the first lines of the code. i am using these at the
    begning.

    use CGI;
    use DBI;
    use DBD::Oracle;
    use LWP::UserAgent;
    use HTTP::Request::Common;
    use HTTP::Headers;
    use HTTP::Headers::Util qw(split_header_words);
     
    , Feb 9, 2006
    #2
    1. Advertising

  3. J. Gleixner Guest

    Re: Probelm to post XML data in a loop. First time XML is posted,second time data is getting truncated. Please help.

    wrote:

    > The problem is, I am able to send first 50, but next 50 is
    > causing problem and the asp application is receiving the truncated XML.
    > When I send 20 after 20, first two times the xml goes fine, but the
    > third time the XML is truncating.


    Where is it being truncated? Before you send it or on the server's side?

    >
    > Any idea what is the problem? I am using post, so the size
    > should not be a problem. And more over the XML size is not more than
    > 50k each time.
    >
    > I am using the below code. I am psing only part of the code
    > where I am posting. if i post the full code, it will be very big.
    >
    >
    > for($x=0;$x<=$count-1;$x++){
    >
    > # getting the XML content.
    > $XML_OUT=$xmlcnt[$x];
    >
    >
    > print 'xml==========='.$XML_OUT;
    > $ua = LWP::UserAgent->new;
    > $ua->timeout(36000);
    > $ua->max_size(1000000000);
    > # post the XML to the below url.
    > $req = HTTP::Request->new(POST =>
    > 'http://viz-ams-001/facility/mapping.aspx');
    > # set the conetnt type of the post.
    > $req->content_type('application/x-www-form-urlencoded');
    > $req->content_length(1000000000);
    >
    > #while posting, assign the XML to Doc variable. This variable acts
    > a form variable.
    > $req->content('Doc='.$XML_OUT);
    > # Pass request to the user agent and get a response back
    > $res = $ua->request($req);
    >
    > # if response is available do the below.
    > if ($res->is_success) {
    >
    > print 'success';
    > }
    > else{
    >
    > print 'error';
    > }
    >


    Untested...

    my $ua = LWP::UserAgent->new;
    my $req = HTTP::Request->new(
    POST=>'http://viz-ams-001/facility/mapping.aspx');
    $req->content_type('application/x-www-form-urlencoded');

    for my $XML_OUT ( @xmlcnt )
    {
    print "xml===========$XML_OUT\n";

    $req->content( "Doc=$XML_OUT" );
    my $res = $ua->request( $req );
    if ($res->is_success)
    {
    print "success\n";
    }
    else
    {
    print 'error: ', $res->as_string, "\n";
    }
    }


    Possibly XML_OUT contains other data? From perldoc HTTP::Request:

    "Note that the content should be a string of bytes. Strings in perl
    can contain characters outside the range of a byte. The Encode
    module can be used to turn such strings into a string of bytes."
     
    J. Gleixner, Feb 9, 2006
    #3
  4. Guest

    ASP application is receiving the truncated xml string. so it should
    have got truncated when the post happens.


    Possibly XML_OUT contains other data?

    I dont understand this statement. XML_OUT contains characters, which
    for a XML string. If the problem is with XML, then first 50 shd also
    fail.

    I think size is what creating the problem. but again, post sholud not
    have a size limit.

    is there a way to flush the data after the post happens? or is there a
    way to create a new request each time, closing the old one.?



    J. Gleixner wrote:
    > wrote:
    >
    > > The problem is, I am able to send first 50, but next 50 is
    > > causing problem and the asp application is receiving the truncated XML.
    > > When I send 20 after 20, first two times the xml goes fine, but the
    > > third time the XML is truncating.

    >
    > Where is it being truncated? Before you send it or on the server's side?
    >
    > >
    > > Any idea what is the problem? I am using post, so the size
    > > should not be a problem. And more over the XML size is not more than
    > > 50k each time.
    > >
    > > I am using the below code. I am psing only part of the code
    > > where I am posting. if i post the full code, it will be very big.
    > >
    > >
    > > for($x=0;$x<=$count-1;$x++){
    > >
    > > # getting the XML content.
    > > $XML_OUT=$xmlcnt[$x];
    > >
    > >
    > > print 'xml==========='.$XML_OUT;
    > > $ua = LWP::UserAgent->new;
    > > $ua->timeout(36000);
    > > $ua->max_size(1000000000);
    > > # post the XML to the below url.
    > > $req = HTTP::Request->new(POST =>
    > > 'http://viz-ams-001/facility/mapping.aspx');
    > > # set the conetnt type of the post.
    > > $req->content_type('application/x-www-form-urlencoded');
    > > $req->content_length(1000000000);
    > >
    > > #while posting, assign the XML to Doc variable. This variable acts
    > > a form variable.
    > > $req->content('Doc='.$XML_OUT);
    > > # Pass request to the user agent and get a response back
    > > $res = $ua->request($req);
    > >
    > > # if response is available do the below.
    > > if ($res->is_success) {
    > >
    > > print 'success';
    > > }
    > > else{
    > >
    > > print 'error';
    > > }
    > >

    >
    > Untested...
    >
    > my $ua = LWP::UserAgent->new;
    > my $req = HTTP::Request->new(
    > POST=>'http://viz-ams-001/facility/mapping.aspx');
    > $req->content_type('application/x-www-form-urlencoded');
    >
    > for my $XML_OUT ( @xmlcnt )
    > {
    > print "xml===========$XML_OUT\n";
    >
    > $req->content( "Doc=$XML_OUT" );
    > my $res = $ua->request( $req );
    > if ($res->is_success)
    > {
    > print "success\n";
    > }
    > else
    > {
    > print 'error: ', $res->as_string, "\n";
    > }
    > }
    >
    >
    > Possibly XML_OUT contains other data? From perldoc HTTP::Request:
    >
    > "Note that the content should be a string of bytes. Strings in perl
    > can contain characters outside the range of a byte. The Encode
    > module can be used to turn such strings into a string of bytes."
     
    , Feb 10, 2006
    #4
  5. J. Gleixner Guest

    Re: Probelm to post XML data in a loop. First time XML is posted,second time data is getting truncated. Please help.

    wrote:
    > ASP application is receiving the truncated xml string. so it should
    > have got truncated when the post happens.


    Why are you top posting??

    Since you are printing out the data, is it 'truncated' before you send it?

    >
    >
    > Possibly XML_OUT contains other data?
    >
    > I dont understand this statement. XML_OUT contains characters, which
    > for a XML string. If the problem is with XML, then first 50 shd also
    > fail.

    What's 'shd'? Please use full words.

    I was only posting a possible issue. You know the data, so if you're
    sure it's OK, then that's fine with me. It seems odd that the
    application would work for a bunch of requests, then seem to fail. That
    usually points to an issue with the data.

    >
    > I think size is what creating the problem. but again, post sholud not
    > have a size limit.
    >
    > is there a way to flush the data after the post happens? or is there a
    > way to create a new request each time, closing the old one.?

    That's not needed. This is an HTTP request. It sends the data and
    receives a response. You are creating a new request each time.

    If you can verify that what you're sending is accurate [ by debugging,
    maybe .. use Data::Dumper; print Dumper( $req ) or print
    $req->content(); after you set it, and looking at what's being POSTed. ]
    then it sounds like an issue on the other side.
     
    J. Gleixner, Feb 10, 2006
    #5
  6. Ian Wilson Guest

    Re: Probelm to post XML data in a loop. First time XML is posted,second time data is getting truncated. Please help.

    Please don't top-post, please intersperse your replies after the
    relevant parts of the quotation - like this ...

    (note I had to reformat the posting after applying "rewrap" in my
    newsreader - consequently the perl scripts may be laid out differently
    than originally posted)

    wrote:
    >
    > J. Gleixner wrote:
    >
    >> wrote:
    >>
    >>
    >>> The problem is, I am able to send first 50, but next 50 is
    >>> causing problem and the asp application is receiving the
    >>> truncated XML. When I send 20 after 20, first two times the xml
    >>> goes fine, but the third time the XML is truncating.

    >>
    >> Where is it being truncated? Before you send it or on the server's
    >> side?
    >>

    > ASP application is receiving the truncated xml string. so it should
    > have got truncated when the post happens.


    It would help if you could better describe the difference between the
    output you expected and the output you actually got.

    I guess your XML is too long to post here (50 lines) - do you get a
    similar problem when you decrease your chunk size, e.g. to 5 lines?

    >>
    >>> Any idea what is the problem? I am using post, so the size should
    >>> not be a problem. And more over the XML size is not more than 50k
    >>> each time.
    >>>
    >>> I am using the below code. I am psing only part of the code where
    >>> I am posting. if i post the full code, it will be very big.
    >>>


    If you read the posting guidelines, you will find that it is recommended
    that you edit your program to produce the smallest working program that
    can be used by others to reproduce your problem.


    >>>
    >>> for($x=0;$x<=$count-1;$x++){
    >>> # getting the XML content.
    >>> $XML_OUT=$xmlcnt[$x];


    I think I'd write the above as
    foreach my $XML_OUT (@xmlcnt) {

    >>>
    >>> print 'xml==========='.$XML_OUT;


    Did this print what you expected? If not, can you describe the difference?

    >>> $ua = LWP::UserAgent->new;
    >>> $ua->timeout(36000);
    >>> $ua->max_size(1000000000); # post the XML to the below url.
    >>> $req = HTTP::Request->new(
    >>> POST => 'http://viz-ams-001/facility/mapping.aspx');
    >>>
    >>> # set the conetnt type of the post.
    >>> $req->content_type('application/x-www-form-urlencoded');
    >>> $req->content_length(1000000000);


    That seems bogus to me. An Internet maxim is "be conservative in what
    you produce but be generous in what you accept". If I could, I'd put the
    correct size into the content_length.

    >>>
    >>> # while posting, assign the XML to Doc variable. This variable
    >>> # acts a form variable.
    >>> $req->content('Doc='.$XML_OUT);
    >>> # Pass request to the user agent and get a response back
    >>> $res =>>> $ua->request($req);
    >>>
    >>> # if response is available do the below.
    >>> if ($res->is_success) {
    >>> print 'success';
    >>> } else {
    >>> print 'error';
    >>> }
    >>>

    >>
    >> Untested...
    >>
    >> my $ua = LWP::UserAgent->new;
    >> my $req = HTTP::Request->new(
    >> POST=>'http://viz-ams-001/facility/mapping.aspx');
    >> $req->content_type('application/x-www-form-urlencoded');
    >>
    >> for my $XML_OUT ( @xmlcnt ) {
    >> print "xml===========$XML_OUT\n";
    >> $req->content( "Doc=$XML_OUT" );
    >> my $res = $ua->request( $req );
    >> if ($res->is_success) { print "success\n"; }
    >> else { print 'error: ', $res->as_string, "\n"; }
    >> }
    >>
    >>
    >> Possibly XML_OUT contains other data? From perldoc HTTP::Request:
    >>
    >> "Note that the content should be a string of bytes. Strings in
    >> perl can contain characters outside the range of a byte. The
    >> Encode module can be used to turn such strings into a string of
    >> bytes."

    >
    >
    > I dont understand this statement. XML_OUT contains characters, which
    > for a XML string. If the problem is with XML, then first 50 shd also
    > fail.


    I'm not sure I understand it either, it may be a reference to multi-byte
    character encodings such as UTF-8. Unless I was certain that my XML is
    in a single-byte encoding such as ISO-8859-1 (Latin-1), I'd read the
    HTTP::Request docs carefully and try to find out whether HTTP standards
    have anything to say on the subject.

    This illustrates why it is useful to post a (small) sample set of data
    that others can examine and/or use for testing your code.

    > I think size is what creating the problem. but again, post sholud not
    > have a size limit.


    You've promised the ASP service that you are sending it one gigabyte of
    XML. Maybe the ASP service hasn't been tested with a gigabyte of data,
    maybe it has some internal limit? I wouldn't assume it is just going to
    ignore the content-length header.

    > is there a way to flush the data after the post happens? or is there
    > a way to create a new request each time, closing the old one.?
    >


    Unless I am mistaken, your original code was doing this.

    It should be relatively simple to create a Perl CGI script that can be
    targetted instead of the ASP URI - this might help in testing.
     
    Ian Wilson, Feb 10, 2006
    #6
  7. Guest

    I figured out the problem.

    I am having an '&' symbol in one of the data elements. I am actually
    replacing that with '&amp;'. But the actual problem was, I did not
    escpae the xml before I post to asp application. Because of this, the
    xml is getting truncated from the place where & symbol is there.

    I changed the code like this.

    $req->content('Doc='.CGI::escape($XML_OUT));

    CGI::escacpe was not there previously.

    Now the code works.

    Thanks everybody for replies.


    Ian Wilson wrote:
    > Please don't top-post, please intersperse your replies after the
    > relevant parts of the quotation - like this ...
    >
    > (note I had to reformat the posting after applying "rewrap" in my
    > newsreader - consequently the perl scripts may be laid out differently
    > than originally posted)
    >
    > wrote:
    > >
    > > J. Gleixner wrote:
    > >
    > >> wrote:
    > >>
    > >>
    > >>> The problem is, I am able to send first 50, but next 50 is
    > >>> causing problem and the asp application is receiving the
    > >>> truncated XML. When I send 20 after 20, first two times the xml
    > >>> goes fine, but the third time the XML is truncating.
    > >>
    > >> Where is it being truncated? Before you send it or on the server's
    > >> side?
    > >>

    > > ASP application is receiving the truncated xml string. so it should
    > > have got truncated when the post happens.

    >
    > It would help if you could better describe the difference between the
    > output you expected and the output you actually got.
    >
    > I guess your XML is too long to post here (50 lines) - do you get a
    > similar problem when you decrease your chunk size, e.g. to 5 lines?
    >
    > >>
    > >>> Any idea what is the problem? I am using post, so the size should
    > >>> not be a problem. And more over the XML size is not more than 50k
    > >>> each time.
    > >>>
    > >>> I am using the below code. I am psing only part of the code where
    > >>> I am posting. if i post the full code, it will be very big.
    > >>>

    >
    > If you read the posting guidelines, you will find that it is recommended
    > that you edit your program to produce the smallest working program that
    > can be used by others to reproduce your problem.
    >
    >
    > >>>
    > >>> for($x=0;$x<=$count-1;$x++){
    > >>> # getting the XML content.
    > >>> $XML_OUT=$xmlcnt[$x];

    >
    > I think I'd write the above as
    > foreach my $XML_OUT (@xmlcnt) {
    >
    > >>>
    > >>> print 'xml==========='.$XML_OUT;

    >
    > Did this print what you expected? If not, can you describe the difference?
    >
    > >>> $ua = LWP::UserAgent->new;
    > >>> $ua->timeout(36000);
    > >>> $ua->max_size(1000000000); # post the XML to the below url.
    > >>> $req = HTTP::Request->new(
    > >>> POST => 'http://viz-ams-001/facility/mapping.aspx');
    > >>>
    > >>> # set the conetnt type of the post.
    > >>> $req->content_type('application/x-www-form-urlencoded');
    > >>> $req->content_length(1000000000);

    >
    > That seems bogus to me. An Internet maxim is "be conservative in what
    > you produce but be generous in what you accept". If I could, I'd put the
    > correct size into the content_length.
    >
    > >>>
    > >>> # while posting, assign the XML to Doc variable. This variable
    > >>> # acts a form variable.
    > >>> $req->content('Doc='.$XML_OUT);
    > >>> # Pass request to the user agent and get a response back
    > >>> $res =>>> $ua->request($req);
    > >>>
    > >>> # if response is available do the below.
    > >>> if ($res->is_success) {
    > >>> print 'success';
    > >>> } else {
    > >>> print 'error';
    > >>> }
    > >>>
    > >>
    > >> Untested...
    > >>
    > >> my $ua = LWP::UserAgent->new;
    > >> my $req = HTTP::Request->new(
    > >> POST=>'http://viz-ams-001/facility/mapping.aspx');
    > >> $req->content_type('application/x-www-form-urlencoded');
    > >>
    > >> for my $XML_OUT ( @xmlcnt ) {
    > >> print "xml===========$XML_OUT\n";
    > >> $req->content( "Doc=$XML_OUT" );
    > >> my $res = $ua->request( $req );
    > >> if ($res->is_success) { print "success\n"; }
    > >> else { print 'error: ', $res->as_string, "\n"; }
    > >> }
    > >>
    > >>
    > >> Possibly XML_OUT contains other data? From perldoc HTTP::Request:
    > >>
    > >> "Note that the content should be a string of bytes. Strings in
    > >> perl can contain characters outside the range of a byte. The
    > >> Encode module can be used to turn such strings into a string of
    > >> bytes."

    > >
    > >
    > > I dont understand this statement. XML_OUT contains characters, which
    > > for a XML string. If the problem is with XML, then first 50 shd also
    > > fail.

    >
    > I'm not sure I understand it either, it may be a reference to multi-byte
    > character encodings such as UTF-8. Unless I was certain that my XML is
    > in a single-byte encoding such as ISO-8859-1 (Latin-1), I'd read the
    > HTTP::Request docs carefully and try to find out whether HTTP standards
    > have anything to say on the subject.
    >
    > This illustrates why it is useful to post a (small) sample set of data
    > that others can examine and/or use for testing your code.
    >
    > > I think size is what creating the problem. but again, post sholud not
    > > have a size limit.

    >
    > You've promised the ASP service that you are sending it one gigabyte of
    > XML. Maybe the ASP service hasn't been tested with a gigabyte of data,
    > maybe it has some internal limit? I wouldn't assume it is just going to
    > ignore the content-length header.
    >
    > > is there a way to flush the data after the post happens? or is there
    > > a way to create a new request each time, closing the old one.?
    > >

    >
    > Unless I am mistaken, your original code was doing this.
    >
    > It should be relatively simple to create a Perl CGI script that can be
    > targetted instead of the ASP URI - this might help in testing.
     
    , Feb 14, 2006
    #7
  8. wrote in
    news::

    [ Please do not top-post and please do quote only the relevant parts of
    previous messages ]

    > I figured out the problem.


    We are happy for you (honestly) but you have failed. The reason you
    posted here was because you someone else to find the problem, thereby
    saving you time and effort. That did not happen for a number of reasons.

    > I am having an '&' symbol in one of the data elements. I am actually
    > replacing that with '&amp;'. But the actual problem was, I did not
    > escpae the xml before I post to asp application. Because of this, the
    > xml is getting truncated from the place where & symbol is there.


    Because you did not post a short but complete program that could run
    easily by others, and a good sample of the data, a lot of us did not
    respond to your question.

    Those who responded had a hard time getting information from you.

    I hope you'll read and follow the posting guidelines the next time.

    Sinan

    --
    A. Sinan Unur <>
    (reverse each component and remove .invalid for email address)

    comp.lang.perl.misc guidelines on the WWW:
    http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
     
    A. Sinan Unur, Feb 14, 2006
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ian Lotinsky

    POST data truncated in Internet Explorer 6

    Ian Lotinsky, Apr 30, 2006, in forum: ASP .Net
    Replies:
    1
    Views:
    696
    Ian Lotinsky
    May 2, 2006
  2. arnuld
    Replies:
    6
    Views:
    635
    arnuld
    Sep 5, 2007
  3. Curt Hibbs
    Replies:
    0
    Views:
    108
    Curt Hibbs
    May 13, 2004
  4. Replies:
    3
    Views:
    439
    Ted Zlatanov
    Aug 10, 2006
  5. Isaac Won
    Replies:
    9
    Views:
    405
    Ulrich Eckhardt
    Mar 4, 2013
Loading...

Share This Page