Curl/Perl http post performanc issue

Discussion in 'Perl Misc' started by wkhedr, Aug 3, 2006.

  1. wkhedr

    wkhedr Guest

    We have a Perl program that has to do thousands of secure http post
    requests to a web server as efficient as we could.
    The web server sends back a big XML structure in every request.

    We used Curl because it allowed us to deal with secured requests.

    Curl by default uses http 1.1.

    When we ask Curl to use http 1.0, we get average response time of 40
    millisecond during our stress tests.

    The response time is good but the problem is it's not releasing the
    CPU's while waiting for the response. It's supposed to be like a disk
    I/O where it should not be using the CPU. So the CPU utilization goes
    up to 99% with multithreads of testing.

    When we run the tests using http 1.1, the response time was 2 seconds
    :)

    Using http 1.1 and asking Curl to disable "Expect: 100-continue" by
    using header "Expect:",
    we were able to reach response time of 120 milliseconds and the CPU
    usage was cut by almost 90%.

    The question is: how can we get the best of the two settings: have the
    same low CPU usage as http 1.1 and reduce the response time from 120 ms
    to 40 ms as http 1.0?

    Thanks
     
    wkhedr, Aug 3, 2006
    #1
    1. Advertising

  2. wkhedr wrote:
    > We have a Perl program that has to do thousands of secure http post
    > requests to a web server as efficient as we could.
    > The web server sends back a big XML structure in every request.
    >
    > We used Curl because it allowed us to deal with secured requests.
    >
    > Curl by default uses http 1.1.
    >
    > When we ask Curl to use http 1.0, we get average response time of 40
    > millisecond during our stress tests.
    >
    > The response time is good but the problem is it's not releasing the
    > CPU's while waiting for the response. It's supposed to be like a disk
    > I/O where it should not be using the CPU. So the CPU utilization goes
    > up to 99% with multithreads of testing.
    >
    > When we run the tests using http 1.1, the response time was 2 seconds
    > :)
    >
    > Using http 1.1 and asking Curl to disable "Expect: 100-continue" by
    > using header "Expect:",
    > we were able to reach response time of 120 milliseconds and the CPU
    > usage was cut by almost 90%.
    >
    > The question is: how can we get the best of the two settings: have the
    > same low CPU usage as http 1.1 and reduce the response time from 120 ms
    > to 40 ms as http 1.0?



    p.s. the version of curl is:
    $ ./curl --version
    curl 7.15.4 (powerpc-ibm-aix5.2.0.0) libcurl/7.15.4 OpenSSL/0.9.8b
    Protocols: tftp ftp telnet dict ldap http file https ftps
    Features: IPv6 Largefile NTLM SSL

    and the Perl module we are using is:
    WWW::Curl::Easy 3.01
     
    it_says_BALLS_on_your forehead, Aug 3, 2006
    #2
    1. Advertising

  3. wkhedr

    Ben Morrow Guest

    Quoth "wkhedr" <>:
    > We have a Perl program that has to do thousands of secure http post
    > requests to a web server as efficient as we could.
    > The web server sends back a big XML structure in every request.
    >
    > We used Curl because it allowed us to deal with secured requests.


    Have you tried using LWP?

    > Curl by default uses http 1.1.
    >
    > When we ask Curl to use http 1.0, we get average response time of 40
    > millisecond during our stress tests.
    >
    > The response time is good but the problem is it's not releasing the
    > CPU's while waiting for the response. It's supposed to be like a disk
    > I/O where it should not be using the CPU. So the CPU utilization goes
    > up to 99% with multithreads of testing.
    >
    > When we run the tests using http 1.1, the response time was 2 seconds
    > :)
    >
    > Using http 1.1 and asking Curl to disable "Expect: 100-continue" by
    > using header "Expect:",
    > we were able to reach response time of 120 milliseconds and the CPU
    > usage was cut by almost 90%.
    >
    > The question is: how can we get the best of the two settings: have the
    > same low CPU usage as http 1.1 and reduce the response time from 120 ms
    > to 40 ms as http 1.0?


    The rest of this is not really relevant to Perl, and would be better
    asked on some Curl-related list or group.

    Ben

    --
    I must not fear. Fear is the mind-killer. I will face my fear and
    I will let it pass through me. When the fear is gone there will be
    nothing. Only I will remain.
    Frank Herbert, 'Dune'
     
    Ben Morrow, Aug 3, 2006
    #3
  4. wkhedr

    wkhedr Guest

    LWP did the same numbers 120ms!


    Ben Morrow wrote:
    > Quoth "wkhedr" <>:
    > > We have a Perl program that has to do thousands of secure http post
    > > requests to a web server as efficient as we could.
    > > The web server sends back a big XML structure in every request.
    > >
    > > We used Curl because it allowed us to deal with secured requests.

    >
    > Have you tried using LWP?
    >
    > > Curl by default uses http 1.1.
    > >
    > > When we ask Curl to use http 1.0, we get average response time of 40
    > > millisecond during our stress tests.
    > >
    > > The response time is good but the problem is it's not releasing the
    > > CPU's while waiting for the response. It's supposed to be like a disk
    > > I/O where it should not be using the CPU. So the CPU utilization goes
    > > up to 99% with multithreads of testing.
    > >
    > > When we run the tests using http 1.1, the response time was 2 seconds
    > > :)
    > >
    > > Using http 1.1 and asking Curl to disable "Expect: 100-continue" by
    > > using header "Expect:",
    > > we were able to reach response time of 120 milliseconds and the CPU
    > > usage was cut by almost 90%.
    > >
    > > The question is: how can we get the best of the two settings: have the
    > > same low CPU usage as http 1.1 and reduce the response time from 120 ms
    > > to 40 ms as http 1.0?

    >
    > The rest of this is not really relevant to Perl, and would be better
    > asked on some Curl-related list or group.
    >
    > Ben
    >
    > --
    > I must not fear. Fear is the mind-killer. I will face my fear and
    > I will let it pass through me. When the fear is gone there will be
    > nothing. Only I will remain.
    > Frank Herbert, 'Dune'
     
    wkhedr, Aug 3, 2006
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. post data using curl

    , Feb 12, 2007, in forum: Python
    Replies:
    0
    Views:
    393
  2. John D Giotta

    HTTP POST File without cURL

    John D Giotta, Sep 9, 2009, in forum: Python
    Replies:
    3
    Views:
    1,860
    John Giotta
    Sep 11, 2009
  3. Ben Johnson

    cURL in ruby? Faster than Net::HTTP?

    Ben Johnson, Aug 23, 2006, in forum: Ruby
    Replies:
    17
    Views:
    375
    Francis Cianfrocca
    Aug 25, 2006
  4. Kad Kerforn
    Replies:
    2
    Views:
    145
    Kad Kerforn
    Aug 28, 2010
  5. Fiaz Idris
    Replies:
    13
    Views:
    1,996
    ifiaz
    Mar 17, 2005
Loading...

Share This Page