Curl/Perl http post performanc issue

[wkhedr]

We have a Perl program that has to do thousands of secure http post
requests to a web server as efficient as we could.
The web server sends back a big XML structure in every request.

We used Curl because it allowed us to deal with secured requests.

Curl by default uses http 1.1.

When we ask Curl to use http 1.0, we get average response time of 40
millisecond during our stress tests.

The response time is good but the problem is it's not releasing the
CPU's while waiting for the response. It's supposed to be like a disk
I/O where it should not be using the CPU. So the CPU utilization goes
up to 99% with multithreads of testing.

When we run the tests using http 1.1, the response time was 2 seconds


Using http 1.1 and asking Curl to disable "Expect: 100-continue" by
using header "Expect:",
we were able to reach response time of 120 milliseconds and the CPU
usage was cut by almost 90%.

The question is: how can we get the best of the two settings: have the
same low CPU usage as http 1.1 and reduce the response time from 120 ms
to 40 ms as http 1.0?

Thanks

 

[it_says_BALLS_on_your forehead]

We have a Perl program that has to do thousands of secure http post
requests to a web server as efficient as we could.
The web server sends back a big XML structure in every request.

We used Curl because it allowed us to deal with secured requests.

Curl by default uses http 1.1.

When we ask Curl to use http 1.0, we get average response time of 40
millisecond during our stress tests.

The response time is good but the problem is it's not releasing the
CPU's while waiting for the response. It's supposed to be like a disk
I/O where it should not be using the CPU. So the CPU utilization goes
up to 99% with multithreads of testing.

When we run the tests using http 1.1, the response time was 2 seconds


Using http 1.1 and asking Curl to disable "Expect: 100-continue" by
using header "Expect:",
we were able to reach response time of 120 milliseconds and the CPU
usage was cut by almost 90%.

The question is: how can we get the best of the two settings: have the
same low CPU usage as http 1.1 and reduce the response time from 120 ms
to 40 ms as http 1.0?

p.s. the version of curl is:
$ ./curl --version
curl 7.15.4 (powerpc-ibm-aix5.2.0.0) libcurl/7.15.4 OpenSSL/0.9.8b
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: IPv6 Largefile NTLM SSL

and the Perl module we are using is:
WWW::Curl::Easy 3.01

 

[Ben Morrow]

We have a Perl program that has to do thousands of secure http post
requests to a web server as efficient as we could.
The web server sends back a big XML structure in every request.

We used Curl because it allowed us to deal with secured requests.

Have you tried using LWP?

Curl by default uses http 1.1.

When we ask Curl to use http 1.0, we get average response time of 40
millisecond during our stress tests.

The response time is good but the problem is it's not releasing the
CPU's while waiting for the response. It's supposed to be like a disk
I/O where it should not be using the CPU. So the CPU utilization goes
up to 99% with multithreads of testing.

When we run the tests using http 1.1, the response time was 2 seconds


Using http 1.1 and asking Curl to disable "Expect: 100-continue" by
using header "Expect:",
we were able to reach response time of 120 milliseconds and the CPU
usage was cut by almost 90%.

The question is: how can we get the best of the two settings: have the
same low CPU usage as http 1.1 and reduce the response time from 120 ms
to 40 ms as http 1.0?

The rest of this is not really relevant to Perl, and would be better
asked on some Curl-related list or group.

Ben

 

[wkhedr]

LWP did the same numbers 120ms!

Have you tried using LWP?


The rest of this is not really relevant to Perl, and would be better
asked on some Curl-related list or group.

Ben

--
I must not fear. Fear is the mind-killer. I will face my fear and
I will let it pass through me. When the fear is gone there will be
nothing. Only I will remain.
(e-mail address removed) Frank Herbert, 'Dune'