how to use POST to call cgi from Perl

M

michal.shmueli

i'm trying to use agent to retrieve data from some websites. the
website is using POST method. so i tried to follow some examples i
found on the web...

Eventually, i did the follow, but i'm getting error that says that it
couldn't fetch any data....
any idea?

#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
my $browser = LWP::UserAgent->new;

$browser->env_proxy;

my $url = 'http://www.carprices.com/cgi-bin/n_page3.cgi';
my $response = $browser->post( $url,
[ "vehicle_type" =>"new,used",
"conf_id" =>"carprices",
"onpage" =>"1",
"frompage" =>"basicsearch",
"bodystyle_buff" =>
"convertible,coupe,hatchback,pickup,sedan,suv,van,wagon",
"price_min_buff" => "1050",
"price_max_buff" => "149990",
"year_min_buff" => "1986",
"year_max_buff" => "2005" ]
);

thanks a lot,
Michal
 
A

A. Sinan Unur

(e-mail address removed) wrote in @g14g2000cwa.googlegroups.com:

[ please provide some context when replying ]

I don't think so. In your original post, you had a variable for body
type, which clearly is not an option with the basic search. So, I guess
you are using the form at

http://inventory.carprices.com/advanced.html

Posting that form, and examining the traffic shows some tracking
happening, and some cookies being set. There is also some Javascript
involved in form submission.

You might want to Google for "web scraping proxy", and use that tool to
figure out exactly what information is being exchanged.

You might also want to look at WWW::Mechanize. I don't think it is going
to be of much help in this case, as the form submission goes through
Javascript, and the site does not provide an alternative when Javascript
is not enabled.

To sum up, I do not have a quick solution. You are going to need to look
at the Javascript to see what is being done with cookies, and form
parameters, and possibly inspect the traffic to figure out exactly what
information the server expects, and submit that.

Sinan
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,904
Latest member
HealthyVisionsCBDPrice

Latest Threads

Top