HTTP::Request/GET response not expected

[Mike Starkie]

I have a script which tries to verify the existence of a BugID by
making a http::get request to a bug tracking url.

http://tracking.edgetrade.com/default.asp?pg=pgEditBug&command=view&ixBug=9999

typing it in manually returns the expected page (of course) but when i
do it in perl i get the default page only
(http://tracking.edgetrade.com)

what am i doing wrong?


#!/usr/bin/perl
use LWP;
print "verifying BugID...\n";
$logFile = shift @ARGV;
open(LOGFILE, $logFile) or die "Can't verify BugID in CVS commit log
file: $logFile: $!\n";
while ($line = <LOGFILE>) {
print "line: $line\n";
if ($line =~ /\s*Bug[zs]*\s*IDs*\s*[#:; ]+((\d+[ ,:;#]*)+)/i)
{
$bugIDstring .= " " . $1;
}
}
print "BugID's: $bugIDstring\n";
@bugIDlist = split("[ ,:;#]+", $bugIDstring);
$ixBug = 0;
foreach (@bugIDlist)
{
if (/\d+/)
{
$ixBug = int($_);
print "Attempting to verify bug info for Bug ID #$ixBug...\n";
$ua = LWP::UserAgent->new();
$url =
"http://tracking.edgetrade.com/default.asp?pg=pgEditBug&command=view&ixBug=$ixBug";
$webdoc = $ua->request(HTTP::Request->new(GET => $url));
$content = $webdoc->content;
print "$content\n";
if ($content =~ /Case #$ixBug does not exist or has been
deleted./)
{
die "BugID: $ixBug does not exist in FogBugz\n";
}
}
}

 

[J. Gleixner]

I have a script which tries to verify the existence of a BugID by
making a http::get request to a bug tracking url.

http://tracking.edgetrade.com/default.asp?pg=pgEditBug&command=view&ixBug=9999

typing it in manually returns the expected page (of course) but when i

Really? When I "type it in manually", I get:

[1] 83664
http://tracking.edgetrade.com/default.asp?pg=pgEditBug: No match.
[2] 83665
ixBug=9999: Command not found.
command=view: Command not found.
[1] - Exit 1
http://tracking.edgetrade.com/default.asp?pg=pgEditBug

do it in perl i get the default page only
(http://tracking.edgetrade.com)

what am i doing wrong?

First, you should provide code can be executed without modification. We
have no idea what's in $logFile. e.g. (based on your code..)

#!/usr/local/bin/perl
use strict;
use LWP;

my $ixBug=999;
print "Attempting to verify bug info for Bug ID #$ixBug...\n";
my $ua = LWP::UserAgent->new();
my $url =
"http://tracking.edgetrade.com/default.asp?pg=pgEditBug&command=view&ixBug=$ixBug";
my $webdoc = $ua->request(HTTP::Request->new(GET => $url));
my $content = $webdoc->content;
print "$content\n";


Second, look at what's returned. Isn't the text "Not Logged On"
helpful? It's asking for authentication, you need to log on to the site,
before you can get to that URL.

Finally, take a look at WWW::Mechanize. It should help you submit your
information to log on, and then to view the bug.

 

[Mike Starkie]

Well it certainly is due to the fact that I'm not logged in. How do I
know what the form expects for authentication (it's a POST)? It seems
to me I would have to sniff the wifre to see the post right?

The form expects two inputs a 'User' and 'Password". i wrote this test
script that fails:

#!/usr/bin/perl
use LWP;
my $browser = LWP::UserAgent->new;
my $url = 'http://tracking.edgetrade.com/default.asp?';
my $res = $browser->post($url, [User => "me", Password =>
"mypasswrod"]);
my $content = $res->content;
if($content =~ /Welcome to FogBUGZ/)
{
print "logged on";
}

 

[J. Gleixner]

Well it certainly is due to the fact that I'm not logged in. How do I
know what the form expects for authentication (it's a POST)? It seems
to me I would have to sniff the wifre to see the post right?

No, simply looking at the correct HTML should be enough.

Ya need to go to the right URL. Selecting "Log On", in the browser
takes me to:

http://tracking.edgetrade.com/default.asp?pg=pgLogon&dest=

View the source to get the names of the fields, and be sure to set the
various hidden variables, you'll also need to store at least one cookie.

Again, do yourself and us a favor and at least read about
WWW::Mechanize. There are examples, and some very nice methods to make
this much easier.

 

[Tad McClellan]

No, simply looking at the correct HTML should be enough.

Ya need to go to the right URL. Selecting "Log On", in the browser
takes me to:

http://tracking.edgetrade.com/default.asp?pg=pgLogon&dest=

View the source to get the names of the fields, and be sure to set the
various hidden variables, you'll also need to store at least one cookie.

Again, do yourself and us a favor and at least read about
WWW::Mechanize. There are examples, and some very nice methods to make
this much easier.

See also

Web Scraping Proxy

http://www.research.att.com/~hpk/wsp/

Which will write the Perl code for you.

 

[Charles DeRykus]

I have a script which tries to verify the existence of a BugID by
...
$webdoc = $ua->request(HTTP::Request->new(GET => $url));
$content = $webdoc->content;
print "$content\n";
if ($content =~ /Case #$ixBug does not exist or has been
deleted./)
{
die "BugID: $ixBug does not exist in FogBugz\n";
}
}
}

In addition to the other answers, you'll normally want to
glean more from the response:

$webdoc = $ua->request(...);
if ($webdoc->is_success) {
print $webdoc->content; }
else {
print "failed: ", $webdoc->status_line; }

hth,
--
Charles DeRykus
#! rnews 1193
Xref: xyzzy comp.unix.programmer:136921
Newsgroups: comp.unix.programmer
Path: xyzzy!nntp
From: "Fred L. Kleinschmidt" <fred.l.kleinschmidt@nospam_boeing.com>
Subject: Re: Folder contents
X-Nntp-Posting-Host: xpc-ps-53.nw.nos.boeing.com
Content-Type: text/plain; charset=us-ascii
Message-ID: <427BE8E4.71D456CB@nospam_boeing.com>
Sender: (e-mail address removed) (Boeing NNTP News Access)
Content-Transfer-Encoding: 7bit
Organization: Boeing
X-Accept-Language: en
References: <[email protected]>
Mime-Version: 1.0
Date: Fri, 6 May 2005 22:00:04 GMT
X-Mailer: Mozilla 4.79 [en]C-CCK-MCD Boeing Kit (Windows NT 5.0; U)

Guys im very new to Unix and scripting as a whole, but can anyone tell
me if its posible to create a script that looks at say 4 folders, then
checks which has the least amount of files in, then copy a file to that
folder?.

First, get used to calling them "directories", instead of "folders".

Look at the 'man' pages for ls, wc, and cp (i.e., type "man ls" or "man
wc" etc.)

 

[Mike Starkie]

For some reason, this tool does not generate any perl for me. The
website uses asp and the results of using wsp are a few files (w0 - w6)
which contain no perl only html and javascript

 

[J. Gleixner]

HTTP::Request does not generate Perl code. It encapsulates an HTTP/GET
request and the response from an HTTP server for some URL. The response
should contain HTML, which can contain javascript code, but that is up
to the server, not your program. Why do you think it should contain
Perl code?

Mike neglected to include information from the post he was referring to
that mentioned "wsp".

See also

Web Scraping Proxy

http://www.research.att.com/~hpk/wsp/

Which will write the Perl code for you.

As for wsp.pl writing code for you, it took a little looking around, but
I found this:

http://hacks.oreilly.com/pub/h/955#code

Very nice!

 

[Mike Starkie]

WooHoo.......sweeeeeet.

./wsp.pl -v | ./translate.pl > test.pl

./test.pl

works.....problem solved !!!

for the annals of time:

1: I didn't realize the perl output of wsp is on STDOUT

2: Use Both wsp and translate together to save a ton of time since you
don't have to encode the session by trial and error and piece together
the perl

4: my complaints should include more context so that new joiners on the
thread can understand them.

 

[Tad McClellan]

For some reason, this tool does not generate any perl for me.

Which tool are you speaking of?

HTTP::Request is a "module", not a "tool", and it isn't _supposed_
to generate any perl.

What made you think that it would generate perl (or Perl, more likely)?

 

[Mike Starkie]

these two 'tools' taken together will generate perl:

../wsp.pl -v | ./translate.pl > test.pl
../test.pl

you're the second person that mis-understood me.....sorry.

see links above for tools: wsp and translate

 

[Tad McClellan]

you're

Who is?

Please quote some context in followups like everybody else does.

the second person that mis-understood me.....sorry.

That's because you are not quoting context.

And you are *still* not quoting context.

I'm doubting the sincerity of your "sorry" there.

If you were truly sorry you would stop the behavior that led
to the misunderstanding, else it will just happen again.

 

[Tad McClellan]

see links above for tools: wsp and translate

There are no "links above", so once again we don't know what
you're talking about.

 

[Mike Starkie]

of course you could always read the entire thread instead in order to
clear up your confusion instead of doubting my sincerity.

 

[Tad McClellan]

of course you could always read the entire thread

Reading the entire thread would not be necessary if you quoted
some context like everybody else does.

You appear to not know much about the propogation of Usenet
articles. It is entirely possible that a followup arrives
*before* the article that it is followup up to.

The original article may even *never* show up. NNTP is not guaranteed.

So, many people don't *have* and entire thread, and they're left
wondering what you're talking about.

instead in order to
clear up your confusion instead of doubting my sincerity.

There is no longer any confusion, you've just confirmed
that your apology was indeed insincere.

*plonk*

 

[skijor]

Actually you are right. There's no need for me to aplogize for you not
understanding my reference to the tool that you suggested in the first
place. My aplogies to others though and to redeem my sincerity I
include some quoted text here:

See also
Web Scraping Proxy

http://www.research.att.com/~hpk/wsp/

Which will write the Perl code for you.


./wsp.pl -v | ./translate.pl > test.pl
./test.pl

works.....problem solved !!!

Which tool are you speaking of?

 

[Tad McClellan]

Actually you are right.

It is rare, but it has been known to happen.

my reference to the tool that you suggested in the first
place.

I answer many many questions each day.

I don't remember who I said what to, they all meld together
after a bit of time has passed.