Help Needed - LWP Redirect problemss

M

mark1.thompson45

Hi,
I am quite new to LWP but familiar with Perl so be kind. I am trying
to pull some data from a webpage using LWP, the actual page I need the
data from is accessed by clicking on a link off the main page, this
is:-

URL1 =
http://www.youthemanager.co.uk/Clev...me=YouTheManager2006&language=MIRROR&aff_id=7

If I take the link I need it starts a servlet and when I view the URL
for the servlet it gives me the following:-

URL2 =
http://www.youthemanager.co.uk/Clever/servlet/PlayerList?gameid=184&catidx=4

So I set about writing some code to get the information from URL2:-

-------------------------------------------------------------------
#!C:\\Perl\\bin\\perl -w
use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Request::Common qw(GET);
use HTTP::Cookies;

my $ua= LWP::UserAgent->new(
requests_redirectable => ['GET', 'HEAD', 'POST'],
cookie_jar => HTTP::Cookies->new(
file => 'mycookies.txt',
autosave => 1
),
);

# Define user agent type
$ua->agent('Mozilla/8.0');


# Request object

my $req = GET
'http://www.youthemanager.co.uk/Clev...=YouTheManager2006&language=MIRROR&aff_id=7';

#my $req = GET
'http://www.youthemanager.co.uk/Clever/servlet/PlayerList?catidx=3';

# Make the request
my $res = $ua->request($req);

# Check the response
if ($res->is_success) {
print ">>".$res->content;
} else {
print ">> FAILED <<\n";
print $res->status_line ."\n";
}

exit 0;
-------------------------------------------------------------------

When I enable URL1 in my code and run it I get the contents back from
the webpage, when I enable URL2 in the code (this is the page I want)
then I get the following when the code is run:-

<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("/");</SCRIPT></HEAD></HTML>

I think this maybe something to do with redirects but I thought the
code above would handle that, I know that the GET is used as apposed to
POST for this page as I have snooped the activity on my LAN.

Can anyone offer any advice on where to go from here ....


cheers, Mark.
 
J

John Bokma

When I enable URL1 in my code and run it I get the contents back from
the webpage, when I enable URL2 in the code (this is the page I want)
then I get the following when the code is run:-

<HTML><HEAD><SCRIPT
LANGUAGE="JAVASCRIPT">location.replace("/");</SCRIPT></HEAD></HTML>

Try to visit the site with JavaScript off, it might be that they use
JavaScript to do some redirection, etc.
 
J

John Bokma

Why? For a lot of folks behind corporate firewalls, googlegroups is the
only usenet access we have.

I did ploink googlegroups, the reason: a large number of people don't use
it properly (no quoting), even not after it's explained to them. And I will
probably do it again in the near future, just for this reason.

I am sure there are other Usenet webinterfaces that are better, or ask your
corporation to change its policy. A firewall sysadmin with clue will {a}
understand how to add port 119 (IIRC), and {b} probably requires Usenet
access him/herself.
 
A

A. Sinan Unur

(e-mail address removed) wrote in @g47g2000cwa.googlegroups.com:
how do you do that !!!!

How do you do what?

If you want to post a followup via groups.google.com,
don't use the broken "Reply" link at the bottom of the
article. Click on "show options" at the top of the
article, then click on the "Reply" at the bottom of
the article headers.

Also, it looks like your keyboard is broken.

Sinan
 
M

mark1.thompson45

A. Sinan Unur said:
(e-mail address removed) wrote in @g47g2000cwa.googlegroups.com:


How do you do what?

If you want to post a followup via groups.google.com,
don't use the broken "Reply" link at the bottom of the
article. Click on "show options" at the top of the
article, then click on the "Reply" at the bottom of
the article headers.

Also, it looks like your keyboard is broken.

Sinan



I am not wanting to get bogged down in trivialities but why would you
have a "Reply" link next to an article if it wasn't to be used. So far
in this thread I have had more replies about posting than I have in
response to my "request for help", If I had been abusive or insulting I
could have seen the irony but please......

p.s. there is nothing wrong with my keyboard !!

I'll try and fix my own problem ---- without google groups.
 
P

Paul Lalli

I am not wanting to get bogged down in trivialities but why would you
have a "Reply" link next to an article if it wasn't to be used.

Who is the "you" to whom you are speaking? You seem to be addressing
Google Groups in general. But instead you are posting to a Usenet
newsgroup, using Google Groups interface. And we agree with you. That
Reply link should either be fixed or removed. Google Groups, however,
seems unwilling or unable to heed the multitudes of complaints that
have been lodged about this.
So far
in this thread I have had more replies about posting than I have in
response to my "request for help", If I had been abusive or insulting I
could have seen the irony but please......

Whether you realized it or not, you were being abusive and insulting,
by ignoring decades of custom in how replies are composed to Usenet
messages, not to mention the Posting Guidelines for the specific group
to which you were posting.
p.s. there is nothing wrong with my keyboard !!

Then why does it keep inserting extra and extraneous '!' characters
towards the end of many of your sentences?
I'll try and fix my own problem ---- without google groups.

That's a good idea. Google Groups is a severely broken interface to
Usenet. You would be far better off using a real newsreader.

Paul Lalli
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads


Members online

No members online now.

Forum statistics

Threads
474,034
Messages
2,570,356
Members
47,002
Latest member
RobertoLip

Latest Threads

Top