unable to get all html code on Yahoo

[tmp1235711]

I have tried for the past several hours to figure out what I need to
get all fo the data on this web page:
http://alaron.com/quotes.php?quote=forex but I am unable to get the
price data. Here is my code:

use warnings;
use Fcntl;
#use strict;
package MAIN;
require LWP::UserAgent; # reads the DATA in as one long string
with embedded logical new lines
$ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->env_proxy;

open (RAW_HTML, ">raw html from lwp_00.txt") ||
die "unable to open the master file $!";

# web site/page desired: http://alaron.com/quotes.php?quote=forex
$market_array[0] = "forex";

### read in next symbol and the expiration dates then push expire
dates to array
foreach $market (@market_array) {

$response =
$ua->get("http://alaron.com/quotes.php?quote=$market");
$html_response = $response->content;
print $html_response;
print RAW_HTML $html_response;
}
# is not getting the price data, any help would be most appreciated
# I have also tried using a socket connection also - partial code:
# $sym_url = "/quotes.php?quote=forex";
# START_SOCKET: $socket = IO::Socket::INET->new
# ( PeerAddr => 'alaron.com',
# PeerPort => 80,
# Proto => 'tcp',
# Type => SOCK_STREAM,
# Reuse => 1)
# ;

 

[Brian Wakem]

I have tried for the past several hours to figure out what I need to
get all fo the data on this web page:
http://alaron.com/quotes.php?quote=forex but I am unable to get the
price data. Here is my code:

use warnings;
use Fcntl;
#use strict;
package MAIN;
require LWP::UserAgent; # reads the DATA in as one long string
with embedded logical new lines
$ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->env_proxy;

open (RAW_HTML, ">raw html from lwp_00.txt") ||
die "unable to open the master file $!";

# web site/page desired: http://alaron.com/quotes.php?quote=forex
$market_array[0] = "forex";

### read in next symbol and the expiration dates then push expire
dates to array
foreach $market (@market_array) {

$response =
$ua->get("http://alaron.com/quotes.php?quote=$market");

The subject line mentions Yahoo. I'm confused.

$html_response = $response->content;
print $html_response;
print RAW_HTML $html_response;
}
# is not getting the price data, any help would be most appreciated

View the source of the page you are retrieving. There is not price data
there, it's inserted by calling some external JS files.

 

[tmp1235711]

The subject line mentions Yahoo. I'm confused.





View the source of the page you are retrieving. There is not price data
there, it's inserted by calling some external JS files.

I made a typo on the subject line - did not intend to type "on yahoo".
Thank you for your reply, I have been teaching myself Perl for a month
and still have not learned as much as needed. How do I call these
external JS files so I can capture the data? I mean I can see the
call "<script language="JavaScript" src="http:", how can I have Perl
automatically execute the call?

 

[A. Sinan Unur]

(e-mail address removed) wrote in

I made a typo on the subject line - did not intend to type "on yahoo".
Thank you for your reply, I have been teaching myself Perl for a month
and still have not learned as much as needed. How do I call these
external JS files so I can capture the data? I mean I can see the
call "<script language="JavaScript" src="http:", how can I have Perl
automatically execute the call?

I don't know if there is a Javascript interpreter module available for
Perl: My guess is there is not.

One option is to look at what the Javascript code is doing, and then re-
write that in Perl.

Sinan

 

[J. Gleixner]

I have tried for the past several hours to figure out what I need to
get all fo the data on this web page:
http://alaron.com/quotes.php?quote=forex but I am unable to get the
price data. Here is my code:

Take a look at the HTML that's returned. The data is generated from
another page 'get.jsp'.

 

[J. Gleixner]

How do I call these
external JS files so I can capture the data? I mean I can see the
call "<script language="JavaScript" src="http:", how can I have Perl
automatically execute the call?

It's being included into the page, via Javascript, that doesn't
mean it's Javascript. Look at the src="...". That's what you
want to call to get the data. With a file name of 'get.jsp' it
would likely not have anything to do with Javascript, and much
more to do with Java. Try it and see.

You could use a regular expression to find the line with 'get.jsp'
in it, and pull out the value for the src attribute, or, if it doesn't
change, just cut & paste it from the HTML source, into your script,
with all of the name=value pairs, and you hopefully will get HTML.

It'd be even easier if they provided a Web service to get the data,
but you'd need to talk to them about that.