What is the best way to "get" a web page?

[Pete]

I have the following code:

The file "temp.html" is created, but it doesn't look like the page at
www.python.org. I'm guessing there are multiple frames and my code did
not get everything. Can anyone point me to a tutorial or other
reference on how to "get" all of the html contents at a particular
page?

Why did Python print the line after "file.close"?

Thanks,
Pete

 

[Paul McGuire]

I have the following code:


The file "temp.html" is created, but it doesn't look like the page at
www.python.org. I'm guessing there are multiple frames and my code did
not get everything. Can anyone point me to a tutorial or other
reference on how to "get" all of the html contents at a particular
page?

Why did Python print the line after "file.close"?

Thanks,
Pete

A. You didn't actually invoke the close method, you simply referenced it,
which is why you got the output line after file.close. Python is not VB.
To call close, you have to follow it with ()'s, as in:

file.close()

This will have the added benefit of flushing the output to temp.html,
probably containing the missing content you were looking for.

B. Don't name variables "file", or "list", "str", "dict", "int", etc. Doing
so masks global names of builtin data types. Try "tempFile" instead.

-- Paul

 

[Pete]

I have the following code:

A. You didn't actually invoke the close method, you simply referenced it,
which is why you got the output line after file.close. Python is not VB.
To call close, you have to follow it with ()'s, as in:

file.close()

Ahhhh. Thank you very much!

This will have the added benefit of flushing the output to temp.html,
probably containing the missing content you were looking for.

B. Don't name variables "file", or "list", "str", "dict", "int", etc. Doing
so masks global names of builtin data types. Try "tempFile" instead.

Oh. Thanks again!
The file "temp.html" is definitely different than the first run, but
still not anything close to www.python.org . Any other suggestions?

Thanks,
Pete

 

[George Sakkis]

The file "temp.html" is definitely different than the first run, but
still not anything close to www.python.org . Any other suggestions?

If you mean that the page looks different in a browser, for one thing
you have to download the css files too. Here's the relevant extract
from the main page:

<link media="screen" href="styles/screen-switcher-default.css"
type="text/css" id="screen-switcher-stylesheet" rel="stylesheet" />
<link media="scReen" href="styles/netscape4.css" type="text/css"
rel="stylesheet" />
<link media="print" href="styles/print.css" type="text/css"
rel="stylesheet" />
<link media="screen" href="styles/largestyles.css" type="text/css"
rel="alternate stylesheet" title="large text" />
<link media="screen" href="styles/defaultfonts.css" type="text/css"
rel="alternate stylesheet" title="default fonts" />

You may either hardcode the urls of the css files, or parse the page,
extract the css links and normalize them to absolute urls. The first is
simpler but the second is more robust, in case a new css is added or an
existing one is renamed or removed.

George

 

[Wolfgang Keller]

Can anyone point me to a tutorial or other reference on how to "get" all

of the html contents at a particular page?

Why not use httrack?

http://www.satzbau-gmbh.de/staff/abel/httrack-py/

Sincerely,

Wolfgang Keller

 

[Pete]

The file "temp.html" is definitely different than the first run, but

If you mean that the page looks different in a browser, for one thing
you have to download the css files too. Here's the relevant extract
from the main page:

<link media="screen" href="styles/screen-switcher-default.css"
type="text/css" id="screen-switcher-stylesheet" rel="stylesheet" />
<link media="scReen" href="styles/netscape4.css" type="text/css"
rel="stylesheet" />
<link media="print" href="styles/print.css" type="text/css"
rel="stylesheet" />
<link media="screen" href="styles/largestyles.css" type="text/css"
rel="alternate stylesheet" title="large text" />
<link media="screen" href="styles/defaultfonts.css" type="text/css"
rel="alternate stylesheet" title="default fonts" />

You may either hardcode the urls of the css files, or parse the page,
extract the css links and normalize them to absolute urls. The first is
simpler but the second is more robust, in case a new css is added or an
existing one is renamed or removed.

George

Thanks for the information on CSS. I'll look into that later, but now
my question is on the first two lines of HTML code. Here's my latest
python code:

Here are the first two lines of temp.html:

1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/x html1/DTD/xhtml1-transitional.dtd">
2 <html lang="en" xml:lang="en"
xmlns="http://www.w3.org/1999/xhtml">

Here are the first two lines of www.python.org as saved from Firefox:

1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/x html1/DTD/xhtml1-transitional.dtd">
2 <html xml:lang="en" xmlns="http://www.w3.org/1999/xhtml"
lang="en"><head>

Lines one are identical. Lines two are different. Why would lines two
differ? Hmmmm...

Thanks,
Pete

 

[Pete]

Can anyone point me to a tutorial or other reference on how to "get" all

Why not use httrack?

http://www.satzbau-gmbh.de/staff/abel/httrack-py/

Sincerely,

Wolfgang Keller

Thanks for the tip. I'll check that out. Is that your code?

 

[Rainy]

Thanks for the information on CSS. I'll look into that later, but now
my question is on the first two lines of HTML code. Here's my latest
python code:


Here are the first two lines of temp.html:

1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/x html1/DTD/xhtml1-transitional.dtd">
2 <html lang="en" xml:lang="en"
xmlns="http://www.w3.org/1999/xhtml">

Here are the first two lines of www.python.org as saved from Firefox:

1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/x html1/DTD/xhtml1-transitional.dtd">
2 <html xml:lang="en" xmlns="http://www.w3.org/1999/xhtml"
lang="en"><head>

Lines one are identical. Lines two are different. Why would lines two
differ? Hmmmm...

Functionally they are the same, but third line included in Firefox.
Opera View Source command produces the same result as Python. It looks
like Firefox will do some cosmetic changes to source but nothing that
would change the way code works. Notice that attributes in second line
are re-arranged in order only?

 

[Felipe Almeida Lessa]

Functionally they are the same, but third line included in Firefox.
Opera View Source command produces the same result as Python.

[snip]

It's better to compare with the result of a downloader-only (instead
of a parser), like wget on Unix. That way you'll get exactly the same
bytes (assuming the page is static).

 

[Pete]

Functionally they are the same, but third line included in Firefox.

Opera View Source command produces the same result as Python.

[snip]

It's better to compare with the result of a downloader-only (instead
of a parser), like wget on Unix. That way you'll get exactly the same
bytes (assuming the page is static).

Ahhhh. wget - most cool. My temp.html matches wget. Now to capture that
pesky css stuff...

Thanks,
Pete

 

[Frithiof Andreas Jensen]

something like:

os.popen("wget -r3 http://juicypornpics.com")

..... wget understands the peculiarities of web pages so you do have to.