Some <head> clauses cases BeautifulSoup to choke?

F

Frank Stutzman

I've got a simple script that looks like (watch the wrap):
---------------------------------------------------
import BeautifulSoup,urllib

ifile = urllib.urlopen("http://www.naco.faa.gov/digital_tpp_search.asp?fldId
ent=klax&fld_ident_type=ICAO&ver=0711&bnSubmit=Complete+Search").read()

soup=BeautifulSoup.BeautifulSoup(ifile)
print soup.prettify()
----------------------------------------------------

and all I get out of it is garbage. Other simular urls from the same site
work fine (use http://www.naco.faa.gov/digital_tpp_search.asp?fldId
ent=klax&fld_ident_type=ICAO&ver=0711&bnSubmit=Complete+Search as one example).

I did some poking and proding and it seems that there is something in the
<head> clause that is causing the problem. Heck if I can see what it is.

I'm new to BeautifulSoup (heck, I'm new to python). If I'm doing something
dumb, you don't need to be gentle.
 
C

Chris Mellon

I've got a simple script that looks like (watch the wrap):
---------------------------------------------------
import BeautifulSoup,urllib

ifile = urllib.urlopen("http://www.naco.faa.gov/digital_tpp_search.asp?fldId
ent=klax&fld_ident_type=ICAO&ver=0711&bnSubmit=Complete+Search").read()

soup=BeautifulSoup.BeautifulSoup(ifile)
print soup.prettify()
----------------------------------------------------

and all I get out of it is garbage. Other simular urls from the same site
work fine (use http://www.naco.faa.gov/digital_tpp_search.asp?fldId
ent=klax&fld_ident_type=ICAO&ver=0711&bnSubmit=Complete+Search as one example).

I did some poking and proding and it seems that there is something in the
<head> clause that is causing the problem. Heck if I can see what it is.

I'm new to BeautifulSoup (heck, I'm new to python). If I'm doing something
dumb, you don't need to be gentle.
Click to expand...

You have the same URL as both your good and bad example.
 
M

Marc Christiansen

Frank Stutzman said:
I've got a simple script that looks like (watch the wrap):
---------------------------------------------------
import BeautifulSoup,urllib

ifile = urllib.urlopen("http://www.naco.faa.gov/digital_tpp_search.asp?fldId
ent=klax&fld_ident_type=ICAO&ver=0711&bnSubmit=Complete+Search").read()

soup=BeautifulSoup.BeautifulSoup(ifile)
print soup.prettify()
Click to expand...

Same for me.
I did some poking and proding and it seems that there is something in the
<head> clause that is causing the problem. Heck if I can see what it is.
Click to expand...

The problem is this line:
<META http-equiv="Content-Type" content="text/html; charset=UTF-16">

Which is wrong. The content is not utf-16 encoded. The line after that
declares the charset as utf-8, which is correct, although ascii would be
ok too.

If I save the search result and remove this line, everything works. So,
you could:
- ignore problematic pages
- save and edit them, then reparse them (not always practical)
- use the fromEncoding argument:
soup=BeautifulSoup.BeautifulSoup(ifile, fromEncoding="utf-8")
(or 'ascii'). Of course this only works if you guess/predict the
encoding correctly ;) Which can be difficult. Since BeautifulSoup uses
"an encoding discovered in the document itself" (quote from
<http://www.crummy.com/software/BeautifulSoup/documentation.html#Beautiful Soup Gives You Unicode, Dammit>)
when the encoding you supply does not work, using fromEncoding="ascii"
should not hurt too much. But this being usenet, I'm sure someone will
tell me that I'm wrong and there is some weird 7bit encoding in use
somewhere on the web...
I'm new to BeautifulSoup (heck, I'm new to python). If I'm doing something
dumb, you don't need to be gentle.
Click to expand...

No, you did nothing dumb. The server sent you broken content.

Ciao
Marc
 
D

Duncan Booth

Frank Stutzman said:
I did some poking and proding and it seems that there is something in
the
<head> clause that is causing the problem. Heck if I can see what it
is.
Click to expand...

Maybe Beautifulsoup believes the incorrect encoding in the meta tags?
 
C

Chris Mellon

Same for me.


The problem is this line:
<META http-equiv="Content-Type" content="text/html; charset=UTF-16">

Which is wrong. The content is not utf-16 encoded. The line after that
declares the charset as utf-8, which is correct, although ascii would be
ok too.

If I save the search result and remove this line, everything works. So,
you could:
- ignore problematic pages
- save and edit them, then reparse them (not always practical)
- use the fromEncoding argument:
soup=BeautifulSoup.BeautifulSoup(ifile, fromEncoding="utf-8")
(or 'ascii'). Of course this only works if you guess/predict the
encoding correctly ;) Which can be difficult. Since BeautifulSoup uses
"an encoding discovered in the document itself" (quote from
<http://www.crummy.com/software/BeautifulSoup/documentation.html#Beautiful Soup Gives You Unicode, Dammit>)
when the encoding you supply does not work, using fromEncoding="ascii"
should not hurt too much. But this being usenet, I'm sure someone will
tell me that I'm wrong and there is some weird 7bit encoding in use
somewhere on the web...


No, you did nothing dumb. The server sent you broken content.
Click to expand...

Correct. However, this is the sort of real-life broken HTML that BS is
tasked to handle. It looks like the major browers handle this by using
the last content type (header or meta tag) encountered before other
content. Right now, it looks like BS has a number of fallback
mechanisms but it's meta-tag fallback only looks at the first tag.

Posting a feature request or whatever through whatever mechanism BS
uses to handle this sort of thing would probably be nice.
 
F

Frank Stutzman

Some kind person replied:
You have the same URL as both your good and bad example.
Click to expand...

Oops, dang emacs cut buffer (yeah, thats what did it). A working
example url would be (again, mind the wrap):

http://www.naco.faa.gov/digital_tpp...t_type=ICAO&ver=0711&bnSubmit=Complete+Search


Marc Christiansen said:
The problem is this line:
<META http-equiv="Content-Type" content="text/html; charset=UTF-16">

Which is wrong. The content is not utf-16 encoded. The line after that
declares the charset as utf-8, which is correct, although ascii would be
ok too.
Click to expand...

Ah, er, hmmm. Take a look the 'good' URL I mentioned above. You will
notice that it has the same utf-16, utf-8 encoding that the 'bad' one
has. And BeautifulSoup works great on it.

I'm still scratchin' ma head...
If I save the search result and remove this line, everything works. So,
you could:
- ignore problematic pages
Click to expand...

Not an option for my application.
- save and edit them, then reparse them (not always practical)
Click to expand...

Thats what I'm doing at the moment during my development. Sure
seems inelegant.
- use the fromEncoding argument:
soup=BeautifulSoup.BeautifulSoup(ifile, fromEncoding="utf-8")
(or 'ascii'). Of course this only works if you guess/predict the
encoding correctly ;) Which can be difficult. Since BeautifulSoup uses
"an encoding discovered in the document itself" (quote from
<http://www.crummy.com/software/BeautifulSoup/documentation.html#Beautiful Soup Gives You Unicode, Dammit>)
Click to expand...

I'll try that. For what I'm doing it ought to be safe enough.

Much appreciate all the comments so far.
 
M

Marc Christiansen

Frank Stutzman said:
Some kind person replied:

Oops, dang emacs cut buffer (yeah, thats what did it). A working
example url would be (again, mind the wrap):

http://www.naco.faa.gov/digital_tpp...t_type=ICAO&ver=0711&bnSubmit=Complete+Search




Ah, er, hmmm. Take a look the 'good' URL I mentioned above. You will
notice that it has the same utf-16, utf-8 encoding that the 'bad' one
has. And BeautifulSoup works great on it.

I'm still scratchin' ma head...
Click to expand...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.5/encodings/utf_16.py", line 16, in decode
return codecs.utf_16_decode(input, errors, True)
UnicodeDecodeError: 'utf16' codec can't decode byte 0x0a in position 41176: truncated data

bad contains the content of the 'bad' url, good the content of the
'good' url. Because of the UnicodeDecodeError, BeautifulSoup tries
either the next encoding or the next step from the url below.
Much appreciate all the comments so far.
Click to expand...

You're welcome.

Marc
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Having trouble with some lists in BeautifulSoup 1

Members online

No members online now.
Total: 26 (members: 0, guests: 26)
Robots: 191

Forum statistics

Threads
473,755
Messages
2,569,534
Members
45,007
Latest member
obedient dusk

Latest Threads

Top