Finding Line numbers of HTML file

[Ramdas]

I am doing some HTML scrapping for a side project.

I need a method using sgmllib or HTMLParser to parse an HTML file and
get line nos of all the tags

I tried a few things, but I am just not able to work with either if
the parsers.



Can someone help

 

[Larry Bates]

I am doing some HTML scrapping for a side project.

I need a method using sgmllib or HTMLParser to parse an HTML file and
get line nos of all the tags

I tried a few things, but I am just not able to work with either if
the parsers.



Can someone help

HTML doesn't really have "lines" it is just a stream of text that can be
formatted with line end characters to make it somewhat easier for humans to
read. Parsers probably won't give you any line numbers. What is the use case
for this (e.g. why do you think you need the line numbers)? Extraction is done
using tags not line numbers. All that said, you should look at Beautiful Soup
module before continuing.

-Larry

 

[Paul McGuire]

I am doing some HTML scrapping for a side project.

I need a method using sgmllib or HTMLParser to parse an HTML file and
get line nos of all the tags

Homework, perhaps? Well, I don't think your instructor will give many
points for a pyparsing solution, but it was an interesting 10-minute
exercise anyway. Once you use pyparsing's built-in expression for
anyOpenTag, and scan through the input html, the rest is just
bookkeeping in a defaultdict.

-- Paul


import urllib
from collections import defaultdict
from pyparsing import anyOpenTag, lineno

# read in a random html page
pg = urllib.urlopen("http://www.yahoo.com")
html = pg.read()
pg.close()

# print out what we got
print html
print

# create a defaultdict to tally up list of line numbers for each tag
tagLocs = defaultdict(list)

# use a parse action to update the tally whenever a tag is found
def tallyTagLineNumber(strg, locn, tagTokens):
line = lineno(locn,strg)
tagLocs[tagTokens[0]].append(line)
anyOpenTag.setParseAction(tallyTagLineNumber)

# scan the input html, and add tag line numbers to the tally dict
anyOpenTag.searchString(html)

# print out the results
tagnames = sorted(tagLocs.keys())
for t in tagnames:
print t, len(tagLocs[t])
print tagLocs[t]
print

-------
Prints:

<... extracted HTML not shown...>

a 46
[54, 68, 96, 97, 98, 99, 110, 111, 112, 113, 114, 115, 116, 117, 120,
121, 122, 123, 124, 125, 126, 127, 130, 131, 132, 133, 134, 135, 136,
139, 140, 141, 142, 143, 150, 159, 160, 161, 162, 163, 164, 165, 166,
168, 169, 170]

b 5
[91, 109, 119, 129, 138]

base 1
[6]

body 1
[17]

br 34
[91, 93, 93, 94, 110, 111, 112, 113, 114, 115, 116, 117, 120, 121,
122, 123, 124, 125, 126, 127, 130, 131, 132, 133, 134, 135, 136, 139,
140, 141, 142, 143, 167, 167]

center 1
[18]

font 15
[30, 36, 54, 68, 90, 96, 97, 98, 99, 109, 119, 129, 138, 158, 168]

form 1
[31]

head 1
[2]

html 1
[1]

img 2
[26, 150]

input 5
[32, 33, 34, 36, 37]

meta 2
[4, 5]

spacer 26
[21, 24, 47, 50, 52, 53, 55, 56, 58, 61, 64, 66, 67, 69, 70, 72, 79,
82, 83, 89, 101, 104, 146, 148, 156, 157]

span 1
[39]

style 2
[7, 12]

table 26
[19, 21, 29, 30, 35, 46, 49, 54, 63, 68, 79, 82, 83, 84, 87, 88, 95,
106, 107, 108, 146, 148, 149, 156, 157, 158]

td 58
[20, 21, 24, 25, 28, 29, 30, 36, 37, 45, 47, 48, 50, 52, 53, 54, 54,
55, 56, 58, 61, 62, 64, 66, 67, 68, 68, 69, 70, 72, 78, 79, 82, 83,
86, 87, 89, 90, 96, 97, 98, 99, 101, 104, 105, 106, 107, 109, 119,
129, 138, 146, 148, 149, 156, 157, 157, 158]

title 1
[3]

tr 37
[20, 21, 23, 29, 30, 35, 44, 46, 49, 51, 54, 57, 63, 65, 68, 71, 78,
79, 82, 83, 85, 87, 88, 96, 97, 98, 99, 106, 107, 108, 146, 148, 149,
156, 157, 157, 158]

 

[Ramdas]

Hey paul,

Thanks a Ton!

Never heard of pyparsing module.

This is more a hobby, than any homework.

This is exactly what I wanted.

I am scrapping a few web pages for data.

I am using Beautiful Soup for tag extraction. However for some quirky
reasons, I need to reference back to the exact line number. BS does
not support line nos right now, and Richardson, says he will include
the same in future.

I need a method using sgmllib or HTMLParser to parse an HTML file and
get line nos of all the tags

Homework, perhaps? Well, I don't think your instructor will give many
points for a pyparsing solution, but it was an interesting 10-minute
exercise anyway. Once you use pyparsing's built-in expression for
anyOpenTag, and scan through the input html, the rest is just
bookkeeping in a defaultdict.

-- Paul

import urllib
from collections import defaultdict
from pyparsing import anyOpenTag, lineno

# read in a random html page
pg = urllib.urlopen("http://www.yahoo.com")
html = pg.read()
pg.close()

# print out what we got
print html
print

# create a defaultdict to tally up list of line numbers for each tag
tagLocs = defaultdict(list)

# use a parse action to update the tally whenever a tag is found
def tallyTagLineNumber(strg, locn, tagTokens):
line = lineno(locn,strg)
tagLocs[tagTokens[0]].append(line)
anyOpenTag.setParseAction(tallyTagLineNumber)

# scan the input html, and add tag line numbers to the tally dict
anyOpenTag.searchString(html)

# print out the results
tagnames = sorted(tagLocs.keys())
for t in tagnames:
print t, len(tagLocs[t])
print tagLocs[t]
print

-------
Prints:

<... extracted HTML not shown...>

a 46
[54, 68, 96, 97, 98, 99, 110, 111, 112, 113, 114, 115, 116, 117, 120,
121, 122, 123, 124, 125, 126, 127, 130, 131, 132, 133, 134, 135, 136,
139, 140, 141, 142, 143, 150, 159, 160, 161, 162, 163, 164, 165, 166,
168, 169, 170]

b 5
[91, 109, 119, 129, 138]

base 1
[6]

body 1
[17]

br 34
[91, 93, 93, 94, 110, 111, 112, 113, 114, 115, 116, 117, 120, 121,
122, 123, 124, 125, 126, 127, 130, 131, 132, 133, 134, 135, 136, 139,
140, 141, 142, 143, 167, 167]

center 1
[18]

font 15
[30, 36, 54, 68, 90, 96, 97, 98, 99, 109, 119, 129, 138, 158, 168]

form 1
[31]

head 1
[2]

html 1
[1]

img 2
[26, 150]

input 5
[32, 33, 34, 36, 37]

meta 2
[4, 5]

spacer 26
[21, 24, 47, 50, 52, 53, 55, 56, 58, 61, 64, 66, 67, 69, 70, 72, 79,
82, 83, 89, 101, 104, 146, 148, 156, 157]

span 1
[39]

style 2
[7, 12]

table 26
[19, 21, 29, 30, 35, 46, 49, 54, 63, 68, 79, 82, 83, 84, 87, 88, 95,
106, 107, 108, 146, 148, 149, 156, 157, 158]

td 58
[20, 21, 24, 25, 28, 29, 30, 36, 37, 45, 47, 48, 50, 52, 53, 54, 54,
55, 56, 58, 61, 62, 64, 66, 67, 68, 68, 69, 70, 72, 78, 79, 82, 83,
86, 87, 89, 90, 96, 97, 98, 99, 101, 104, 105, 106, 107, 109, 119,
129, 138, 146, 148, 149, 156, 157, 157, 158]

title 1
[3]

tr 37
[20, 21, 23, 29, 30, 35, 44, 46, 49, 51, 54, 57, 63, 65, 68, 71, 78,
79, 82, 83, 85, 87, 88, 96, 97, 98, 99, 106, 107, 108, 146, 148, 149,
156, 157, 157, 158]

 

[Ramdas]

Hi Paul,

I am cross posting the same to grab your attention at pyparsing forums
too. 1000 apologies on the same count!

I am a complete newbie to parsing and totally new to pyparsing.

I have adapted your code to store the line numbers as below.
Surprisingly, the line numbers printed, when I scrap some of the URLs,
is not accurate and is kind of way off.


page = urlli2b.urlopen("www.....com).read()

def tallyTagLineNumber(strg, locn, tagTokens):
line = lineno(locn,strg)
tagLocs[tagTokens[0]].append(line)



def getlinenos(page):
anyOpenTag.setParseAction(tallyTagLineNumber)
anyOpenTag.searchString(page.lower()) # changing the entire string to
lower case to get INPUT
tagnames = sorted(tagLocs.keys())
taglinedict={}
for t in tagnames:
taglinedict[t]= unique(tagLocs[t])
return taglinedict

 

[Ramdas]

Hi Paul,


I am cross posting the same to grab your attention at pyparsing forums
too. 1000 apologies on the same count!

I am a complete newbie to parsing and totally new to pyparsing.

I have adapted your code to store the line numbers as below.
Surprisingly, the line numbers printed, when I scrap some of the URLs,
is not accurate and is kind of way off.

page = urlli2b.urlopen("www.....com).read()

def tallyTagLineNumber(strg, locn, tagTokens):
line = lineno(locn,strg)
tagLocs[tagTokens[0]].append(line)

def getlinenos(page):
anyOpenTag.setParseAction(tallyTagLineNumber)
anyOpenTag.searchString(page.lower()) # changing the entire
string to lowercase, to grab
# input and INPUT from html as input tag ONLy

tagnames = sorted(tagLocs.keys())
taglinedict={}
for t in tagnames:
taglinedict[t]= unique(tagLocs[t])
return taglinedict


What did I do wrong and why this problem!

Ramdas

 

[Paul McGuire]

Hi Paul,

I am cross posting the same to grab your attention at pyparsing forums
too. 1000 apologies on the same count!

I am a complete newbie to parsing and totally new to pyparsing.

I have adapted your code to store the line numbers as below.
Surprisingly, the line numbers printed, when I scrap some of the URLs,
is not accurate and is kind of way off.

<snip>

Ramdas -

You will have to send me that URL off-list using e-mail, Google Groups
masks it and I can't pull it up. In my example, I used the Yahoo home
page. What is the URL you used, and which tags' results were off?

Just some comments:
- I did a quasi-verification of my results, using a quick-and-dirty re
match. This did not give me the line numbers, but did give me counts
of tag names (if anyone knows how to get the string location of an re
match, this would be the missing link for an alternative solution to
this problem). I added this code after the code I posted earlier:

print "Quick-and-dirty verify using re's"
import re
openTagRe = re.compile("<([^ >/!]+)")

tally2 = defaultdict(int)
for match in openTagRe.findall(html):
tally2[match] += 1

for t in tally2.keys():
print t,tally2[t],
if tally2[t] != len(tagLocs[t]):
print "<<<"
else:
print

This crude verifier turned up no mismatches when parsing the Yahoo
home page.

- Could the culprit be your unique function? You did not post the
code for this, so I had to make up my own:

def unique(lst):
return sorted(list(set(lst)))

This does trim some of the line numbers, but I did not try to validate
this.

- In your getlinenos function, it is not necessary to call
setParseAction every time. You only need to do this once, probably
right after you define the tallyTagLineNumber function.

- Here is an abbreviated form of getlinenos:

def getlinenos(page):
# clear out tally dict, so as not to get crossover data from
# a previously-parsed page
tagLocs.clear()
anyOpenTag.searchString(page)
return dict((k,unique(v)) for k,v in tagLocs.items())

If you wanted, you could even inline the unique logic, without too
much obfuscation.

-- Paul

 

[Stefan Behnel]

I am doing some HTML scrapping for a side project.

I need a method using sgmllib or HTMLParser to parse an HTML file and
get line nos of all the tags

Try lxml.html, it provides line numbers for each element.

http://codespeak.net/lxml/dev/

Stefan