Yet another Hpricot question

R

Rick DeNatale

I'm trying to scan an html file using Hpricot to produce a table of
links within the file.

Right now I've got something like this.

doc = Hpricot(open(url)).
doc.search('a').each do | element |
puts "#{element.inner_html}
puts " #{element.attributes['href']
end

This works, but in this document some of the a tags use markup on
their contents. Something like
<a href="http://blah.org/blah.htm"><b>blah blah</b> blah</a>

I'd like to strip out the markup tags so that I'd get

blah blah blah
http://blah.org/blah.htm

Is there some way to search for or iterate over the leaf elements of
the tree rooted by an element in Hpricot?
 
A

Aaron Patterson

I'm trying to scan an html file using Hpricot to produce a table of
links within the file.

Right now I've got something like this.

doc = Hpricot(open(url)).
doc.search('a').each do | element |
puts "#{element.inner_html}
puts " #{element.attributes['href']
end

This works, but in this document some of the a tags use markup on
their contents. Something like
<a href="http://blah.org/blah.htm"><b>blah blah</b> blah</a>

I'd like to strip out the markup tags so that I'd get

blah blah blah
http://blah.org/blah.htm

Is there some way to search for or iterate over the leaf elements of
the tree rooted by an element in Hpricot?

I had to do something similar in Mechanize, and this is what I came up
with:

class Hpricot::Elem
def all_text
text = ''
children.each do |child|
if child.respond_to? :content
text << child.content
end
if child.respond_to? :all_text
text << child.all_text
end
end
text
end
end

doc = Hpricot("<a href=\"http://blah.org/blah.htm\"><b>blah blah</b> blah</a>")
doc.search('a').each do |e|
puts "#{e.all_text}"
puts " #{e.attributes['href']}"
end

Hope that helps!

--Aaron
 
G

Gregory Seidman

13:07AM +0900, Rick DeNatale wrote:
} > I'm trying to scan an html file using Hpricot to produce a table of
} > links within the file.
} >
} > Right now I've got something like this.
} >
} > doc = Hpricot(open(url)).
} > doc.search('a').each do | element |
} > puts "#{element.inner_html}
} > puts " #{element.attributes['href']
} > end
} >
} > This works, but in this document some of the a tags use markup on
} > their contents. Something like
} > <a href="http://blah.org/blah.htm"><b>blah blah</b> blah</a>
} >
} > I'd like to strip out the markup tags so that I'd get
} >
} > blah blah blah
} > http://blah.org/blah.htm
} >
} > Is there some way to search for or iterate over the leaf elements of
} > the tree rooted by an element in Hpricot?
}
} I had to do something similar in Mechanize, and this is what I came up
} with:
}
} class Hpricot::Elem
} def all_text
} text = ''
} children.each do |child|
} if child.respond_to? :content
} text << child.content
} end
} if child.respond_to? :all_text
} text << child.all_text
} end
} end
} text
} end
} end
}
} doc = Hpricot("<a href=\"http://blah.org/blah.htm\"><b>blah blah</b> blah</a>")
} doc.search('a').each do |e|
} puts "#{e.all_text}"
} puts " #{e.attributes['href']}"
} end

There is a simpler implementation of all_text:

class Hpricot::Elem
def all_text
text = ''
traverse_text {|t| text << t.content }
text
end
end

} Hope that helps!
} --Aaron
--Greg
 
R

Rick DeNatale

13:07AM +0900, Rick DeNatale wrote:
} > I'm trying to scan an html file using Hpricot to produce a table of
} > links within the file.
} >
} > Right now I've got something like this.
} >
} > doc = Hpricot(open(url)).
} > doc.search('a').each do | element |
} > puts "#{element.inner_html}
} > puts " #{element.attributes['href']
} > end
} >
} > This works, but in this document some of the a tags use markup on
} > their contents. Something like
} > <a href="http://blah.org/blah.htm"><b>blah blah</b> blah</a>
} >
} > I'd like to strip out the markup tags so that I'd get
} >
} > blah blah blah
} > http://blah.org/blah.htm
} >
} > Is there some way to search for or iterate over the leaf elements of
} > the tree rooted by an element in Hpricot?
}
} I had to do something similar in Mechanize, and this is what I came up
} with:
}
} class Hpricot::Elem
} def all_text
} text = ''
} children.each do |child|
} if child.respond_to? :content
} text << child.content
} end
} if child.respond_to? :all_text
} text << child.all_text
} end
} end
} text
} end
} end
}
} doc = Hpricot("<a href=\"http://blah.org/blah.htm\"><b>blah blah</b> blah</a>")
} doc.search('a').each do |e|
} puts "#{e.all_text}"
} puts " #{e.attributes['href']}"
} end

There is a simpler implementation of all_text:

class Hpricot::Elem
def all_text
text = ''
traverse_text {|t| text << t.content }
text
end
end

} Hope that helps!
} --Aaron
--Greg

Thanks Aaron and Greg, works a treat!
 
W

why the lucky stiff

Thanks Aaron and Greg, works a treat!

If three of you have independantly used this, let's check it in. Elements#text
which parallels Elements#html (I think jQuery also has this) and Elem#inner_text
as well.

_why
 
G

Gregory Seidman

29:54PM +0900, Rick DeNatale wrote:
} > >There is a simpler implementation of all_text:
} > >
} > >class Hpricot::Elem
} > > def all_text
} > > text = ''
} > > traverse_text {|t| text << t.content }
} > > text
} > > end
} > >end
} >
} > Thanks Aaron and Greg, works a treat!
}
} If three of you have independantly used this, let's check it in.
} Elements#text which parallels Elements#html (I think jQuery also has
} this) and Elem#inner_text as well.

Actually, I haven't used it. I just knew how to go about it. What I would
find much more useful is an #inject_text (and a corresponding
#inject_elements, though that isn't as important since the / notation
retrieves an Enumerable).

} _why
--Greg
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,951
Messages
2,570,113
Members
46,698
Latest member
alexxx

Latest Threads

Top