Use C++ to parse HTML

[Bo Yang]

Hi, guys. I am now developing an application
in which I need to fetch some html page, and then
parsing it to get some intended content in it.

Because HTML is not a standard XML format, so I
am curious about how should it be parsed?

Any help and suggestion will be appreciated very
much, thanks in advance!

 

[Alan Johnson]

Hi, guys. I am now developing an application
in which I need to fetch some html page, and then
parsing it to get some intended content in it.

Because HTML is not a standard XML format, so I
am curious about how should it be parsed?

Any help and suggestion will be appreciated very
much, thanks in advance!

The same way you'd parse anything else.
- Specify a grammar for what you want to parse.
- Split the input into a stream of tokens (I suggesting using flex or a
similar tool).
- Read in tokens until you can match a production in the grammar (I
suggest using bison or a similar tool).

If you don't have any previous experience with lexical analysis and
parsing then writing an HTML parser is probably too huge of a first
step. The book "Compilers: Principles, Techniques, and Tools" is the
place most people go to start learning about this stuff.

 

[Ian Collins]

Hi, guys. I am now developing an application
in which I need to fetch some html page, and then
parsing it to get some intended content in it.

Because HTML is not a standard XML format, so I
am curious about how should it be parsed?

Any help and suggestion will be appreciated very
much, thanks in advance!

Run it though HTML tidy first, that gives you clean XHTML to parse.

 

[Ian Collins]

The same way you'd parse anything else.
- Specify a grammar for what you want to parse.
- Split the input into a stream of tokens (I suggesting using flex or a
similar tool).
- Read in tokens until you can match a production in the grammar (I
suggest using bison or a similar tool).

If you don't have any previous experience with lexical analysis and
parsing then writing an HTML parser is probably too huge of a first
step. The book "Compilers: Principles, Techniques, and Tools" is the
place most people go to start learning about this stuff.

That's easier said than done with HTML, it breaks a lot of the XML
grammar rules (optional closing tags) in conforming markup and there is
a lot of realy crap (frontpage) generated html out there which violates
HTML grammar.

I had the unfortunate task of having to parse some of this before I
discovered HTML tidy.

 

[Alan Johnson]

That's easier said than done with HTML, it breaks a lot of the XML
grammar rules (optional closing tags) in conforming markup and there is
a lot of realy crap (frontpage) generated html out there which violates
HTML grammar.

I had the unfortunate task of having to parse some of this before I
discovered HTML tidy.

My view on that is that if it can't be parsed using the HTML grammar,
then it, by definition, is not HTML.

Of course being pragmatic one typically realizes that what they REALLY
want is something that is almost, but not entirely unlike an HTML
parser. Still, I think trying to first define a grammar for whatever it
is you really want to parse (this thing that is sort of like HTML) is
still the best approach. But then, I've never tried it myself.

 

[Ian Collins]

My view on that is that if it can't be parsed using the HTML grammar,
then it, by definition, is not HTML.

Of course being pragmatic one typically realizes that what they REALLY
want is something that is almost, but not entirely unlike an HTML
parser. Still, I think trying to first define a grammar for whatever it
is you really want to parse (this thing that is sort of like HTML) is
still the best approach. But then, I've never tried it myself.

I agree with that, my suggestion to use HTML tidy is based on my own
unfortunate experiences with parsing almost, but not entirely unlike
HTML HTML!

The XML grammar used by XHTML is way easier to define and parse.

 

[Gernot Frisch]

Hi, guys. I am now developing an application

in which I need to fetch some html page, and then
parsing it to get some intended content in it.

Because HTML is not a standard XML format, so I
am curious about how should it be parsed?

Take a look at boost:spirit. It's a parser generator that's pretty
easy to use once you got the idea.

 

[sasoon]

Hi, guys. I am now developing an application
in which I need to fetch some html page, and then
parsing it to get some intended content in it.

Because HTML is not a standard XML format, so I
am curious about how should it be parsed?

Any help and suggestion will be appreciated very
much, thanks in advance!

If tidy can convert this to xml, use it. Then use the apache software
foundation sex(sax) parser to do the parsing. Spend your time having
fun

 

[adams7807]

If tidy can convert this to xml, use it. Then use the apache software
foundation sex(sax) parser to do theparsing. Spend your time having
fun

See how Visual BNF tool parses code at www.intralogic.eu

 

[Emmanuel Deloget]

My view on that is that if it can't be parsed using the HTML grammar,
then it, by definition, is not HTML.

That's not totally true: if it can't be parsed using the HTML grammar,
then it's not really HTML but you have to treat it as HTML anyway. So
you'll have to add the missing </table> tags and so on. This is why
HTML is such a nightmare, because your parser should try to understand
the HTML to the best of its ability, often by trying to understand
what the HTML writer wanted to do.