Unstructured HTML extraction

[dayzman]

Hi,

I'm interested in a program that extracts the structure of unstructured HTML documents. The program should be able to make good estimates about different font styles used to represent headings, for example, some may use <font size = 24> for headings and some may use <h1>, in the end, both should output the same structure. The output can be in XML or other formats. Manual driving should remain minimal. Does anyone know of such program (preferably open-source)?

Cheers,
Michael

 

[Nick Kew]

Hi,

I'm interested in a program that extracts the structure of unstructured HTML documents. The program should be able to make good estimates about different font styles used to represent headings, for example, some may use <font size = 24> for headings and some may use <h1>, in the end,

Not quite sure what you mean, but does http://valet.webthing.com/access/ seem relevant to you?

both should output the same structure. The output can be in XML or other formats.

Output is XML. What you see on the Web is transformed with XSLT to HTML or RDF.

 

[dayzman]

Hi,

Sorry for not making myself clear. What I'm trying to find is a program that handles the inconsistent styles used on webpages to imply, e.g., headings -- where some people may use large fonts, some may use the <h1> tag, and some may use block letters etc. The program needs to be able to recognise the implication of the different styles used, and "cleans up" the structure and outputs it in a readable format. I hope it is more clear this time.

Cheers,
Michael

 

[dayzman]

Hi,

Sorry for not making myself clear. What I'm trying to find is a program that handles the inconsistent styles used on webpages to imply, e.g., headings -- where some people may use large fonts, some may use the <h1> tag, and some may use block letters etc. The program needs to be able to recognise the implication of the different styles used, and "cleans up" the structure and outputs it in a readable format. I hope it is more clear this time.

Cheers,
Michael

 

[Nick Kew]

Sorry for not making myself clear. What I'm trying to find is a program that handles the inconsistent styles used on webpages to imply, e.g., headings -- where some people may use large fonts, some may use the <h1> tag, and some may use block letters etc. The program needs to be

That's a hard AI problem. Incompetent authors abuse presentational markup when they mean a heading, but someone might also legitimately present something that isn't a heading as big and bold. That's why tools like AccessValet highlight cases that look like a bogus heading (e.g. <p><big>FOO</big></p>) for a human user to decide.

This topic has been done to death on appropriate newsgroups, as google would have told you. Redirecting this.

P.S. Please fix your news client not to post everything twice.