Extract Content from HTML ?

mark4 · Feb 28, 2005

Hello,

Are there any utilities to help me extract Content from HTML ?

I'd like to store this data in a database.

The HTML consists of about 10,000 files with a total size of
about 160 Mb. Each file is a thread from a message forum. Each
thread has several contributions. The threads are in linear
order of date posted with filenames such as 000125633.html. The
HTML is marked up with <table>, etc tags. This HTML is very
badly formed with crucial tags missing (such as <TR>, <BODY>,
etc.). There is no coherence to this; no system - sometimes tags
are missing and sometimes they are present. Despite this, the
threads seem to render correctly; such is the forgiving nature
of modern browsers.

Fields for each post are usually identified by an attribute tag.
(usually an attribute of a <TD> or <SPAN>.

Sometimes I need to actually store HTML with the content (for
instance when a post includes a link, colored writing or text
formatted with <PRE> tags.

My purpose in storing this in a database is to make the content
(a) easier to search and (b) use a more efficient storage
medium.

The original database from which these web-forum posts were
taken is no longer available on the web nor does it look like it
ever will be again. Nor can I contact the person who 'owns' it.
If I did contact them, they would be unlikely to release the
data.

Despite this, there are no copyright issues here. Every single
post made to the forum was made using an alias and no forum
poster wants to be identified, nor do any posters wish to claim
"ownership" of their contributions.

Toby Inkster · Feb 28, 2005

mark4 said:
Are there any utilities to help me extract Content from HTML ?
I'd like to store this data in a database.

Looks to me like you'd have to write your own customised program to
extract the data.

To do that, I recommend using Perl. Perl has a module called HTML:

arser
which is apparently pretty good at extracting information from malformed
HTML files. Whatsmore, it is generally very good at text handling and has
decent database modules too.

Nor can I contact the person who 'owns' it. If I did contact them, they
would be unlikely to release the data.

Despite this, there are no copyright issues here. Every single post made
to the forum was made using an alias and no forum poster wants to be
identified, nor do any posters wish to claim "ownership" of their
contributions.

Sounds to me like there are *major* copyright issues!

mark4 · Feb 28, 2005

Looks to me like you'd have to write your own customised program to
extract the data.

I expected as much.

To do that, I recommend using Perl. Perl has a module called HTML:arser
which is apparently pretty good at extracting information from malformed
HTML files. Whatsmore, it is generally very good at text handling and has
decent database modules too.

Thanks. Being a microserf, I don't normally code in Perl but I
may look into this. It's either that or WSH Javascript with
it's regular expressions. Fortunately I already have a top
level design and it looks pretty simple. I may look into this
Perl module but it will probably be easier to use microserf
technology with which I'm intimate with. I shall probably store
it in MSSQL.

Sounds to me like there are *major* copyright issues!

I can't see what those issues are. Who owns the data? Not the
original forum provider. The data posted to a forum is copyright
of the original author - no matter what ToS my be specified in
the forum. All those original authors have an alias and don't
actually want to be identified. What I'm doing is no more a
violation of copyright than someone keeping newspaper clippings.

So long as I don't republish it.

Sherm Pendley · Feb 28, 2005

Mark's right. I don't do the whole "language cheerleader" thing - but for
this particular problem, Perl's an ideal fit.

Thanks. Being a microserf, I don't normally code in Perl but I
may look into this. It's either that or WSH Javascript with
it's regular expressions.

There's Perl for Windows, you know. It integrates nicely with WSH too.

<http://www.activestate.com>

sherm--

Philip Herlihy · Feb 28, 2005

Access can link to HTML (direct from the web) and will recognise tables.
You might be lucky! It would make a very quick solution. File > Get
External Data > Link... and then choose HTML. I was surprised how well it
worked when I tried it on a table I'd created in FrontPage.

Jim Royal · Feb 28, 2005

mark4 said:
Are there any utilities to help me extract Content from HTML ?

BBEdit has a simple menu command to remove markup from an HTML page,
leaving only the content. You should then perform any kind of regex
operation to massage the data before saving it.

To process all those files, it should be a pretty simple matter to
write an AppleScript to automate this procesure.

However, this solution is Macintosh-only.

Chrissy Cruiser · Feb 28, 2005

I can't see what those issues are.

By law, those posts are copyrighted and owned by the posters.

John Fitzsimons · Feb 28, 2005

Hello,

Are there any utilities to help me extract Content from HTML ?

< snip >

Notetab ? Modify - Strip HTML tags ?

http://www.notetab.com/

Not sure whether that is in the freeware version or not.

Regards, John.

Toby Inkster · Feb 28, 2005

mark4 said:
Thanks. Being a microserf, I don't normally code in Perl but I
may look into this.

I am told ActiveState's Windows port of Perl is pretty good. Alternatively
there is also a Cygwin version of Perl.

I can't see what those issues are. Who owns the data?

Its original authors, unless they explicitly signed away the copyright.

All those original authors have an alias and don't actually want to be
identified.

Publishing anonymously or under a pseudonym does not mean you forgo
copyright.

So long as I don't republish it.

If you are keeping the database for private use, then you can probably
"get away with it", but the natural assumption on alt.html is that posters
are wanting to publish their efforts to the web, unless it's explicitly
stated otherwise.

ggrothendieck · Mar 1, 2005

To do that, I recommend using Perl. Perl has a module called
HTML:

arser

Thanks. Being a microserf, I don't normally code in Perl but I
may look into this. It's either that or WSH Javascript with
it's regular expressions. Fortunately I already have a top
level design and it looks pretty simple. I may look into this
Perl module but it will probably be easier to use microserf
technology with which I'm intimate with. I shall probably store
it in MSSQL.

You could use the InternetExplorer.Application COM object.
That would give you the facilities for performing HTML
parsing without regexps. It would therefore be
more robust and readily doable in your favorite language.
Try google for examples.

mbstevens · Mar 1, 2005

mark4 said:
Hello,

Are there any utilities to help me extract Content from HTML ?

lynx -dump http://whateverTheHeck.com > temp.txt

.... is the shortest program I know of for this kind of thing.
The '>' redirection to temp.txt may vary somewhat between operating systems.

Python client/server that reads HTML body from server	1	Apr 12, 2023
Html Js forum	2	Apr 27, 2021
Batch modifying text - content and context based	5	Jan 19, 2023
Hi From Canada	3	Nov 26, 2023
CORS/Express: Getting data from server from domain html	2	Sep 3, 2022
HTML Site Problems	11	Nov 25, 2019
Digital Signature field form in PDF generated document from HTML	5	Nov 16, 2022
Generate one HTML from API based on the object key language and their value	2	Aug 19, 2022

Extract Content from HTML ?

mark4

Toby Inkster

mark4

Sherm Pendley

Philip Herlihy

Jim Royal

Chrissy Cruiser

John Fitzsimons

Toby Inkster

ggrothendieck

mbstevens

Ask a Question

Similar Threads

Members online

Forum statistics

Latest Threads