On the contrary. All we have to do is find a programmer who advocates HTML
as a valid universal format, and ask him to produce an HTML-aware version
of grep.
If one's only goal is to test for spam, there is an easier method:
just remove (or separate out) the HTML tags.
There are two ways to go about this; one works better than another.
The "dumb" way is to split an input into two outputs based only
on the "<" and ">" characters:
/*
* Copy input from "in" to "txt" and "tag" output files.
* Text in <angle brackets> goes to the "tag" file, the
* rest goes to the "txt" file.
*/
int separate(FILE *in, FILE *txt, FILE *tag) {
int c;
int in_tag = 0, next_tag;
while ((c = getc(in)) != EOF) {
next_tag = in_tag;
if (c == '<')
in_tag = 1;
else if (c == '>')
next_tag = 0;
putc(c, in_tag ? tag : txt);
in_tag = next_tag;
}
}
The "smart" way is to process the HTML and recognize the parts
of text that browser-type readers discard. This works better
for spam-detection, because some spammers have switched from:
s<!--junk>pamkeyword
to something more like:
s<script language="unknown">junk</script>pamkeyword
The spammers are doing this because people are stripping HTML
comments before feeding text to spam-detectors.
(None of this really addresses the fundamental problem, which is
the fact that spam is theft. As long as the spammer's ill-gotten
profits exceed his or her cost and risk, the spammer comes out
ahead on average. Since the cost and risk is miniscule -- especially
with spammers hijacking Windows-based boxes to send the actual spam
-- the required "per-spam profit" is likewise miniscule. The only
real solution is to raise the cost of spamming, either directly
[e.g., in dollars] or indirectly [e.g., by jailing the thieves for
their hijacking of cable-connected home PCs]. This will not make
the spam stop completely, but will reduce it to manageable levels.
Content-filtering is an attempt to increase the cost, but because
spammers *are* thieves, the increased costs are mainly borne by
those being stolen from, rather than the spammers.)
, at least