can I know how to write a html parser in C

W

WUV999U

Hi

I am fairly familiar in C but not much.

I want to know how I can write a html parser in C that only parses for
the image file in the html file and display or print
all the images found in the html file.

How to go about it?

Should I have a file pointer and store the html file into an array
first and then look for the img src..
like do some string compare...

Is there a sample on the net(not a hifi code,, a simple one) that I can
look at to give me an idea on what I need to do.

Thanks again
 
J

Jarmo

WUV999U said:
I want to know how I can write a html parser in C that only parses for
the image file in the html file and display or print
all the images found in the html file.

You need to detect and print all img references that are found in an HTML
file? I'd use Perl, not C.
 
W

WUV999U

well,
that was a great suggestion Jarmo. As you said, I can use Perl.
But m afraid m not used to it.
I need to get this done in a day or so..
If I use C, how do I go about it?

Thanks
 
I

infobahn

WUV999U said:
Hi

I am fairly familiar in C but not much.

I want to know how I can write a html parser in C that only parses for
the image file in the html file and display or print
all the images found in the html file.

How to go about it?

Should I have a file pointer and store the html file into an array
first and then look for the img src..
like do some string compare...

That's certainly a valid approach, if you are sure you have the
RAM to get the whole HTML file into memory (for this to be a problem,
it would have to be a tiny computer and one mother of a Web page!).

Use strchr to find a '<' character. Now you know you have a tag.
I don't recall whether whitespace is allowed before the 'i' or 'I'
of img; to be certain, skip past whitespace. isspace() will help
you there. When you get past the whitespace, compare the next
three characters, case-insensitively, to "img". If you have a
match, press on and look for "src", which isn't necessarily just
one whitespace away from "img", so be careful. Don't forget it
might be "SRC" or even "sRc". The rest of this bit should be
obvious.

If the first non-whitespace char after '<' is /not/ 'i' or 'I',
simply look for another '<'.

Keep going until you run out of file.
Is there a sample on the net(not a hifi code,, a simple one) that I can
look at to give me an idea on what I need to do.

Have a go at it yourself. If you get stuck, post your best-effort
code here, and I expect someone will help you get unstuck again.
 
W

Walter Roberson

:that was a great suggestion Jarmo. As you said, I can use Perl.
:But m afraid m not used to it.

Too bad, it'd be faster to write the program.

:I need to get this done in a day or so..
:If I use C, how do I go about it?

State machine.

set state = 0
On each iteration of the loop, fetch one character

State 0: if the character is < then set matchlen = 0 and transit to state 1
otherwise discard the character and stay in state 0

State 1: If you are in state 1 and tolower(character) is
"img"[matchlen] then matchlen++; if matchlen=3 then transit to
state 2 else stay in state 1
else if the character is ! then transit to state 3 else transit to state 4

State 2: recognize and discard whitespace (including newline).
When you get the first non-whitespace character, then if you had
no whitespace or if tolower(character) is not 'h' then transit to state 4
else transit to state 5

State 3: you might be in a comment. Do what you need to to figure out
if you have a valid start of comment. When you have determined that you
do, go to state 6; if you don't, go to state 4

State 4: you are either not in an IMG tag or you are recovering from
an error. In either case, you are not presently in quotes. accept and
discard characters until you either get a '>' or you hit quotes; if you
hit quotes, transit to a quote-absorbtion state

State 5: you have recognized up to "<img h". recognize and accept
characters that match "ref=\"" and then enter url acceptance mode;
if you hit something else, go to state 4

State 6: you are in a comment. accept and discard all characters until
you find an end-of-comment marker or you find quotes. At end of comment
go back to state 0; at quotes, go to state 7; otherwise stay in state 6

State 7: you are inside quotes inside a comment. accept and discard
all characters until you find an unescaped end of quote. When you
do, go back to state 6; until then stay in state 7


And so on. You can see the general outline -- and you can see some
of the complications. You must account for comments! You must account
for the possibility that what looks like the end of a comment is in
the middle of a quoted string! You should probably take into account
whether you are in an OBJ or javascript, since any IMG in those are
not necessarily going to be shown. You should probably take into
account that if you are within a LAYER that the layer might not be
visible. You should probably take into account that if you are
inside a FRAMES section that nothing there will ever be displayed:
FRAMES sections can only have references to the frame files they import.
You should probably take into account that if you are within a
FRAMES section that you should be chasing the URLs named there because
images referenced in them will be shown. You should probably take
into account that an IMG reference in a HEAD section will not be
displayed. And probably two or three fortnights worth of more complications.


Frankly, if your C and programming experience is not strong enough
that you didn't know how to go about starting this, then there is
virtually no chance that you can properly impliment it in C within
your "day or two" timeframe. HTML parsing has lots of Gotcha!'s.

It would probably be faster for you to learn the rudiments of Perl and
call upon the LWP moudle to extract the IMG tags for you, then it would
be for you to write the parser in C.
 
M

Michael Mair

WUV999U said:
well,
that was a great suggestion Jarmo. As you said, I can use Perl.
But m afraid m not used to it.
I need to get this done in a day or so..
If I use C, how do I go about it?

One simple way:
- fopen() the file
- read characters with getc() until EOF
- implement a state stack with the following states:
1 outside of body
2 inside of body, normal text
3 dito, tag opens
4 dito, img tag
5 dito, not img tag
6 dito, comment starts
7 dito, inside comment
8 dito, comment ends
9 dito, tag closes
for 4 and 5, you also need
a) inside tag, outside quoted
b) dito, quote starts
c) dito, inside quote
d) dito, quote ends
for 4), you also need e)-h) which is like a)-d) for src="path/to/pic"
- enable only certain transitions
- add special code for 4g)

If you are certain that the only quoted part within an img tag
is the path to the image, or that no graphics are commented out,
etc., you can leave out the respective states.

Cheers
Michael
 
J

Jarmo

WUV999U said:
well,
that was a great suggestion Jarmo. As you said, I can use Perl.
But m afraid m not used to it.
I need to get this done in a day or so..
If I use C, how do I go about it?

Thanks

If you have one day to do it then use Perl. Or a combination of existing
tools such as grep and sed. The fact that you don't currently know how to
do it in C indicates that you have little to no chance to learning how to do
it in C in one day.
 
W

WUV999U

oh OK,

well let me check if I can use PERL and write back soon.
Thank you sooooooo much for all your help.
 
J

Jarmo

WUV999U said:
oh OK,

well let me check if I can use PERL and write back soon.
Thank you sooooooo much for all your help.

A Perl newsgroup will be able to help pretty quickly. Finding and isolating
pieces of text is something that Perl's very good at.
 
R

Randy Howard

Hi

I am fairly familiar in C but not much.

A sentence that contradicts itself.
I want to know how I can write a html parser in C that only parses for
the image file in the html file and display or print
all the images found in the html file.

How to go about it?

Numerous examples should be available on www.sf.net. You may find
searching on freshmeat.net slightly easier, but they seem to all
go to the same place in the end.
Should I have a file pointer and store the html file into an array
first and then look for the img src..
like do some string compare...

That's certainly one way. It sort of depends upon how big the HTML
files might be, and how much horsepower your target system has.
Is there a sample on the net(not a hifi code,, a simple one) that I can
look at to give me an idea on what I need to do.

Google for "HTML parser C" found lots of them. Is there some reason
you prefer Usenet to google for web searching?
 
D

Daniel Bruce

Walter said:
State 2: recognize and discard whitespace (including newline).
When you get the first non-whitespace character, then if you had
no whitespace or if tolower(character) is not 'h' then transit to state 4
else transit to state 5
State 5: you have recognized up to "<img h". recognize and accept
characters that match "ref=\"" and then enter url acceptance mode;
if you hit something else, go to state 4
<snip>

Just a slight nitpick on a seemingly good text(I have no idea about the subject
myself, so can't really say anything about the quality of the text :)
I was under the impression that image URLs were stored in the src attribute, and
not the href one. :) Easy to switch anyways.
 
A

Alan Connor

Hi

I am fairly familiar in C but not much.

I want to know how I can write a html parser in C that only
parses for the image file in the html file and display or print
all the images found in the html file.

How to go about it?

Should I have a file pointer and store the html file into an
array first and then look for the img src.. like do some string
compare...

Is there a sample on the net(not a hifi code,, a simple one)
that I can look at to give me an idea on what I need to do.

Thanks again

If you use linux/unix, something like this could work:

----------

#!/bin/sh

$tmp=$HOME/images.html

echo "<HTML><HEAD><TITLE>Images</TITLE></HEAD><BODY>" >> $tmp

wget -O - http://www.foo.com/whatever.foo | sed -n "s/\(.*\)\
\(<[iI][mM][gG] [sS][rR][cC]="[^>]*">\)\(.*\)/\
<P>\2/p" >> $tmp

echo "</BODY></HTML>" >> $tmp
 
W

Walter Roberson

:> State 5: you have recognized up to "<img h". recognize and accept
:> characters that match "ref=\"" and then enter url acceptance mode;

:I was under the impression that image URLs were stored in the src attribute, and
:not the href one. :) Easy to switch anyways.

You are right, I was thinking of anchors when I wrote that.
 
W

WUV999U

/**************HTML PARSER*************/

void htmlparse(FILE *);

int main(int argc, char * argv[])
{

FILE * op;
op = fopen(argv[1],"r");
if (op == NULL)
{
printf("Error opening file\n");
exit(0);
}

htmlparse(op);
return 1;

}

void htmlparse(FILE * op)
{
char line[81];
char images[250];
if (fgets(line,81,op) == NULL)
{
printf("Error reading data");
exit(0);
}

puts(line);

if(line == "<img src")
{
 
W

Walter Roberson

: op = fopen(argv[1],"r");

argv[1] might be NULL. You should be checking that you have the right
number of parameters before you use any of them.

:void htmlparse(FILE * op)
:{
: char line[81];

Are the lines truly limited to 80 characters of text? It is not
at all uncommon to encounter HTML in which the lines go on for
several hundred characters.

: char images[250];

That declares a single character array named 'images' with a maximum
null-terminated character string size of 249 characters. However,
since you are only fetching 80 characters per line, the maximum
image file name you are going to be able to extract is about 68
characters (once you remove the tag and quotes.)

If you want to allow for 250 images, then you should be declaring
either an array of char * pointers or else a "two dimensional"
array of characters.

: if (fgets(line,81,op) == NULL)

There's that magic number again, 81. Any time you have a number whose
meaning is not obvious and which is repeated, you should either
use a #define or store the value in a variable [which would have
implications on how you would write the code.]

: {
: printf("Error reading data");
: exit(0);
: }

Eventually you are going to run out of input and get NULL returned.
That isn't an error: it is a signal that your function should
finish up and return. As you have named the function 'htmlparse',
the reader would tend to assume that -all- the function does is
parse the input and extract certain information from it, but would
not act upon that information, so the reader would tend to assume
that you would return the list of images to the calling routine
and let it do whatever should be done with the list.

: puts(line);

Why do you need to output the line at that point? The input file
isn't going anywhere, so you are unlikely to need to duplicate the
input.


: if(line == "<img src")

That is never going to be true. That is going to compare the
*address* of the string "<img src" to the address of the character
array 'line'. Since "<img src" is a literal string, it is not going
to have the same address as your buffer.

You also cannot fix this just by using strcmp() instead of testing
the pointer: you need to be looking inside the line to find a place
on the line (not necessarily at the beginning) where the string
"<img src" occurs. Try strstr(). But watch out for comments and
for the possibility that you might be within a quoted string...

Note too that in the general case it is perfectly acceptable in HTML
for there to be a linebreak between the "<img" and "src". Are you
working with a very restricted subset of HTML? If so then it would
help a lot to describe what the subset is. Some HTML subsets are
very easy to parse, whereas HTML in general is fairly complex to
parse.


:well,, thats all i hav......... and m stuck here...

Ekkk!

No offense intended but you really haven't gotten very far
at all and have made a number of mistakes in what you posted.
Looking at this, we would tend to conclude that you are very
much a beginner at C (and possibly a beginner at programming
in general). Parsing general HTML is something that requires a
fair bit of experience to program correctly; if what you posted
is indeed representative of your C skills then you have no hope of
writing a generalized HTML img file name extractor in any reasonable
amount of time. Even a well-experienced programmer would take more
than "a day or two" to write a proper HTML parser from scratch.

[Of course, a well-experience programmer would know to *not*
write it from scratch if it could be avoided: there are a number
of already-written HTML parser libraries out there, and there
are programs such as "lynx" which could be canablized. Writing
from scratch would usually be reserved for instances in which
there were notable copyright or patent issues at stake.]
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,901
Latest member
Noble71S45

Latest Threads

Top