can I know how to write a html parser in C

Discussion in 'C Programming' started by WUV999U, Feb 23, 2005.

  1. WUV999U

    WUV999U Guest

    Hi

    I am fairly familiar in C but not much.

    I want to know how I can write a html parser in C that only parses for
    the image file in the html file and display or print
    all the images found in the html file.

    How to go about it?

    Should I have a file pointer and store the html file into an array
    first and then look for the img src..
    like do some string compare...

    Is there a sample on the net(not a hifi code,, a simple one) that I can
    look at to give me an idea on what I need to do.

    Thanks again
    WUV999U, Feb 23, 2005
    #1
    1. Advertising

  2. WUV999U

    Jarmo Guest

    "WUV999U" <> wrote in message
    news:...
    >
    > I want to know how I can write a html parser in C that only parses for
    > the image file in the html file and display or print
    > all the images found in the html file.


    You need to detect and print all img references that are found in an HTML
    file? I'd use Perl, not C.
    Jarmo, Feb 23, 2005
    #2
    1. Advertising

  3. WUV999U

    WUV999U Guest

    well,
    that was a great suggestion Jarmo. As you said, I can use Perl.
    But m afraid m not used to it.
    I need to get this done in a day or so..
    If I use C, how do I go about it?

    Thanks
    WUV999U, Feb 23, 2005
    #3
  4. WUV999U

    infobahn Guest

    WUV999U wrote:
    >
    > Hi
    >
    > I am fairly familiar in C but not much.
    >
    > I want to know how I can write a html parser in C that only parses for
    > the image file in the html file and display or print
    > all the images found in the html file.
    >
    > How to go about it?
    >
    > Should I have a file pointer and store the html file into an array
    > first and then look for the img src..
    > like do some string compare...


    That's certainly a valid approach, if you are sure you have the
    RAM to get the whole HTML file into memory (for this to be a problem,
    it would have to be a tiny computer and one mother of a Web page!).

    Use strchr to find a '<' character. Now you know you have a tag.
    I don't recall whether whitespace is allowed before the 'i' or 'I'
    of img; to be certain, skip past whitespace. isspace() will help
    you there. When you get past the whitespace, compare the next
    three characters, case-insensitively, to "img". If you have a
    match, press on and look for "src", which isn't necessarily just
    one whitespace away from "img", so be careful. Don't forget it
    might be "SRC" or even "sRc". The rest of this bit should be
    obvious.

    If the first non-whitespace char after '<' is /not/ 'i' or 'I',
    simply look for another '<'.

    Keep going until you run out of file.

    > Is there a sample on the net(not a hifi code,, a simple one) that I can
    > look at to give me an idea on what I need to do.


    Have a go at it yourself. If you get stuck, post your best-effort
    code here, and I expect someone will help you get unstuck again.
    infobahn, Feb 23, 2005
    #4
  5. In article <>,
    WUV999U <> wrote:
    :that was a great suggestion Jarmo. As you said, I can use Perl.
    :But m afraid m not used to it.

    Too bad, it'd be faster to write the program.

    :I need to get this done in a day or so..
    :If I use C, how do I go about it?

    State machine.

    set state = 0
    On each iteration of the loop, fetch one character

    State 0: if the character is < then set matchlen = 0 and transit to state 1
    otherwise discard the character and stay in state 0

    State 1: If you are in state 1 and tolower(character) is
    "img"[matchlen] then matchlen++; if matchlen=3 then transit to
    state 2 else stay in state 1
    else if the character is ! then transit to state 3 else transit to state 4

    State 2: recognize and discard whitespace (including newline).
    When you get the first non-whitespace character, then if you had
    no whitespace or if tolower(character) is not 'h' then transit to state 4
    else transit to state 5

    State 3: you might be in a comment. Do what you need to to figure out
    if you have a valid start of comment. When you have determined that you
    do, go to state 6; if you don't, go to state 4

    State 4: you are either not in an IMG tag or you are recovering from
    an error. In either case, you are not presently in quotes. accept and
    discard characters until you either get a '>' or you hit quotes; if you
    hit quotes, transit to a quote-absorbtion state

    State 5: you have recognized up to "<img h". recognize and accept
    characters that match "ref=\"" and then enter url acceptance mode;
    if you hit something else, go to state 4

    State 6: you are in a comment. accept and discard all characters until
    you find an end-of-comment marker or you find quotes. At end of comment
    go back to state 0; at quotes, go to state 7; otherwise stay in state 6

    State 7: you are inside quotes inside a comment. accept and discard
    all characters until you find an unescaped end of quote. When you
    do, go back to state 6; until then stay in state 7


    And so on. You can see the general outline -- and you can see some
    of the complications. You must account for comments! You must account
    for the possibility that what looks like the end of a comment is in
    the middle of a quoted string! You should probably take into account
    whether you are in an OBJ or javascript, since any IMG in those are
    not necessarily going to be shown. You should probably take into
    account that if you are within a LAYER that the layer might not be
    visible. You should probably take into account that if you are
    inside a FRAMES section that nothing there will ever be displayed:
    FRAMES sections can only have references to the frame files they import.
    You should probably take into account that if you are within a
    FRAMES section that you should be chasing the URLs named there because
    images referenced in them will be shown. You should probably take
    into account that an IMG reference in a HEAD section will not be
    displayed. And probably two or three fortnights worth of more complications.


    Frankly, if your C and programming experience is not strong enough
    that you didn't know how to go about starting this, then there is
    virtually no chance that you can properly impliment it in C within
    your "day or two" timeframe. HTML parsing has lots of Gotcha!'s.

    It would probably be faster for you to learn the rudiments of Perl and
    call upon the LWP moudle to extract the IMG tags for you, then it would
    be for you to write the parser in C.
    --
    If a troll and a half can hook a reader and a half in a posting and a half,
    how many readers can six trolls hook in six postings?
    Walter Roberson, Feb 23, 2005
    #5
  6. WUV999U

    Michael Mair Guest

    WUV999U wrote:
    > well,
    > that was a great suggestion Jarmo. As you said, I can use Perl.
    > But m afraid m not used to it.
    > I need to get this done in a day or so..
    > If I use C, how do I go about it?


    One simple way:
    - fopen() the file
    - read characters with getc() until EOF
    - implement a state stack with the following states:
    1 outside of body
    2 inside of body, normal text
    3 dito, tag opens
    4 dito, img tag
    5 dito, not img tag
    6 dito, comment starts
    7 dito, inside comment
    8 dito, comment ends
    9 dito, tag closes
    for 4 and 5, you also need
    a) inside tag, outside quoted
    b) dito, quote starts
    c) dito, inside quote
    d) dito, quote ends
    for 4), you also need e)-h) which is like a)-d) for src="path/to/pic"
    - enable only certain transitions
    - add special code for 4g)

    If you are certain that the only quoted part within an img tag
    is the path to the image, or that no graphics are commented out,
    etc., you can leave out the respective states.

    Cheers
    Michael
    --
    E-Mail: Mine is an /at/ gmx /dot/ de address.
    Michael Mair, Feb 23, 2005
    #6
  7. WUV999U

    Jarmo Guest

    "WUV999U" <> wrote in message
    news:...
    > well,
    > that was a great suggestion Jarmo. As you said, I can use Perl.
    > But m afraid m not used to it.
    > I need to get this done in a day or so..
    > If I use C, how do I go about it?
    >
    > Thanks


    If you have one day to do it then use Perl. Or a combination of existing
    tools such as grep and sed. The fact that you don't currently know how to
    do it in C indicates that you have little to no chance to learning how to do
    it in C in one day.
    Jarmo, Feb 24, 2005
    #7
  8. WUV999U

    WUV999U Guest

    oh OK,

    well let me check if I can use PERL and write back soon.
    Thank you sooooooo much for all your help.
    WUV999U, Feb 24, 2005
    #8
  9. WUV999U

    Jarmo Guest

    "WUV999U" <> wrote in message
    news:...
    > oh OK,
    >
    > well let me check if I can use PERL and write back soon.
    > Thank you sooooooo much for all your help.


    A Perl newsgroup will be able to help pretty quickly. Finding and isolating
    pieces of text is something that Perl's very good at.
    Jarmo, Feb 24, 2005
    #9
  10. WUV999U

    Randy Howard Guest

    In article <>,
    says...
    > Hi
    >
    > I am fairly familiar in C but not much.


    A sentence that contradicts itself.

    > I want to know how I can write a html parser in C that only parses for
    > the image file in the html file and display or print
    > all the images found in the html file.
    >
    > How to go about it?


    Numerous examples should be available on www.sf.net. You may find
    searching on freshmeat.net slightly easier, but they seem to all
    go to the same place in the end.

    > Should I have a file pointer and store the html file into an array
    > first and then look for the img src..
    > like do some string compare...


    That's certainly one way. It sort of depends upon how big the HTML
    files might be, and how much horsepower your target system has.

    > Is there a sample on the net(not a hifi code,, a simple one) that I can
    > look at to give me an idea on what I need to do.


    Google for "HTML parser C" found lots of them. Is there some reason
    you prefer Usenet to google for web searching?

    --
    Randy Howard (2reply remove FOOBAR)
    "Making it hard to do stupid things often makes it hard
    to do smart ones too." -- Andrew Koenig
    Randy Howard, Feb 25, 2005
    #10
  11. WUV999U

    Daniel Bruce Guest

    Walter Roberson wrote:
    > State 2: recognize and discard whitespace (including newline).
    > When you get the first non-whitespace character, then if you had
    > no whitespace or if tolower(character) is not 'h' then transit to state 4
    > else transit to state 5

    <snip>
    > State 5: you have recognized up to "<img h". recognize and accept
    > characters that match "ref=\"" and then enter url acceptance mode;
    > if you hit something else, go to state 4

    <snip>

    Just a slight nitpick on a seemingly good text(I have no idea about the subject
    myself, so can't really say anything about the quality of the text :)
    I was under the impression that image URLs were stored in the src attribute, and
    not the href one. :) Easy to switch anyways.
    Daniel Bruce, Feb 26, 2005
    #11
  12. WUV999U

    Alan Connor Guest

    On 23 Feb 2005 13:00:29 -0800, WUV999U
    <> wrote:

    > Hi
    >
    > I am fairly familiar in C but not much.
    >
    > I want to know how I can write a html parser in C that only
    > parses for the image file in the html file and display or print
    > all the images found in the html file.
    >
    > How to go about it?
    >
    > Should I have a file pointer and store the html file into an
    > array first and then look for the img src.. like do some string
    > compare...
    >
    > Is there a sample on the net(not a hifi code,, a simple one)
    > that I can look at to give me an idea on what I need to do.
    >
    > Thanks again
    >


    If you use linux/unix, something like this could work:

    ----------

    #!/bin/sh

    $tmp=$HOME/images.html

    echo "<HTML><HEAD><TITLE>Images</TITLE></HEAD><BODY>" >> $tmp

    wget -O - http://www.foo.com/whatever.foo | sed -n "s/\(.*\)\
    \(<[iI][mM][gG] [sS][rR][cC]="[^>]*">\)\(.*\)/\
    <P>\2/p" >> $tmp

    echo "</BODY></HTML>" >> $tmp

    --------


    AC
    Alan Connor, Feb 26, 2005
    #12
  13. In article <gVUTd.10342$>,
    Daniel Bruce <> wrote:
    :Walter Roberson wrote:
    :> State 5: you have recognized up to "<img h". recognize and accept
    :> characters that match "ref=\"" and then enter url acceptance mode;

    :I was under the impression that image URLs were stored in the src attribute, and
    :not the href one. :) Easy to switch anyways.

    You are right, I was thinking of anchors when I wrote that.
    --
    WW{Backus,Church,Dijkstra,Knuth,Hollerith,Turing,vonNeumann}D ?
    Walter Roberson, Feb 26, 2005
    #13
  14. WUV999U

    WUV999U Guest

    /**************HTML PARSER*************/

    void htmlparse(FILE *);

    int main(int argc, char * argv[])
    {

    FILE * op;
    op = fopen(argv[1],"r");
    if (op == NULL)
    {
    printf("Error opening file\n");
    exit(0);
    }

    htmlparse(op);
    return 1;

    }

    void htmlparse(FILE * op)
    {
    char line[81];
    char images[250];
    if (fgets(line,81,op) == NULL)
    {
    printf("Error reading data");
    exit(0);
    }

    puts(line);

    if(line == "<img src")
    {

    ------------------
    well,, thats all i hav......... and m stuck here...
    WUV999U, Feb 28, 2005
    #14
  15. In article <>,
    WUV999U <> wrote:
    : op = fopen(argv[1],"r");

    argv[1] might be NULL. You should be checking that you have the right
    number of parameters before you use any of them.

    :void htmlparse(FILE * op)
    :{
    : char line[81];

    Are the lines truly limited to 80 characters of text? It is not
    at all uncommon to encounter HTML in which the lines go on for
    several hundred characters.

    : char images[250];

    That declares a single character array named 'images' with a maximum
    null-terminated character string size of 249 characters. However,
    since you are only fetching 80 characters per line, the maximum
    image file name you are going to be able to extract is about 68
    characters (once you remove the tag and quotes.)

    If you want to allow for 250 images, then you should be declaring
    either an array of char * pointers or else a "two dimensional"
    array of characters.

    : if (fgets(line,81,op) == NULL)

    There's that magic number again, 81. Any time you have a number whose
    meaning is not obvious and which is repeated, you should either
    use a #define or store the value in a variable [which would have
    implications on how you would write the code.]

    : {
    : printf("Error reading data");
    : exit(0);
    : }

    Eventually you are going to run out of input and get NULL returned.
    That isn't an error: it is a signal that your function should
    finish up and return. As you have named the function 'htmlparse',
    the reader would tend to assume that -all- the function does is
    parse the input and extract certain information from it, but would
    not act upon that information, so the reader would tend to assume
    that you would return the list of images to the calling routine
    and let it do whatever should be done with the list.

    : puts(line);

    Why do you need to output the line at that point? The input file
    isn't going anywhere, so you are unlikely to need to duplicate the
    input.


    : if(line == "<img src")

    That is never going to be true. That is going to compare the
    *address* of the string "<img src" to the address of the character
    array 'line'. Since "<img src" is a literal string, it is not going
    to have the same address as your buffer.

    You also cannot fix this just by using strcmp() instead of testing
    the pointer: you need to be looking inside the line to find a place
    on the line (not necessarily at the beginning) where the string
    "<img src" occurs. Try strstr(). But watch out for comments and
    for the possibility that you might be within a quoted string...

    Note too that in the general case it is perfectly acceptable in HTML
    for there to be a linebreak between the "<img" and "src". Are you
    working with a very restricted subset of HTML? If so then it would
    help a lot to describe what the subset is. Some HTML subsets are
    very easy to parse, whereas HTML in general is fairly complex to
    parse.


    :well,, thats all i hav......... and m stuck here...

    Ekkk!

    No offense intended but you really haven't gotten very far
    at all and have made a number of mistakes in what you posted.
    Looking at this, we would tend to conclude that you are very
    much a beginner at C (and possibly a beginner at programming
    in general). Parsing general HTML is something that requires a
    fair bit of experience to program correctly; if what you posted
    is indeed representative of your C skills then you have no hope of
    writing a generalized HTML img file name extractor in any reasonable
    amount of time. Even a well-experienced programmer would take more
    than "a day or two" to write a proper HTML parser from scratch.

    [Of course, a well-experience programmer would know to *not*
    write it from scratch if it could be avoided: there are a number
    of already-written HTML parser libraries out there, and there
    are programs such as "lynx" which could be canablized. Writing
    from scratch would usually be reserved for instances in which
    there were notable copyright or patent issues at stake.]
    --
    IEA408I: GETMAIN cannot provide buffer for WATLIB.
    Walter Roberson, Feb 28, 2005
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. ZOCOR

    XML Parser VS HTML Parser

    ZOCOR, Oct 3, 2004, in forum: Java
    Replies:
    11
    Views:
    810
    Paul King
    Oct 5, 2004
  2. Zach Dennis

    HTML-Parser / SGML-Parser

    Zach Dennis, Oct 1, 2003, in forum: Ruby
    Replies:
    5
    Views:
    402
    Bernard Delmée
    Oct 1, 2003
  3. Replies:
    2
    Views:
    117
  4. Himanshu Garg
    Replies:
    2
    Views:
    116
    Himanshu Garg
    Feb 3, 2004
  5. Andries

    I know, I know, I don't know

    Andries, Apr 23, 2004, in forum: Perl Misc
    Replies:
    3
    Views:
    228
    Gregory Toomey
    Apr 23, 2004
Loading...

Share This Page