Web crawler

Discussion in 'C Programming' started by Oscarian, Jan 11, 2007.

  1. Oscarian

    Oscarian Guest

    Does anyone have any tips/tutorials on how to write a web crawler using C?

    On that note, is C a "good" language to use for a web crawler?

    -O
    Oscarian, Jan 11, 2007
    #1
    1. Advertising

  2. Oscarian

    Guest

    Oscarian wrote:
    > Does anyone have any tips/tutorials on how to write a web crawler using C?


    If you have to ask, you're probably not ready to do it.

    > On that note, is C a "good" language to use for a web crawler?


    What do you mean by "good"? It wouldn't be my first choice. Other
    languages probably provide more infrastructure for the job "out of the
    box".
    , Jan 11, 2007
    #2
    1. Advertising

  3. Oscarian

    Oscarian Guest

    On Thu, 11 Jan 2007 01:09:17 -0800, mark_bluemel wrote:

    > If you have to ask, you're probably not ready to do it.
    >


    Indeed, but I enjoy having goals, even if it is a long term goal.

    >> On that note, is C a "good" language to use for a web crawler?

    >
    > What do you mean by "good"? It wouldn't be my first choice. Other
    > languages probably provide more infrastructure for the job "out of the
    > box".



    "Good", as in "providing best efficiency". I guess.

    -O
    Oscarian, Jan 11, 2007
    #3
  4. Oscarian said:

    > On Thu, 11 Jan 2007 01:09:17 -0800, mark_bluemel wrote:
    >
    >> If you have to ask, you're probably not ready to do it.
    >>

    >
    > Indeed, but I enjoy having goals, even if it is a long term goal.
    >
    >>> On that note, is C a "good" language to use for a web crawler?

    >>
    >> What do you mean by "good"? It wouldn't be my first choice. Other
    >> languages probably provide more infrastructure for the job "out of the
    >> box".

    >
    >
    > "Good", as in "providing best efficiency". I guess.


    Language isn't the efficiency bottleneck with Internet applications. The
    Internet is!

    Any language that can do the job is a good language. Sort them in order of
    your preference. If you would rather write this in SNOBOL++, then write it
    in SNOBOL++. If you'd rather write it in Perl, write it in Perl. If you'd
    rather write it in C, write it in C. And if you'd rather write it in MS-DOS
    batch script, you're a braver man than me!

    Different languages exist because not everybody thinks the same way. You
    have a choice. Use it. Don't let others dictate it.

    --
    Richard Heathfield
    "Usenet is a strange place" - dmr 29/7/1999
    http://www.cpax.org.uk
    email: rjh at the above domain, - www.
    Richard Heathfield, Jan 11, 2007
    #4
  5. Oscarian

    Guest

    Oscarian wrote:
    > On Thu, 11 Jan 2007 01:09:17 -0800, mark_bluemel wrote:
    >
    > > If you have to ask, you're probably not ready to do it.
    > >

    >
    > Indeed, but I enjoy having goals, even if it is a long term goal.


    Fine, but if you want to research writing a Web Crawler, asking on a C
    language newsgroup is probably not the best starting point.
    <Off-topic>

    The principle is fairly simple:-
    1. Make a list of URLs
    2. Access the next URL from the list
    3. Find any links in the URL, add them to your list, discarding
    duplicates
    4. Go to 2

    What language would you find it easiest to do this in? As I've already
    stated, I probably wouldn't choose C - as I work with Java regularly, I
    might choose that (out of the box, it has useful classes for accessing
    URLs, maintaining lists, etc), or I might use a scripting language such
    as Python or even Perl. As Richard has pointed out, the choice of
    language is unlikely to have a huge impact on performance.

    There is a discussion of WebCrawlers on Wikipedia, with links to a
    variety of implementations in a range of languages. Entering "writing a
    webcrawler" into Google leads to an assortment of papers, tutorials
    etc.
    </Off-topic>
    , Jan 11, 2007
    #5
  6. Oscarian

    Oscarian Guest

    OT: Re: Web crawler

    <Off Topic>

    Thanks guys, I will do some playful stuff and once I have learned enough I
    will plug away at doing something like a web crawler.

    On Thu, 11 Jan 2007 09:52:37 +0000, Richard Heathfield wrote:

    > Different languages exist because not everybody thinks the same way. You
    > have a choice. Use it. Don't let others dictate it.


    Richard, you sound like an old hacker trying to open the minds of us
    inexperienced guys. Thanks for the encouragement.

    On Thu, 11 Jan 2007 02:24:32 -0800, mark_bluemel wrote:

    > There is a discussion of WebCrawlers on Wikipedia, with links to a
    > variety of implementations in a range of languages. Entering "writing a
    > webcrawler" into Google leads to an assortment of papers, tutorials
    > etc.


    I know, I did a search and read the wiki page and also looked at the
    google results. None of them mentioned C, which is why I posted to the
    group. I'll dig through the tutorials in Java and see if I can come up
    with some psuedocode to do the same thing in C.

    Thanks though.

    Cheers-
    O

    > </Off-topic>
    Oscarian, Jan 12, 2007
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Paul Morrison

    Web Crawler

    Paul Morrison, Oct 17, 2005, in forum: Java
    Replies:
    3
    Views:
    4,910
    lamantpirate
    Jun 30, 2012
  2. Sanjay Patra

    Web Crawler

    Sanjay Patra, Nov 17, 2004, in forum: C++
    Replies:
    2
    Views:
    713
  3. abhinav

    web crawler in python or C?

    abhinav, Feb 16, 2006, in forum: Python
    Replies:
    13
    Views:
    1,265
  4. Sanjay Patra

    C Web crawler code

    Sanjay Patra, Nov 18, 2004, in forum: C Programming
    Replies:
    1
    Views:
    1,426
    Raymond Martineau
    Nov 18, 2004
  5. abhinav

    web crawler in python or C?

    abhinav, Feb 16, 2006, in forum: C Programming
    Replies:
    1
    Views:
    1,405
Loading...

Share This Page