Web Spider

[Chase Preuninger]

If I was parsing a web page and extracting data from it in order to
make a search engine, what should I extract?

 

[Lord Zoltar]

If I was parsing a web page and extracting data from it in order to
make a search engine, what should I extract?

That depends on what you are interested in.

 

[Daniel Pitts]

That depends on what you are interested in.

Instead of a "that depends" answer...

You extract the information you want!

 

[Jeff Higgins]

If I was parsing a web page and extracting data from it in order to
make a search engine, what should I extract?

<http://en.wikipedia.org/wiki/Search_engine_(computing)>

 

[Chase Preuninger]

<http://en.wikipedia.org/wiki/Search_engine_(computing)>

just usefull stuff for a web search

 

[Jeff Higgins]

<http://en.wikipedia.org/wiki/Search_engine_(computing)>

just usefull stuff for a web search

<http://en.wikipedia.org/wiki/Sitemaps>

Both the links I've provided,
I've found using a web search engine
and the search terms: search, engine, wiki.

You could try searching on:
"most frequent search query", or
"most interesting search query", or
"most useful data for a web search engine".

 

[timjowers]

just usefull stuff for a web search

<http://en.wikipedia.org/wiki/Sitemaps>

Both the links I've provided,
I've found using a web search engine
and the search terms: search, engine, wiki.

You could try searching on:
"most frequent search query", or
"most interesting search query", or
"most useful data for a web search engine".

Chase. The basic is the words. Then you correlate the words into
clusters. Historically these are called Information Retrieval Systems
("IRS" oh my). The simplest idea is pages with words in common must be
like one another and like your topic. Imagine what would happen if you
had one list with all URLs. One list with all words. And one list
connecting the two. Then you could lookup all matching URL's for each
word. These lists might be large though! Then you could find the set
matching the search phrase by intersecting each set for each word.

Second thing to know is words have forms so maybe you'd work off of
all lower case and reduce all words to a base form. Well, what about
"Farenheit 451"? Do you also store numbers? What about "(...)"? Can
you also search on computerese? So, it starts to get complicated. One
idea is the "edit distance" or number of changes to get from the word
entered to a base word. That might tell if it might be the same word.
What about synonyms (I haven't seen a search engine do this). What
about bigrams and n-grams? That is, multi-word combinations. If one
types super computer then maybe any occurrences of "super computer"
should be matched higher than a page with just the word super or
computer.

OK, so a real search engine uses ranking and bases this on many
things. Things like how long the site has been up. How many other
sites link to them. How stuffed full of links their pages are. Maybe
if they buy ads from teh search engine? Nah, that wouldn't b
right. Etc. Also, by clustering a person's past searches or areas
of interest then you can greatly increase your precision.

In 2001 I took an IR course and we studied MSN, Google, and Yahoo.
Everyone found Google to have about the same recall (document
universe) but superior precision (accuracy). Now if I'd had the common
sense to buy stock!!!!

 

[timjowers]

Check this out: http://sourceforge.net/projects/webwordcnt/

Also see the Apache Lucene project. Its the base search engine used in
many commercial products.

 

[Roedy Green]

If I was parsing a web page and extracting data from it in order to
make a search engine, what should I extract?

you DON'T want the html tags
You DON'T want the header.
you DON'T want header/footer info common to all pages at a website.
you DON'T want common words like the that then is a ...
you DON'T want URLs