Search engine bots - questions

[...D.]

Please bear with me. I know about .001% of search engines and the way they
work.;

I am curious as to what search engine bots actually read from a website? All
I have looked at is Google, and it seems to me it reports <h2> or <H-whatever>
from a page to show up when a search is done - or is it just reporting what is
at the top of a page.. Or is it looking at the meta tags?

So my real question is what do search engines look for when they look at a
website, & what do they display to a searcher? Or do different ones look for
different things?

Also - a pure HTML website using frames - is a search engine able to see the
sub-pages and report them, or does the site using frames not let a search
engine see beyond the index.html ?

Thanks...


...D.

 

[Andrew Urquhart]

*...D.* wrote in alt.html:

I am curious as to what search engine bots actually read from a website? All
I have looked at is Google, and it seems to me it reports <h2> or <H-whatever>
from a page to show up when a search is done - or is it just reporting what is
at the top of a page.. Or is it looking at the meta tags?

So my real question is what do search engines look for when they look at a
website, & what do they display to a searcher? Or do different ones look for
different things?

Also - a pure HTML website using frames - is a search engine able to see the
sub-pages and report them, or does the site using frames not let a search
engine see beyond the index.html ?

Read the Google Guidelines, they apply more or less equally well to
other search engine robots.

http://www.google.com/webmasters/guidelines.html

 

[I told you before]

go to www.kirwood.com for basics on search engines

x-no-archive:yes

 

[...D.]

...D. wrote in alt.html:

Read the Google Guidelines, they apply more or less equally well to
other search engine robots.
http://www.google.com/webmasters/guidelines.html

Ok thanks for the link, but I still do not know if a frames website can be
seen beyond the index.html file by bots, specifically Google's web bots. Here
is what it says, it is very vague:

I am somewhat new to HTML & having my own website. I cannot find the answer
to a question I have on any of your web pages.

----------- begin quote:
Use a text browser such as Lynx to examine your site, because most search
engine spiders see your site much as Lynx would. If fancy features such as
Javascript, cookies, session ID's, frames, DHTML, or Flash keep you from
seeing all of your site in a text browser, then search engine spiders may have
trouble crawling your site.
------------ end quote

So you can see that this is pretty vague, since Javascript is (or was) very
common, and of course everyone has cookies.

I question whether sub-pages can be seen, because the links from the menu
frame (all of my sub-pages are linked from there) do not use a conventional
link coding sequence. the part of the code sequence I am talking about says:
<A target="start", for every link, rather than a normal <A target="_blank" or
"_top". Everything else though, is the same. (see part of my menu frame code
below, part of a table in the frame).

Also I do have normal looking links in a <noframes> section of index.html, as
well as listing the same in my menu.html (left frame). I am wondering if a web
crawling bot looks inside of a <noframes> </noframes> section?

Thanks for your input. (website: www.bnt-blacknova.com )

<!-- ------- ------- -->
<table border="6" bgcolor="#FDF5E6" cellPadding="1" cellSpacing="3"
width="192" align="right">
<TBODY>

<TR><TD align=center><font size="3" face="arial black" color="#8B0000">
- Navigation -</A></font></TD></TR>

<TR><TD align=center><font size="2" face="microsoft sans serif">
<A target="start" href="start.html"><b>Home</b></A></font></TD></TR>

<TR><TD align=center><font size="2" face="microsoft sans serif">
<A target="start" href="2-chatter&news.html"><b>Chatter &
News</b></A></font></TD></TR>

<TR><TD align=center><font size="2" face="microsoft sans serif">
<A target="start" href="3-bnttips.html"><b>Tips &
Strategy</b></A></font></TD></TR>

<TR><TD align=center><font size="2" face="microsoft sans serif">
<A target="start" href="4-macroprograms.html"><b>* the gamers
edge</b></A></font></TD></TR>

etc etc

BTW I don't have have Lynx to examine my website.

...D.

 

[lostinspace]

<snip>
<snip>
<snip>

"I question whether sub-pages can be seen,"

ONLY if you include a link to an alternative page in the NOFRAMES section.

" I am wondering if a web crawling bot looks inside of a <noframes>
</noframes> section?"
Assuredly.

"BTW I don't have have Lynx to examine my website."

http://www.delorie.com/web/lynxview.html

 

[Art Sackett]

Please bear with me. I know about .001% of search engines and the way they
work.;

Keeping that in mind...

I am curious as to what search engine bots actually read from a website?

All of the markup, and in some cases *some* of the URL's embedded in
flash animations. Some search engines can also read any of a broad
range of document types.

All
I have looked at is Google, and it seems to me it reports <h2> or <H-whatever>
from a page to show up when a search is done - or is it just reporting what is
at the top of a page.. Or is it looking at the meta tags?

Google indexes everything, and pays very little attention to the META
descriptions and keywords. Whether or not a heading appears in their
site synopsis depends upon whether or not terms in the user-provided
search phrase occur within such an element.

So my real question is what do search engines look for when they look at a
website, & what do they display to a searcher? Or do different ones look for
different things?

Different search engines look at documents in different ways, but my
most recent research indicates that, for the most part, the most
popular search engines are far more similar than they'd have you
believe.

As for the site summary shown, some will just use any META description
provided, or fall back to the first readable line(s) of content. Google
shows the context of the search phrase and/or individual terms of your
search phrase as they actually appear within the document. Some don't
pay any attention to any of that, and rely upon the description
submitted by the webmaster.

Also - a pure HTML website using frames - is a search engine able to see the
sub-pages and report them, or does the site using frames not let a search
engine see beyond the index.html ?

The major (popular) search engines index frames just fine. Some older
"also-rans" don't, so if you really really really want to maximize the
potential for a new site to draw from everywhere, avoid frames.

 

[...D.]

Hey thanks you guys for your answers. They really helped out.

Now I have just one final question, I know that Google's bots automatically
found my website soon after I installed it. But what about Yahoo & Microsoft,
and others? Do I have to submit my website to them, or do they also
automatically crawl around the web & find & use?

...D.

 

[Big Bill]

Hey thanks you guys for your answers. They really helped out.

Now I have just one final question, I know that Google's bots automatically
found my website soon after I installed it. But what about Yahoo & Microsoft,
and others? Do I have to submit my website to them, or do they also
automatically crawl around the web & find & use?

Assuming that Google found you through an incoming link, the other
major engines will find you the same way. If you build a specialised
site, then you'll need to find specialist directories to submit to,
but engines will find you on their own.

BB