Jumping into a conversation with random web factoid:
While it isn't 100% you can set up some roadblocks to keep the index spiders
out of pages/directories. Here's some info:
http://www.robotstxt.org/wc/exclusion.html If I recall correctly, somewhere
it points out that the spider's author can code them to ignore the no-index
requests and just index it all.
----- Original Message -----
From: "Tom A. Trottier" <TomATrottier@xxxxxxxx>
> On 7 Apr 2001, at 13:01, sayeth Andrew Dacey <olympus@xxxxxxxxxxxxxxx>
> > On 4/7/01 10:25 AM, "Skip Williams" <skipwilliams@xxxxxxxxx> wrote:
<snip> If you're talking about search engine spiders, normally they are
setup to<snip>
< This message was delivered via the Olympus Mailing List >
< For questions, mailto:owner-olympus@xxxxxxxxxxxxxxx >
< Web Page: http://Zuiko.sls.bc.ca/swright/olympuslist.html >
|