_
                           _ __   ___  ___| | _____
                          | '_ \ / _ \/ _ \ |/ / _ \
                          | | | |  __/  __/   < (_) |
                          |_| |_|\___|\___|_|\_\___/
                                                 _
                  _ __  _ __ ___  ___  ___ _ __ | |_ ___
                 | '_ \| '__/ _ \/ __|/ _ \ '_ \| __/ __|
                 | |_) | | |  __/\__ \  __/ | | | |_\__ \_ _ _
                 | .__/|_|  \___||___/\___|_| |_|\__|___(_|_|_)
                 |_|

    [This document best viewed at 80x25-- *cough* lynx]

[The Problem]-------------------|
The various coders of webspiders(robots) created a sort of "standard",
many webservers will contain a /robots.txt (http://example.com/robots.txt)...
In it are defined a list of directories a spider should not access.
Well, although not a problem within itself, lazy admins may add otherwise
unprotected directories to robots.txt believing noone would stumble across
them.  (ie. your favorite porn site.)

[Vulnerability Level]-------------------|
Not horribly high, it's more of a "you can find wierd shit scanning for
robots.txt" kind of problem than a script kiddie rootshell.com issue.
You _will_ find some wierd things if you look...

[What to do]-------------------|
Admins: Make sure all the directories you don't want accessed by random
surfers are .htaccess'd (or the equivalent).
Kids: Modify some of those lame cgi-bin scanners and see if you can find a 
few of the "Reporters and Stock holders Only" sections I've found.
(More harmless than rooting boxes, right?)

[Oops!  I'm sorry]-------------------|
I've been sitting on this for ages for several reasons,
for one, I'm lazy.  And the other (much more acceptable) reason would be
that I thought it to be rather common knowledge...

[References]
Only one,
   Linkname: A Standard for Robot Exclusion
   URL: http://info.webcrawler.com/mak/projects/robots/norobots.html
More information on the format of robots.txt, etc.

   [[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]

            Another (quality?) advisory from neeko. 
            Contact me at: neeko@mc2.nu
  Chill with some poetry.... http://www.angelfire.com/ca/optik9/poems.html