The "/robots.txt" file usually contains a record looking like this: 
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/
Disallow: /createmenu/