Make sure the URL has the http:// in front of it

Make sure to include the filename to spider (ie index.html)

This system looks for the information in the <HEAD> section of your documents: <TITLE> for the Link Title, and the ... <META> TAGS for keywords, description to build the description and search terms. It certainly helps if the documents the spider finds has them.