Robots Meta Tag
If your web host prohibits you from uploading “robots.txt” to the root directory, or you simply wish to restrict crawlers from a few select pages on your site, an alternative to “robots.txt” is to use the robots meta tag.
Creating your “robots” meta tag
The “robots” meta tag looks similar to any meta tag, and should be added between the HEAD section of your page(s) in question:
<meta name=”robots” content=”noindex,nofollow” />
Here’s a list of the values you can specify within the “contents” attribute of this tag:
Value Description
(no)index Determines whether crawler should index this page. Possible values: “noindex” or “index”
(no)follow Determines whether crawler should follow links on this page and crawl them. Possible values: “nofollow” and “follow.”
