+1 vote

What is a meta robots tag and do I need one?

asked Dec 22, 2016 in SEO Tips by anonymous

1 Answer

+2 votes
Best answer

About the Robots <META> tag

In a nutshell

You can use a special HTML <META> tag to tell robots not to index the content of a page, and/or not scan it for links to follow.

For example:

<html>
<head>
<title>...</title>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>

There are two important considerations when using the robots <META> tag:

  • robots can ignore your <META> tag. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
  • the NOFOLLOW directive only applies to links on this page. It's entirely likely that a robot might find the same links on some other page without a NOFOLLOW (perhaps on some other site), and so still arrives at your undesired page.

Don't confuse this NOFOLLOW with the rel="nofollow" link attribute.

The details

Like the /robots.txt, the robots META tag is a de-facto standard. It originated from a "birds of a feather" meeting at a 1996 distributed indexing workshop, and was described in meeting notes.

The META tag is also described in the HTML 4.01 specification, Appendix B.4.1.

The rest of this page gives an overview of how to use the robots <META> tags in your pages, with some simple recipes. To learn more see also the FAQ.

How to write a Robots Meta Tag

Where to put it

Like any <META> tag it should be placed in the HEAD section of an HTML page, as in the example above. You should put it in every page on your site, because a robot can encounter a deep link to any page on your site.

What to put into it

The "NAME" attribute must be "ROBOTS".

Valid values for the "CONTENT" attribute are: "INDEX", "NOINDEX", "FOLLOW", "NOFOLLOW". Multiple comma-separated values are allowed, but obviously only some combinations make sense. If there is no robots <META> tag, the default is "INDEX,FOLLOW", so there's no need to spell that out. That leaves:

<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
answered Dec 23, 2016 by anonymous
selected Dec 23, 2016 by admin
Welcome to ARTOFSEO!
Introducing SEO Q&A , where you can ask questions related to SEO and the industry and receive answers from other members of the SEO community.
We recommend you include a link to your website to increase your relevance and showcase your company in the community.
Play Nice!
274 questions
56,021 answers
5,164 users