Use arrow keys to navigate

WordPress is truly a great piece of software. It really changed up dramatically the way people create, use and manage blogs or websites. But when it comes to duplicate content, boy they really messed it up.

It just scares me how Wordpress makes the categorization of items. Without a few tweaks in terms of dealing with duplicate content after a fresh WordPress you are dead. Normally the number of pages indexed by Google should be equal to the number written on posts and pages …  well, things are not that simple when using Wordpress. WordPress creates pages for paging, search, trackback, authors, dates, categories and archive, which can contain only the excerpt of the post or the entire post.

The way the content is arranged and structured in WordPress is useful for the users, no doubt, but becomes a problem when search engines index three times more pages than they should. Literally, on average, for a fresh WordPress installation with 10 published posts, Google ends up indexing close to 30 URLs, all pointing back to just 10 unique articles.

You might say you’ll get more traffic from Google if you have more pages in Google index, but there is a bigger chance of getting a penalty rather than a boost in traffic. There are obvious advantages of clearing up this mess, like better crawling results, better Page Rank distribution and original content.

The problem can be solved pretty simple, just by adding “noindex, nofollow” to unwanted pages. These pages can still be crawled by spiders; however the search engines will not include them in the index anymore.

Using the built-in WordPress functions you can create and add the code below inside your header.php file, before the </head> tag.

<?php if ( $paged > 1 ) {
echo '<meta name="robots" content="noindex,follow" /> ';
<?php if (is_author() ) {
echo '<meta name="robots" content="noindex,follow" /> ';
<?php if (is_trackback() ) {
echo '<meta name="robots" content="noindex,follow" /> ';
<?php if (is_search() ) {
echo '<meta name="robots" content="noindex,follow" /> ';
<?php if (is_date() ) {
echo '<meta name="robots" content="noindex,follow" /> ';

For my personal blog for example, I am allowing Google Bot and also the other search engine spiders to crawl and index my categories, which contain only the excerpts of my blog posts, so the content is not really the same as the content on the pages displaying the single posts.

You can use the following code if you want to exclude also the categories from the Google index.

<?php if (is_category() ) {
echo '<meta name="robots" content="noindex,follow" /> ';

Google likes pages that have large amounts of content, therefore pages like categories or archives, will most likely receive more credit then single post pages for example.

After using the code above you might notice a small decrease in traffic but it should be temporary, until the link juice gets redistributed between the pages that remained in the index.

You can check to see if everything went well by doing a search query site: and see if the pages you wanted to remove from Google index are still there or not. Depending on the crawl rate of your website it can take anywhere from a couple of hours to a few days for the changes appear in the results page.

We hope you will find the tips above useful. We would love to hear your feedback regarding this article, so please comment below! For more useful articles like this please don’t forget to subscribe to the RSS-feed and follow Inspirationfeed on TwitterFacebook! If you enjoyed the following article we humbly ask you to comment, and help us spread the word!

Use arrow keys to navigate

Posted by Claire

This post is written by Claire, an expert in Brazilian waxing and blogging. When she is not online she enjoys walking in the park or mountain biking.


  1. Hello Sir/Madam

    I need a suggestion/clarification from your side.

    Can I ALLOW crawling of Categories on my site? The category pages on my site shows only a part of post i.e. excerpt

    But still, many people says you shouldn’t allow Bots to Crawl Categories, even if they are showing excerpt

    Please suggest what to do?, as I want to ALLOW crawling for categories

    Thanks and Regards

    Mrinmay 🙂

  2. I just discovered that I had this problem with my blog. I’ve applied your suggestions – I hope it works! Many, many thanks for posting this.

  3. nice one for WP cms 🙂

  4. This trick is better than relying on plugins for managing robots.txt but we need might loose the code when we update or change the theme.

  5. I agree with Sreejesh. A great WordPress plugin achieving all this (and much more) is wpSEO.

    Kind regards,

  6. Well Written!

    You clarify several things that were not made exactly as clear in other posts I’ve researched on the subject.

    The code snippets were good, too.


  7. Any idea how to add noindex to query string pages like print and mobile redirect or multiple paged article with page number?

Leave a reply

Your email address will not be published. Required fields are marked *

Powered by Creative Market