Paginated archives have long been a topic of discussion in the SEO community. Over time, best practices for optimization have evolved, and we now have pretty clear definitions. This post explains what these best practices are. It’s good to know that Yoast SEO applies all these rules to every archive with pagination.

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO: the #1 WordPress SEO plugin Info

Indicate that an archive has pagination

When a search engine crawls page one of an archive, it needs to know it’s a paginated archive. For the longest time, the only way for it to know that something was a paginated archive is when it found a “next” or “previous link”. This was solved by the introduction of rel="next" and rel="prev" link-elements, to be applied in the head of a page, a topic we’ve written about before.

For a while, there was a discussion in the SEO community about how to combine this with rel canonical. Should page 2 and further of an archive have a canonical link to page 1, or to itself? The idea was that you mostly want visitors to end up on page 1 of an archive. That page is usually the most relevant for the majority of users.

Google is very clear now: each page within a paginated series should canonicalize to itself, so /page/2/ has a canonical pointing to /page/2/.

Should page 2 etc. be in the search results?

For a while, SEOs thought it might be a good idea to add a noindex robots meta tag to page 2 and further of a paginated archive. This would prevent people from finding page 2 and further in the search results. The idea was that the search engine would still follow all these links, so all the linked pages would still be properly indexed.

The problem is that at the end of last year, Google said something that caught our attention: long-term noindex on a page will lead to them not following links on that page. This makes adding noindex to page 2 and further of paginated archives a bad idea, as it might lead to your articles no longer getting the internal links they need.

Because of what Google said about long-term noindex, in Yoast SEO 6.3 we removed the option to add noindex to subpages of archives.

Annoying SEO side effects

So you can no longer keep page 2 and further out of the search results. This has the annoying side effect that Google Search Console might start to give you warnings. Specifically, it might warn you about duplicate titles and duplicate meta descriptions. You can safely ignore these warnings, a fact I’ve confirmed with Google this week:

I guess, in time, Google will stop showing these warnings for paginated archives in Google Search Console.

Read on: ‘Why every website needs Yoast SEO’ »

The post Pagination & SEO: best practices appeared first on Yoast.

Some of the pages of your site serve a purpose, but that purpose isn’t ranking in search engines or even getting traffic to your site. These pages need to be there as glue for other pages, or simply because whatever regulations require them to be accessible on your website. As a regular visitor to our website, you know what noindex or nofollow can do to these pages. If you are new to these terms, please read on and let me explain what they are and what pages they might apply to!

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO: the #1 WordPress SEO plugin Info

What is noindex or nofollow?

Both are settings you can add to your robots meta tag. We did quite an extensive ultimate guide on the robots meta tag that you should read.

In short:

  • It looks like this in most cases:
    <meta name="robots" content="[VALUE1,VALUE2]">
  • VALUE1 and VALUE2 are set to index, follow by default, meaning the page at hand can be indexed and all links on that page can be followed by search engine spiders to index the pages they link to.
  • VALUE1 and VALUE2 can be set to noindex, nofollow as well. noindex means that the page shouldn’t be indexed by search engines, but doesn’t mean the search engines shouldn’t follow the links on the page. nofollow means that it also shouldn’t follow the links.

Pages that you might want to noindex

Author archives on a one-author blog

If you are the only one writing for your blog, your author pages are probably 90% the same as your blog homepage. That’s of no use to Google and can be considered duplicate content. To keep these out of the search results, you can noindex them.

Certain (custom) post types

Sometimes a plugin or a web developer adds a custom post type that you don’t want to be indexed. At Yoast, we use custom pages for our products, as we are not a regular online shop that sells, for instance, kitchen appliances. We don’t need a product image, filters like dimensions and technical specifications on a tab next to the description. Therefore, we noindex the regular product pages WooCommerce outputs and are using our own pages. Indeed, we noindex the product post type.

By the way, I have seen shop solutions that added things like dimensions and weight as a custom post type as well. These pages are considered to be low-quality content. You will understand that these pages have no use for a visitor or Google, so need to be kept out of the search result pages.

Thank you pages

That page serves no other purpose than to thank your customer/newsletter subscriber. Usually thin content, or upsell and social share options, but no added value content-wise.

Admin and login pages

Of course, your login pages are not in Google. But these are. Keep them out of the index by adding that noindex. Exceptions are the login pages that serve a community, like Dropbox or similar services. Just ask yourself if you would google for one of your login pages if you were not in your company. If not, it’s probably safe to say that Google doesn’t need to index these pages.

Internal search results

Internal search results are like the last pages Google wants to point its visitors to. If you want to ruin a search experience, you link to other search pages. But the links on that search result page are still very valuable for Google, so all links should be followed. The robots meta setting should be:
<meta name="robots" content="noindex, follow">

The same setting goes for all the examples mentioned above, there is no need to nofollow the links on these pages. Now, when should you add a nofollow to your robots meta tag?

Pages that you might want to nofollow

Google roughly indicates that there are three reasons to nofollow links:

  1. Untrusted content
  2. Paid links
  3. Crawl prioritization

For instance, we add a nofollow tag to links in comments. We don’t know what all of you are dropping in there, right. It could be anything from the #1 and #2 above. With regards to number 3, this could, for instance, apply to login links, that we sometimes find on WordPress websites, see image on the right. It’s no use having a Googlebot go over these links, as for search engines, they add no value. These are nofollowed.

All of the above is very much on a link level. But if you have for instance a page that shows SEO books, with a surplus of Amazon affiliate links, these might add value to your site for your users. But I’d nofollow that entire page if there’s nothing else that matters on the page. You might have it indexed, though. Just make sure you cloak your links the right way.

To be honest, on a regular website, I don’t think there are a lot of pages I’d set to nofollow. Check for yourself if you have any content that mainly contains links like the ones Google indicated, and decide if Google should follow them or not.

Changing SEO insights

At Yoast, we always try to keep you on top of your SEO game, without per se bugging you about it. One of the settings in Yoast SEO that we have had for years, the “Noindex subpages of archives” checkbox is one of those. It made all the sense in the world to noindex, follow these, and have Google index just the main page, the first page of your (f.i.) category archive.

We were always aware that Google was getting better and better at understanding rel="next" and rel="prev" on these subpages of archives. Yoast SEO adds these tags as well. At this point, we know that rel="next" and rel="prev" cover the way archives should be indexed and noindex-ing subpages isn’t necessary anymore, so we’ve removed that setting from our plugin altogether to make sure it’s done right on your site!

Read on: ‘Prevent your site from being indexed, the right way’ »

The post Which pages should I noindex or nofollow? appeared first on Yoast.

Whenever you make some big changes to your website, for instance to your brand name, you’re probably eager for these changes to show in the search results. Unfortunately, it can take a while for Google to crawl your site again and until then, it will show the indexed version of your site in the results, without the changes. Of course, when people click to your site they see’ll the changes, but you want them to be visible in the results pages too.

So, is there anything you can do to help or speed up this process? And is there anything else to bear in mind when making these changes? Let’s get into that in this Ask Yoast!

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO: the #1 WordPress SEO plugin Info

Jolene Moody emailed us her question on this topic:

“I recently changed the name of my business. We have changed it in the WordPress dashboard too. But I didn’t see the change yet in the search results. How long does it take Google to show this change?”

Watch the video or read the transcript further down the page for my answer!

Helping Google pick up changes on your site

“It depends on how often Google visits your site and you probably don’t know how often that is. Now, what you can do is go to Google search console and go to ‘Fetch & render’ and then fetch and render your homepage. Then, after it’s done, there’s an option to submit to index. At that point, Google will have already crawled your site and will use that data to show your site in the index, so when people search for your brand, at least your homepage will have the proper brand name.

But it’s very important, if you change the name of your business and people are still searching for the old name of your business, that you also have the old name of your business on your site somewhere. That way people can still find you for that, when they don’t know that you’ve renamed your business. Good luck!”

Ask Yoast

In the series Ask Yoast we answer SEO questions from our readers. Have an SEO-related question? Let us help you out! Send an email to

(Note: please check our blog and knowledge base first, the answer to your question may already be out there! For urgent questions, for example about our plugin not working properly, we’d like to refer you to our support page.)

Read more: ‘SEO Basics: What is Googlebot?’ »

The post Ask Yoast: Changes to your site and the search results appeared first on Yoast.

We recently made some changes to how is run as a shop and how it’s hosted. In that process, we accidentally removed our robots.txt file and caused a so-called spider trap to open. In this post, I’ll show you what a spider trap is, why it’s problematic and how you can find and fix them.

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO: the #1 WordPress SEO plugin Info

What is a spider trap?

A spider trap is when you create a system that creates unlimited URLs. So Google can spider a page and find 20 new URLs on it. If it then spiders those 20 URLs, it finds 20 * 20 new URLs. If it then spiders those 400 URLs, it finds 400 * 20 = 8,000 new URLs. This escalates quickly, as you can see. If each and every one of these URLs were unique and wonderful, this would not be a problem, but usually, they’re not. So this causes a massive duplicate content problem.

A spider trap is bad for your SEO because every time Google crawls (or “spiders”) a page in your trap, it’s not crawling actual content on your site. Your new, high quality, super-valuable content might get indexed later, or not at all, because Google is spending its precious time in your trap. And the content it is crawling is deemed as duplicate and lessens how Google sees your site overall. This is why solving spider traps is important for SEO, and especially if you’re thinking about crawl budget optimization.

What do spider traps look like?

Our spider was one of a very particular type. We have a tool here on called Yoast Suggest. It helps you mine Google Suggest for keyword ideas. When you enter a word into it, it returns the suggestions Google gives when you type that word into Google. The problem is: Google, when given a search box, will start throwing random words into it. And the results then have links for more results. And Google was thus trapping itself.

You might think that this is a nice story and spider traps never happen in real life. Unfortunately, they do. Faceted navigation on web shops often creates hundreds of thousands of combinations of URL parameters. Every new combination of facets (and thus URL parameters) is a new URL. So faceted navigation done poorly very often results in trapping the spider.

Another common cause of spider traps is when a site has date pages. If you can go back one day, to get a new date, and then back, and back, and back, you get a lot of pages. In my time as a consultant for the Guardian, we found Google spidering a date in the year 1670. It had gone back through our online archives, which went back almost 20 years at that point, to find nothing for more than 300 years…

How to recognize a spider trap

The easiest way to recognize a spider trap is by looking at your access logs. These logs contain a line for every visit to your site. Now as you can imagine, on larger sites your access logs get big very quickly. Here at Yoast, we use a so-called ELK-stack to monitor our website’s logs, but I’ve personally also used SEO log file analyzer by Screaming Frog to do this.

Logs from an ELK stack, showing a graph of indexing, URLs, timestamp, user agents and more

An example of logs in our ELK stack

What you’re looking to do is look at only Googlebot’s visits, and then start looking for patterns. In most cases, they’ll jump straight at you. It’s not uncommon for spider traps to take up 20-30% or even larger chunks of all the crawls. If you can’t find them immediately, start grouping crawls, looking for patterns within URLs. You can start from the beginning of the URL, provided you have clean URLs. If your URLs are slightly more cumbersome, you’ll have to create groups manually.

An ELK stack makes this very easy because you can search and segment quickly:

An example of filtering for the word "SEO" within our Googlebot hits in our ELK stack

An example of filtering for the word “SEO” within our Googlebot hits

How do you solve a spider trap?

Solving a spider trap can be a tricky thing. In our case, we don’t want /suggest/ to be indexed at all, so we just blocked it entirely with robots.txt. In other cases, you cannot do that as easily. For faceted navigation, you have to think long and hard about which facets you’d like Google to crawl and index.

In general, there are three types of solutions:

  1. Block (a section of) the URLs with robots.txt.
  2. Add rel=nofollow and noindex,follow on specific subsets of links and pages and use rel=canonical wisely.
  3. Fix the trap by no longer generating endless amounts of URLs.

In the case of the Guardian, we could simply prevent linking to dates where we had no articles. In the case of’s suggest tool, we simply blocked the URL in robots.txt. If you’re working with faceted search, the solution is, usually and unfortunately, not that simple. The best first step to take is to use a form of faceted search that doesn’t create crawlable URLs all the time. Checkboxes are better than straight links, in that regard.

In all, finding and closing a spider trap is one of the more rewarding things an SEO can do to a website. It’s good fun, but can certainly also be hard. If you have fun examples of spider traps, please do share them in the comments!

Read more: ‘Robots.txt: the ultimate guide’ »

The post Closing a spider trap: fix crawl inefficiencies appeared first on Yoast.

How does a new website start ranking? Does it just magically appear in Google after you’ve launched it? What things do you have to do to start ranking in Google and get traffic from the search engines? Here, I explain the first steps you’ll need to take right after the launch of your new website. Learn how to start working on the SEO for a new website!

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO for WordPress pluginBuy now » Info

First: you’ll need to have an external link

One of my closest friends launched a birthday party packages online store last week. It’s all in Dutch and it’s not WordPress (wrong choice of course, but I love her all the same :-)). After my friend launched her website, she celebrated and asked her friends, including me, what they thought of her new site. I love her site, but couldn’t find her in Google, not even if I googled the exact domain name. My first question to my friend was: do you have another site linking to your site? And her answer was ‘no’. I linked to her site from my personal site and after half a day, her website popped up in the search results. The very first step when working on SEO for a new website: getting at least one external link.

Why do you need an external link?

Google is a search engine that follows links. For Google to know about your site, it has to find it by following a link from another site. Google found my friend’s site because I put a link to that site on my personal site. When Google came around to crawl my site after I put the link there, it discovered the existence of my friend’s site. And indexed it. After indexing the site, it started to show the site in the search results.

Read more: ‘What does Google do?’ »

Next step: tweak your settings…

After that first link, your site probably will turn up in the search results. If it doesn’t turn up, it could be that the settings of your site are on noindex or is still blocked by robots.txt. If that’s the case, you’re telling Google not to index your site. Sometimes developers forget to turn either of these off after they finished working on your site.

Some pages are just not the best landing pages. You don’t want people landing on your check out page, for instance. And you don’t want this page to compete with other – useful – content or product pages to show up in the search results. Pages you don’t want to pop up in the search results ever (but there aren’t many of these) should have a noindex.

Yoast SEO can help you to set these pages to noindex. That means Google will not save this page in the index and it’ll not turn op in the search results.

Keep reading: ‘The ultimate guide to the robots meta tag’ »

Important third step: keyword research

My friend’s site now ranks on her domain name. That’s about it. She’s got some work to do to start ranking on other terms as well. When you want to improve the SEO for a new website you have carry out some proper keyword research. So go find out what your audience is searching for! What words do they use?

If you execute your keyword research properly, you’ll end up with a long list of search terms you want to be found for. Make sure to search for those terms in Google yourself. What results are there already? Who will be your online competitors for these search terms? What can you do to stand out from these results?

Read on: ‘Keyword research: the ultimate guide’ »

Learn how to write awesome and SEO friendly articles in our SEO Copywriting training »

SEO copywriting training$ 199 - Buy now » Info

And then: write, write, write

Then you start writing. Write about all those topics that are important to your audience. Use the words you came up with in your keyword research. You need to have content about the topics you want to rank for to start ranking in the search results.

Read more: ‘How to write a high quality and seo-friendly blog post’ »

But also: improve those snippets

Take a look at your results in the search engines once you start ranking (the so called snippets). Are those meta descriptions and the titles of the search results inviting? Are they tempting enough for your audience to click on them? Or should you write better ones?

Yoast SEO helps you to write great titles and meta descriptions. Use our snippet preview to create awesome snippets. That’ll really help in attracting traffic to your site.

Keep reading: ‘The snippet preview: what it means and how to use it?’ »

Think about site structure

Which pages and posts are most important? These should have other pages and posts linking to them. Make sure to link to the most important content. Google will follow your links, the post and pages that have the most internal links will be most likely to rank high in the search engines. Setting up such a structure, is basically telling Google which articles are important and which aren’t. Our brand new text link counter can be a great help to see if you’re linking often enough to your most important content.

Read on: ‘Internal linking for SEO: why and how’ »

Finally: do some link building

Google follows links. Links are important. So get the word out. Reach out to other site owners – preferably of topically related websites – and ask them to write about your new site. If Google follows multiple links to your website, it’ll crawl it more often. This is crucial when you do the SEO for a new website, and will eventually help in your rankings. Don’t go overboard in link building for SEO though, buying links is still a no-go:

Read more: ‘Link building: what not to do?’ »

We’ve said it in 2009, and we’ll say it again: it keeps amazing us that there are still people using just a robots.txt files to prevent indexing of their site in Google or Bing. As a result their site shows up in the search engines anyway. You know why it keeps amazing us? Because robots.txt doesn’t actually do the latter, even though it does prevents indexing of your site. Let me explain how this works in this post.

For more on robots.txt, please read robots.txt: the ultimate guide.

Become a technical SEO expert with our Technical SEO 1 training! »

Technical SEO 1 training$ 199 - Buy now » Info

There is a difference between being indexed and being listed in Google

Before we explain things any further, we need to go over some terms here first:

  • Indexed / Indexing
    The process of downloading a site or a page’s content to the server of the search engine, thereby adding it to its “index”.
  • Ranking / Listing / Showing
    Showing a site in the search result pages (aka SERPs).

So, while the most common process goes from Indexing to Listing, a site doesn’t have to be indexed to be listed. If a link points to a page, domain or wherever, Google follows that link. If the robots.txt on that domain prevents indexing of that page by a search engine, it’ll still show the URL in the results if it can gather from other variables that it might be worth looking at. In the old days, that could have been DMOZ or the Yahoo directory, but I can imagine Google using, for instance, your My Business details these days, or the old data from these projects. There are more sites that summarize your website, right.

Now if the explanation above doesn’t make sense, have a look at this 2009 Matt Cutts video explanation:

If you have reasons to prevent indexing of your website, adding that request to the specific page you want to block like Matt is talking about, is still the right way to go. But you’ll need to inform Google about that meta robots tag.  So, if you want to effectively hide pages from the search engines you need them to index those pages. Even though that might seem contradictory. There are two ways of doing that.

Prevent listing of your page by adding a meta robots tag

The first option to prevent listing of your page is by using robots meta tags. We’ve got an ultimate guide on robots meta tags that’s more extensive, but it basically comes down to adding this tag to your page:

<meta name="robots" content="noindex,nofollow>

The issue with a tag like that is that you have to add it to each and every page.

Or by adding a X-Robots-Tag HTTP header

To make the process of adding the meta robots tag to every single page of your site a bit easier, the search engines came up with the X-Robots-Tag HTTP header. This allows you to specify an HTTP header called X-Robots-Tag and set the value as you would the meta robots tags value. The cool thing about this is that you can do it for an entire site. If your site is running on Apache, and mod_headers is enabled (it usually is), you could add the following single line to your .htaccess file:

Header set X-Robots-Tag "noindex, nofollow"

And this would have the effect that that entire site can be indexed. But would never be shown in the search results.

So, get rid of that robots.txt file with Disallow: / in it. Use the X-Robots-Tag or that meta robots tag instead!

Read more: ‘The ultimate guide to the meta robots tag’ »

Every website should have a decent internal search functionality that shows the visitors search results that fit their search query. However, those search results pages on your site don’t need to be shown in Google’s search results. In fact, Google advises against this too; it’s not a great user experience to click on a Google search result, just to end up on a search result page of your site. Learn what’s best practice to prevent this from happening!

User experience is not the only reason to prevent Google from including these pages in their search results. Spam domains can also abuse your search results pages, which is what happened to Krunoslav from Croatia. He therefore emailed Ask Yoast:

“Some spam domains were linking to the search results pages on my WordPress site. So what could I do to block Google from accessing my site search results? Is there any code that I could put in robots.txt?”

Check out the video or read the answer below!

Become a technical SEO expert with our Technical SEO 1 training! »

Technical SEO 1 training$ 199 - Buy now » Info

Block your search results pages?

In the video, we explain what you could do to prevent Google from showing your site’s search results:

“Well, to be honest, I don’t think I would block them. What you could do, is try two different things:

1. One is do nothing and run our Yoast SEO plugin. We’ll automatically noindex all the search result pages on your site. But if that leads to weird rankings or to other stuff that is not really working for you, then you could do another thing:

2. The second way is to block them and put a disallow:/?=s* in your robots.txt. This basically means that you’re blocking Google from crawling your entire search query. I don’t know whether that’s the best solution though.

I would try noindex first and see if that does anything. If it doesn’t, then use the method of blocking your search results in your robots.txt.

Good luck!”

Ask Yoast

In the series Ask Yoast we answer SEO questions from followers. Need some advice about SEO? Let us help you out! Send your question to

Read more: ‘Block your site’s search results pages’ »

Why should you block your internal search result pages for Google? Well, how would you feel if you are in dire need for the answer to your search query and end up on the internal search pages of a certain website? That’s one crappy experience. Google thinks so too. And prefers you not to have these internal search pages indexed.

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO for WordPress pluginBuy now » Info

Google considers these search results pages to be of lower quality than your actual informational pages. That doesn’t mean these internal search pages are useless, but it makes sense to block these internal search pages.

Back in 2007

10 Years ago, Google, or more specifically Matt Cutts, told us that we should block these pages in our robots.txt. The reason for that:

Typically, web search results don’t add value to users, and since our core goal is to provide the best search results possible, we generally exclude search results from our web search index. (Not all URLs that contains things like “/results” or “/search” are search results, of course.)
– Matt Cutts (2007)

Nothing changed, really. Even after 10 years of SEO changes, this remains the same. The Google Webmaster Guidelines still state that you should “Use the robots.txt file on your web server to manage your crawling budget by preventing crawling of infinite spaces such as search result pages.” Furthermore, the guidelines state that webmasters should avoid techniques like automatically generated content, in this case, “Stitching or combining content from different web pages without adding sufficient value”.

However, blocking internal search pages in your robots.txt doesn’t seem the right solution. In 2007, it even made more sense to simply redirect the user to the first result of these internal search pages. These days, I’d rather use a slightly different solution.

Blocking internal search pages in 2017

I believe nowadays, using a noindex, follow meta robots tag is the way to go instead. It seems Google ‘listens’ to that meta robots tag and sometimes ignores the robots.txt. That happens, for instance, when a surplus of backlinks to a blocked page tells Google it is of interest to the public anyway. We’ve already mentioned this in our Ultimate guide to robots.txt.

The 2007 reason is still the same in 2017, by the way: linking to search pages from search pages delivers a poor experience for a visitor. For Google, on a mission to deliver the best result for your query, it makes a lot more sense to link directly to an article or another informative page.

Yoast SEO will block internal search pages for you

If you’re on WordPress and using our plugin, you’re fine. We’ve got you covered:

Block internal search pages

That’s located at SEO › Titles & Metas › Archives. Most other content management systems allow for templates for your site’s search results as well, so adding a simple line of code to that template will suffice:
<meta name="robots" content="noindex,follow"/>

Become a technical SEO expert with our Technical SEO 1 training! »

Technical SEO 1 training$ 199 - Buy now » Info

Meta robots AND robots.txt?

If you try to block internal search pages by adding that meta robots tag and disallowing these in your robots.txt, please think again. Just the meta robots will do. Otherwise, you’ll risk losing the link value of these pages (hence the follow in the meta tag). If Google listens to your robots.txt, they will ignore the meta robots tag, right? And that’s not what you want. So just use the meta robots tag!

Back to you

Did you block your internal search results? And how did you do that? Go check for yourself! Any further insights or experiences are appreciated; just drop us a line in the comments.

Read more: ‘Robots.txt: the ultimate guide’ »

Ranking in the search engines requires a website with flawless technical SEO. Luckily, the Yoast SEO plugin takes care (of almost) everything on your WordPress site. Still, if you really want to get most out of your website and keep on outranking the competition, some basic knowledge of technical SEO is a must. In this post, I’ll explain one of the most important concepts of technical SEO: crawlability.

What is the crawler again?

A search engine like Google consists of a crawler, an index and an algorithm. The crawler follows the links. When Google’s crawler finds your website, it’ll read it and its content is saved in the index.

A crawler follows the links on the web. A crawler is also called a robot, a bot, or a spider. It goes around the internet 24/7. Once it comes to a website, it saves the HTML version of a page in a gigantic database, called the index. This index is updated every time the crawler comes around your website and finds a new or revised version of it. Depending on how important Google deems your site and the amount of changes you make on your website, the crawler comes around more or less often.

Read more: ‘SEO basics: what does Google do’ »

And what is crawlability?

Crawlability has to do with the possibilities Google has to crawl your website. Crawlers can be blocked from your site. There are a few ways to block a crawler from your website. If your website or a page on your website is blocked, you’re saying to Google’s crawler: “do not come here”. Your site or the respective page won’t turn up in the search results in most of these cases.
There are a few things that could prevent Google from crawling (or indexing) your website:

  • If your robots.txt file blocks the crawler, Google will not come to your website or specific web page.
  • Before crawling your website, the crawler will take a look at the HTTP header of your page. This HTTP header contains a status code. If this status code says that a page doesn’t exist, Google won’t crawl your website. In the module about HTTP headers of our (soon to be launched!) Technical SEO training we’ll tell you all about that.
  • If the robots meta tag on a specific page blocks the search engine from indexing that page, Google will crawl that page, but won’t add it to its index.

This flow chart might help you understand the process bots follow when attempting to index a page:

Want to learn all about crawlability?

Although crawlability is just the very basics of technical SEO (it has to do with all the things that enable Google to index your site), for most people it’s already pretty advanced stuff. Nevertheless, if you’re blocking – perhaps even without knowing! – crawlers from your site, you’ll never rank high in Google. So, if you’re serious about SEO, this should matter to you.

If you really want to understand all the technical aspects concerning crawlability, you should definitely check out our Technical SEO 1 training, which will be released this week. In this SEO course, we’ll teach you how to detect technical SEO issues and how to solve them (with our Yoast SEO plugin).

Keep reading: ‘How to get Google to crawl your site faster’ »


There are several reasons for cloaking or redirecting affiliate links. For instance, it’s easier to work with affiliate links when you redirect them, plus you can make them look prettier. But do you know how to cloak affiliate links? We explained how the process works in one of our previous posts. This Ask Yoast is about the method of cloaking affiliate links we gave you in that post. Is it still a good idea to redirect affiliate links via the script we described?

Elias Nilson emailed us, saying that he read our article about cloaking affiliate links and he’s wondering if the solution is still up-to-date.

“Is it still a good idea to redirect affiliate links via the script you describe in your post?”

Check out the video or read the answer below!

Get the most out of Yoast SEO, learn every feature and best practice in our Yoast SEO for WordPress training! »

Yoast SEO for WordPress training€ 99 - Buy now » Info

Redirect affiliate links

Read this transcript to figure out if it is still a valid option to redirect affiliate links via the described script. Want to see the script directly? Read this post: ‘How to cloak affiliate links’:

Honestly, yes. Recently we updated the post about cloaking affiliate links, so the post and therefore the script is still up to date. Link cloaking, which sounds negative, because we use the word cloaking, is basically hiding from Google that you’re an affiliate. And if you’re an affiliate, that’s still the thing that you want to do, because usually Google ranks original content that is not by affiliates better than it does affiliates.

So, yes, I’d still recommend that method, the link will be below this post, so you can see the original post that we are referencing to. It’s a very simple method to cloak your affiliate links and I think it works in probably the best way that I know.

So, keep going. Good luck.

Ask Yoast

In the series Ask Yoast we answer SEO questions from followers. Need help with SEO? Let us help you out! Send your question to

Read more: ‘How to cloak your affiliate links’ »