Ask Yoast: why connect GSC with Yoast SEO?

If you use our Yoast SEO plugin you’ve got the opportunity to connect it to Google Search Console (GSC). With GSC you can monitor the SEO health of your site, while Yoast SEO helps you to optimize your site. Connecting the two, so they can work together, will allow you to be more efficient when maintaining your site.

In this Ask Yoast we’ll give you the answer to the following question:

“How does implementing Google Search Console in Yoast SEO help me to optimize my site?”

Check out the video or read the answer below!

Get the most out of Yoast SEO, learn every feature and best practice in our Yoast SEO for WordPress training!

$ 79 - Start learning today »Yoast SEO for WordPress training

Google Search Console in Yoast SEO

Read this transcript to learn how you’ll benefit from connecting GSC to Yoast SEO:

“Well, honestly, connecting the two doesn’t help if you don’t make any mistakes on your site. But you’re human, right? So you’re going to make mistakes. GSC tells you when Google has found errors on your site that you should fix. 

By implementing GSC in Yoast SEO, you’ll see those errors in Yoast SEO. It makes it possible for you to very easily fix those errors and make sure that your site is as error free as possible. This is the best thing you can do for your site’s SEO.

So, if you don’t make any mistakes ever, then it’s not going to help you. If you do make some mistakes sometimes, even if it’s only a couple of mistakes over the years, it really helps to find these errors, fix them, and then be done with them.

Good luck!

Ask Yoast

In the series Ask Yoast we help you with your SEO question! Did you get stuck on an SEO issue? Are you in doubt between two SEO strategies? Don’t fret, just ask us your question! You can send your question to

Read more: ‘Google Search Console Crawl’ »

Weekly SEO recap: Happy new year!

First of all, a happy new year to you all! This is the first weekly recap of 2016 and the first weekly recap of the last 3 weeks so we have a few weeks of news to cover. Luckily, we weren’t the only ones taking it slow, so it’s about a normal week’s worth of news.

Joost's weekly SEO Recap

Google “No country redirect” stops working, then comes back

For a short while I was slightly freaked out. You see, Google broke the NCR feature. NCR stands for No Country Redirect and allows you to go to and it will not redirect you to your country’s local Google (in my case Breaking that would force me into using a proxy all the time. Luckily, it’s back working now.

Google Search Analytics update

Google did an update to the Search Analytics in Google Search Console, a feature I love. The “full” report reads:

An update to web search logs analysis. This change may increase the total number of clicks and impressions.

It seems to have indeed increased the number of clicks & impressions on several sites I’ve seen, while changing nothing for some other sites.

Keep reading: Search Analytics in Google Search Console.

Google Knowledge Graph Search API

Google released an API to query its Knowledge Graph, replacing its old Freebase APIs. This could be a very useful API in specific fields, I’ll certainly be looking into what kind of interesting data it offers.

That’s it, see you next week!

joost signature

Weekly SEO Recap: Panda U-Turns, Yandex penalizes and Bing launches new tools

Joost's weekly SEO recapNew iPhones! Search news? Oh right… So… About that search news. Panda’s making U-turns, Yandex penalizing sites for selling links, Bing offering new keyword tools and more!

What happened to Panda?

Google announced a Panda 4.2 update that’d “very slowly” roll out a while back, we covered that. It seems as though, according to this post by Barry Schwartz at SearchEngineLand, the update has been reversed.

It could be that Google’s continuous user testing started giving lower ratings for their search results, indicating that Panda 4.2 was, in fact, not an improvement. Google didn’t comment, but it seems a plausible hypothesis. Looking at some sites myself in SearchMetrics, I definitely see… Weird behavior, visibility going up and coming down again, which could support this hypothesis.

Yandex penalizing link sellers

Yandex has announced that they’ve penalized a bunch of sites for selling links. I’ll spare you a link to the Russian blog post (oops you got it anyway) and give you a link to SearchEngineLand instead. It makes a ton of sense to make these kinds of adjustments, but it’s funny to see them come out of Yandex, who actually announced with some fanfare at the end of 2013 that they would stop using links (for certain commercial queries).

I think it shows how important links are to ranking algorithms and how hard they are to replace. So much that it’s worth cleaning up the signal by making adjustments like this.

Bing launched a new keyword planner

We write a lot about keyword research, as it is, truly, the basis of SEO. So when new tools arrive that help you do keyword research, I always get excited. Even more so when created by Bing as, historically, even though Bing drives far less search traffic, their keyword research tools have always been among the best in the business.

Bing announced a new keyword planner on Wednesday, available in your Bing Ads account. It’s worth creating a Bing Ads account just for that. There are historical statistics in there, suggestions for more keywords etc. etc. Of course, the best tool in doing keyword research is still your brain, but tools like these can help you get new ideas.

Index count in Google Search Console

One of the things you might be keeping an eye on while optimizing a site is the Index Status in Google’s Search Console. It shows the number of indexed pages for a site, and will usually remain static if you’re not doing too much. So it’s weird when that graph spikes or dips suddenly. Recently, it did, this is the graph for

search console status

Turns out that Google had a bug (the dip) and then changed how they calculate this number (the spike). There’s an “explanation” by Google here, but it doesn’t make sense, they just changed it. I’m hoping it’ll stay stable at the “new normal”, as that ways it’s the most reliable.

That’s it, see you next week!

joost signature

This post first appeared as Weekly SEO Recap: Panda U-Turns, Yandex penalizes and Bing launches new tools on Yoast. Whoopity Doo!

Phantom Quality Panda and Google Search Console

stop being bambooAt Yoast, we’re all really busy in the preparation for YoastCon next Wednesday, as you will understand. When us SEO people are really busy focussing on other stuff, Google tends to ‘surprise’ the online community with breaking news, algorithm updates or other developments that really need our attention as well. How is that, you might say. Well, remember the Panda updates that immediately affected 5% of the internet? That for instance also meant an increase in site review business for us. We want to be prepared :)

Quality Update, Phantom 2 or Reverse Panda?

Google did another one of these algorithm updates on the 3rd of May this year, just a couple of weeks ago. Hubspot mentions mainly the less contributing How-to sites that lost rankings, like HubPages and eHow. By less contributing we mean in terms of contribution of new, quality content to Google. Google claims it wasn’t Panda. Nevertheless, it seems that mainly sites that already dealt with a Panda penalty dropped in rankings.

There are many names that are given to this update, as none was given by Google. Did anyone already mention that Google is less talkative about these updates since Matt left? That on a side note. The name for this update we like least is ‘Quality Update‘ or ‘Quality Algo’. That’s not a distinctive name whatsoever. For years, Google’s wish has been for webmasters to focus on quality. Panda is about quality, and so is Penguin. Google’s entire penalty system could be called Quality Algo. Glenn Gabe called it the ‘Phantom 2‘ update, it being the second large, unnamed algorithm change. I like that one a lot better, but have to say that, as it is clearly Panda related, Reverse Panda is my personal favorite.

reverse pandaThe update is told to positively influence rankings of websites that provide quality content, instead of Panda punishing sites that lack quality content. That would indeed make it a reverse Panda. Call a spade a spade, right?
The algorithm update isn’t targeting low quality sites, but a side effect of an update like that is of course a decline in rankings for these Panda candidates. Just stop being bamboo, as we often say in the office. It’s all about quality content. Google’s search result pages are stuffed with websites and if your website simply isn’t contributing on a larger scale (e.g. in the search results), you’d better up your game fast.

Google Search Console!?

google search consoleYou’ll probably not find notifications on this reverse Panda update in the Manual Spam Action section in Google Webmaster Tools, by the way. For two obvious reasons:

  1. It’s not a manual update, but could be the start of even more real time algorithm updates
  2. It’s not Google Webmaster Tools anymore.

What was that all about, right? Since not all users of Webmaster Tools are webmasters, they renamed it into Google Search Console. To target ‘more users’. Seriously. But hey, it brings in some extra traffic, right. That is why we decided to rename our WordPress SEO Premium plugin into Page Analysis And General SEO That Includes Redirect Options For Crawl Errors In Google Search Console Plugin. That just makes clear that it’s not just for people that want green bullets. Now excuse me, I’ll have to go and rewrite all pages on this website about that plugin.

This post first appeared as Phantom Quality Panda and Google Search Console on Yoast. Whoopity Doo!

Search Analytics in Google Webmaster Tools

Yesterday, Google confirmed what we’ve been anticipating for years: mobile search has overtaken desktop search. On their Inside AdWords blog, they stated the following:

more Google searches take place on mobile devices than on computers in 10 countries including the US and Japan.

Let’s let that sink in for a moment, combining it in our heads with the recent Mobilegeddon roll-out. The question for most people and companies is: how does this affect my/our website? Luckily, Google released the new Search Analytics report in Google Webmaster Tools today that answers just that question (and many more) for you.

Using Search Analytics to measure impact

If you go to Google Webmaster Tools you’ll find a new report under Search Traffic called Search Analytics. For, it shows that the impact of the mobile update was absolutely negligible:

yoast com mobile traffic

As you can see, this new report allows you to do a lot of awesome filtering. One of the features I like a lot is the ability to filter by a keyword, like “mobilegeddon”, compare it to a previous date range and also, filtering deeper on top of that. For instance, we can see how many people reached our site searching for that keyword, comparing desktop versus mobile search:

Mobilegeddon search traffic - desktop vs mobile search

We could then filter these by country as well, should we want to.

Comparing brand versus non-brand searches

It’s obvious that when people use your brand name in their searches, they’re more likely to click on your site in the results than otherwise. Just how big that difference is something Search Analytics can make very clear for you. For instance, compare these results: [yoast seo] versus [wordpress seo]:

compare yoast seo wordpress seo

As you can see, the already high 10.62% for [wordpress seo] is completely obliterated by the 62.33% for [yoast seo]. This is why we focus on building a brand as a part of our holistic view on SEO campaign as much as we do.

Search Analytics – Search Type report

Another very interesting new feature in this report is the Search Type filter. This allows us to look specifically for clicks from Video Search, something that’s very hard to measure in other ways. For this shows us that we rank very well (and thus get many clicks) with the video on our Google Analytics plugin page, in part due to the optimization we do with our Video SEO plugin:

search analytics video

We also rank fairly well for the term [google analytics] in “normal” Web search. The problem is that that doesn’t really mean anything: when someone search for google analytics in web search, that is a highly navigational search: they probably want to log into Google Analytics. The difference becomes very clear when we compare the CTR of web searches with video searches:

google analytics search analytics type comparison

A CTR of over 12% compared to a CTR of 0.14%. Of course, we do rank lower on web and the CTR drops off a cliff as soon as you get below position #3, but it’s obvious how valuable these types of reports can be.

Up to you now: go play with Search Analytics!

It’s really up to you now, go play with Search Analytics and drop your favorite queries in the comments. If you all share nice combinations, we’ll figure out all the cool uses of this great new tool together.

This post first appeared as Search Analytics in Google Webmaster Tools on Yoast. Whoopity Doo!

Bing Webmaster Tools: Security, Widgets and Messages

Bing Webmaster Tools: Security, Widgets and MessagesCompared to the first three sections, the last three are relatively small. In this final post in our series on Bing Webmaster Tools we will go over Security, Malware and Messages and share our findings.


Making sure your website is secure is of course really important. We have already emphasized this on our page on WordPress security, of course. This section is about the security alerts in Bing Webmaster Tools.


Malware is software that is intended to damage or disable computers and computer systems. For a website, this usually means software that is left on your website, usually in a file on your server. It’s software that is left there without your knowledge. If you find an alert in Bing Webmaster Tools about malware on your website, you should clean your site as soon as possible.

Bing Webmaster Tools divides the malware into two categories: malware found on the page and malware reference found on the page. In the first case, there is an immediate issue to be solved on the page with the listed URL, in the second case, there is a resource linked on the page at that URL that has malware. The page itself isn’t infected in that last case.

Bing Webmaster Tools lists a couple of malware issues:

  • Malware Network Reference: any trace of a known malware distribution network is detected on your website.
  • Browser Exploit: malicious browser exploit detected, which may cause unsolicited execution of external code.
  • Malicious JavaScript: in a page or one of its attached script of frames, malicious JavaScript is detected (f.i. a spammy redirect).
  • Malicious ActiveX: ActiveX interactions seem to trigger malicious activity.
  • Heapspray: Bing detected a potential preparation for a browser exploit via a heapspray.  Heapspraying is a technique used in exploits to facilitate arbitrary code execution.
  • Malware Found on Adjacent Pages: URL is in a folder or subdomain containing malware.
  • Malware Reported by External Source: external sources reported that malware, obviously.

Sucuri is our and your friend in this. Hire these guys to clean up your site. After that, you want to address the vulnerability that allowed the malware to be installed. Simply download the Sucuri plugin and follow their step by step instructions and guidance on how to secure your website (sections Hardening and Post-Hack in that plugin).

And only after this, you should Request a Review in Bing Webmaster Tools to have them check again for the presence of malware.

Track Certificates beta

Yet another beta in Bing Webmaster Tools: Track Certificates. This page will tell you all about the certificates that were requested by people visiting your site. The main purpose of that list is so you’ll be able to spot unexpected or suspicious certificates, so you can report them to Microsoft using the Report link.

This will also include security certificates like the ones we have on our website:

Bing Webmaster Tools: Track Certificates


In their competition struggle with Google, Bing has released a number of additions for your website that should make your life better. Two of these are in Bing Webmaster Tools and we’ll discuss both briefly.

Knowledge Widget beta

Bing explains the knowledge widget like this:

For the first time, we empowered every webmaster to use the entity data from the Bing Knowledge repository. Since then, webmasters have added the embed code to thousands of pages to enhance their websites with the rich entity information from the Bing Knowledge system.

Yes, the Bing Knowledge information is similar to the Knowledge Graph in Google (Bing added it first, by the way). It’s a separate block of content but now on your website itself! It works really simple: while the visitor goes over your website, the widget detects related entities in real time, marking them with a little Bing charm. Now I do understand the social value, but feel that this is a bit like the pop-up ads in texts / on links that were briefly popular a couple of years ago. But hey, I might be wrong. Bing tells us that this is improving engagement, time-on-site, and user satisfaction.

Adding the widget is simple: add your URL, copy the provided code and paste it for instance right before the </body> tag in your template. Options are available for displaying images, images and links, just links. Besides that, there is an option to only activate the Bing Knowledge information on text selection by a user. It looks like this (example from Bing itself):

Bing Webmaster Tools: Knowledge Widget

It’s a collapsible sidebar on your website.

At the moment of writing, this is only available for English entities.

Translator Widget

This Translator Widget could be useful, and is similar to the Google Translate option that is on a lot of websites. It only requires a simple copy/paste action and you are good to go. There is even a WordPress plugin to help you out.

You can set the language of your website, and tell Bing to automatically translate based upon the visitor’s browser language, or have the visitor pick the language himself.

I’m not a big fan of these widgets, Google or Bing. I understand that these are ‘convenient’, but rather see people putting some effort (or money) in a decent translation. Go read our post on hreflang. It’s not that hard.


Preferably this section looks like this:


Bing sends five types of messages:

  • Administrator: if anything changes to the Bing Webmaster Tools service, the administrator will inform you about it.
  • Crawl errors: if an error occurs during the crawl of your website, Bing Webmaster Tools will automatically tell you about it.
  • Index issues: if Bingbot has any problem indexing your site, a message will be sent as well.
  • Malware: following the section on Malware, you’ll receive a message here that malware has been found.
  • Bing Ads: automatically generated messages about Bing Ads.

The Current and Archived sections at Messages are self-explanatory.

Bing Webmaster Tools: It’s a wrap!

With this post, we conclude our series on Bing Webmaster Tools. We have compared Bing Webmaster Tools to Google Webmaster Tools a couple of times during these four posts. Is that a fair comparison? For me it was like having Skippy sandwiches every day and then trying a jar of Peanut Butter & Co. It’s different, but the same product. One could say variety is the spice of life, but in this case I tend stick to Google Webmaster Tools for the larger scale and user base, although the Markup Validator tool in Bing Webmaster Tools will come in handy now and then for quick checks!

For more reading on Webmaster Tools, please find all our related articles here.

This post first appeared as Bing Webmaster Tools: Security, Widgets and Messages on Yoast. Whoopity Doo!

Bing Webmaster Tools: Diagnostics & Tools

Bing Webmaster Tools: Diagnostics & ToolsBing Webmaster Tools provides a number of tools to analyze your website. Somehow, they managed to squeeze these into one page in Bing Webmaster Tools as the dashboard page for the section Diagnostics & Tools. Fortunately, all tools also have a separate page…
In this post, we will go over all tools and tell you how to use these to your advantage.

This is already our third article on Bing Webmaster Tools. In case you have missed the first two, go read these here:

Keyword Research beta

Out of the five tools described in this article, Bing classifies most as ‘beta’. With all these tools, we also have to keep in mind that we are dealing with data from organic search at Bing. Let’s start with the Keyword Research tool:

Bing Webmaster Tools: Keyword Research

I have tested this tool using a number of WordPress SEO related keywords for our website. As you can see, you can set a country and language, as well as a time frame from your research. I chose the US, US English and changed the default last 30 days to January 2015. Note that, as with Google Webmaster Tools, there is a gap between today and the end of your data; Bing Webmaster Tools goes up to 3 days ago.

The tiny ‘Strict’ option in there is to determine whether you want the research to be done for exact phrases or phrases containing the keyword. As I’d like to investigate general keywords (WordPress, plugin, seo), I have left this option unchecked:

Bing Webmaster Tools: Keyword Planner

All results can be sorted, by the way. Just click the (blue) title of a column. First thing that comes to mind is that the Google Keyword Planner shows 550K searches for WordPress in January 2015. Bing Webmaster Tools shows 60K. We’ve already mentioned that the search volume in Google’s keyword planner isn’t always useful, and the same must apply to these Bing numbers.

Buy KeywordThe $-icon behind the keyword is a hint. You can actually buy the keyword here directly. That’s pretty straightforward :)

I still feel that if you properly optimize your website, you can get more clicks as well from organic results. Not trying to ruin your revenue model here, Bing. Paid results seem to work fine if you have narrowed down your niche and target these specific visitors. If the ad / paid result tells me what I need (to know), I’ll click it without a doubt.

In the first table, you see search volume and trends. These are not just going up, by the way :) If you choose a longer period of time, it also won’t be just a straight line :) It will tell you what the rise or drop in search volume is as well.

You can click all the results, but doing so will just filter the results, it won’t give you more details. By the way, just to be complete, you must have seen the Suggestions as well in the screenshots above. These keywords are additional keywords, related to your original keyword by relevancy.

It’s a nice tool, but being used to the keyword tools we normally use, this isn’t one I’ll use very often. Perhaps the release version will have some features that will make it more attractive?

Link Explorer

This is Bing’s alternative for MajesticSEO, OpenSiteExplorer and more backlink tools like that. It allows you to find all backlinks known to Bing for any site. It has a few filters:

  • Filter by site: only pages on your website linking to that URL.
  • Anchor text: only links that use this anchor text.
  • Additional query: only pages that link and that rank for the search keyword as well.
  • Scope: links to the URL entered only or all pages from that domain.
  • Source: only links from the site at hand (internal) or all links to that URL (external).

For that last one, you probably want to use the inbound link tool at Reports & Data instead. It’s more comprehensive and for instance provides you with the exact anchor text for each link.

The result of the Link Explorer tool is something like this:

Bing Webmaster Tools: Link Explorer

Clicking the Source URL will perform a link analysis in Bing Webmaster Tools for that domain, clicking the title will get you to the website itself.

What I am missing here is total search volumes, and some more details on the domain. MajesticSEO adds Trust and Citation flow, Moz’ OpenSiteExplorer adds both page and domain authority. Right now, the filters are what making this section on Bing Webmaster Tools interesting, not per se the (outcome of the) general exploration. if I wanted to check all the pages that link to our domain using “WordPress plugins”, this tool works fine.

Fetch as Bingbot beta

Like Fetch as Google, this will tell you how the search engine bot sees your website. Simply enter your URL and click Fetch:

Bing Webmaster Tools: Fetch as Bingbot

After the fetch is done, status will change from Pending to Completed. Now you can click the Status and see how Bing sees your website (clicking the URL will simply get you to the page itself). Where Google Webmaster Tools actually renders your website as well, Bing displays the server response and your source code, as seen by Bingbot, of course. It will tell you if a page is redirected, or blocked by your robots.txt, for example.

Markup Validator beta

I really like this one, to be honest. As it is totally unrelated to search traffic in terms of volume, this tools is a nice substitute for Google’s Rich Snippet Test tool. I don’t know if it is just getting used to something new, but I liked the clean setup of that Google tool like it was a few weeks ago. Bing tells me just what I need to know; is implemented correctly?

Bing Webmaster Tools: Rich Snippets Test

It is. It will also tell you about RDFa, which you still want to use for breadcrumbs, for instance. Nice extra is that this Bing Webmaster tool also extracts the OpenGraph data.

And yes, I understand the new Google test tool explains what is wrong (according to Google, that is*), but if you just want to see what / RDFa / OpenGraph is on a page, this tool actually works just fine!

* The Rich Snippet Test tool keeps on telling me to link the last element in my breadcrumbs, but why should I link to the page the visitor is already on? From a UX stand, that seems odd…

SEO Analyzer beta

There we go: Bing Webmaster Tools gives you a free SEO analyzer to cover all the basics (for more in depth analysis, check our site reviews).

You can simply insert a URL and see what basics can be improved. This is very much like the HTML Improvements in Google Webmaster Tools. There is also some overlap with the SEO Reports in Bing Webmaster Tools, but the SEO analyzer is a single page analyzing tool.

Bing Webmaster Tools: SEO Analyzer

It will tell you if H1’s are missing, which images lack ALT tags and for instance if a page is missing language information. I especially like the way it is displayed, with a direct reference to the ‘error’ on the Analyzed tab on the right:

Bing Webmaster Tools: Missing ALT tag

Two things to keep in mind when using the SEO Analyzer:

  • SEO Analyzer, unlike Bingbot, will ignore robots.txt directives, so you can basically check any page on your site.
  • SEO Analyzer follows any redirect and analyzes the page it end up on. It will tell you that a redirect is detected and followed (right below the SEO Suggestions).

Verify Bingbot Tool

So why would you want to know what IP address the Bingbot is? Well, the Bingbot might be ‘overcrawling’ your website, unintendedly sending more than the usual server load your way. With this tool, you can verify if the IP address that is causing this load, is indeed Bingbot, which is needed for your support request. By the way, Bing does respect that crawl-delay line in your robots.txt. That might already help.

It seems to me that this tools should be incorporated in a workload. The fact that it is displayed here as a separate section, seems a bit too much honor for that tool.

Site Move

If you are moving your site to a new domain, be sure to let Bing Webmaster Tools know. It’s pretty easy:

Bing Webmaster Tools: Site Move

Just make sure both sites are verified. This can be done for entire domains, subdomains and directories. If you are moving a larger part of your website, like to, this is also the section where you can see if all went well. Note that this is just a notification, it has nothing to do with actually moving a site or directory.

That’s section three of Bing Webmaster Tools

When comparing Bing Webmaster Tools to Google Webmaster Tools (and of course we do), this section holds some nice surprises. As mentioned, I like the simplicity of the Rich Snippet test, and the way the SEO Analyzer highlights the improvements on your page.

As mentioned in our earlier posts, I’d never replace Google Webmaster Tools entirely by Bing Webmaster Tools, but a combination of might have its benefits. In the next post in this series, we will cover the remaining three sections of Bing Webmaster Tools we’d like to discuss: Security, Widgets and Messages.

This post first appeared as Bing Webmaster Tools: Diagnostics & Tools on Yoast. Whoopity Doo!

Bing Webmaster Tools: Reports and Data

Bing Webmaster Tools: Reports & DataIn this second Bing Webmaster Tools post in our series on Webmaster Tools, the focus is on reporting. We’ll be going over a number of pages, and try to explain what information is available and how this can help you improve your website. Or get new insights about your audience, of course. This section is called: Reports and Data.

Note that although we very much like the ‘broader’ setup of Bing Webmaster Tools, I personally tend to use Google Webmaster Tools first for one simple reason. Google dominates the search engine market. There is data from much more users in Google Webmaster Tools. Having said that, there are a lot of things in Bing Webmaster Tools that could be used to further improve Google Webmaster Tools. Now let’s go over the different chapters in this section of Bing Webmaster Tools.

Site activity

As you can see in the graph below (upper right corner), Bing Webmaster Tools combines data from Yahoo! and Bing for these stats. It also painfully highlights the difference in use of the webmaster tools: as you can see, our website gets around 80 clicks a day from Bing and Yahoo!, compared to about 9,000 clicks on average per day from Google.

Bing Webmaster Tools: Site Activity

In the chart, you can also choose to show a number of other things:

  • Clicks from Search
  • Appeared in Search
  • Pages Crawled
  • Crawl Errors
  • Total Pages indexed

It’s nice to align the number of pages crawled and the crawl errors, but as with most graphs like this, it’s the trend that matter the most to me here. If all of a sudden the graph flatlines at zero (except at the crawl errors), there must be something wrong.

Page Traffic

Let’s start with the main table here:
Bing Webmaster Tools: Page Traffic
There is a lot going on here, right? All the arrows and numbers tell you what happened with for instance the Clicks from Search, and how often you appeared in search. For your convenience, Bing has divided the clicks by the appearances, giving you the click-through rate (CTR).

The average search click position tells you the position your result was when clicked, on average. The trend is key here. If the average position gets closer to #1, you know you are doing well, as your page is increasing it’s ranking for ‘that’ search term. This highlights the main issue: you need to combine this with the Search Keywords in the last column to be useful. Just click View, as Bing Webmaster Tools presents these keywords in a convenient popup:

Bing Webmaster Tools: Traffic Details

Who’s ‘youst’, right? I can’t get used to that downwards pointing arrow with a number, it’s like we dropped 675 clicks. But that’s not what is meant. We had 675 clicks from Bing search result pages to our homepage where the search keyword was ‘yoast’, and that is a drop from the beginning of the selected time period.

This is also where you can check the performance per position in the search result pages. Just click the ‘+’ in front of the keyword:

Bing Webmaster Tools: Traffic Details, positions

Back to the main table for your Page Traffic. The last column we haven’t mentioned yet is Average Search Appearance Position. In most if not all cases, this number is higher than the average click position, simply because we all know the higher the position, the more clicks.

Index Explorer

This is your Mac Finder or Windows Explorer for all things Bing found in your website or site’s structure. It’s how Bing sees your site. You can see files Bing considers 301 Redirects or 404 Error pages, for instance. There are a number of filters here, listed as the blue links below. Next to that, you can use some custom options:

Bing Webmaster Tools: Index Explorer, robots.txt

Strange thing is that our robots.txt is actually updated a while ago already. Where Google tends to find these updates within a day, sometimes hour, Bing still shows some old content in here. We re-fetched the robots.txt during the writing of this article, just to make sure. The fetch shows the current content of our robots.txt. I’m sure Bing Webmaster Tools will find that eventually.

Search Keywords

Much like the Page Traffic table, this table shows you the number of clicks from search results in Bing, as well as all other things explained there. It are keywords instead of URLs in this table, but that is the main difference.

Bing Webmaster Tools: Search Keywords

As you can see, that last column is different. You can view the pages instead of keywords (at Page Traffic). So it’s basically the same table, but the other way around :) Use to your preference, so to speak.

SEO Reports

Why order a site review when you have Bing SEO Reports. [promo] Well, for one thing, we tell you much more about stuff from speed to design, from content to social. [/promo] Bing SEO Reports in beta, but does provide some nice insights. We need to work on our meta descriptions, according to this overview:

Bing Webmaster Tools: SEO Reports

I especially like the fact that Bing tells you how they consider these suggestions to influence your SEO. Bing agrees with Google that meta descriptions are like invites to your website that should have a clear message about the contents of the page and tells that the severity of these meta descriptions is high. The fact that the meta description is missing, isn’t bad* per se, but it’s good to know where these are missing. You can easily click the suggestion title to find the pages these should be added.

Bing Webmaster Tools has taken the time to add a proper explanation to all these suggestions:

Bing Webmaster Tools: SEO Analysis Detail

If you click one of the (50, it’s limited to that) links on that Detail page, you’ll be taken to a SEO Analyzer (also in beta). More on that SEO Analyzer in our next post on Bing Webmaster Tools.

It seems our knowledge base – we’re using HelpScout’s awesome Docs – is missing meta language information. I can imagine that being so due to their global user base. In that case it’s a good thing language isn’t templated.

All in all, this is a very useful section you can use for some basic SEO check of your website.

* in case you can’t come up with a proper meta description, let Bing (or Google) decide. Both will create a meta description that in most cases is constructed from a text snippet containing the keyword used in search. This can be beneficial for conversion, as you will understand.

Again, the details make this a useful section. On the dashboard page of this section, the total number of inbound links (links to your website), doesn’t say much, unless there is a break in the trend. Right below the graph, you’ll find target pages. Clicking one of these gives you an overview of the pages linking your specific page:

Bing Webmaster Tools: Link details

As you can see, this includes the anchor text. There are up to 20,000 external pages listed. As the popup isn’t suitable for that number of links, the export option in the upper right of the popup comes in really handy for further analysis. Bing Webmaster Tools offers that export option on almost all pages, by the way.

Crawl Information

This is a quick overview of error codes (like 404, 502), Redirects (301, 302), DNS failures, connection timeouts and robots.txt exclusions. All is grouped per category:

Bing Webmaster Tools: Crawl Information

Click the row marked with * in Bing Webmaster Tools to find all URLs that resulted in that error, are redirected, etc. It’s nice to see if all pages you have excluded are indeed excluded, for instance. Clicking a link in this table isn’t providing more information, but gets you to the page at hand to check f.i. if the error or redirect still exists. As mentioned right above the table, Bing refers you to the Index Explorer for more details. It would be nice if I could mark a 404 as fixed here, by the way (in case any of the Bing people reads this).



Bing Webmaster Tools: Malware

In case Bing detects any malware on your website, it will list the URLs of potentially infected pages here.
Malware (short for malicious software) is “any software used to disrupt computer operation, gather sensitive information, or gain access to private computer systems.” (Wikipedia). Of course you don’t want that on your website.

In the unwanted case you have some URLs listed here, fix the malware or have our friends at Sucuri clean this up, and Request a Review via this same section in Bing Webmaster Tools.

That concludes this section in Bing Webmaster Tools

To round things up, the Reports & Data section in Bing Webmaster Tools gives you a lot to work with. Be sure to add your site to Bing Webmaster Tools and check this section to learn more about what can be improved for your website.

If you have any additions or remarks, these are more than welcome. If you are an absolute Bing adept, please let us know what hidden gems you have found in the Reports & Data section in Bing Webmaster Tools!

This post first appeared as Bing Webmaster Tools: Reports and Data on Yoast. Whoopity Doo!

Bing Webmaster Tools: Configure My Site

Bing Webmaster Tools - Configure My SiteWe recently did a series about Google Webmaster Tools. So we thought it was time to do a series about Bing Webmaster Tools as well. Bing isn’t that big in The Netherlands, or Europe, for that matter, but it still holds some ground in the US. The most important reason, however, is we think Bing Webmaster Tools is pretty awesome. It offers some really nice tools and details you, unfortunately, won’t find in Google Webmaster Tools (anymore).

In this post we’ll go into the first main menu item of Bing Webmaster Tools and its subitems. But first, we’ll explain how you can set it up for your own site!

Setting up Bing Webmaster Tools

Since Bing Webmaster Tools is probably less known than its Google counterpart, we’ll let you know how to set it up! So lets start with the beginning.

You need a Microsoft Live account

Don’t worry, this doesn’t mean you’ll need an or email address. You can actually use any email address you like. Go here to sign up for a Live account and just follow the steps.

Once you have a Live account, you can go to Bing Webmaster Tools and sign in with the account you just created.

Add your site

After logging in to your Bing Webmaster Tools, you obviously need to add your site:

Bing Webmaster Tools - Add your site

Fill in your website’s URL here and click ADD. This will take you to this screen:

Bing Webmaster Tools - Add your site and sitemap

If you already have a sitemap and you’re aware of it, you can fill in the sitemap here as well. We recommend just leaving the last option on the default, for now. Now you can click the “ADD” button again. Doing this will take you to the following screen:

Bing Webmaster Tools - Verify ownership

And this is where the fun comes in. All of this might seem very technical and way over your head. However, if you’re using our WordPress SEO plugin, this is all very easy. Simply copy the entire part in the grey bar and paste it in our plugin here:

WordPress SEO Setting up Bing Webmaster Tools

Hit “Save Changes” and it’ll remove everything it won’t need, just your ‘key’ will remain. After that hit verify back in the Bing Webmaster Tools setup. If you have some caching, this might not work instantaneously; clear your cache or wait for a bit until you can verify it.

Once you’ve verified it, you’ll be able to access your Bing Webmaster Tools!

Configure My Site

When clicking the ‘Configure My Site’ menu item, you’ll be taken to a dashboard overview of all that you can do in this menu item. You can even do most things right there in the dashboard, which makes it the go to place, once you know what it all does ;)

So lets get to explaining what you can play with in Bing Webmaster Tools.


The sitemaps menu item is about just that: sitemaps. Here you can (re)submit, remove or export sitemaps. Adding a sitemap is simple, just copy the sitemap’s URL, paste it in the appropriate bar and hit Submit. As with Google, if you have multiple sitemaps under a sitemap index (as our WordPress SEO plugin does for you), you just have to add the sitemap index.

If you’ve added a sitemap index, just clicking that index sitemap’s link will take you to all underlying sitemaps. You can also see whether any errors occurred when crawling your sitemap, when it was last crawled and how many URLs were submitted through the sitemaps, for instance.

Submit URLs

In this section you can add URLs of pages that you think are important directly into the Bing index. This feature is limited to 10 links a day and 50 links a month per domain. Whether this actually makes sure your content is indexed more quickly than through your sitemaps, for instance, remains a bit unclear.

Ignore URL Parameters

URL Parameters are anything after the normal URL, so usually anything after a question mark in your URL. If you want Bing to ignore any of these parameters, you can tell them here. Just to be clear: this will ignore only the parameters, it’ll literally remove the parameters from the URL before adding the page to the index. So this does not mean any URLs with these parameters will not be indexed, you should do that in the Block URLs section.

A URL with parameters is actually viewed as a different URL as the same one without parameters. So “” etc… would actually be considered a different URL than just “”, even though they end up at the same place. So using this feature can prevent duplicate content.

Crawl Control

This is one of the more nifty tools in the Configure My Site section of Bing Webmaster Tools. You can tell Bing when to crawl your site more and when to crawl it less:

Crawl Control Bing Webmaster Tools

They have some preset crawl rates, which are made to prevent an overload of your server. So when your site is busy, they’ll crawl your site less, and when it’s less busy, they’ll crawl more. However, you can also set a custom crawl rate. Just click in the graph to increase or decrease the crawl rate.

Note: If you have any info on crawl rate specified in your robots.txt, that will take precedence over anything you set up here.

Deep links are the Bing equivalent of Google’s Site Links. If you don’t know what those are either, they’re all the links in the pink block:

Bing Webmaster Tools - Deep Links

In the Deep Links section you can block pages from showing up there. What’s nice about Bing here, is that they allow you to remove the URL showing as a deep link for all URLs, a specific URL or even for a specific country or region.

What’s not so nice about Bing here is that they limited this block to 90 days. The block can be extended, but that means you have to revisit your webmaster tools every 3 months to extend the block you set.

Block URLs

The Block URLs section is very similar to the Deep Links section, in that it blocks URLs from showing up in the Bing search results. Adding a URL here will block the URL from showing up anywhere in the search results of Bing, though.

Annoyingly, they’ve again set the block to a maximum of 90 days, so you should only use this feature if you’ve just deleted a page from your site. Any permanent pages you want to block should be blocked in robots.txt.

Page Preview

The page preview is just that: a preview of your page. Bing doesn’t show this page preview everywhere, but they do in for example Bing Smart Search in Windows. You can tell Bing to refresh or remove the preview of a specific URL here. Just fill in the URL and click Fetch. You’ll then be given a choice to either block the preview or refresh it.

Disavow Links

Here you can disavow links you got from other websites. If you don’t know what disavowing a link means, read the post by Michiel on cleaning up bad backlinks. The best feature here by far is that you’re able to disavow not only a single page link, but also complete directories or domains. This can save you a lot of time when cleaning up your backlinks.


In this section you can set a geo-targeting for your domain, subdomain, directory or page. Geo-targeting means you tell Bing the part of your website you selected is meant for a specific region or country. For instance, if we had a German version of under, we’d set the geo-targeting for the directory to Germany. This doesn’t mean it won’t show in Bing results in other countries, it just makes it more clear to Bing what people should be seeing the content.

One note about this: although you can set geo-targeting for your domain, if your domain has a country-code (.nl, .de, for instance), geo-targeting your entire domain won’t work. Directories and pages can still be geo-targeted.

Verify ownership

If you need to verify your website again, you can do so here.

Connected Pages

Under Connected Pages you can tell Bing any social media presences you have elsewhere. By connecting these pages to Bing Webmaster Tools, you’ll be given quite some nice insights. For instance, we’re able to see how many times our twitter account appeared in search and how many times people clicked those appearances. We can even see what keywords people used to get our twitter account to appear, how many backlinks our twitter account (or status updates) has and what anchor texts were used for those links.

App Linking

Here you can link any apps you’ve connected in the Connected Pages section to Bing search results on Windows and Windows Phone. As we don’t have any experience with apps, I’d love to hear your thoughts on what this does exactly! ;)


Here you can manage the users that have access to Bing Webmaster Tools for the current site.

What do you think?

Although Bing is definitely used less than Google, their Webmaster Tools are actually pretty awesome. So if Bing gives you a relevant amount of traffic to your site, it’s definitely something you should check out!

Do you agree? Let me know in the comments!

The next post on the next main menu item in Bing Webmaster Tools, written by Michiel, will be published tomorrow.

This post first appeared as Bing Webmaster Tools: Configure My Site on Yoast. Whoopity Doo!

Google Webmaster Tools: Crawl

Google Webmaster Tools: CrawlThe section in Google Webmaster Tools that works most closely with our WordPress SEO Premium plugin, is the Crawl section. In our premium plugin, you’ll find a Webmaster Tools section, that lists all pages Google somehow did not find on your website. You can easily import these into the plugin and redirect the ones that need redirecting, so the crawl error is resolved.

But there is more in that fourth ‘chapter’ in Google Webmaster Tools. Following our posts on Search Appearance, Search Traffic and Google Index, this article digs into for instance crawl errors and stats, finding out how Google sees your website and your XML sitemaps.

Crawl Errors

This section lists two types of errors, being site errors and URL errors. Site errors simply lists whether your DNS works, the server can easily be reached (no timeouts, for instance) and if Google can access your robots.txt file.

Google Webmaster Tools: Crawl / Site error

Google provides background information on the error (when did it occur, how often in the last 90 days). If things like this happen too much (as in more than once or twice a year without advance warning), be sure to contact your hosting company or switch hosts altogether.

The URL error section is divided into multiple sections and subsections. First, you can check for errors that Google gets when acting a number of different devices. On for instance the desktop tab, we find the number of server errors, access denied errors and page not found errors.

Google Webmaster Tools: crawl / URL errors

Please monitor these errors, as too many errors could send a signal of low quality (bad maintenance) to Google. A 404 can be redirected, like mentioned for instance in our WordPress SEO Premium plugin, or with a bit more work straight in your .htaccess file. After that, check the checkbox in front of the URL and click Mark As Fixed (this will just make sure the list is cleaned up, it won’t do much besides that).

We have found in our reviews, that a lot of people either ignore the errors here, or forget to mark errors as fixed. This will only lead to a very long list of errors. Clean that list up now and then. If you want to check if any of these URLs are already fixed, click the link to find more information on the error and use Fetch as Google to see if the URL is now accessible for Google (or click Marked as fixed, of course).

Soft 404s

These tabs can also show Soft 404s. A soft 404 occurs, when a page as such exists, but has an empty ‘content area’. Let me elaborate on that. Google does a fine job on locating content on a page. It understands your header, sidebar, footer and content area. That also means that you can have a fully packed sidebar and footer, Google will still return a Soft 404 when the content area is empty. And by empty we also mean a category page with no posts, or any other page stating there is no content available. Or a page that just says 404 Not Found and returns a 200 OK server response anyway (happens more often than you think). This also occurs when you would for instance link an internal search page that just doesn’t return any results. There is no page, but an almost empty page is returned anyway.

Although the server will return a 200 OK message for that page, Google will consider it a (Soft) 404. You don’t want these pages on your website for a number of reasons. For one, these pages are obviously not very user-friendly. But besides that, Googlebot will have to go over all these pages for no reason at all (as they lack content), which will prevent your site from being crawled in an efficient way. After all, you don’t want Googlebot to spend time trying to see what’s on all these non-existing pages. Either add content to these pages or noindex them completely. In case of an empty category or tag page, consider removing the category or tag. You’re not using it anyway.

Smartphone visits

For your smartphone visits, Google also tests for faulty redirects and blocked URLs. A blocked URL is a page that is blocked for Googlebot-mobile for smartphones in robots.txt. Simply check if these are intentionally blocked and otherwise change robots.txt to allow access.

Faulty redirects occur when a page redirects to “an irrelevant landing page instead of the equivalent, smartphone-optimized version of the desktop page”. That could for instance be a redirect to your homepage instead of a redirect to the exact same page on your mobile site. Check the question mark next to labels in Google Webmaster Tools for more information on the terminology in Google Webmaster Tools, by the way. These explanations really come in handy, sometimes :)

Lastly, there can also be a tab for News at URL sections. That shows crawl errors for Google News content. Of course that is only for News sites. See the Google News requirements to see if your website qualifies at all for Google News, otherwise don’t bother looking for this tab.

In the end, you’ll just want this to be the only text in the Crawl Error section:

Google Webmaster Tools: No crawl errors

Crawl Stats

This is your handy overview of Googlebot activity on your website. It shows you the pages it crawled per day, the number of bytes downloaded per day and the time spent downloading a page. Depending on the number of pages of your website, this might be handy information.

Google Webmaster Tools: crawled per day

We think this is great for trend watching. If you have made changes to your site structure, robots.txt or for instance added XML sitemaps for the very first time, that should show in the provided graphs.

In case these stats show a drastically declining line, of even a flat line at zero, there’s probably something really wrong with either the website (robots.txt might be blocking Googlebot), or your server (it could be down, for instance). Again, monitor this.

Fetch as Google

As mentioned, in case of any crawl errors, you should look into what happened and why. One of the tools Google Webmaster Tools provides, is a way to view your website as Google does. You can fetch a page as Google. You could either click the link at Crawl Errors and click the Fetch as Google link in the pop-up, or go to the Fetch as Google section in Google Webmaster Tools to enter an URL manually:

Google Webmaster Tools: Fetch as Google

Note that in the image above, all rendered pages have been rendered quite a while ago – perhaps these already can be fetched the right way – so visit them to check this and perhaps even do a refetch. In the image, we see three different kinds of statuses (these go for both Fetch and Fetch and Render commands):

  • Partial: The page can be partially rendered, as some elements are probably not displayed as intended or not at all, for instance because you are blocking CSS or JS in your robots.txt file. If you click the line with the partial status in the overview, you’ll actually be taken to a snapshot of how Google rendered your page. On that page, Webmaster Tools will also tell you which resources it could not get, so you’ll be able to fix this.
  • Not Found: The page can’t be found. This might be because a redirect isn’t set yet in case of an URL / structure change, or perhaps you simply deleted the page (server returns a 404 error).
  • Unreachable: Googlebot didn’t have the patience to wait for your website to load (make it faster), or your server simply replied that it could not allow the request for the URL.

Of course there are more. Other statuses you might find here are:

  • Complete: That’s the one you want. Google managed to crawl the entire page.
  • Redirected: Either the server or your website itself (HTML/JS) told Google to visit another URL.
  • Not authorized: Your server tells Google that URL access is restricted or has been blocked from crawling (server returns a 403 error).
  • DNS not found: Perhaps you entered the wrong URL? The domain name seems incorrect.
  • Blocked: Your robots.txt tells Google to bugger off.
  • Unreachable robots.txt: Google can’t reach your robots.txt at all. More on testing the robots.txt below.
  • Temporarily unreachable: Either the server took too long to reply or too many consecutive requests were made to the server for different URLs.
  • Error: An error occurred when trying to complete the fetch (contact Webmaster Tools product support in this case).

By clicking the URL, you can see the rendered page as seen by both Googlebot and a visitor, so you can make an judgement on the impact of the blocked file:

Google Webmaster Tools: Render

In this case, the impact is clearly low.

Robots.txt Tester

Last week, Joost explained a lot about the Partial status at Fetch as Google in his post WordPress robots.txt Example. You really need to make sure your robots.txt is in order.

By the way, you might be wondering if you really need that robots.txt file. Actually, you don’t. If you think Google should crawl all sections on your server, you could leave it out. The Robots.txt tester will return this message in that case:

Google Webmaster Tools: No robots.txt

In the Robots.txt tester, Google will show the robots.txt you are using and tell you any and all issues Google finds:

Google Webmaster Tools: robots.txt warning

That’s a warning: Googlebot ignores that crawl delay. If for some reason you do want to set a crawl delay, please do so using the gear icon in the upper right in Google Webmaster Tools (at Site Settings > Crawl Rate – new crawl rates are valid for 90 days). Please note that this is not how often Google visits your site, it’s the speed of Googlebot’s requests during the crawl of your website.

Google Webmaster Tools: robots.txt error

Hm. And what do you want Google to do with /wordpress/wp-includes/? By the way, like in this example, we see a lot of webmasters adding a link to the sitemap to their robots.txt. No problem, but why not simply add that in Google Webmaster Tools instead? More on that later.

gwt robots txt wrong comment

This is another syntax not understood. Comments in robots.txt can be done using hashtags instead:

Google Webmaster Tools: robots.txt comment

Works like a charm.

gwt robots txt error

Ouch. The horror. Google could not find a user-agent in this robots.txt – but it’s there, right? It’s actually preceded by a space, and that immediately triggers a ‘syntax not understood’ for the robots.txt test.
It doesn’t mean all 20 restrictions in that robots.txt will be ignored, by the way. This seems to be a strict test, but Google is very capable of filtering that space. Google actually encourages whitespace for readability. But strictly speaking, it shouldn’t be in there.

Visit Webmaster Tools for even more information on the robots.txt syntax.

Test allowed / disallowed

One more thing. When you want to test whether a page or directory on your site can or can not be reached by Googlebot, or for instance Googlebot-News or Googlebot-Mobile, you can test that as well in this section, right below the robots.txt code.

If we take the last example above, and test the /Tests/ part of it, you’ll see that that indeed can be indexed if we follow the strict rules of the Robots.txt tester:

Google Webmaster Tools: robots.txt allowed

Although the text ‘Allowed’ is green, it’s not a good thing that this directory can be indexed. As mentioned, the space isn’t allowing Googlebot in this case, according to the Google search result pages:

Robots.txt tester isn't flawless

Feel free to add any insights on this particular issue in the comments below.

If you test a page or directory and find that it is blocked like this:

gwt robots txt url blocked

the test tool will also tell you what line in your robots.txt is causing this:

Google Webmaster Tools: robots.txt blocked by line

All in all, if you are using your robots.txt actively, make sure your robots.txt will do what you intended it to do. The Robots.txt tester will help you a lot.


It doesn’t matter if you are setting up a new WordPress site, or have just installed our WordPress SEO plugin: activate XML sitemaps and remove any other plugin that does this as well. Don’t forget to redirect the old XML sitemap, probably at /sitemap.xml, to ours at /sitemap_index.xml. If you experience any issues, check our knowledge base.

Having said that, if you have a proper XML sitemap, go to Google Webmaster Tools and test and add it at Sitemaps. Unfortunately Google doesn’t allow you to add a tested sitemap immediately after testing. Yes, this is my feature request, Google ;)

Google Webmaster Tools: Tabs on sitemaps

These sitemaps can be added manually, but perhaps Google already found some. These are listed on the All tab:

Google Webmaster Tools: sitemaps found by Google

We often get questions about image XML sitemaps. Images are actually already inserted by our plugin in for instance post-sitemap.xml and page-sitemap.xml:

Google XML Sitemaps

Back to Google Webmaster Tools. First you want to make sure /sitemap_index.xml contains all the content types you want Google to index. Please check the XML sitemap section in our plugin and see if you can exclude any post types or taxonomies from the XML sitemaps; this usually already fixes a lot of warnings and errors. Especially on shops, where sitemaps can be created for shipping classes and cloth sizes, for instance, that would be my first advice.

Second, you add the /sitemap_index.xml, which will be added immediately, along with any sitemaps listed on that sitemap. If for some reason that sitemap still contains content you’d rather not have indexed, simply change that in our plugin and resubmit it in Google Webmaster Tools. Note that you can manually add and delete sitemaps, but the ones that are automatically added for instance by an index sitemap, can only be deleted by a resubmit.

I thought about adding a list of possible errors and warnings to this article as well. Seriously. But when I found that Webmaster Tools actually added internal navigation on their Sitemap Errors page, it seemed to make sense to simply link that page.

Common warnings are warnings about Google not being able to reach a page, due to a long response time or for instance an URL included in an XML sitemap, but excluded in robots.txt.

Errors vary from invalid date formats to sitemaps not being found (a 404 on your sitemap is never a good idea). A sitemap can also be empty, or a required tag can be missing.

Indexed versus submitted content

Another thing you might wonder about is the difference between index and submitted content types (pages, video, images). These are the red and blue bars in this section. The red bar (indexed types) is usually a bit lower, as Google isn’t crawling your entire site at once. Time is precious, so Google spiders a (large) number of pages at a time, but if your site structure goes a gazillion levels deep, chances are Googlebot isn’t getting to these deepest pages in a crawl. It’s not that Google bookmarks where they end up and start from there the next time it crawls your website. This emphasizes the need for a good internal link structure, well formatted sitemaps and things like that.

URL Parameters

Let’s start with Google’s warning here:

Use this feature only if you’re sure how parameters work. Incorrectly excluding URLs could result in many pages disappearing from search.

If you are using URL parameters, like the default s for WordPress search, please check this section. When discussing this post with Joost, he told from experience that f.i. in case of a site migrations, things might go terribly wrong if this isn’t done properly.

In this section, you can tell Google how to handle your parameters. When clicking Add a Parameter, you’ll get a pop-up with these options:

gwt url parameters

I entered the ‘s’ for search, and have to decide on that parameter to affect page content or just change the way content is displayed on the page. Google respectively calls this passive and active URL parameters. Active parameters can for instance be used for sorting, pagination and sometimes even translations or categorization. Passive parameters are usually for tracking or referrals, like Magento’s SID (session ID) and Google’s own utm_source.

Now if you feel the parameter “changes, reorders or narrows page content”, and pick Yes in the above select box, you’ll be presented with four more options. You can set here how you want Google to handle the parameter:

  1. Let Googlebot decide: a general option if you’re not sure what to choose here.
  2. Every URL: Every URL using this parameter is an entirely new page or product.
  3. Only URLs with specified value: This will indicate to Google that you only want URLs crawled with a specific value for this parameter and forget the rest, for instance to avoid duplicate content due to sorting options.
  4. No URLs: Don’t crawl pages with this parameter at all. Avoiding duplicate content is a reason to use this one as well.

Note that instead of using URL parameters for option 3 and 4, you could also set the right canonical on all of these pages.


Sorry to disappoint you. There is no Too Long; Didn’t Read in this. In the previous posts on Google Webmaster Tools, we have already emphasized the importance of checking your site now and then, or monitoring it actively. Google Webmaster Tools helps a lot with that, and this is one of the longest (and in my opinion most interesting) sections in Google Webmaster Tools.

Feel free to drop any addition or question related to this section in the comments. Looking forward to it!

This post first appeared as Google Webmaster Tools: Crawl on Yoast. Whoopity Doo!