Update or delete? Cleaning up old content on your site

Sometimes, content on your website becomes irrelevant or out of date, and you need to decide whether to update it or delete it. It’s part of your regular content maintenance activities. There are several ways to go about this and this article helps you decide what’s the best solution for your old content!

Update old content that is still valid

Let’s start with an example: On our blog, we have an article on meta descriptions that needs constant updating to keep it relevant. We just have to make sure it stays up to date with all the changes Google keeps making to the way it handles meta descriptions. Sometimes it seems they can be a bit longer and sometimes they seem to go back to the old length again.

Our post helps writers and editors to write meta descriptions, even though the advice changes over time. Although the article itself might be what we call cornerstone content, its content must be updated to keep up with the latest standards – constantly.

You can easily create new, valuable content from your old posts if you update it and make it current again: old wine in new bottles, as the saying goes. You could, for example, replace older parts of that content with updates, or you could merge three old blog posts about the same subject into one new post. If you do this, please remember to redirect the old post URLs to the new post, using a 301 Redirect. More on that later.

Read more: Keep your content fresh and up to date »

Delete irrelevant posts or pages

It’s likely that you have old posts or pages on your site that you don’t need anymore. Think along the lines of a blog post about a product you stopped selling a while ago and have no intention of ever selling again, an announcement of an event that took place a long time ago or old pages with little or no content – so-called thin content pages.

These are just some examples, but I’m sure you know which posts and/or pages I’m talking about. This old content adds no value anymore, now or for the foreseeable future. In that case, you need to either tell Google to forget about these old posts or pages or give the URL another purpose.

When I talk about deleting old content, I don’t mean just pressing “delete” and then forgetting about it. If you do that, the content might show up in Google for weeks after deletion. The URL might actually have some link value as well, which would be a shame to waste.

So, what should you do? Here are two options:

“301 Redirect” the old post to a related one

When a URL still holds value because, say, you have a number of quality links pointing to that page, you want to leverage that value by redirecting the URL to a related one. With a 301 Redirect you’ll tell search engines and visitors there’s a better or newer version of this content elsewhere on your site. The 301 redirect automatically sends people and Google to this page.

Say you have an old post on a specific dog breed. You need to delete it, so the logical next step would be to redirect that post to a newer post about this dog breed. If you don’t have that post, choose a post about the closest breed possible. If that post isn’t available, you could redirect it to the category page for these posts (e.g. “dog breeds”) and if that is also not an option, redirect to the homepage. That last one might be about “pets”, for example. It’s a bit of a last resort though, there probably are better options on your site.

Creating a 301 Redirect (for instance in WordPress) isn’t hard, but doing it with Yoast SEO Premium is easy as pie. If you don’t have it yet, find out about all the extras that are in Yoast SEO Premium here.

Tell search engines the content is intentionally gone

If there isn’t a relevant page on your site you can redirect to, it’s wise to tell Google to forget about your old post entirely by serving a “410 Deleted” status to Google. This status code will tell Google and visitors the content didn’t just disappear; you’ve deleted it with a reason.

When Google can’t find a post, the server will usually return a “404 Not Found” status to the search engine’s bot. You’ll also find a 404 crawl error in your Google Search Console for that page. Eventually, Google will work it out and the URL will gradually vanish from the search result pages. But this takes time.

The 410 is more powerful in the sense that it tells Google that the page is gone forever, never to return. You deleted it on purpose, period. Google will act on that faster than with a 404. Read up about the server status codes if this is all gibberish to you.

Keep reading: How to properly delete a page from your site »

Do you have old content to deal with?

Cleaning up old content should be part of your content maintenance routine. If you don’t go through your old posts regularly, you’re bound to run into issues sooner or later. You might show incorrect information to visitors or hurt your own rankings by having too many pages about the same topic, increasing chances of keyword cannibalization, which is a lot of work to fix later on. Therefore, go through your old posts, and decide what to do: update, merge or delete!

Good luck cleaning up your site.

Read on: Should you keep old content? »

The post Update or delete? Cleaning up old content on your site appeared first on Yoast.

Should you keep old content?

Writing a blog post can be a challenge. It is hard work, but afterward, you’re probably proud of what you have created. No way you are ever going to throw those beautiful articles away, right? But what should you do with blog posts that are really, really old? Should you keep all of those?

In this blog post, I’ll explain why you cannot keep all of your old content. Also, I’ll explain what types of content you should keep on your site and which kinds of articles should be deleted.

Why you cannot keep all your content

Even if your content is really awesome, you need to do some cleaning. Otherwise, you’ll be hurting your own chances of ranking in Google. You see, there are only a limited number of places in Google’s search results pages. Google will only show 1 or 2 results from the same domain in the search results for any specific query. If you’re a high authority domain, you might get away with three results.

If you have written 3 articles focussing on the same – or very similar – keywords, you are competing with yourself for those limited spaces in the search results. You’ll be confusing Google. That’s why you cannot blog endlessly about the same content and leave it be. You need to do some content management.

Read more: What is keyword cannibalization? »

Update, delete or merge?

There are three things you can do with old content. You can keep it, you can delete it or you can merge it. Not sure what to do? It all depends upon your content.

1. Update valuable content

Is an article still very valuable? Does it get a lot of traffic from Google? Is the post still in line with your site and your company? Old content that is still very valuable should, of course, be kept on your website. Do make sure that this content is updated on a regular basis. Your most important articles should never contain any outdated information. Setting reminders for yourself to update those evergreens every now and then is a great way to make sure your content is always up to date. 

Keep reading: Keep your content fresh and up to date »

Solve it with site structure

Keeping content on your website does come with a price, especially if you write a lot about similar topics. Make sure you add some structure and hierarchy to your website. If one of your pages or posts is much more important than the other one, you should treat it as such. Place that important page higher in your hierarchy. Link from less important pages to your most important page. that way you’ll be telling Google which article you want to rank highest with and you can keep both articles.

Read on: How to set up a cornerstone content strategy? »

2. Delete (and redirect) outdated content

Is an article outdated? Does it contain invalid information? Does it contain information that’s no longer helpful? Every now and then you write about an upcoming event or you announce something new. After some time, these articles are pretty much useless. These types of articles should be deleted. Do make sure to redirect the article to something similar (or to the homepage if you cannot find an alternative).

By the way, did you know redirecting is incredibly easy with Yoast SEO Premium?

3. Merge content

Have you written multiple articles about the same topic? Are they pretty much the same? Are they ranking for the same topics? These types of articles should be merged. Make one really awesome article out of the two (or three) you have written. Then delete (but do not forget to redirect) the old articles. I would write the new merged article on the URL that attracted the most traffic from Google.

Keep on reading: Content maintenance for SEO: Research, merge & redirect »

Repost your updated content

Updating and merging your old blog posts is never a waste of time. But if you really want to reap the benefit of your refurbished content you can republish it on your blog, repost it on your social channels and mention it in your newsletter. This way you’ll make sure your content won’t be forgotten and you probably even attract new visitors to your site.

If you make minor changes to a recent article you can choose to just hit the Update button. In WordPress, the Last-Modified date will then change, but the Publish date will stay the same. When you’ve completely merged or rewritten content you can choose to also change the Publish date. By doing so, your post will pop up on top of your blog again. Share it on your other marketing channels too and it’ll surely get the attention it deserves!

If you do the latter, don’t forget to delete the old comments on the article! It might come across odd to publish an article with 5-years-old comments further down the page.

Conclusion: continue to clean up

Checking, updating, structuring and deleting old content should be part of a process. Just like you need to clean up your kitchen closet every now and then, you also need to clean up your old content. As your site grows, you need to clean out the content and maintain the structure. This really needs to be a core element in every SEO strategy.

Read more: Update or delete: Cleaning up old content on your site »

The post Should you keep old content? appeared first on Yoast.

Ask Yoast: Old static website, move to WordPress?

Imagine, you created a website a few years ago. It’s still out there, but you didn’t make any changes or updates ever since. So, your site probably needs a major – SEO – update. If you have a static website, you might consider to move your site to a CMS, like WordPress. What’s the best choice? I’ll help you out and explain in which case it would be better to start all over using WordPress.

In this Ask Yoast, we’ll answer a question from Richard Millstein:

“My website was created 10 years ago in HTML, it needs a major SEO update and has other issues. I think it would be better to start over using WordPress. What do you think?”

Check out the video or read the answer below!

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO for WordPress pluginBuy now » Info

Major SEO update

Read this transcript to learn more about choosing a CMS or not, when your website needs a major SEO update:

“Well, you get plus points for using WordPress, of course, no questions asked. Also, if your website was created 10 years ago and not much has happened to it since, then, you really need to think about, “Okay, what will I do once I re-create it? “Will I not do anything with it again for 10 years or will I keep updating it?”

If you want to keep updating it, then yes, you should really go for WordPress, because that makes that an awful lot easier. Of course, with WordPress you also get Yoast SEO and a lot of other advantages or things that you don’t have to build, that will work automatically for you. So, yes, you should probably do that.

The funny thing is, the output from WordPress will still be HTML, so you could probably get your theme of your site to look like your old site very easily. If you just hire someone to copy that into a WordPress theme and maybe do some optimizations as they do that. So, it could be a very simple job on Upwork or some other rental site, where you just go in and say “Hey I want you to change this theme to a WordPress theme and then input my content in it.” That could be a very simple job for someone and might save you an awful lot of time.

Good luck!”

Ask Yoast

In the series Ask Yoast we answer SEO questions from followers! Need help with SEO? Let us help you out! Send your question to ask@yoast.com.

Read more: ‘WordPress SEO’ »

How to create a 301 redirect in WordPress

There will be times where you want to redirect visitors to a different part of your website when they visit a particular page or post. Reasons for this can be that you renamed a post and its URL, a page was removed or you want a different page to rank.

Redirects in a nutshell

The name ‘redirect’ pretty much says it all: It sends visitors traveling to a specific page to an alternative one. But what does this 301 mean and how does it differ from a 302 redirect? Both send your users to a different page. The only subtle (yet very important) difference is that a 301 will permanently send visitors and search engines to the new destination. 302 redirects indicate that you only temporarily want visitors to be sent to a different page.

Creating a 301 redirect on the server

One of the most basic methods of adding a 301 redirect, is by editing your .htaccess file on the server. This method is only available on Apache servers. Nginx has their own way of defining redirects in the server configuration and requires extensive knowledge of system administration.

These configurations can get quite unmaintainable over time, especially if you’re an avid blogger or you’re trying to improve the SEO of your posts. On top of that, you would have to log in on your server over FTP, edit the files and re-upload them every time you add a new redirect. That’s why, generally speaking, using this method is not considered the way to go.

Creating a 301 redirect with PHP

As a WordPress developer, you have two options: Either you make a redirect by altering the headers of a file in the code -or- you make use of WordPress’ built-in
wp_redirect function.

An example of plain PHP could be as follows:

<?php
// MyExampleFile.php
header("HTTP/1.1 301 Moved Permanently"); 
header("Location: http://www.my-blog.com/a-new-destination"); 
?>

And this is how you’d do the same, but now by using WordPress’ built-in function:

wp_redirect( get_permalink( http://www.my-blog.com/a-new-destination ), 301 );

If you forget to add the 301, both WordPress and PHP will both assume that it’s a 302 redirect, which isn’t always the case.

This method is a bit easier than editing files on the server, but can also become cumbersome once the amount of redirects increases.

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO for WordPress pluginBuy now » Info

Creating a 301 redirect with Yoast SEO

Our Yoast SEO Premium plugin offers you a helping hand when it comes to creating these redirects. Our built-in Redirect manager assists you whenever you change the URL of a post, page or any of the taxonomies that may result in a possible 404 if you don’t properly redirect visitors.

In addition, we also offer you an interface to edit or remove these redirects at a later point in time. The plugin also tells you when you’re about to create a redirect that will result in a redirect loop. This looping is something you want to avoid at all costs.

Read more: ‘How to properly delete pages from your site’ »

Keep your content fresh and up to date!

How do you make sure that all content on your website is still relevant? Why is that even important? For some of you, coming up with new ideas for blog posts seems the biggest challenge of blogging. However, you want your existing posts to show up-to-date content as well. And that could even be a bigger challenge! Here, I’ll explain why it’s worth investing time in having fresh content on your site. I’ll also give you some tips on how to achieve this.

Why is up-to-date content important?

Updating content is essential, because you don’t want people to find information on your website that isn’t valid anymore. If you’ve written a high quality blog post and optimized it well, chances are that people will find that post in the search engines. That was the whole point of writing and optimizing that post in the first place. Imagine a blog that hands out SEO advice to its readers. Older posts shouldn’t contain advice that isn’t valid anymore, because you want people to invest time in SEO strategies that work nowadays.

Why is fresh content important for SEO?

Keeping your content fresh is important for SEO reasons as well. If you rewrite an article, Google will notice changes in that particular article. If you update your content regularly, you’re actually showing Google that your site is alive. You could either update an article – with or without changing the date – or you could republish the post.

Read more: Should you republish old content? »

Tips on how to keep your content fresh

1. Crucial: update those cornerstones

My most important tip on keeping content up to date is to focus on your cornerstone articles. These articles should be the best articles on your website. These are the articles you want to be found for. So, these should be on the top of your list, while updating content.

In Yoast SEO Premium you can now turn on a beta version of the new SEO analysis, including a stale content filter. This handy feature will let you know whether your cornerstone content hasn’t been updated for 6 months or longer. On top of the post overview you can click on the filter and see which cornerstones you should go through and update. Learn all about testing the new version of the Yoast SEO analysis here

Let me give you an example from Yoast.com. I mostly write about blogging and about content SEO. The most important article about content writing is SEO copywriting: the ultimate guide. Regularly, I write a new blog post about content SEO or about copywriting. I do some research for each and every one of those (I really do!). I discover some new things. These new discoveries should be covered – briefly – in my cornerstone article as well. So, I (should) take the time to update my ultimate guide now and then too. And you should do the same!

2. Update posts that generate a lot of traffic

Updating can be a lot of work. And if your site becomes really large, this could be a day job! If you have to make choices which articles to tackle and update first, take a look at which pages generate a lot of traffic. Start with those. These are the pages that are actually seen by your audience. These pages have the highest priority on showing fresh content.

3. Delete really outdated posts

As your blog is growing and you’re writing tons of blog posts, some of your content slowly becomes really old. Some posts are evergreens, but some posts just aren’t. If these articles aren’t read by anyone either, you could decide to delete them all together. This will clean up your site nicely! If you do decide to delete a post, make sure to read Joost’s post about properly deleting a post.

Keep reading: The site structure of a growing blog »

4. Merge posts that are much alike

If you publish lots of posts – like we do – you’ll end up with some posts that are quite similar. When you’re updating old content, check if you have posts that are much like the one you’re updating. Those posts might diminishes the chance of ranking for the one you’re working on! This phenomenon is called keyword cannibalization. You can solve this by checking which of all these posts on the same topic perform best in the search engines. Update the one that ranks highest and merge the other posts into that one, high quality and complete, blog post.

5. Don’t forget ‘other’ pages

Not only your blog posts should have up-to-date content, other pages are important as well. Don’t forget to update those FAQ. Perhaps new questions have emerged among your audience. Make sure to add those. While updating your website, don’t forget the ‘about page’, the ‘contact page’ and other static pages on your website. I understand that this content doesn’t change that often, but make sure that changes are actually being made!

How to keep content fresh?

Keeping the content of your website up to date and fresh is important if you want to attract readers from the search engines with (somewhat) older posts. While rewriting and refreshening old content, make sure to start with those cornerstones and most popular articles first!

Read on: Website maintenance: clean up old posts & pages »

The post Keep your content fresh and up to date! appeared first on Yoast.

Ask Yoast: why connect GSC with Yoast SEO?

If you use our Yoast SEO plugin you’ve got the opportunity to connect it to Google Search Console (GSC). With GSC you can monitor the SEO health of your site, while Yoast SEO helps you to optimize your site. Connecting the two, so they can work together, will allow you to be more efficient when maintaining your site.

In this Ask Yoast we’ll give you the answer to the following question:

“How does implementing Google Search Console in Yoast SEO help me to optimize my site?”

Check out the video or read the answer below!

Get the most out of Yoast SEO, learn every feature and best practice in our Yoast SEO for WordPress training!

$ 79 - Start learning today »Yoast SEO for WordPress training

Google Search Console in Yoast SEO

Read this transcript to learn how you’ll benefit from connecting GSC to Yoast SEO:

“Well, honestly, connecting the two doesn’t help if you don’t make any mistakes on your site. But you’re human, right? So you’re going to make mistakes. GSC tells you when Google has found errors on your site that you should fix. 

By implementing GSC in Yoast SEO, you’ll see those errors in Yoast SEO. It makes it possible for you to very easily fix those errors and make sure that your site is as error free as possible. This is the best thing you can do for your site’s SEO.

So, if you don’t make any mistakes ever, then it’s not going to help you. If you do make some mistakes sometimes, even if it’s only a couple of mistakes over the years, it really helps to find these errors, fix them, and then be done with them.

Good luck!

Ask Yoast

In the series Ask Yoast we help you with your SEO question! Did you get stuck on an SEO issue? Are you in doubt between two SEO strategies? Don’t fret, just ask us your question! You can send your question to ask@yoast.com.

Read more: ‘Google Search Console Crawl’ »

Feature highlight: WP SEO Premium redirect manager

One of the things we are understanding more and more, is that we have failed to explain why our WordPress SEO Premium plugin is awesome. Today, I’d like to tell you about the redirect manager that’s contained in it.

First, let me show you this video:

This is one of the 13 videos that you get along with WordPress SEO Premium as well, that explain how to use the plugin.

Redirect when you need to

WordPress SEO Premium makes sure you don’t leave holes in your site. When you’re changing a posts URL, you should redirect the old URL to the new one. When you’re deleting a post, in most cases, you should redirect the URL somewhere. Both of these are common things to do when you’re cleaning up your old content. In both cases, WordPress SEO Premium will give you notices like this:

Image of the redirect manager notice that says: WordPress SEO Premium detected that you moved a post to the trash. Click here to create a redirect from the old post URL. Image of the notice that says: WordPress SEO Premium created a redirect from the old post URL to the new post URL. Click here to undo this.

Similarly, when you’re cleaning out your tags or categories, you want to redirect them when you delete or change them:

Image of the redirect manager notice saying: WordPress SEO Premium detected that you deleted a term. Click here to create a redirect from the old term URL.

Note that these notice have been improved slightly from what they look like right now, you’ll all get this update next Wednesday when we release WordPress SEO 2.2.

Solve crawl issues with the redirect manager

The redirect manager is heavily tied into the Google Search Console (formerly known as Google Webmaster Tools) integration, which allows you easily solve all Google’s crawl issues. Simply click “Redirect” and tell the plugin where the redirect should go.

WordPress SEO, even the free version, helps you to optimize your site. WordPress SEO Premium, with this functionality, helps you make sure your site stays optimized.

Interested? Go get WordPress SEO Premium. It includes a year of support and comes with a month-long money back guarantee, no questions asked.

This post first appeared as Feature highlight: WP SEO Premium redirect manager on Yoast. Whoopity Doo!

Website Maintenance: clean up old posts & pages

website maintenance: clean up old postsWhen you maintain a blog, you write different types of posts. Of course, you try to write evergreen cornerstone content posts all the time. The reality is different. Some posts have historical value, others only have momentary value. You should go through your archive every once in a while, and clean it up. This should be a part of your regular website maintenance routines.

One way I like to go through our old posts is by checking our XML sitemap. You open the post or page sitemap and go through the posts from old to new. There are three possible decisions for each post: keep, re-do and delete. Let’s go into what they mean you should do.

Keep your old post

You should keep a post if all the below are true:

  1. The posts’s content is still valid and true.
  2. The post is well optimized for its focus keyword.
  3. The post is receiving decent traffic. Decent being relative to how big your site is and how such a post could do if it was well optimized and posted now.

If you’re going to keep the post, check if it has all the appropriate tags and categories. Maybe it should link to posts you’ve written later on to tie them together.

Then, update the post to make it look more timely. Even a short update will do wonders for how people perceive the value of the post. You could for instance add something like:

“2015-05-13: this post is still as valid as it was when I wrote it. Here are some more posts about this topic: ”

This shouldn’t be more than a few minutes work for each post.

Re-do your old post

You should re-do a post if any of the following are true:

  1. The posts’s content is no longer completely true.
  2. The post is poorly optimized for its focus keyword.
  3. The post is doing far worse than what it would do if you wrote a new one. This is often tied to point 2 being true.

At this point it’s usually best to start fresh. Read your old post, copy some of the good bits into a new post and start writing. Optimize the content for the focus keyword. Check your tags and categories. Do everything we tell you to do in our Content SEO book and in Marieke’s posts.

Now, change the URL of your old post (adding for instance -old to it) and hit update. Publish your new post under the URL of your old post. Link to your new post from other posts more recent than your old post. Then delete the old post and redirect it to the new one.

You delete a post by clicking “move to trash” in WordPress. Our WordPress SEO Premium plugin will give you a notice to redirect the post to a new URL when you do. If you don’t use our WordPress SEO Premium plugin… Well, you should!

Merge old posts

I often find myself merging several old posts into one newer post. The reason is that posts we write here are longer now than they were in the past. Also, sometimes something has become easier to explain. You can still do all of the above and just redirect all your old posts into your single new post.

Delete the old post

If the value of the post was truly only temporary, don’t be afraid to delete it. When we delete old content, we redirect the URL to the most appropriate URL on the site. You could also just decide to let the old post 404. If people are linking to your old post and you delete it, you’re throwing away link value. At that point, it’s better to redirect the old URL, if only to your homepage.

Don’t throw away history though: historical posts are fun to link back to later on. For instance, in our infographic post earlier this week, I linked to my original gravatar post. This post belongs to the history of this site, deleting it would be a shame.

Ready? There’s more website maintenance to do!

Once you’re done with this (or have had enough for a while), make sure to do the other website maintenance tasks too. For starters, clean up your 404’s!

This post first appeared as Website Maintenance: clean up old posts & pages on Yoast. Whoopity Doo!

Crawl efficiency: making Google’s crawl easier

crawl efficiencySearch engines crawl your site to get the contents into their index. The bigger your site gets, the longer this crawl takes. It’s important that the time spent crawling your site is well spent. If your site has a 1,000 pages or less, this is not a topic you’ll need to think about much. If  you intend on growing your site though, keep on reading. Acquiring some good habits early on can safe you from huge headaches later on. In this article we’ll cover what crawl efficiency is and what you can do about it.

All search engines crawl the same way. In this article we’ll refer to Google and Googlebot.

How does a crawl of your site work?

Google finds a link to your site somewhere on the web. At that point that URL is the beginning of a virtual pile. The process is pretty easy after that:

  1. Googlebot takes one page from that pile;
  2. it crawls the page and indexes all the contents for use in Google;
  3. it then adds all the links on that page to the pile.

During the crawl, Googlebot might encounter a redirect. The URL it’s redirected to goes on the pile.

Your primary goal is to make sure Googlebot can get to all pages on the site. A secondary goal is to make sure new and updated content gets crawled fast. Good site architecture will help you reach that goal. It is imperative though that you maintain your site well.

Crawl depth

An important concept while talking about crawling is the concept of crawl depth. Say you had 1 link, from 1 site to 1 page on your site. This page linked to another, to another, to another, etc. Googlebot will keep crawling for a while. At some point though, it’ll decide it’s no longer necessary to keep crawling. When that point is, depends on how important the link pointing at that first page is.

This might seem theoretical, so let’s look at a practical example. If you have 10,000 posts, all in the same category and you show 10 articles per page. These pages only link to “Next »” and
“« Previous”. Google would need to crawl 1,000 pages deep to get to the first of those 10,000 posts. On most sites, it won’t do that.

This is why it’s important to:

  • Use categories / tags and other taxonomies for more granular segmentation. Don’t go overboard on them either. As a rule of thumb, a tag is only useful when it connects more than 3 pieces of content. Also make sure to optimize those category archives.
  • Link to deeper pages with numbers, so Googlebot can get there quicker. Say you link page 1 to 10 on page 1 and keep doing that. In the example above, the deepest page would only be 100 clicks away from the homepage.
  • Keep your site fast. The slower your site, the longer a crawl will take.

XML Sitemaps and crawl efficiency

Your site should have one or more XML sitemaps. Those XML sitemaps tell Google which URLs exist on your site. A good XML sitemap also indicates when you’ve last updated a particular URL. Most search engines will crawl URLs in your XML sitemap more often than others.

In Google Webmaster Tools, XML sitemaps give you an added benefit. For every sitemap, Google will show you errors and warnings. You can use this by making different XML sitemaps for different types of URLs. This means you can see what types of URLs on your site have the most issues. Our WordPress SEO plugin does this for you automatically.

Problems that cause bad craw efficiency

Many 404s and other errors

While it crawls your site, Google will encounter errors. It’ll usually just pick the next page from the pile when it does. If you have a lot of errors on your site during a crawl, Googlebot will slow down. It does that because it’s afraid that it’s causing the errors by crawling too fast. To prevent Googlebot from slowing down, you thus want to fix as much errors as you can.

Google reports all those errors to you in its Webmaster Tools, as do Bing and Yandex. We’ve covered errors in Google Webmaster Tools (GWT) and Bing Webmaster Tools before. If you have our WordPress SEO Premium plugin, you can import and fix the errors from GWT with it. You can do that straight from your WordPress admin.

You wouldn’t be the first client we see that has 3,000 actual URLs and 20,000 errors in GWT. Don’t let your site become that site. Fix those errors on a regular basis, at least every month.

Excessive 301 redirects

I was recently consulting on a site that had just done a domain migration. The site is big, so I used one of our tools to run a full crawl of the site and see what we should fix. It became clear we had one big issue. A large group of URLs on this site is always linked to without a trailing slash. If you go to such a URL without the trailing slash, you’re 301 redirected. You’re redirected to the version with the trailing slash.

If that’s an issue for one or two URLs on your site it doesn’t really matter. It’s actually often an issue with homepages. If that’s an issue with 250,000 URLs on your site, it becomes a bigger issue. Googlebot doesn’t have to crawl 250,000 URLs but 500,000. That’s not exactly efficient.

This is why you should always try to update links within your site when you change URLs. If you don’t, you’ll get more and more 301 redirects over time. This will slow down your crawl and your users. Most systems take up to a second to server a redirect. That adds another second onto your page load time.

Spider traps

If your site is somewhat more authoritative in Google’s eyes, fun things can happen. Even when it’s clear that a link doesn’t make sense, Google will crawl it. Give Google the virtual equivalent of an infinite spiral staircase, it’ll keep going. I want to share a hilarious example of this I encountered at the Guardian.

At the Guardian we used to have daily archives for all our main categories. As the Guardian publishes a lot of content, those daily archives make sense. You could click back from today, to yesterday and so on. And on. And on. Even to long before the Guardian’s existence. You could get to December 25th of the year 0 if you were so inclined. We’ve seen Google index back to the year 1,600. That’s almost 150,000 clicks deep.

This is what we call a “spider trap”. Traps like these can make a search engines crawl extremely inefficient. Fixing them almost always leads to better results in organic search. The bigger your site gets, the harder issues like these are to find. This is true even for experienced SEOs.

Tools to find issues and improve crawl efficiency

If you’re intrigued by this and want to test your own site, you’re going to need some tools. We use Screaming Froglot during our site reviews. It’s the Swiss army knife of most SEOs. Some other SEOs I know swear by Xenu, which is also pretty good (and free). Be aware: these are not “simple” tools. They are power tools that can even take down a site when used wrong, so take care.

A good first step is to start crawling a site and filter for HTML pages. Then sort descending by HTTP status code. You’ll see 500 – 400 – 300 type responses on the top of the list. You’ll be able to see how bad your site is doing, compared to the total number of URLs. See an example below:

Screaming Frog screenshot

How’s your sites crawl efficiency?

I’d love to hear if you’ve had particular issues like these with crawl efficiency and how you solved them. Even better if this post helped you fix something, come tell us below!

This post first appeared as Crawl efficiency: making Google’s crawl easier on Yoast. Whoopity Doo!

Website maintenance: Check and fix 404 error pages

If your website is important to your business, it’s essential to schedule time to keep it running smoothly. Therefore we regularly write about the things you should do to keep your site in shape. In this post, we’ll write about the most basic of all: checking for 404 errors.

Note: this post does not cover the required elements of a good 404 page, we do have a post on that, though: Thoughts on 404 error pages.

404 errors and broken links

One of the most annoying things that can happen to a visitor is to hit a 404 “page not found” error on your website. Search engine spiders tend to not like such errors much either. Annoyingly, search engines often encounter other types of 404s than your visitors, which is why the first section of this post is split in two:

1. Measuring visitor 404 error pages

If you use the MonsterInsights plugin, it’ll automatically tag your 404 pages for you. So then, if you go into your Google Analytics account and go to Behavior → Site Content → Content Drilldown and search for 404.html, you’ll find a ton of info about your 404s (click for larger version):

Google Analytics report showing 404 error pages

You’ll see URLs like this:

/404.html?page=/wordpress/plugin/local-seo/&from=https://yoast.com/articles/wordpress-seo/

This tells you two things:

  • The 404 URL was /wordpress/plugin/local-seo/ (it lacks an s after plugin)
  • It was linked to from our WordPress SEO article.

Using this info, you can fix the 404 and go into the article and fix the link.

As you can see from the above screenshot, we actually get 404s too. We break things all the time because our website is a constant work in progress! Making sure that you notice it when you’re breaking things is a good way of not looking stupid for too long though.

2. Measuring bot 404 error pages

Next to 404s for visitors, search engines will also encounter 404s on your site that can be quite different. You can find the 404s that search engine spiders encounter by logging into their respective Webmaster Tools programs. There are three webmaster tools programs that can give you indexation reports, in which they tell you which 404s they encountered:

  1. Bing Webmaster Tools under Reports & Data → Crawl Information
  2. Google Search Console under Coverage → Errors
  3. Yandex Webmaster under Indexing → Excluded Pages → HTTP Status: Not Found (404)

One of the weird things you’ll find if you’re looking into those Webmaster Tools programs is that search engine spiders can encounter 404s that normal users would never get to. This is because a search spider will crawl just about anything on most sites, so even links that are hidden will be followed.

If you’re serious about website maintenance, you might want to find these 404s before search engines encounter them. In that case, spidering your site with a tool like Screaming Frog will give you a lot of insight. These tools are built specifically to behave just like search engine spiders and will, therefore, help you find a lot of issues.

Fixing 404 errors

Now that we’ve found all these 404 errors, it’s time to fix them. If you know what caused the 404 and you can fix the link that caused it, it’s best to do that. This will be the best indication of the quality of your site for both users and search engines.

As search engines will continue to hit those URLs for quite a while, it actually makes sense to still redirect those faulty URLs to the right pages as well. To create those redirects, there are several things you can do:

  • Create them manually in your .htaccess or your NGINX server config
    While this is not for the faint of heart, it’s often one of the fastest methods available if you have the know-how and the access to do it.
  • Create them with a redirect plugin
    There are several redirect plugins on the market, the most well-known one being Redirection. This is a lot easier but has the disadvantage of being a lot slower as to do the redirect, the entire WordPress install has to load first. This usually adds half a second to a second to the load time for that particular redirect.
  • Create them with our Yoast SEO Premium plugin
    Our Yoast SEO Premium plugin has a redirect module that allows you to make redirects with the ease of the WordPress interface, but also allows you to save those to your .htaccess file or a NGINX include file, so they get executed with the speed of the first option above. It actually also has another few nifty options: you can get the 404 errors from Google Search Console straight in your WordPress install and redirect them straight away, and it’ll add a nice button in your WordPress toolbar if you’re on a 404 page:

Check for image / embed errors

If you’d look at your server logs, you’d get 404 errors of a different type too: 404s for broken images or broken video embeds. You might also have errors that don’t show up in your logs, like broken YouTube video embeds. They don’t cause the entire page not to work, but they do look sloppy. These types of errors are harder to find because webmaster tools programs don’t report them as reliably and you can’t track them with something like Google Analytics either.

The easiest method to find these broken images and embeds is using one of the aforementioned spiders. Screaming Frog, in particular, is very good at finding broken images. Another method is to check your server logs and go through them searching for a combination of 404 and “.jpg” and “.png”.

How often should you check for 404 errors?

You should be checking your 404s at least once every month and on a bigger site, every week. It doesn’t really depend on how much visitors you have but much more on how much content you have and create and how much can go wrong because of that. The first time you start looking into and trying to fix your 404 error pages you might find out that there are a lot of them and it can take quite a bit of time… Try to make it a habit so you’ll at least find the important ones quickly.

Read more: Content maintenance for SEO: research, merge and redirect »

The post Website maintenance: Check and fix 404 error pages appeared first on Yoast.