How to get your new WordPress site indexed

When you roll out the red carpet on a new website, you might be expecting visitors to start coming eagerly. In reality, before people start visiting your site, search engines need to find, index and rank it. In this article, I share some tips that could help you get your site indexed faster. 

A quick note before we go on. Although in the short term, there are some things you can do to get your site indexed faster, you mustn’t forget the long game. A sustainably high-ranking website depends in large part on creating remarkable content. Why? Because search engines want to find the best answer to the queries their users make. The site with the best content wins the race to the top of the search results. 

How do search engines work? 

To understand how to get your site indexed, it’s useful to know how search engines work. Search engines generate results in three main steps: crawling, indexing, and ranking. 

Crawling is the process of discovery done by crawlers, bots, or spiders. A computer program instructs crawlers on what pages to crawl and what to look for. When crawlers land on a page, they gather information and follow links. Whatever they find, they report back to the search engine servers. Then, the search engine tries to make sense of the page in order to index it. It looks at the content and everything it finds, it puts in a giant database; their ‘index’. 

Finally, ranking begins when you search for something online. So, the search engine algorithm looks through the index and filters the pages to find the best ones. We do not know the exact mechanics of the algorithm. Still, we know that search engines are especially enthusiastic about high-quality content and user-friendly, up-to-date pages.

Read more: What does Google do? »

Crawling, indexing and ranking never stop. As new pages keep pouring in, and as old ones get updated, the crawlers continuously crawl, and the search engines get new and improved ways to gather and display results

So, how can you speed up this process and help search engines find you quicker? An excellent way to start is submitting an XML sitemap in Google Search Console.

How to get your site found and indexed by Google:

  1. Create an XML sitemap with Yoast SEO

    An XML sitemap is a file that contains information about your website. In plain language, it is a list of your most important pages. It is a useful tool that helps Google find and explore your site. Yoast SEO can help you to create a sitemap. All you need to do is enable the XML sitemap option and the sitemap will be automatically generated. It’s quite a time saver!

  2. Set up an account with Google Search Console

    After you’ve created the sitemap, you need to tell Google about it. Google Search Console is a tool that can help you with that. To add your sitemap to the Console, you need to create an account. Yoast SEO can also help you get your site verified with Google Search Console.

  3. Add your sitemap to Google Search Console

    In Google Search Console, you will find the XML sitemap tab. There, you can add the sitemap you created, so that Google will know where to find it. If you update content on your site, your XML sitemap will be updated automatically.

  4. And/or, submit your most important individual pages in Google Search Console

    On top of this, you can ask Google to crawl individual pages too. In Google Search Console you’ll find the URL inspection tool, where you can ask Google to crawl or recrawl a URL. There’s a quota, so think about which pages are crucial for your business in terms of ranking and submit those here.

Why do you need an XML sitemap?

We mentioned that crawlers discover pages by following links. When you have a new website, you may face at least two issues. First, there are likely not that many external sites that point to your website. Second, you probably still don’t have a lot of content, so your internal linking and your site structure are not (yet) stellar. Without links, how can the crawlers come to your site? 

One solution to getting your site indexed is to create a sitemap right from the start and add it to Google Search Console. However, since you might still not have a lot of content, you should be careful with what you include in it. Although you can create sitemaps for videos, images, categories, and tags, it does not mean that you should necessarily do it. For example, you might have already set up some categories. But, for each category, you have just one post. In that case, creating a sitemap for your categories is not so useful, since the content does not give a lot of information, both to visitors and crawlers. 

It is important to note that Google may not crawl and index all the items in your sitemap. Still, we encourage you to create one, as we believe that you will benefit from it.

Getting your site indexed beyond Google

We told you how you could submit a sitemap with Google Search Console. But, it’s not the only search engine out there. So, how can you submit your sitemap to other search engines? It is easy with Yoast SEO. Other search engines also have Webmaster tools, where you can submit a sitemap and follow the performance of your site. Currently, you use Yoast SEO to add your site to:

What’s next? 

After you’ve created a sitemap and connected it to search engines with Yoast SEO, can you finally sit back, relax, and watch as visitors pour in? Not really.  As we said, you will have to continue making high-quality content. Don’t forget that you can also use social media to your advantage and strategically share your content there. Another important thing is getting links from other, preferably high-ranking websites. That means that you will need to work on your link building. Of course, don’t forget to apply holistic SEO strategies to your website to cover all SEO fronts and ensure high rankings.

Keep reading: How to rank high in Google »

The post How to get your new WordPress site indexed appeared first on Yoast.

Help, I’ve accidentally noindexed a post. What to do?

It can happen to anyone: You’re working on your site, fiddling on some posts here and there, and hit update when you’re done. After a while, you check back on how a post is doing and, to your dismay, it disappeared completely from the search engines! It turns out you’ve accidentally set a post or page to noindex on your site. Here, we’ll share a – pretty funny – story about how it happened to us, and what to do if you’ve made a similar mistake.

How to unintentionally noindex a post

Let’s start with a short story. We have a post called Noindex a post in WordPress, the easy way on yoast.com. In this post, we – surprise, surprise – explain how to noindex a post with Yoast SEO in WordPress. To show how easy that is, we added some screenshots of setting a post to noindex. A picture says more than a thousand words, right?

setting a post to noindex with Yoast SEO
Some of the copy and the screenshot in the ‘How to noindex a post’ post

Now, we’ll reveal a little secret. Oftentimes, when we want to illustrate a certain feature of Yoast SEO, we’ll just take a screenshot of that feature in the post we’re working on. So, in this case, we went to the Advanced tab in the Yoast SEO meta box, clicked No in the dropdown, took a screenshot, and added the screenshot to our post. We checked the copy we’ve written, added images, checked the SEO and readability scores and previewed our post. All looked fine, so we hit publish, shared it on social and in our newsletter and went on with other tasks.

Sometime later, we were checking how our content performed on the query [how to noindex a post] in Google. Surprisingly, we didn’t encounter this article, while we were pretty sure we already had a post like this. We started looking for it in our post overview, and there it was! Waiting in vain for visitors to learn more about this handy feature of Yoast SEO.

So, while we were happily typing away, making sure people understand what this feature is about, we forgot one thing… removing the noindex from this post. Therefore, accidentally and ironically, we’ve set our post about setting posts to noindex to… noindex.

How to reverse noindexing a post

In our case, reverting the noindex on that post wasn’t very difficult. The post, although it describes a nifty feature of Yoast SEO, wasn’t crucial for our business. Therefore, we decided to just remove the noindex and republish and share it again. But there’s more you can do; the options to get your article back in the search engines are listed here below. Depending on the severity of the issue you can choose to follow all steps or select some of them.

1. Remove the noindex tag

This is an essential step. You can easily remove the noindex tag by Google and other search engines in the Advanced tab of the Yoast SEO meta box. Just click on Yes here and you’ve removed the noindex tag:

Remove the noindex tag in the Yoast SEO meta box

In the search appearance section, you can set multiple posts or pages on your site to noindex. If you did that by accident or forgot to reverse that after temporarily setting it to noindex, you can set it to index again there too:

Remove the noindex tag on a post type in the Search appearance section of Yoast SEO

If you’ve added a meta robots tag in the code to noindex your post, please remove it from the code. There’s no need to set it to index though since that is the default value when nothing is set.

2. Google Search Console is your friend

If you’ve accidentally noindexed a valuable post or maybe even an important part of your website, there are some things you can do to make Google retrieve your content faster. Google Search Console can help you do this. So if you didn’t sign up for Google Search Console yet, now’s the time to do it. Yoast SEO will help you to verify your site, as you can read in this guide on how to add your site to Google Search Console.

Request for reindexing of a URL

In the URL Inspection Tool of Google Search Console there’s an option to ask Google to crawl or recrawl a URL. This might speed up the process and allows you to follow the progress. There is a quota for submitting individual URLs with this tool. So, if you’ve noindexed (a part of) your site it might be wise to select the posts or pages that are most crucial for your business and request to index those again.

Resubmit your XML sitemap

Another option is to resubmit your XML sitemap in Google Search Console. If you’re using Yoast SEO you don’t have to worry about this though. In that case, when you publish or update content on your site, Yoast SEO automatically pings Google with your sitemap.

If you didn’t submit your XML sitemap to Google Search Console yet, you’ll find a step-by-step guide to submitting your XML sitemap to GSC here.

3. Republish and share it again

Lastly, you can share the reindexed content in your newsletter, on social and other marketing channels. This way, you’ll generate some traffic and exposure, especially if other people start sharing it too. In case of a blog post, you can republish it on your blog. If it considers important pages of your site quickly thinking up a campaign and publishing new blog posts that link to the reindexed content could also help to get the initial traffic and rankings back.

It’s not the end of the world

Finding out you’ve accidentally set a post or even (parts of) your site might give you a big scare. But, fortunately, it’s not the end of the world and there are various things you can do to get it back in the search engines again. Depending on the size of the issue and the frequency your site gets crawled, it will take some time to recover, but eventually, it probably will.

Now, let’s hope I haven’t accidentally set anything to noindex when creating this post…

Read more: The ultimate guide to the meta robots tag »

The post Help, I’ve accidentally noindexed a post. What to do? appeared first on Yoast.

What’s technical SEO? 8 technical aspects everyone should know

An SEO Basics post about technical SEO might seem like a contradiction in terms. Nevertheless, some basic knowledge about the more technical side of SEO can mean the difference between a high ranking site and a site that doesn’t rank at all. Technical SEO isn’t easy, but here we’ll explain – in layman’s language – which aspects you should (ask your developer to) pay attention to when working on the technical foundation of your website.

What is technical SEO?

Technical SEO refers to improving the technical aspects of a website in order to increase the ranking of its pages in the search engines. Making a website faster, easier to crawl and understandable for search engines are the pillars of technical optimization. Technical SEO is part of on-page SEO, which focuses on improving elements on your website to get higher rankings. It’s the opposite of off-page SEO, which is about generating exposure for a website through other channels.

Why should you optimize your site technically?

Google and other search engines want to present their users with the best possible results for their query. Therefore, Google’s robots crawl and evaluate web pages on a multitude of factors. Some factors are based on the user’s experience, like how fast a page loads. Other factors help search engine robots grasp what your pages are about. This is what, amongst others, structured data does. So, by improving technical aspects you help search engines crawl and understand your site. If you do this well, you might be rewarded with higher rankings or even rich results.

It also works the other way around: if you make serious technical mistakes on your site, they can cost you. You wouldn’t be the first to block search engines entirely from crawling your site by accidentally adding a trailing slash in the wrong place in your robots.txt file.

But it’s a misconception you should focus on technical details of a website just to please search engines. A website should work well – be fast, clear and easy to use – for your users in the first place. Fortunately, creating a strong technical foundation often coincides with a better experience for both users and search engines.

What are the characteristics of a technically optimized website?

A technically sound website is fast for users and easy to crawl for search engine robots. A proper technical setup helps search engines to understand what a site is about and it prevents confusion caused by, for instance, duplicate content. Moreover, it doesn’t send visitors, nor search engines, into dead-end streets by non-working links. Here, we’ll shortly go into some important characteristics of a technically optimized website.

1. It’s fast

Nowadays, web pages need to load fast. People are impatient and don’t want to wait for a page to open. In 2016 already, research showed that 53% of mobile website visitors will leave if a webpage doesn’t open within three seconds. So if your website is slow, people get frustrated and move on to another website, and you’ll miss out on all that traffic.

Google knows slow web pages offer a less than optimal experience. Therefore they prefer web pages that load faster. So, a slow web page also ends up further down the search results than its faster equivalent, resulting in even less traffic.

Wondering if your website is fast enough? Read how to easily test your site speed. Most tests will also give you pointers on what to improve. We’ll guide you through common site speed optimization tips here.

2. It’s crawlable for search engines

Search engines use robots to crawl or spider your website. The robots follow links to discover content on your site. A great internal linking structure will make sure that they’ll understand what the most important content on your site is.

But there are more ways to guide robots. You can, for instance, block them from crawling certain content if you don’t want them to go there. You can also let them crawl a page, but tell them not to show this page in the search results or not to follow the links on that page.

Robots.txt file

You can give robots directions on your site by using the robots.txt file. It’s a powerful tool, which should be handled carefully. As we mentioned in the beginning, a small mistake might prevent robots from crawling (important parts of) your site. Sometimes, people unintentionally block their site’s CSS and JS files in the robot.txt file. These files contain code that tells browsers what your site should look like and how it works. If those files are blocked, search engines can’t find out if your site works properly.

All in all, we recommend to really dive into robots.txt if you want to learn how it works. Or, perhaps even better, let a developer handle it for you!

The meta robots tag

The robots meta tag is a piece of code that you won’t see on the page as a visitor. It’s in the source code in the so-called head section of a page. Robots read this section when finding a page. In it, they’ll find information about what they’ll find on the page or what they need to do with it.

If you want search engine robots to crawl a page, but to keep it out of the search results for some reason, you can tell them with the robots meta tag. With the robots meta tag, you can also instruct them to crawl a page, but not to follow the links on the page. With Yoast SEO it’s easy to noindex or nofollow a post or page. Learn for which pages you’d want to do that.

Read more: https://yoast.com/what-is-crawlability/

3. It doesn’t have (many) dead links

We’ve discussed that slow websites are frustrating. What might be even more annoying for visitors than a slow page, is landing on a page that doesn’t exist at all. If a link leads to a non-existing page on your site, people will encounter a 404 error page. There goes your carefully crafted user experience!

What’s more, search engines don’t like to find these error pages either. And, they tend to find even more dead links than visitors encounter because they follow every link they bump into, even if it’s hidden.

Unfortunately, most sites have (at least) some dead links, because a website is a continuous work in progress: people make things and break things. Fortunately, there are tools that can help you retrieve dead links on your site. Read about those tools and how to solve 404 errors.

To prevent unnecessary dead links, you should always redirect the URL of a page when you delete it or move it. Ideally, you’d redirect it to a page that replaces the old page. With Yoast SEO Premium, you can easily make redirects yourself. No need for a developer!

Read more: https://yoast.com/what-is-a-redirect/

4. It doesn’t confuse search engines with duplicate content

If you have the same content on multiple pages of your site – or even on other sites – search engines might get confused. Because, if these pages show the same content, which one should they rank highest? As a result, they might rank all pages with the same content lower.

Unfortunately, you might have a duplicate content issue without even knowing it. Because of technical reasons, different URLs can show the same content. For a visitor, this doesn’t make any difference, but for a search engine it does; it’ll see the same content on a different URL.

Luckily, there’s a technical solution to this issue. With the so-called, canonical link element you can indicate what the original page – or the page you’d like to rank in the search engines – is. In Yoast SEO you can easily set a canonical URL for a page. And, to make it easy for you, Yoast SEO adds self-referencing canonical links to all your pages. This will help prevent duplicate content issues that you’d might not even be aware of.

5. It’s secure

A technically optimized website is a secure website. Making your website safe for users to guarantee their privacy is a basic requirement nowadays. There are many things you can do to make your (WordPress) website secure, and one of the most crucial things is implementing HTTPS.

HTTPS makes sure that no-one can intercept the data that’s sent over between the browser and the site. So, for instance, if people log in to your site, their credentials are safe. You’ll need a so-called SSL certificate to implement HTTPS on your site. Google acknowledges the importance of security and therefore made HTTPS a ranking signal: secure websites rank higher than unsafe equivalents.

You can easily check if your website is HTTPS in most browsers. On the left hand side of the search bar of your browser, you’ll see a lock if it’s safe. If you see the words “not secure” you (or your developer) have some work to do!

Read more: SEO Basics: What is HTTPS?

6. Plus: it has structured data

Structured data helps search engines understand your website, content or even your business better. With structured data you can tell search engines, what kind of product you sell or which recipes you have on your site. Plus, it will give you the opportunity to provide all kinds of details about those products or recipes.

Because there’s a fixed format (described on Schema.org) in which you should provide this information, search engines can easily find and understand it. It helps them to place your content in a bigger picture. Here, you can read a story about how it works and how Yoast SEO helps you with that.

Implementing structured data can bring you more than just a better understanding by search engines. It also makes your content eligible for rich results; those shiny results with stars or details that stand out in the search results.

7. Plus: It has an XML sitemap

Simply put, an XML sitemap is a list of all pages of your site. It serves as a roadmap for search engines on your site. With it, you’ll make sure search engines won’t miss any important content on your site. The XML sitemap is often categorized in posts, pages, tags or other custom post types and includes the number of images and the last modified date for every page.

Ideally, a website doesn’t need an XML sitemap. If it has an internal linking structure which connects all content nicely, robots won’t need it. However, not all sites have a great structure, and having an XML sitemap won’t do any harm. So we’d always advise having an XML site map on your site.

8. Plus: International websites use hreflang

If your site targets more than one country or countries where the same language is spoken, search engines need a little help to understand which countries or language you’re trying to reach. If you help them, they can show people the right website for their area in the search results.

Hreflang tags help you do just that. You can define for a page which country and language it is meant for. This also solves a possible duplicate content problem: even if your US and UK site show the same content, Google will know it’s written for a different region.

Optimizing international websites is quite a specialism. If you’d like to learn how to make your international sites rank, we’d advise taking a look at our Multilingual SEO training.

Want to learn more about this?

So this is technical SEO in a nutshell. It’s quite a lot already, while we’ve only scratched the surface here. There’s so much more to tell about the technical side of SEO! If you want to take a deep-dive into technical SEO, we’d advise our Technical SEO training or Structured data training. With these courses, you’ll learn how to create a solid technical foundation for your own website.

PS You’re the ambitious type? Get both training courses together and save $59!

Read more: https://yoast.com/wordpress-seo/

The post What’s technical SEO? 8 technical aspects everyone should know appeared first on Yoast.

Indexing in Yoast SEO: Show x in search results?

Before a search engine can rank a page or a post, it needs to index it. A crawler must discover a piece of content before it can evaluate if it is a valuable addition to its index. One of the ways crawlers discover pages, is by crawling XML sitemaps. After a page has been indexed, a search engine can rank the piece of content if it fits the users search query best. Yoast SEO makes it easy for you to determine what should be indexable.

Show x in search results?

Determining what has to be indexed by crawlers and what not tends to be hard to understand and it’s easy to make a mistake. You wouldn’t be the first to have unknowingly set a whole post type to noindex, making it unavailable to search engines. We’ve thought long and hard about this and drastically simplified this process for you. Now it all boils down to asking you a straightforward question: Do you want x to appear in search engines?

Yoast SEO makes it easy for you to determine what appears in the search results

You can find the individual settings for making your content available for indexing in the corresponding parts of Yoast SEO. You’ll find the settings for posts and pages in the Content Types part of the Search Appearance tab. Taxonomies like categories and tags can be found in the Taxonomies tab.

By saying Yes to the ‘Show Posts in search results’ question in the post settings, for instance, you make sure that your posts will appear in the XML sitemap and, therefore, in the search results.

If you want to exclude something, you can switch this toggle to No, and the taxonomy or post type will not appear in the XML sitemap. Because of that, it will not appear in the search results. Whenever you set something here to not appear in search engines, it will be noindexed and kept from the XML sitemap. 

Read more: Why doesn’t Google index my content properly? »

Exclude individual posts

If you really don’t want a specific URL to show up in the search results you’ll need to add a noindex, follow tag. Leaving it out of your XML sitemap doesn’t automatically mean Google won’t index the URL. If Google can find it by following links, Google can index the URL.

You can also exclude individual posts from the XML sitemaps from the Yoast SEO meta box in the post editor. Click on the Advanced tab and select No to the ‘Allow search engines to show this Post in search results?’ question.

You can determine on a per-post basis if something should appear in the search results

View your XML sitemap

You should always check your sitemap to see if the content you want to include appears in the XML sitemap. While you’re there, you should also check if the content you want to exclude from the sitemap doesn’t appear in it.

You can find your XML sitemap by going to General > Features > XML Sitemaps > ? (click on the question mark).

Check your XML sitemap to see if everything is as it should be

We’ve taken away a lot of the confusion around indexing content and XML sitemaps by simplifying things. But, most importantly, it is now so much easier to determine what should and should not appear in search results.

More on XML sitemaps

XML sitemaps are a kind of treasure map for search engine robots. They crawl them to discover new or updated content on your site. Every site benefits from a sitemap. Your rankings won’t soar if you add one, but it does help the crawlers to discover your content that much easier. If you need more information about the use of XML sitemaps on your site, we have some further reading for you:

Keep reading: What is an XML sitemap and why should you have one? »

Read on: The sense and nonsense of an XML sitemap »

Keep on reading: XML sitemap insights for developers »

The post Indexing in Yoast SEO: Show x in search results? appeared first on Yoast.

What is an XML sitemap and why should you have one?

A good XML sitemap acts as a roadmap of your website that leads Google to all your important pages. XML sitemaps can be good for SEO, as they allow Google to quickly find your essential website pages, even if your internal linking isn’t perfect. This post explains what they are and how they help you rank better.

We hope you’ll enjoy reading #5 of our best-read posts this year! Find out all about XML sitemaps and why it’s important that you have one. Don’t forget to check back tomorrow for another holiday countdown surprise!

What are XML sitemaps?

You want Google to crawl every important page of your website, but sometimes, pages end up without any internal links pointing to them, making them hard to find. An XML sitemap lists a website’s important pages, making sure Google can find and crawl them all, also helping it understand your website structure:

XML sitemap Yoast
Yoast.com’s XML sitemap

Above is Yoast.com’s XML sitemap, created by the Yoast SEO plugin and later on we’ll explain how our plugin helps you create the best XML sitemaps. If you don’t use our plugin, your sitemap may look a little different but will work the same way.

As you can see, the Yoast.com XML sitemap shows several ‘index’ sitemaps: post-sitemap.xml, page-sitemap.xml, video-sitemap.xml etc. This categorization makes a site’s structure as clear as possible, so if you click on one of the index sitemaps, you’ll see all URLs in that particular sitemap. For example, if you click on post-sitemap.xml you’ll see all Yoast.com’s post URLs (click on the image to enlarge):

XML Post Sitemap Yoast
Yoast.com’s post XML sitemap

You’ll notice a date at the end of each line. This tells Google when each post was last updated and helps with SEO because you want Google to crawl your updated content as soon as possible. When a date changes in the XML sitemap, Google knows there is new content to crawl and index.

If you have a very large website, sometimes it’s necessary to split an index sitemap. A single XML sitemap is limited to 50,000 URLs, so if your website has more than 50,000 posts, for example, you’ll need two separate ones for the post URLs, effectively adding a second index sitemap. The Yoast SEO plugin sets the limit even lower – at 1.000 URLs – to keep your sitemap loading as fast as possible

What websites need an XML sitemap?

Google’s documentation says XML sitemaps are beneficial for “really large websites”, for “websites with large archives”, for “new websites with just a few external links to it” and for “websites which use rich media content”.

While we agree that these kinds of websites will definitely benefit the most from having one, at Yoast, we think XML sitemaps are beneficial for every website. Every single website needs Google to be able to easily find the most important pages and to know when they were last updated, which is why this feature is included in the Yoast SEO plugin.

Which pages should be in your XML sitemap?

How do you decide which pages to include in your XML sitemap? Always start by thinking of the relevance of a URL: when a visitor lands on a particular URL, is it a good result? Do you want visitors to land on that URL? If not, it probably shouldn’t be in it. However, if you really don’t want that URL to show up in the search results you’ll need to add a ‘noindex, follow’ tag. Leaving it out of your XML sitemap doesn’t mean Google won’t index the URL. If Google can find it by following links, Google can index the URL.

Example 1: A new blog

Say, for example, you are starting a new blog. You will want Google to find new posts quickly to make sure your target audience can find your blog in the search results, so it’s a good idea to create an XML sitemap right from the start. You might create a handful of first posts and categories for them as well as some tags to start with. But there won’t be enough content yet to fill the tag overview pages, making them “thin content” that’s not valuable to visitors – yet. In this case, you should leave the tag’s URLs out of the sitemap for now. Set the tag pages to ‘noindex, follow’ because you don’t want people to find them in search results.

Example 2: Media and images

The ‘media’ or ‘image’ XML sitemap is also unnecessary for most websites. This is because your images are probably used within your pages and posts, so will already be included in your ‘post’ or ‘page’ sitemap. So having a separate ‘media’ or ‘image’ sitemap would be pointless and we recommend leaving it out. The only exception to this is if images are your main business. Photographers, for example, will probably want to show a separate ‘media’ or ‘image’ XML sitemap to Google.

How to make Google find your sitemap

If you want Google to find your XML sitemap quicker, you’ll need to add it to your Google Search Console account. In the ‘Sitemaps’ section, you’ll immediately see if your XML sitemap is already added. If not, you can add your sitemap at top of the page:

Add sitemap index to Search Console
Yoast.com’s XML sitemap added to Google Search Console

As you can see in the image, adding your XML sitemap can be helpful to check whether all pages in your sitemap really have been indexed by Google. If there is a big difference in the ‘submitted’ and ‘indexed’ number on a particular sitemap, we recommend looking into this further. There could be an error preventing some pages from being indexed or maybe you need more content or links pointing to the content that’s not been indexed yet.

Yoast SEO and XML sitemaps

Because they are so important for your SEO, we’ve added the ability to create your own XML sitemaps in our Yoast SEO plugin. They are available in both the free and premium versions of the plugin.

Yoast SEO creates an XML sitemap for your website automatically. Click on ‘SEO’ in the sidebar of your WordPress install and then select the ‘Features’ tab:

XML sitemaps in Yoast SEO
XML Sitemaps

In this screen, you can enable or disable the different XML sitemaps for your website. Also, you can click on the question mark to expand the information and see more possibilities, like checking your sitemap in your browser:

You can exclude content types from your XML sitemap in the ‘Search Appearance’ tab. If you select ‘no’ as an answer to ‘show X in the search results?’ then this type of content won’t be included in it.

Read more about excluding content types here.

Check your own XML sitemap!

Now, you know how important it is to have an XML sitemap because having one can really help your site’s SEO. Google can easily access your most important pages and posts if you add the right URLs to it. Google will also be able to find updated content easily, so they know when a URL needs to be crawled again. Lastly, adding your XML sitemap to Google Search Console helps Google find your sitemap fast and it allows you to check for sitemap errors.

Now go check your own XML sitemap and make sure you’re doing it right!

The post What is an XML sitemap and why should you have one? appeared first on Yoast.

The sense and nonsense of XML sitemaps

Fact: if your website is set up the right way, you shouldn’t need an XML sitemap at all. You shouldn’t need to think about your category’s XML sitemaps or about including images in your post’s XML sitemap. But why do we keep talking about them like it’s the most important thing ever for SEO? It’s an almost daily subject in our support. That might be, because it’s a convenient list of all the pages on your website. It makes sense that Google is able to crawl all pages of your website if you list them on a page, right?

Google is almost human

Over the last years, we have been talking a lot about Google becoming more ‘human’, so to say. Google is quite good at mimicking the user’s behavior on a website and uses this knowledge in their ranking methods. If your website is user-friendly and gives users the answers they were looking for in Google, chances are your website will do well in the search result pages.

Structure is a sitemap within your website

In the process of setting up your website, you should look at the keywords you’d like to address and translate that to a proper site structure. Using, for instance, the internal linking tool in our Yoast SEO plugin, you are able to create structured links to all the pages of your website. That simply means that Google is able to follow all links and find all pages. That means you have set up a great infrastructure within your website for search engines.

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO for WordPress pluginBuy now » Info

But why should I use XML sitemaps in that case?

Sitemaps, both HTML and XML, come in handy when your site structure and internal linking structure really aren’t that good, to be honest. When you’re dealing with a huge, messy inheritance of the previous owner, years of writing (more or less unstructured) content, or if you simply haven’t thought about internal links that much, your XML sitemap is probably a life saver.

In addition to pointing Google to all your content, XML sitemaps can also optimize crawling of your website by a search engine bot. XML sitemaps should include the last modified date. This date will immediately tell a search engine which pages should be crawled and which haven’t changed since the last crawl and can be excluded from this crawl. This is a huge benefit of using XML sitemaps.

Analyzing your XML sitemaps

In Google Search Console’s Sitemap section, you can find errors in your sitemaps. Pages that are linked there, but don’t work. There’s a huge list of possible errors in the Google Search Console Help documentation.

Apart from that, an important thing to analyze is the types of XML sitemaps you have. You can find these in Google Search Console and in the SEO › XML Sitemaps section in our plugin. In WordPress, sitemaps are created for post types and taxonomies, where sometimes you just don’t need an XML sitemap for all of these. In Yoast SEO Care, we find websites that have XML sitemaps for filter types (in eCommerce shops for instance), or for dimensions and things like that. If these ‘pages’ don’t make sense for the user, by all means, disable that XML sitemap in our Yoast SEO plugin. Only serve sitemaps that matter.

There is a reason Google included an XML sitemap section in Google Search Console. Google likes to know every page of your website. They want to see everything, to see if it contains interesting information to answer their user’s search queries. Your XML sitemap is like a roadmap to all the different POI’s on your site, to all the tourist attractions. And yes, some are more interesting than others. Last year, the XML sitemaps served by our plugin contained a priority percentage. Heavy users of our plugin sometimes requested an option to alter that percentage and we never got to that. We decided to remove the percentage altogether as it just did not work as intended – on Google’s side. That emphasizes even more, that it’s just a list of pages. A convenient list, nevertheless.

Should every website have an XML sitemap?

Perhaps I have already answered this question. Yes, I think every website should have an XML sitemap. Or multiple XML sitemaps to provide a lot of links in a better format. It’s a way to make sure search engines find every page on your website, no matter how much of a mess you make of your website. But you should really put your best effort in making that XML sitemap an extra and not a necessity.

If the crawlability of your website depends on your XML sitemap, you have a much larger problem on your hands. I really do think so. Hopefully, you can still go back to the drawing board, invest a bit in a good keyword research training. Restructure the site. Use our internal linking tool when going over your most visited pages again and insert the right links. And then, when most of your pages can be reached via your website itself, rely on that nice, comforting XML sitemap to serve Google any forgotten leftovers and help you to further optimize the crawling of your website.

Read on: ‘WordPress SEO tutorial: definite guide to higher ranking’ »

Ask Yoast: include taxonomies in XML sitemap?

Your tag and category pages could be very important pages on your site. If you’re using these pages in the right way, they can boost your site’s SEO. You might wonder if you should also include these taxonomy pages in your XML sitemap. In this Ask Yoast we explain whether you should, and why!

Our support team received the following question from a Yoast SEO user:

“I just installed Yoast SEO. Should I include taxonomies, like tags and categories in my XML sitemap?”

Check out the video or read the answer below!

Get the most out of Yoast SEO, learn every feature and best practice in our Yoast SEO for WordPress training!

$ 129 – Start learning today »

Yoast SEO for WordPress training

Include taxonomies in XML sitemap

Read this transcript to learn how you’ll benefit from including taxonomies in your XML sitemap:

“Well, of course you should, that’s what we do by default. We do that for a reason. Your tags and your categories could be very important pages on your site.
We have a post on yoast.com about category SEO, that you should really have a look at. We think that if you do your job well, then your category pages could rank for terms that your posts might never rank for. Because every post that you write in a specific category or tag, links back to that category or that tag. So those pages become very powerful. And if you optimize them well, they can very well become pages that rank in the search results. Good luck!”

Ask Yoast

In the series Ask Yoast we help you with your SEO question! Did you get stuck on an SEO issue? Are you in doubt between two SEO strategies? Don’t fret, just ask us your question! You can send your question to email hidden; JavaScript is required.

Read more: ‘ Using category and tag pages for your site’s SEO’ »

Yoast SEO 3.5

We’ve just pushed out a new release of Yoast SEO, our flagship plugin. The new version, 3.5, mostly has a metric ton of small bugfixes. In this post, we’ll discuss the most notable changes, but you should mostly be aware that this is what we would call a bugfix release.

XML Sitemap changes

We’ve decided to remove the change frequency and priority variables from our XML Sitemaps. Google has said publicly that they don’t use them on most sites. Private discussions with Googlers have led us to believe there’s really no compelling reason to keep them around. This change makes XML sitemaps faster and easier to generate.

There are plugins out there that allow you to change the priority and change frequency of just about everything. We really do not believe that adds any value whatsoever.

Flesch reading ease for German and Dutch

We’ve added Flesch reading ease compatibility for German and Dutch. This doesn’t just mean we’ve enabled it for those languages. The Flesch reading ease test needs to change because each language is different. If you’re interested in the technicalities, this ticket has the formulas.

To be able to do this, we need to be able to recognize syllables in words. We’re working on adding this for more languages, but as you can imagine that’s a fair bit of work. This comes on top of the changes in 3.4, where we added support for transition words checks for German as well. This means German and Dutch writers will now get a score that is meaningful for their language.

Add @id to schema.org output

Yoast SEO outputs JSON+LD metadata on the page. This metadata informs Google about whether this is a site for a person or a company, what your social profiles are, etc.

This change, which is admittedly a bit more on the technical side allows other plugins to tie into our metadata. They can output pieces of JSON+LD metadata and combine them with ours. For instance saying “this is extra info related to this organization”, where “this organization” is a pointer for our organization info. They can do this by referencing the @id‘s we’ve added, something that wasn’t possible before this change.

i18n and a11y improvements

We’ve, again, made many internationalization (i18n) and accessibility (a11y) improvements in this release. From making sure everything is translatable to adding descriptions and adding more explanation everywhere.

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium

Buy this plugin now

Yoast SEO for WordPress plugin

A new Premium page

For the last few years, Yoast SEO has had an extensions page. We’ve now replaced that extensions page with a Go Premium page. It explains the features of Yoast SEO Premium and our extensions a lot better. It’s also much more in line with our current style, as you can see:

Screen Shot 2016-09-06 at 15.18.34

Full changelog

The full changelog is available on WordPress.org, if you see anything in there that you’d like clarification on, let us know in the comments!

Weekly SEO recap: Let’s disambiguate!

This week we had a few interesting things happening, from a potential update to some new features and an important change in Google Suggest. Let’s dive in!

Joost's weekly SEO Recap

Google adds disambiguation to Suggest

In a very interesting move this week, Google added disambiguation results to their suggest box, basically forcing you to choose for a specific topic when you’re searching. A search for [mercury] for instance shows this:

Google disambiguation results for Mercury

The 3 disambiguation results are marked with a long dash. Clicking on either 3 shows a knowledge graph panel on the right hand side, and, interestingly, none of the 3 results has Wikipedia as the number 1 result.

The disambiguation page for Mercury on Wikipedia has different options, so Google is at least not using that as its only source. A search for [ocampo] has 3 options, all of which do have a Wikipedia result as #1:

Disambiguation result for Ocampo

One more example, a search for [yahudi], delivers this:

yahudi disambiguation result

Whether you’re looking for the Jewish people or a film… Quite the difference of course in results. This makes it easier for Google to determine what it is you’re searching for, but it also makes it clear that you should optimize enough for Google to understand not just your keyword, but to understand your topic, which SearchEngineLand had a nice post about as well. This has profound implications for how you write articles, and for how you do keyword research.

Google to prefer HTTPS versions of URLs

Google has been giving a slight ranking boost to HTTPS URLs for a while now, at least, so they tell us. But now they’re going even further. When there is a working HTTPS version of your site, regardless of whether you link to it or not, Google will prefer the HTTPS URLs.

Google search update on the 16th?

Barry reported an update on the 16th, that sounded more to me like something had gone wrong somewhere. This sort of stuff happens all the time, and it seems in the comments that people were bouncing back after a few days. To me it’s always interesting to see when stuff goes wrong so badly.

Why having multiple XML Sitemaps can be useful

In an article on the SEM Post, they’re discussing whether you should have multiple or one single XML sitemap and what’d be the benefit of both. The most important thing John Mueller from Google says, in my opinion, is this:

I like to split up the sitemap file because in the Search Console you can look at the index stats by sitemap file and that sometimes makes it a little bit easier to understand what types of pages are currently being indexed and what pages aren’t being indexed.

That’s the exact reason why our Yoast SEO plugin has an XML Sitemap feature that creates a single index with XML sitemaps per post type. This makes it a lot more interesting to look at the XML sitemaps section of Google Search Console.

That’s it, see you next week!

joost signature

PS If you find disambiguation a nasty word, you’re not alone. I must have mistyped that about a dozen times while writing this post 🙂