Imagine visiting an online shop, and receiving a notification from your browser that this site is not secure. You’re probably not so eager to buy something there and provide this website with private or sensitive information. Well, both your visitors and Google most likely agree. In 2014, Google announced that HTTPS would become a ranking signal. So it’s definitely something to take into account as a website owner. But getting an SSL certificate and moving your website to HTTPS can be difficult, as we at Yoast know all too well. If you don’t do it right, it can have a negative impact on your rankings. But what about the other way around? In this Ask Yoast, I discuss what to do if you suspect your rankings have lowered because you don’t have an SSL layer.

Become a technical SEO expert with our Technical SEO 1 training! »

Technical SEO 1 training Info

Larry Launstein Jr. emailed us a question on this subject.

How do you optimize a site that has been lowered in the rankings because of lack of an SSL layer? Is there a workaround
for this, or does it need to get an SSL layer no matter what?

Watch the video or read the transcript further down the page for my answer!

Lower rankings without HTTPS?

“So what can you do to optimize a site that has lost rankings, because of lack of an SSL layer or basically because of lack of HTTPS? Well, if all of your competitors have HTTPS and you don’t, at some point you might start losing some rankings because of that.

But I honestly do not believe that that is your only ranking problem if you have a ranking problem. But does every website need HTTPS in the future? Yes. Should you thus be doing that right now if you’re working on your site? Yes, by all means.

If you do migrate, though, make sure that all the old URLs redirect properly to the new ones and, basically, get an SSL certificate and thus HTTPS for your entire site as soon as possible. Good luck!”

Ask Yoast

In the series Ask Yoast we answer SEO questions from our readers. Have an SEO-related question? Let us help you out! Send an email to ask@yoast.com.
(note: please check our knowledge base first, the answer to your question may already be there! For urgent questions, for example about our plugin not working properly, we’d like to refer you to our support page.)

Read more: ‘Ask Yoast: redirecting your site to non-www and HTTPS’ »

The post Ask Yoast: Lower rankings without HTTPS? appeared first on Yoast.

If you make a website from scratch, you need to take a few SEO related things into account. It’s incredibly important to do this right from the start, as that will prevent a vast number of future headaches. Things like speed optimization and the right use of heading tags help to improve your website for both your visitors and Google. Now, I am sure you have covered the technical basics we described in the first part of this article. Just be aware of what you are doing, add focus and you’ll be fine. The greatest challenge begins when you start adding content to your website.

In this article, we’ll go over a number of steps everyone who makes a website should take when optimizing content.

Learn how to write awesome and SEO friendly articles in our SEO Copywriting training »

SEO copywriting training Info

Filling the website with content

You can optimize the entire technical side of your site and still find it lost on page two or more in Google. SEO isn’t a trick. It isn’t something your web developer can do for you. It’s something you can get guidance in, for instance with our SEO courses, or in our Yoast SEO (Premium) plugin. But first and foremost, SEO is Seriously Effortful Optimization. A continuous process, and something you, as a website owner, should make a strategy for. If you make a website, be prepared to write valuable content about the topic/purpose of your website. And that process starts with a bit of research.

Keyword research

Speaking from experience, I have often seen product manufacturers describe a product from their point of view. Let me give you an example: our Yoast SEO Premium plugin has an internal linking feature, which analyzes what posts on your website best match the content you are writing in your new post. You can copy that link from the WordPress sidebar and paste it into your post, to optimize your site structure. How awesome is that? Well, it might not sound so awesome to the user. They’re probably wondering what exactly they’re gaining with this feature. From a developer point of view, the description matches the feature. But for the user, the description should be:

Our internal linking tool allows you to create valuable links to all pages of your website, which will help these pages to rank in Google.

And even that might be a bit technical. In your keyword research you should focus on customer lingo first: how do people call your product? Find the right keywords and start writing. Want to learn more about keyword research? Take our SEO copywriting course for more insights.

Setting up the menu and site structure

In that same SEO copywriting training, we continue the SEO process for your website with the next step: site structure. You can even take our site structure course for more on this subject. Don’t underestimate the importance of a good structure. It’s the foundation of you visible site. If you set up a proper site structure right when you start a website, you’ll make optimizing your content so much easier.

As Marieke put it:

Your users need the structure to navigate through your site, to click from one page to the other. And Google uses the structure of your site in order to determine what content is important and what content is less important.

That’s a quote from our ultimate guide to site structure. It’s as simple as that. Optimizing the site structure influences SEO, UX, crawling of your pages and, let’s not forget, the right structure makes maintenance so much easier.

Good site structure will also help highlight the most important pages for your users. Include these pages in your (main, sub or footer) menu. As with customer lingo, use the data collected from Google Analytics to find the pages your visitors like most, and use these in your menu. Read more about optimizing your menu here.

Learn how to write engaging copy and how to organize it well on your site: Combine our SEO copywriting and Site structure training. »

Content SEO training bundle Info

One page per topic

Keyword research is done, the structure is set up, so now we can start writing. And we’ll create one page per topic. Need more pages to elaborate on a topic? Then feel free to do so, but use a new, long tail keyword for every new page.

Using one focus keyword per page, you force yourself to make a hierarchy in your collection of keywords, something you already started when doing your keyword research.

Small step back to keyword research

Now I see you thinking about this focus keyword and the analyses in our plugin during your writing process. You know that feeling when you check keyword density and wonder what to do with synonyms? The first thing to do here is to check Google Trends. Let’s say we optimize for (just an example!) “create a website”. A synonym is “start a website”. Google Trends tells us the main keyword should be “create a website”, see graph here:

Make a website - Google Trends

But we use “start a website” a lot ourselves and want to include that phrase in our post as well. In the Yoast SEO Premium plugin, I can simply insert a second focus keyword. Overall keyword density should be right in that case (check both keyword tabs), as we know Google treats them as the same keyword: If I do a search for “start a website”, “create a website” is also a highlighted (bold) keyphrase in the results. “Make a website” is also a valid synonym. Just a small peek into the way I approach this myself.

Title tags

Using the focus keyword we mentioned earlier, creating a great title for your page is easy. Google still values that title highly, so put some effort in improving it. We usually use the title as the main article heading (or the H1 we mentioned in part 1 of this post). Besides that, it’s used as the most important part of the actual <title> tag, the tag that also defines your title in Google’s search result pages. This tag is not only visible when someone shares your post on Facebook or Twitter, it’s also used when someone bookmarks your site.

Optimize the title page according to these guidelines » 

Optimize your content

Learn how to write engaging copy and how to organize it well on your site: Combine our SEO copywriting and Site structure training. »

Content SEO training bundle Info
Here we go: use Yoast SEO to optimize your page’s content. I’m just going to tell you that again. And if you don’t use WordPress, that’s not even an excuse anymore. We have Yoast SEO for Magento 2 and Yoast SEO for TYPO3 these days (created with our partner MaxServ).

Our SEO analysis will tell you, every time you write a post or page, what can be improved on your content:

Yoast SEO analysis

Read our SEO blog, as there’s a ton of free information about content optimization in there. And subscribe to our newsletter to keep your knowledge up to date.

How about meta descriptions?

I felt the need to at least mention meta descriptions here. Most SEO plugins, plans or whatever, mention meta descriptions as a must-do. I agree to a certain level. If you can write a nice, optimized ‘invitation’ to your website, you should most definitely add a meta description to your product pages. Usually, there is so much unrelated content on a product page (dimensions, manufacturer info, terms of service, etc.) that there’s always a risk that Google creates a meta description for that product page that doesn’t give the right information. So it makes sense to serve your own.

On regular pages and posts, Google will most probably grab a piece of related content, including the keyword used in the search query. That makes a lot of sense for news sites, for example. Still, I recommend adding a meta description to all your important pages. Facebook will use it as a description as well. And usually, when you are setting up a page on your website, it’s not that much work to add it, right?

Go make a website!

So, with this second part about content optimization, we have covered the very basics of SEO related things you should take into account when you make a website. I am sure you can come up with more insights, more tools or any other help for people that want to create a new website. Please feel free to add these in the comment section below this post, our readers (and I) will appreciate that!

Now go start that website!

Read on: ‘The ultimate guide to SEO copywriting’ »

 

The post Create a website with SEO in mind: Content optimization appeared first on Yoast.

After a lot of its and buts, you have finally decided to create a website. A personal website for yourself, or that long overdue website for your business. You know you have to think about design and should supply well-written texts. You’ve already been inquiring about that thing called hosting. You may even have called someone that can build your website for you. All in all, you’re pretty confident that you can now start a website without any problems. But wait. Have you thought about this little thing called SEO?

Become a technical SEO expert with our Technical SEO 1 training! »

Technical SEO 1 training Info

Create your website with SEO in mind

A large part of the search engine optimization process starts with focus: what is your website about? You have to focus on what we sometimes call ‘top tasks’. It’s a term used in mobile UX but it most certainly also goes for that moment when you decide to create a website from scratch. What is the purpose of a visitor coming to your website? And how can we make the journey of that visitor a pleasant journey?

When we discuss SEO with people that want to create a website, we focus on two main areas:

  • The technical side of things
  • Filling the website with content

In this post, I’ll highlight a number of important technical issues. In a follow-up post, I will go into the content side of things.

The technical side of things

With WordPress, it’s easy to build a website yourself. But a lot of you have probably hired a web agency to construct your website for you. That doesn’t mean you can sit back and wait for them to finish. There are a lot of things you can check and optimize yourself.

Speed optimization

One thing you want to pay attention to is the speed of your future website. You can easily check that on websites like Google PageSpeed Insights, or Pingdom. In an ideal world, your web agency has already tested things and your own check of your site’s speed will result in nothing but greens and great ratings. If not, these tools will tell you exactly where you can improve. That could be an image of several MBs that slows down the loading of a page, or the loading of an excessive number of JavaScript files, just to name a few.

If you want to learn more about speed optimization, please read Site Speed: tools and suggestions. And make sure your new website is as fast as possible.

URL Structure

Since you are setting up a new site, you still have full control over your URL structure. In most cases, focus is your friend here. Including dates in blog URLs that aren’t related to dates is my favorite example of what not to do. You just don’t need a date in there, unless you are a news website and that date does matter.

For shop websites, focus the URL on your product. Do people use SKUs to find your products? Include one in the URL. If not, please leave them out. Ground-rule: strip your URLs from anything that’s unnecessary. And if we’re honest: /blog/ isn’t a useful addition to your URL, neither is /shop/.

A small remark about the length of your URL: if you use focus, your URL will never need to be too long. I’m not against long URLs, especially since Google seems to leave them out of the search result pages in a growing number of cases. But a shorter, logical URL is easier to remember. And easier to share offline, for that matter.

Heading tags

One of my favorite subjects: heading tags. HTML5 allows for one H1 per block element, am I right? I still recommend against that. If you use one H1 and one H1 only, you need to make very clear for yourself what the subject or focus keyword of that page is. By restricting yourself to that one H1, you most definitely will add focus to that page. It’ll help you to properly optimize – read more about that further down in this post.

It’s simple:

Make it responsive

The mobile version of your website is equally important, if not more important than the desktop version. Mobile-first, they say. Fact is, that your website probably has as many mobile visitors as it has desktop visitors, of course depending on the type of site you have. I think, therefore, that a responsive site should be the default for every website that has been built in 2010 or later. We all use our mobile devices to browse the web, and your website should be ready for that.

If your web developer tells you that the website is accessible from a mobile device, don’t just trust him/her. Go over your mobile website yourself and check if you, as a visitor, can do all you want and need to do there. I already mentioned our article on mobile UX; use that as a reference when testing your mobile site yourself.

Read more: ‘Mobile SEO: the ultimate guide’ »

Local optimization

We have written a lot about structured data. Using structured data, you can serve Google your address details in the most convenient way. With for instance JSON, or using our Local SEO for WordPress plugin, you can insert a snippet that will help Google to fix your website/business to a location. This information is used for local searches, but will also end up in Google’s Knowledge Graph:

Apple NY Knowledge Graph

So if your business or website is related to an actual location, be sure to optimize for that part of your site right from the start as well. And definitely add your LocalBusiness data right now, if you haven’t done that already :)

Track your traffic

Let’s not forget this one. I have seen my share of websites where the owner told me that conversion was low or that nobody filled out their contact form. But the owner had no idea how many people got to his/her website. No idea what the main landing pages or exit pages were.

If you are serious about your website, at least install Google Analytics or any other preferred statistics app. Collect data about your visitors, and find out what the customer journey on your website is. Find out what pages people like and which pages they dislike. If you want to know more about Google Analytics, please visit our Google Analytics archives for related posts. When you start a website, don’t wait too long before adding Google Analytics, so you can see your traffic grow from day one.

Get your technical aspects right

If you have covered the technical issues of a new website, you’ll have properly prepared your site for all the great content you’ll be adding. Adding content is the next big step in building an awesome website! We’ll deal with how to approach that in a follow-up post tomorrow.

Keep reading: ‘WordPress SEO: The definitive guide to higher rankings for WordPress sites’ »

The post Create a website with SEO in mind: Technical pointers appeared first on Yoast.

WordPress 4.9 Beta 3 is now available!

This software is still in development, so we don’t recommend you run it on a production site. Consider setting up a test site just to play with the new version. To test WordPress 4.9, try the WordPress Beta Tester plugin (you’ll want “bleeding edge nightlies”). Or you can download the beta here (zip).

For more information on what’s new in 4.9, check out the Beta 1 blog post. Since the Beta 1 release, we’ve made 70 changes in Beta 2 and 92 changes in Beta 3. A few of these newest changes to take note of in particular:

  • The plugin/theme editors now show files in a scrollable expandable tree list. See #24048.
  • Backwards compatibility has been improved for MediaElement.js, which is upgraded from 2.2 to 4.2. See #42189.
  • When you create post stubs in the Customizer (such as for nav menu items, for the homepage or the posts page), if you then schedule your customized changes or save them as a draft, then these Customizer-created posts will appear in the admin as “Customization Drafts”; these drafts can be edited before your customized changes are published, at which time these posts (or pages) will also be automatically published. See #42220.
  • Theme browsing and installation experience in the Customizer has seen some bugfixes (e.g. #42215 and #42212), with some known remaining issues outstanding in Safari.
  • There is now a callout on the dashboard to install and activate Gutenberg. See #41316.
  • Menus in the Customizer have seen additional usability improvements. See #36279 and #42114.

Do you speak a language other than English? Help us translate WordPress into more than 100 languages!

If you think you’ve found a bug, you can post to the Alpha/Beta area in the support forums. We’d love to hear from you! If you’re comfortable writing a reproducible bug report, file one on WordPress Trac, where you can also find a list of known bugs.

Many refinements
Exist within this release;
Can you find them all?

HTTPS secures the connection to the website you are visiting. I’m sure you have seen this in action; look at the address bar in the browser and find the lock icon on the left-hand side. Is the lock closed? Then the connection is secure. Is it open or is there another type of icon or message? Then it’s not secure and vulnerable to attack. Using a site over a non-secure connection means hackers/criminals could intercept the data you send to the site, like your password and email address. Here, I’ll explain what HTTPS is and why it plays a role in SEO.

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO: the #1 WordPress SEO plugin Info

HTTP vs HTTPS

When you type in a URL in the search bar, your browser asks the site for its IP address – for instance 123.456.7.89. This number is the actual address that a site gets online. The browser connects to this number in the hopes this is the correct site. This is all done in plain sight and there is no encryption to be seen, so everyone can intercept this traffic. So when you want to log in to a site that you connect to via an HTTP connection, the data you enter – username and password – is sent in plain text. Trust me, that’s really bad. Think about what would happen if you’d connect to your bank this way.

HTTPS secures this process. HTTPS encrypts the connection between the browser and the site, therefore making sure that no one can intercept the data sent between those two. Every site that wants to secure itself needs a so-called SSL certificate.  The browser checks the certificate of the site and verifies its legitimacy with the company that issued it. If you want to see who issued the certificate, please click on the lock icon. By using HTTPS, sites not only secure your login procedure and personal data but also what you do on a site and which sites you visit.

Besides securing the web, HTTPS is necessary for sites that want to upgrade to a new, safer and much faster internet protocol called HTTP/2. HTTP/2 includes different new technologies that make sites a lot faster to load.

yoast ssl certificate

Value of HTTPS for the user

Everyone has the right to privacy on the web. We are doing so many mission-critical things on the web these days that we can use any kind of security we can get. An ever-increasing number of websites is making the move to HTTPS. In the screenshot below, you can see that at the moment, 61% of the sites that Firefox loads are being sent over HTTPS (stats by Let’s Encrypt). HTTPS is a must for any type of site, even if you own the bakery around the corner and don’t send or request sensitive data via your website.

lets encrypt https usage

Value of HTTPS for SEO

In 2014, Google announced that HTTPS would become a ranking signal. Today, your rankings will hardly change when you activate HTTPS. But it’s not just about rankings as much as it is about user experience and gaining trust with your future customers. It’s inevitable that we are moving to an all-HTTPS web. It is, therefore, incredibly important that your site makes the switch to HTTPS in the coming year.

Several browsers now show ‘not secure’ messages when your site doesn’t have an HTTPS connection or when you try to send data via HTTP on your HTTPS site. Don’t forget, it’s easy to scare off visitors! Wouldn’t you switch over to the site of a competitor when you’d see something like the ‘not secure’ message in the screenshot below?

https not secure message

Make the switch to HTTPS

A few years ago, switching to HTTPS was a major undertaking. Some big sites waited years to do it because it came with several challenges, like speed issues and the cost/benefit issue. These days, while still not easy, it’s manageable. If you’re planning to make the switch to HTTPS, be sure to make a checklist so you don’t forget anything during the process.

Joost shared some advice in a recent Ask Yoast video on moving to HTTPS:

Forcing HTTPS is something that you need to test really well. There are all sorts of things in your site that probably aren’t HTTPS ready that you should know of upfront. I know it was a lot of hard work to get yoast.com to HTTPS and we don’t even have ads. Especially ad services can be really tough to get working on HTTPS.

The Let’s Encrypt project issues free certificates to anyone wanting to secure their site. Several web hosts even offer free Let’s Encrypt services that make the installation of a certificate as easy as pie. That is, however, only one piece of the puzzle. On Google’s Secure your site with HTTPS site you can find more information on best practices. We’ve written a guide on moving your site from HTTP to HTTPS. If this stuff scares you it is best to hire an expert!

Read more: ‘Moving your website to HTTPS: tips & tricks’ »

The post SEO basics: What is HTTPS? appeared first on Yoast.

With only a few weeks left until YoastCon 2017, it’s time we introduced another of our amazing speakers. Laura Crimmons is Communications Director at Branded3, an award-winning SEO and digital marketing agency in the UK. Laura herself also has an admirable amount of achievements and awards under her belt, for example being named PR Moment’s Young Professional of the Year 2017. At YoastCon, she’s going to talk about link building in a successful online campaign, and successfully structuring a link building campaign. We asked her a few questions about links and link building to give you a little preview!

Don’t miss the opportunity to see Laura in action! Get your ticket now for YoastCon 2017!
Tickets

Come see Laura Crimmons at YoastCon 2017 on November 2 »banner YoastCon

Tell us a bit about yourself and your background. How did you end up at Branded3? And what is the accomplishment you are most proud of while working at that agency?

My background prior to joining Branded3 was in PR; I did a PR degree and had some experience in more traditional PR agencies and in-house roles but it was always digital that appealed to me, so I decided to join Branded3. I joined about a week after manual penalties and Penguin first rolled out, so it was at a time when the agency (and the SEO industry as a whole) was trying to find its feet, with how to build links now that the old ways were (rightfully in many cases) being penalized. Thankfully PR seemed to be part of that solution.

You focus a lot on link building for larger clients. Link building, of course, is a science in itself. Could you share your tactics for starting a – hopefully – successful link building project?

The starting point always has to be the audience, and plenty of research. You need to understand:

  • Who is my audience?
  • Where do they hang out online?
  • What are they interested in?

From here you’re able to start brainstorming ideas that will engage the audience. At this point you should also have started to develop a list of sites that will be your targets for link acquisition.

Links are still incredibly important, even in this day and age. Anyone is looking for high-value links from relevant sites in their industry. What are your favorite tips for getting these kinds of quality links?

We use PR as a way to generate these kinds of links i.e. working with journalists who usually work for higher quality sites (publishers) than say bloggers who would generally have lower quality domains.

That said, there are lots of other high quality sites that you can attract links from without PR, for example by looking at any genuine resource sites in your industry that link to competitors but not you.

Every site-owner needs to gather links and local business owners would probably benefit even more for good links. Could you explain the impact of link building for local SEO?

Link building is important for local SEO in the same sense that it is for any SEO, however, when specifically looking at local SEO we place more emphasis on citations, data accuracy and proximity.

Do you see the importance of links changing anytime soon?

We all know that search engines have been trying for years to decrease their reliance on links as a ranking factor. But they haven’t got there yet and I don’t necessarily see that happening in the next year or so.

Even if they do manage to find a way to determine a site’s authority better than links, I still think the practice of Digital PR/Content Marketing that we do now for link acquisition will remain important, as it goes beyond just acquiring links. It’s about building brand awareness, affinity and ultimately does play a part in assisted conversions.

Why shouldn’t people miss your talk at YoastCon?

I’ve spent the last five and a half years working in link acquisition and have had a lot of success over that time gaining links from some of the biggest publishers in the UK and globally. So anyone that wants to up their game or learn some tips would probably take something away from it.

Get your ticket for YoastCon 2017 now!
Tickets

Read more: ‘YoastCon 2017: Practical SEO’ »

The post Laura Crimmons on link building appeared first on Yoast.

There are several kinds of pages that you’ll expect to find on most websites: a home page, a contact page or an about page, for example. In this Ask Yoast, I’m going to discuss another type of page: the privacy page. This is a page where you put your privacy policy, which allows your visitors to check, for instance, how their information is handled. Not every website will need a such a page, but what about privacy pages and SEO? Is there a benefit to having a privacy page on your site? Read on to find out!

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO: the #1 WordPress SEO plugin Info

Derek Little sent us an email with the following question:

I’ve heard that having (or not having ) a privacy page is or was a big factor in SEO, and important to Google. Is this still the case?

Watch the video or read the transcript further down the page!

Privacy pages and SEO?

So, does having or not having a privacy page on your site have an impact on SEO?

“Well, as far as I know, this was only ever a factor for the AdWords quality score, not for SEO itself. So, it was important when you were advertising in AdWords, and not in SEO.  

At the same time, having a privacy page on your site makes you look all that more professional, which can help, of course, if you’re selling something. So, I would say, make it, think about what you put on there, think about how you deal with the privacy of your visitors and customers. Put that on there, and show that to the world: that’s always a good thing.

Good luck!”   

Ask Yoast

In the series Ask Yoast we answer SEO questions from our readers. Have an SEO-related question? Let us help you out! Send an email to ask@yoast.com.
(note: please check our knowledge base first, the answer to your question may already be there! For urgent questions, for example about our plugin not working properly, we’d like to refer you to our support page.)

Read more: ‘Holistic SEO’ »

The post Ask Yoast: having a privacy page and SEO appeared first on Yoast.

We recently made some changes to how yoast.com is run as a shop and how it’s hosted. In that process, we accidentally removed our robots.txt file and caused a so-called spider trap to open. In this post, I’ll show you what a spider trap is, why it’s problematic and how you can find and fix them.

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO: the #1 WordPress SEO plugin Info

What is a spider trap?

A spider trap is when you create a system that creates unlimited URLs. So Google can spider a page and find 20 new URLs on it. If it then spiders those 20 URLs, it finds 20 * 20 new URLs. If it then spiders those 400 URLs, it finds 400 * 20 = 8,000 new URLs. This escalates quickly, as you can see. If each and every one of these URLs were unique and wonderful, this would not be a problem, but usually, they’re not. So this causes a massive duplicate content problem.

A spider trap is bad for your SEO because every time Google crawls (or “spiders”) a page in your trap, it’s not crawling actual content on your site. Your new, high quality, super-valuable content might get indexed later, or not at all, because Google is spending its precious time in your trap. And the content it is crawling is deemed as duplicate and lessens how Google sees your site overall. This is why solving spider traps is important for SEO, and especially if you’re thinking about crawl budget optimization.

What do spider traps look like?

Our spider was one of a very particular type. We have a tool here on yoast.com called Yoast Suggest. It helps you mine Google Suggest for keyword ideas. When you enter a word into it, it returns the suggestions Google gives when you type that word into Google. The problem is: Google, when given a search box, will start throwing random words into it. And the results then have links for more results. And Google was thus trapping itself.

You might think that this is a nice story and spider traps never happen in real life. Unfortunately, they do. Faceted navigation on web shops often creates hundreds of thousands of combinations of URL parameters. Every new combination of facets (and thus URL parameters) is a new URL. So faceted navigation done poorly very often results in trapping the spider.

Another common cause of spider traps is when a site has date pages. If you can go back one day, to get a new date, and then back, and back, and back, you get a lot of pages. In my time as a consultant for the Guardian, we found Google spidering a date in the year 1670. It had gone back through our online archives, which went back almost 20 years at that point, to find nothing for more than 300 years…

How to recognize a spider trap

The easiest way to recognize a spider trap is by looking at your access logs. These logs contain a line for every visit to your site. Now as you can imagine, on larger sites your access logs get big very quickly. Here at Yoast, we use a so-called ELK-stack to monitor our website’s logs, but I’ve personally also used SEO log file analyzer by Screaming Frog to do this.

Logs from an ELK stack, showing a graph of indexing, URLs, timestamp, user agents and more

An example of logs in our ELK stack

What you’re looking to do is look at only Googlebot’s visits, and then start looking for patterns. In most cases, they’ll jump straight at you. It’s not uncommon for spider traps to take up 20-30% or even larger chunks of all the crawls. If you can’t find them immediately, start grouping crawls, looking for patterns within URLs. You can start from the beginning of the URL, provided you have clean URLs. If your URLs are slightly more cumbersome, you’ll have to create groups manually.

An ELK stack makes this very easy because you can search and segment quickly:

An example of filtering for the word "SEO" within our Googlebot hits in our ELK stack

An example of filtering for the word “SEO” within our Googlebot hits

How do you solve a spider trap?

Solving a spider trap can be a tricky thing. In our case, we don’t want /suggest/ to be indexed at all, so we just blocked it entirely with robots.txt. In other cases, you cannot do that as easily. For faceted navigation, you have to think long and hard about which facets you’d like Google to crawl and index.

In general, there are three types of solutions:

  1. Block (a section of) the URLs with robots.txt.
  2. Add rel=nofollow and noindex,follow on specific subsets of links and pages and use rel=canonical wisely.
  3. Fix the trap by no longer generating endless amounts of URLs.

In the case of the Guardian, we could simply prevent linking to dates where we had no articles. In the case of Yoast.com’s suggest tool, we simply blocked the URL in robots.txt. If you’re working with faceted search, the solution is, usually and unfortunately, not that simple. The best first step to take is to use a form of faceted search that doesn’t create crawlable URLs all the time. Checkboxes are better than straight links, in that regard.

In all, finding and closing a spider trap is one of the more rewarding things an SEO can do to a website. It’s good fun, but can certainly also be hard. If you have fun examples of spider traps, please do share them in the comments!

Read more: ‘Robots.txt: the ultimate guide’ »

The post Closing a spider trap: fix crawl inefficiencies appeared first on Yoast.

WordPress 4.9 Beta 2 is now available!

This software is still in development, so we don’t recommend you run it on a production site. Consider setting up a test site just to play with the new version. To test WordPress 4.9, try the WordPress Beta Tester plugin (you’ll want “bleeding edge nightlies”). Or you can download the beta here (zip).

For more information on what’s new in 4.9, check out the Beta 1 blog post. Since then, we’ve made 70 changes in Beta 2.

Do you speak a language other than English? Help us translate WordPress into more than 100 languages!

If you think you’ve found a bug, you can post to the Alpha/Beta area in the support forums. We’d love to hear from you! If you’re comfortable writing a reproducible bug report, file one on WordPress Trac, where you can also find a list of known bugs.

Let’s test all of these:
code editing, theme switches,
widgets, scheduling.

Have you ever done a fresh Yoast SEO for WordPress install on your WordPress website? Have you ever found yourself wondering what’s hidden in the general SEO section of Yoast SEO? In the SEO section, in the bottom half of the WordPress menu on the left of the page? Perhaps the better question would be: have you ever tried our Yoast SEO configuration wizard? Our wizard takes care of all the little things that you should configure. Things that you might forget in your eagerness to get started with your newly set up website.

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO: the #1 WordPress SEO plugin Info

Where can I find the Yoast SEO configuration wizard?

Of course, you want to jump right in and configure our plugin, using that Yoast SEO configuration wizard. Once you have installed our plugin, you’ll see a notification on your Yoast SEO dashboard:

The configuration wizard helps you to easily configure your site to have the optimal SEO settings.
We have detected that you have not finished this wizard yet, so we recommend you to start the configuration wizard to configure Yoast SEO.

There is a link in this message, leading you to a tab that’s located next to your SEO dashboard: “General”. You can set a number of things here, but you’ll also find a button that takes you to the Yoast SEO configuration wizard:

Where to find the Yoast SEO configuration wizard

The wizard

Once you’ve opened the wizard, we’ll guide you through the steps via a few questions. If you answer these, we’ll implement the right settings for your website, based specifically on your answers.

Step 1: Welcome to the Yoast SEO configuration Wizard

Let’s look at the first screen of the configuration wizard:
Yoast SEO configuration wizard: Welcome

You have two options here. You can start the wizard by clicking the purple button in the left box. This will continue the process as described below in this article.

The other option, on the right, will take you to our shop. Because we can do the configuration process for you, if you feel that there’s more to configure, and want to be sure it is done right for your particular site. We’ll check all the things in the wizard, but first, we will have a quick look at your website to see how you implemented things. And how we can optimize these settings for your specific business. Especially the technical side of things may feel challenging for the average site owner. This is where the configuration service is of great help. For the best result, you can also purchase our configuration package, which also includes our Yoast SEO Premium plugin, and the installation of that plugin.

For this post, let’s assume you want to use the Yoast SEO configuration wizard first.

Step 2: Is your site ready to be indexed?

The first question determines whether you want your site to be indexed or not. Perhaps you are working on a development site, on a staging server or just don’t want the public to see your site yet:
Yoast SEO configuration wizard

The reason we ask, is that one of the most important checks in our plugin determines whether Google can index your site or not. Google needs to be able to reach your website and index it unless you don’t want that. If you don’t want that, we merely need to know. You can set your preference for this in the second step of our wizard.

Step 3: What kind of site do you have?

In the next step, we will ask you about the type of site you have. It could be a blog or an online shop, but might as well be a news site or a portfolio.
Yoast SEO configuration wizard

One of the reasons we ask this question is because it’s essential you take a moment and think about this. What is your site about? Let’s take yoast.com, for example. We have two different sections on our website yoast.com:

  • Our blogs: an SEO blog and a dev blog. In these blogs, we share knowledge about both SEO and software development in all its facets.
  • Our online shop. We run an online shop and you’ll find our premium plugins and online courses in there.

What makes this question hard for ourselves, is the fact that following our mission “SEO for everyone”, both are equally important. Sharing knowledge is our main goal. Making sure all companies large and small and all individuals rich and poor, wherever on this planet, can optimize their websites and have an equal chance to rank in the search result pages. We use our products to provide even more insights or to deliver our knowledge to you in a structured package. That is also the reason we charge prices for our software that fit well into the offers of most online agencies. Charging $5,000 for a website, and including a mere $89 for our Yoast SEO Premium plugin seems like a no-brainer. Especially since it just makes your work / the work of your client so much easier. But enough with the promotional talk.

Think for yourself what your answer to this question should be. That’ll make it easier to configure several features of our plugin and, in fact, of your website later on.

For us, as plugin developers, the information we get from this question is also useful for future improvements. For instance, it can help us to prioritize future additions to our plugin for specific types of sites.

Step 4: Is it you or your company?

For the right metadata, we ask you to choose between company and person here. Is your website about you, or the company you represent? If you are a person, we would like to include your name. If you are a company, please add the name and logo.
Yoast SEO configuration wizard

This information will be included in the metadata of your website, with the goal to provide Google with the right information for their Knowledge Graph. The Knowledge Graph is the block of information you see on the right-hand side of the search results, for instance when you do a company search for Sony or Apple. My search for “Apple” actually returned details for our local Apple premium reseller, with a sort of ‘footer’ about the global Apple company details.

To make sure you give Google proper suggestions for that Knowledge Graph, we have added this question.

Step 5: Tell us your social profiles

In addition to your name or company name, we also ask you to let us know which social profiles you have. Again, so we can provide Google with the right information for their Knowledge Graph. Google seems keen on delivering answers to their visitors right away, so you’d better make sure your information is on Google.
Yoast SEO configuration wizard

With social being a part of the Knowledge Graph, and your website being linked on all your social profile pages, be sure to fill this out as completely as possible.

Step 6: To show or not show certain post types

The description in the image below is pretty clear: this is where you can set posts and pages to hidden or visible. Besides that, you can also choose to hide the Media post type.
Yoast SEO configuration wizard

If you set your Media post type to ‘visible’, WordPress will generate separate pages for your images, and we will generate an XML sitemap for your images as well. Now, unless you have a very specific reason to generate these pages, we recommend setting this to ‘hidden’ instead. That way, most websites prevent the generation of a ton of pages that just contain an image and no further content. Google will spend time indexing all these pages, but they add little value to your content. Keep in mind that when an image is on your page, post, or a specific gallery, Google will find it anyway.

Step 7: How many people are publishing content on your site?

We absolutely want to know if your website has multiple authors. There’s a reason for that: when your site only has one author, WordPress will still generate author pages. But if you write all the content on your blog yourself, your blog page will show the exact same collection of posts as your author page. And that, indeed, is duplicate content.
Yoast SEO configuration wizard

We call something duplicate content when the majority of a page is the same as the content on another page. Google will notice this, get confused, won’t know what page to rank first, and might decide to rank both a bit less. You obviously want to prevent that from happening. As we can guide you in this case, we added this check to our Yoast SEO configuration wizard.

Step 8: Google Search Console integration

There is a ton of information about your website in Google Search Console. We have written many posts about webmaster tools like Google Search Console, but did you know we also have an integration for it in our plugin? It connects your website to Google Search Console and allows you to keep a keen eye on your 404 Not Found errors. In our Yoast SEO Premium plugin, we’ll even guide you in preventing these 404 errors by helping you change them to for instance a 301 (so redirect the page to another page), or a 410 status code (which tells Google the page is gone forever).
Yoast SEO configuration wizard

In this step of the Yoast SEO configuration wizard, we guide you in connecting Google Search Console to our plugin. After that, we’ll start showing you your 404s so you can monitor and fix them!

Step 9: Optimizing your page title

At the title settings step in the wizard, we ask you to think about your branding. The website name you enter here is the name that our default page title template will use to put at the end of each page title. The default page title template looks like this:
%%title%% %%page%% %%sep%% %%sitename%%

The last part of that template is %%sitename%% and that’s what you fill out here. Be sure to add it, but keep it short, so the focus will be on the page or post title. It’s nice to have some of your branding in here so people will recognize your pages in the search result pages. If they already know you and your site, they’re more likely to click on one of your links.
Yoast SEO configuration wizard

The third part of the page template is %%sep%%, which stands for separator. A page title that follows our template can be “Some title of a post – Yoast”. The hyphen in there is the separator you can set at this step in the Yoast SEO configuration wizard. Using another separator than the average person might make you stand out from your competitors in the search result pages. But beside that, you can also pick the smallest separator, which could mean you can squeeze in another character or two.

Read more: ‘Titles and meta variables in Yoast SEO’ »

Step 10: Awesome tips and new products in your inbox

As SEO is an ongoing process, our goal is to keep you up-to-date on any changes in Google’s search result pages or Google’s algorithm. We do that by posting on our SEO blog, but also with our newsletter. In the newsletter, we highlight new developments in search, in WordPress and in our company – if relevant.
Yoast SEO configuration wizard

Simply insert your email address, and we’ll keep you in the loop on all things SEO!

Step 11: Upsell: buy our Premium plugin

Call it whatever you want (upsell, spam, useful information), but we have to tell you about our premium plugin in our configuration. Because we deliver incredibly useful SEO extras with that premium plugin, for a reasonable price. To name but a few:

  • What about a redirect manager? We’ll not only show you your 404s, but will also make it very easy to redirect, and thereby fix ’em.
  • An internal linking tool that will help you optimize your site structure to the max. Link suggestions and an easy way to copy these into your text. Optimize your cornerstone content even further.
  • Social previews, so you’ll know exactly what your website will show on Facebook and Twitter, and the option to tweak that.
  • A year of updates for all premium features, so your entire plugin will always be 100% up-to-date.
  • Email support for as long as you have Premium. This means you can email our 24/7 support team with any questions you have about the plugin.

Yoast SEO configuration wizard

Next to that, we offer some hands-on online courses to improve your SEO game even more. Be sure to check them out; you can always decide later if they add value for you, right? We think they do :)

Step 12: Even more free information about the plugin: then you’re ready to get started!

All the steps above have one goal: prepare you and your website for SEO. These steps are focused on the general settings of our plugin.

If you have used our plugin before, you’ll know it also provides a thorough SEO analysis in real time, while you write your posts or pages. On the page/post edit screen, where you write your content, you’ll find a so-called meta box with our SEO and readability analysis. For more insights on both, we finish our Yoast SEO configuration wizard with a helpful video, which tells you more about that specific part of the plugin. Be sure to watch that video!
Yoast SEO configuration wizard: finished

The configuration wizard makes things easier for all of us

All in all, I trust this article gives you a pretty good insight in why you should give our Yoast SEO configuration wizard a spin. And why we ask what we ask in there!

And again, if you want us to configure the plugin for you, feel free to use our Yoast SEO configuration service instead.

Keep reading: ‘Why every website needs Yoast SEO’ »

The post The Yoast SEO Configuration Wizard appeared first on Yoast.