People often mistakenly believe that the more your site is exposed to search engines, the better for SEO. But in fact, there are some pages should be excluded from search results, or restrict the way search engines use it.
For SEO, the robots meta tag is often considered one of the most useful tools that allow you to give special instructions directly to search engines. This helps you to control how and if your content appears on Google, Bing and other search engines.
Interested in Magento 2 SEO? Come to Magento 2 SEO Guide now.
READ MORE Magento 2 SEO Suite extension to give the complete SEO solution for your site.
To better understand Robots meta tags, let’s explore the information below!
What is Robots Meta Tag?
Magento 2 Robots Meta Tag is a piece of code that is in the tag of the HTML document. It is used to Indexation-controlling the Magento 2 web page.
Magento 2 Robots Meta Tag tells what pages you want to hide (noindex) and what pages you want them to index from search engine crawlers. It also tells search engines what to follow and what not to follow the content of the Magento 2 web pages or Magento 2 website.
Why Robots Meta Tag Is Important for SEO in Magento 2?
Not all of your content needs to be indexed and shown in Google. To get in the high ranking for your Magento 2 website, you need a wise instruction for the bots about what NOT to index. If all of your content is crawled, the performance of SEO will not be good for sure.
A few examples of what you should restrict from crawlers: 404 not found pages, duplicate content, out-dated product pages, pages in the staging environment, thank-you pages, internal search results and more.
In short, the bigger your Magento 2 website is, the more you’ll deal with managing crawlability and indexation. If you want Google and other search engines to crawl and index your pages as efficiently as possible, you have to know how to correctly combine page-level directives with Magento 2 robots.txt and sitemaps.
How to Setup Robots Meta Tag in Magento 2 Default?
- Log in to Admin Panel.
- Navigate to Content →Design→ Configuration
- Expand the Search Engine Robots.
- Default Robots: Choose a combination from the drop-down.
There are 4 combinations among robots meta tag nofollow, noindex, follow and index:
INDEX, FOLLOW: Allow crawlers to index and follow the links.
NOINDEX, FOLLOW: Disallow indexing but allows crawlers to follow the links.
INDEX, NOFOLLOW: Allow crawlers to index the current pages but blocks it from following the links.
NOINDEX, NOFOLLOW: Disallow indexing and following of current page and links in that page.
- In the Edit custom instruction of robots.txt File, you can write the needed custom instructions for Magento 2 robots.txt.
Take a look at the following examples for customizing Magento 2 Robots.txt:
Allows Full Access
Disallows Access to All Folders
User-agent:* Disallow: /
Disallow: /lib/ Disallow: /*.php$ Disallow: /pkginfo/ Disallow: /report/ Disallow: /var/ Disallow: /catalog/ Disallow: /customer/ Disallow: /sendfriend/ Disallow: /review/ Disallow: /*SID=
- Click Save Configuration and flush cache.
However, it is only applied to the website. If you want to hide or restrict indexation of a specific page you have to edit the code using HTML editors. For example, you can restrict indexation with the following code:
<html > <head > <meta name=”robots” content=”noindex, follow”/ > <title>page title> </head >
How to Setup Robots Meta Tag in BSS Magento 2 SEO Extension?
By default, admins have to change the “INDEX, FOLLOW” by fixing the source code of the pages manually. But with BSS Commerce lastest extension, Magento 2 SEO extension, you can complete your need of changing the robots meta tag for any pages within a few clicks at the backend.
Back to our topic, there are 3 combinations you can set up for with our robots meta tag:
- NOINDEX, NOFOLLOW
- NOINDEX, FOLLOW
- INDEX, NOFOLLOW
And here is how easily you can configure:
- Choose the robots meta tag for specific pages, categories or folder
Just fill in the URL pattern of the page you want to set up. Choose the tag combination you want. Click “Save” then flush cache to complete. Admins can have as many pages changed as you want without any seek for developers.
- Set NOINDEX, FOLLOW for a list of regular pages
We select a collection of pages that we think should be set to noindex, follow for a better SEO performance such as checkout, CMS page, customer account pages, etc. You can choose the page based on your purpose.
Guiding search engines to crawl and index your Magento 2 website as the way you want can be a challenging task. You may have trouble when setting up robots meta tags for specific pages if you don’t know about code. Feel free to ask us if you have any questions!
Or you can continue reading our next SEO guide for Magento 2, which is about the secret 301 Redirect Technique!
Remember to check our SEO package with more utilities for your Magento 2 websites:
- Gather 9 Magento 2 extensions in one.
- Solve duplicate content issue by Canonical URLs and Hreflang Tags.
- Improve UX/UI by using SEO Advanced Breadcrumbs, 301 Redirect, Open Graph, Rich snippets and Twitter Cards.
- Get higher control over indexation with Magento 2 HTML/XML sitemap and Robots Meta Tag.
- Add meta information faster with Meta Tags Template.