Skip Navigation

This thread is resolved. Here is a description of the problem and solution.

Problem:
The client has a multilingual site with Spanish as the root language and English as an additional language. They have configured a robots.txt file for the Spanish version, which includes SEO configurations for both languages. However, they noticed that the English version (/en/) has a robots.txt file without any configurations and lacks a sitemap file. The client is concerned about the potential SEO impact.
Solution:
We confirmed that search engines do not look for robots.txt files in language-specific folders, so this will not impact SEO. However, to address the issue of the unconfigured robots.txt file in the English directory, we provided a temporary workaround:
1) Ensure you have a full site backup.
2) Open the

functions.php

file in your Child theme for editing.
3) Add the following code to the bottom of the file:

add_filter( 'rewrite_rules_array', 'remove_robots_rewrite_rules' );

function remove_robots_rewrite_rules( $rules ) { 
unset( $rules['robots\.txt$'] );
return $rules;
}

4) Save the changes.
5) Go to Settings >> Permalinks and resave the settings.
We are currently working on a permanent fix and will update the client as soon as it is available.

If this solution does not resolve your issue, or if it seems outdated or irrelevant to your case, we highly recommend checking related known issues at https://wpml.org/known-issues/, verifying the version of the permanent fix, and confirming that you have installed the latest versions of themes and plugins. If further assistance is needed, please open a new support ticket at WPML support forum.

This is the technical support forum for WPML - the multilingual WordPress plugin.

Everyone can read, but only WPML clients can post here. WPML team is replying on the forum 6 days per week, 22 hours per day.

Tagged: 

This topic contains 4 replies, has 2 voices.

Last updated by Bigul 4 years ago.

Assisted by: Bigul.

Author Posts
May 17, 2021 at 5:34 pm #8783205

ievgeniiB

Hello,

I have my site in two different languages Spanish, which is my root language, and english /en/. For my root (Spanish) language I specified robots.txt with sitemap which includes pages/posts/categories also for English version of my site. And I realised that for added (English) /en/ language there are also exists file robot.txt but without any configs and there are not sitemap file. So my question is: should I ignore /en/robots.txt with sitemap if I have already configured everything for my root language and how it can impact on SEO?

FYI: You can check these files by accessing them through my website. I'm also using Yoast SEO plugin.

Thank you!

May 18, 2021 at 4:34 pm #8793601

Bigul
WPML Supporter since 01/2013

Languages: English (English )

Timezone: Europe/Vienna (GMT+02:00)

Hello,

Welcome to the WPML support forum. I will do my best to help you to resolve the issue.

To help you faster, I've enabled debug information for this support ticket. Please see this link for how to get this information from your site and give it to us: http://wpml.org/faq/provide-debug-information-faster-support/

Normally it will not be an issue. But I will consult about this with our developers. It requires further checking.

I hope you are following this documentation for multilingual SEO - https://wpml.org/documentation/related-projects/using-wordpress-seo-with-wpml/

--
Thanks!

Bigul

May 18, 2021 at 4:49 pm #8793671

ievgeniiB

Here is my debug information.

May 19, 2021 at 4:22 pm #8802297

Bigul
WPML Supporter since 01/2013

Languages: English (English )

Timezone: Europe/Vienna (GMT+02:00)

Hello,

Thank you for the Debug information. Search engines will not look for robots.txt in the language folder. So it will not have an SEO impact.

But I have reproduced the issue in a fresh installation on your sandbox site and escalated the ticket to our second-tier team for further checking. Please check the following URLs for more details. We will get back to you as early as possible. Please wait.

Default language URL - hidden link

Secondary language URL - hidden link

--
Thanks!

Bigul

May 31, 2021 at 7:29 am #8878859

Bigul
WPML Supporter since 01/2013

Languages: English (English )

Timezone: Europe/Vienna (GMT+02:00)

Hello,

The ticket is escalated to our developers for further debugging and fixing. But we have a workaround for this bug for the time being. Please try the following steps after a full site backup and let us know your feedback.

1) Open the functions.php file your Child theme for edit
2) Add the following code to the bottom of the file

add_filter( 'rewrite_rules_array', 'remove_robots_rewrite_rules' );

function remove_robots_rewrite_rules( $rules ) {
	unset( $rules['robots\.txt$'] );
	return $rules;
}

3) Save changes
4) Visit Settigs>>Permalinks and resave the settings

We will get back to you soon when we have an update from our developers on this. Please wait.

--
Thanks!

Bigul