|
Hello! I'm trying to set up a robots.txt file for every different domain on our website, which consists of a main website and 4 translations. Every domain is an unique one for 5 different countries, EN, NL, DE, FR and IT.
I found the following two methods as a solution for similar questions about this topic in the past:
----------------------------------------------------------------------------------
METHOD 1
----------------------------------------------------------------------------------
# Seperate robots.txt for each individual domain
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
#EN
RewriteCond %{HTTP_HOST} ^uniquedomain1\.eu.*$ [NC]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*robots\.txt.*\ HTTP/ [NC]
RewriteRule ^robots\.txt /robots-en.txt [NC,L]
#NL
RewriteCond %{HTTP_HOST} ^uniquedomain2\.nl.*$ [NC]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*robots\.txt.*\ HTTP/ [NC]
RewriteRule ^robots\.txt /robots-nl.txt [NC,L]
#DE
RewriteCond %{HTTP_HOST} ^uniquedomain3\.de.*$ [NC]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*robots\.txt.*\ HTTP/ [NC]
RewriteRule ^robots\.txt /robots-de.txt [NC,L]
#FR
RewriteCond %{HTTP_HOST} ^uniquedomain4\.fr.*$ [NC]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*robots\.txt.*\ HTTP/ [NC]
RewriteRule ^robots\.txt /robots-fr.txt [NC,L]
#IT
RewriteCond %{HTTP_HOST} ^uniquedomain5\.it.*$ [NC]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*robots\.txt.*\ HTTP/ [NC]
RewriteRule ^robots\.txt /robots-it.txt [NC,L]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress
<IfModule mod_security.c>
SecFilterRemove 000220
</IfModule>
-----------------
#After that create and edit individual robots txt for each domain, using the file names as referred in .htaccess:
robots-en.txt
robots-nl.txt
robots-de.txt
robots-fr.txt
robots-it.txt
----------------------------------------------------------------------------------
METHOD 2
----------------------------------------------------------------------------------
RewriteCond %{HTTP_HOST} ^www.uniquedomain1.eu [NC]
RewriteRule ^robots\.txt /robots-en.txt [NC,L]
RewriteCond %{HTTP_HOST} ^www.uniquedomain2.nl [NC]
RewriteRule ^robots\.txt /robots-nl.txt [NC,L]
RewriteCond %{HTTP_HOST} ^www.uniquedomain3.de [NC]
RewriteRule ^robots\.txt /robots-de.txt [NC,L]
RewriteCond %{HTTP_HOST} ^www.uniquedomain4.fr [NC]
RewriteRule ^robots\.txt /robots-fr.txt [NC,L]
RewriteCond %{HTTP_HOST} ^www.uniquedomain5.it [NC]
RewriteRule ^robots\.txt /robots-it.txt [NC,L]
#To prevent WP from rendering a virtual robots.txt:
touch robots.txt
-----------------
#After that create and edit individual robots txt for each domain, using the file names as referred in .htaccess:
robots-en.txt
robots-nl.txt
robots-de.txt
robots-fr.txt
robots-it.txt
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
Can you help me which one is the right method to use? The Google robots.txt-tester returns for both methods only data of the original robots.txt file (which I deleted after generating the 5 seperate robots.txt files).
When using method 2, adding the line 'touch robots.txt' results in a '500 Internal Server Error' (see attachment) when visiting the website. When this line is deleted the server error disappears.
Thanks!
|
