Jump to content

Multi Sites with multi Robots.txt


chrizz
 Share

Recommended Posts

perhaps this is the wrong subforum, perhaps there are better waays to do this, but I wanted to share some thoughts regarding multi sites together with multiple robots.txt

If you are running multiple sites with one PW setup you can't place multiple robots.txt files into your root. As long as all robots.txt files are identical there is no problem with it. You can stop reading right here. 

In my robots.txt I wanted to include a link to the current sitemap, e.g.

Sitemap: http://www.domain.com/sitemap.xml

I put each robots.txt into the "site-" directories. Search engines expect the robots.txt file directly in the root so I added some lines to my htaccess file (for each domain you have to do this)

# domain 1  
  RewriteCond %{REQUEST_URI} /robots\.txt
  RewriteCond %{HTTP_HOST} www\.domain\. [NC]
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d  RewriteRule ^(.*)$ /site-domain/robots.txt [L] 

# domain 2  
  RewriteCond %{REQUEST_URI} /robots\.txt
  RewriteCond %{HTTP_HOST} www\.domain\-2\. [NC]
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d  RewriteRule ^(.*)$ /site-domain-2/robots.txt [L] 

Another possible approach: create a PW page within each site :)

  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...