Multi Sites with multi Robots.txt

[chrizz]

perhaps this is the wrong subforum, perhaps there are better waays to do this, but I wanted to share some thoughts regarding multi sites together with multiple robots.txt

If you are running multiple sites with one PW setup you can't place multiple robots.txt files into your root. As long as all robots.txt files are identical there is no problem with it. You can stop reading right here. 

In my robots.txt I wanted to include a link to the current sitemap, e.g.

Sitemap: http://www.domain.com/sitemap.xml

I put each robots.txt into the "site-" directories. Search engines expect the robots.txt file directly in the root so I added some lines to my htaccess file (for each domain you have to do this)

# domain 1  
  RewriteCond %{REQUEST_URI} /robots\.txt
  RewriteCond %{HTTP_HOST} www\.domain\. [NC]
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d  RewriteRule ^(.*)$ /site-domain/robots.txt [L] 

# domain 2  
  RewriteCond %{REQUEST_URI} /robots\.txt
  RewriteCond %{HTTP_HOST} www\.domain\-2\. [NC]
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d  RewriteRule ^(.*)$ /site-domain-2/robots.txt [L] 

Another possible approach: create a PW page within each site