Jump to content
Sign in to follow this  
chrizz

Multi Sites with multi Robots.txt

Recommended Posts

perhaps this is the wrong subforum, perhaps there are better waays to do this, but I wanted to share some thoughts regarding multi sites together with multiple robots.txt

If you are running multiple sites with one PW setup you can't place multiple robots.txt files into your root. As long as all robots.txt files are identical there is no problem with it. You can stop reading right here. 

In my robots.txt I wanted to include a link to the current sitemap, e.g.

Sitemap: http://www.domain.com/sitemap.xml

I put each robots.txt into the "site-" directories. Search engines expect the robots.txt file directly in the root so I added some lines to my htaccess file (for each domain you have to do this)

# domain 1  
  RewriteCond %{REQUEST_URI} /robots\.txt
  RewriteCond %{HTTP_HOST} www\.domain\. [NC]
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d  RewriteRule ^(.*)$ /site-domain/robots.txt [L] 

# domain 2  
  RewriteCond %{REQUEST_URI} /robots\.txt
  RewriteCond %{HTTP_HOST} www\.domain\-2\. [NC]
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d  RewriteRule ^(.*)$ /site-domain-2/robots.txt [L] 

Another possible approach: create a PW page within each site :)

  • Like 1

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Mike Rockett
      Docs & Download: rockettpw/markup-sitemap
      Modules Directory: MarkupSitemap
      Composer: rockett/sitemap
      MarkupSitemap is essentially an upgrade to MarkupSitemapXML by Pete. It adds multi-language support using the built-in LanguageSupportPageNames. Where multi-language pages are available, they are added to the sitemap by means of an alternate link in that page's <url>. Support for listing images in the sitemap on a page-by-page basis and using a sitemap stylesheet are also added.
      Example when using the built-in multi-language profile:
      <url> <loc>http://domain.local/about/</loc> <lastmod>2017-08-27T16:16:32+02:00</lastmod> <xhtml:link rel="alternate" hreflang="en" href="http://domain.local/en/about/"/> <xhtml:link rel="alternate" hreflang="de" href="http://domain.local/de/uber/"/> <xhtml:link rel="alternate" hreflang="fi" href="http://domain.local/fi/tietoja/"/> </url> It also uses a locally maintained fork of a sitemap package by Matthew Davies that assists in automating the process.
      The doesn't use the same sitemap_ignore field available in MarkupSitemapXML. Rather, it renders sitemap options fields in a Page's Settings tab. One of the fields is for excluding a Page from the sitemap, and another is for excluding its children. You can assign which templates get these config fields in the module's configuration (much like you would with MarkupSEO).
      Note that the two exclusion options are mutually exclusive at this point as there may be cases where you don't want to show a parent page, but only its children. Whilst unorthodox, I'm leaving the flexibility there. (The home page cannot be excluded from the sitemap, so the applicable exclusion fields won't be available there.)
      As of December 2017, you can also exclude templates from sitemap access altogether, whilst retaining their settings if previously configured.
      Sitemap also allows you to include images for each page at the template level, and you can disable image output at the page level.
      The module allows you to set the priority on a per-page basis (it's optional and will not be included if not set).
      Lastly, a stylesheet option has also been added. You can use the default one (enabled by default), or set your own.
      Note that if the module is uninstalled, any saved data on a per-page basis is removed. The same thing happens for a specific page when it is deleted after having been trashed.
          
    • By franciccio-ITALIANO
      Hi, we can choose the "headline" and "title" and "summery" in panel page of processwire, but we can't write the "metadecriptions" and "tags".
       I can write mdescropt and tags in templates, but I've same templates for many articles... so, how I can change mdescription and tags?

      Thanks...
    • By iipa
      Hi everybody!
      I have been reading about Multisite, but it kinda bugs me that every topic talks about having both admin and database same for multiple sites.
      I have a project where customer tests it by adding content to the site, while I still need to do some changes here and there in code, maybe some in database. If something crashes for a while, customer can't keep testing, which is a bit problematic.
      Is there any way that I could have two separate versions of one site ("production" and development) that share the same database, but are otherwise independent? Just the thought of having to migrate database every time I want to show client something new gives me anxiety 😁
    • By sins7ven
      Hi community, I am wondering if its possible to display languages based on what host/domain has been requested.
      The use case is the following: I have one installation of PW - and within this installation I set up 5 different languages (default, english, spanish, french, dutch). Now I have 3 different domains (domainA.com, domainB.com, domainC.com) and I want to decide which languages to make available based on what domain has been navigated to. 
      For example:
      domainA.com
       - default (domainA.com)
       - english (domainA.com/en)
      domainB.com
        - default (domainB.com)
        - french (domainB.com/fr)
        - dutch (domainB.com/nl)
      domainC.com
        - default (domainC.com)
        - dutch (domainC.com/nl)
      So what I want to achieve is making only these particular languages selectable in the language switcher on the frontend. Since all pages share the same templates and overall site structure I don't see the benefits of running this installation as a multisite setup or would it be better to do so? But I assume that this would be difficult to handle since the default language might change as well. (On the french version of the page, french should be the default language and so on). Any suggestions how to accomplish that? Thanks!
    • By Peter Knight
      Hi guys
      Is it possible to have a PW site display content from another PW site?
      I've been reading about ProcessWire's Multi-site options and I don't think either is a way forward for me.
      Both PW sites are separate installs with separate databases but crucially they are on the same VPS. So I'm hoping that's a good thing.
      Basically, Site A has approx 100 reviews consisting of these fields
      Reviewer name Review Date Instead of setting up the fields onto Site B and duplicating work and creating 100 pages, I want to *somehow* get site B to use a selector to fetch the necessary content from Site A.
      In future, if I need to update the reviews, I can goto the 'source' Site A to do this.
       
×
×
  • Create New...