MarkupSitemap

[Mike Rockett 1,160]

Docs & Download: rockettpw/seo/markup-sitemap
Modules Directory: MarkupSitemap
Composer: rockett/sitemap

MarkupSitemap is essentially an upgrade to MarkupSitemapXML by Pete. It adds multi-language support using the built-in LanguageSupportPageNames. Where multi-language pages are available, they are added to the sitemap by means of an alternate link in that page's <url>. Support for listing images in the sitemap on a page-by-page basis and using a sitemap stylesheet are also added.

Example when using the built-in multi-language profile:

<url>
    <loc>http://domain.local/about/</loc>
    <lastmod>2017-08-27T16:16:32+02:00</lastmod>
    <xhtml:link rel="alternate" hreflang="en" href="http://domain.local/en/about/"/>
    <xhtml:link rel="alternate" hreflang="de" href="http://domain.local/de/uber/"/>
    <xhtml:link rel="alternate" hreflang="fi" href="http://domain.local/fi/tietoja/"/>
</url>

It also uses a locally maintained fork of a sitemap package by Matthew Davies that assists in automating the process.

The doesn't use the same sitemap_ignore field available in MarkupSitemapXML. Rather, it renders sitemap options fields in a Page's Settings tab. One of the fields is for excluding a Page from the sitemap, and another is for excluding its children. You can assign which templates get these config fields in the module's configuration (much like you would with MarkupSEO).

Note that the two exclusion options are mutually exclusive at this point as there may be cases where you don't want to show a parent page, but only its children. Whilst unorthodox, I'm leaving the flexibility there. (The home page cannot be excluded from the sitemap, so the applicable exclusion fields won't be available there.)

As of December 2017, you can also exclude templates from sitemap access altogether, whilst retaining their settings if previously configured.

Sitemap also allows you to include images for each page at the template level, and you can disable image output at the page level.

The module allows you to set the priority on a per-page basis (it's optional and will not be included if not set).

Lastly, a stylesheet option has also been added. You can use the default one (enabled by default), or set your own.

Note that if the module is uninstalled, any saved data on a per-page basis is removed. The same thing happens for a specific page when it is deleted after having been trashed.

    

[Mike Rockett 1,160]

Bumped to 0.1.1-beta

• Added image support:
  • You can now specify fields in module config that should be included when rendering the sitemap. Simply choose one or more image fields from the list, and any Page that uses those fields will have their images listed in the sitemap. If a description is present, it will be used as the image caption.
  • Also supports ImageExtra. To add specific details to your sitemap's image entries, use ImageExtra to add the meta fields accordingly:
    • Add a license field to set a license for the image (it must be a URL, or it won't be included)
    • Add a title, if needed
    • Add geo, location, or geolocation if you'd like to specify the geo-location of your images
• The template field in ModuleConfig was changed from includeTemplates to sitemap_include_templates.
• The ModuleConfig fields now show template and field labels, and not their names.
• Small changes to the per-Page config field descriptions and labels

[mel47 67]

Hi

Thanks for this module. I was googling tonight to find an how-to for building sitemap and see this. But I have 2 stupid questions:

1. "Home" template is not available, is it because it's included by default?
2. Where is the sitemap? domain.com/sitemap.xml brings me to my home page. Did I did something wrong? 

Thanks

Mel

[Mike Rockett 1,160]

Hi Mel, You're most welcome.

1 hour ago, mel47 said:

"Home" template is not available, is it because it's included by default?

The list on the configuration page defines templates which get the sitemap_ignore_page and sitemap_ignore_children fields. As the home page should not be ignored, the home template does not get the fields (the module would be pretty much useless otherwise). The presentation of the sitemap works out of the box without any configuration.

Edit: I think the module config should be a little clearer with the explanation. I will sort that out now.

1 hour ago, mel47 said:

Where is the sitemap? domain.com/sitemap.xml brings me to my home page. Did I did something wrong?

The sitemap should be working at that URI. If you're looking for a simple sitemap, maybe use MarkupSitemapXML and see if it renders. If it does, then it means I've done something wrong with my module, which is based on that one.

[Mike Rockett 1,160]

Bumped to 0.1.2-beta

• Fixed descriptions to better explain the purpose of the fields added to templates.
  If additional non-related fields are ever added, these descriptions will be amended accordingly.
   
• Removes sitemap cache on module-save.
  This is already done when pages are saved, unless the option in MarkupCache's config dictates otherwise.

[mel47 67]

Thanks for clarification.

To be honest, I already tried MarkupSitemapXML before this module. I was unable to see sitemap in any of them, so I guess it's my config who failed.

[Mike Rockett 1,160]

2 hours ago, mel47 said:

Thanks for clarification.

To be honest, I already tried MarkupSitemapXML before this module. I was unable to see sitemap in any of them, so I guess it's my config who failed.

You're welcome.

Are you actually redirected to your home page, or does it not redirect but does show the home page?

[mel47 67]

Solved. Found why. Url segments was activated on home template.

Thanks!

[Mike Rockett 1,160]

12 hours ago, mel47 said:

Solved. Found why. Url segments was activated on home template.

Thanks!

Ah, of course. Url segments is something that can't be supported because of how they're created... No easy way to keep track of them.

[maxf5 133]

Hi Mike, wonderful module for languages and image integration!
But what can i do to fix this case?

<url>
<loc>http://www.xxx.de/</loc>
<lastmod>2017-09-10T00:03:12+02:00</lastmod>
<xhtml:link rel="alternate" hreflang="home" href="http://www.xxx.de/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://www.xxx.de/en/"/>
</url>

[Mike Rockett 1,160]

7 hours ago, maxf5 said:

Hi Mike, wonderful module for languages and image integration!
But what can i do to fix this case?

<url>
<loc>http://www.xxx.de/</loc>
<lastmod>2017-09-10T00:03:12+02:00</lastmod>
<xhtml:link rel="alternate" hreflang="home" href="http://www.xxx.de/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://www.xxx.de/en/"/>
</url>

@maxf5 - Glad you like it, and thanks for the post. I still need to work better multi-language support. Even at this point on the multi-lang site profile that PW ships with, the first alternate is identifcal to the main URL in terms of what they contain. Yours is a little different, and so I'd need to envisage the possibilities before working on it properly.

[PWaddict 354]

@Mike Rockett There are 3 problems with the module:

1. The iso for the default language as @maxf5 mentioned.
2. Some pages are completely ignored
3. Wrong sitemap markup. Each link must have it's alternate links not just the default one.

I'm currently using the below method for a proper multi-language sitemap.

1st step
Create a php file "sitemap-xml-multilanguage.php" with the below code and add it on /site/templates/ folder.

<?php namespace ProcessWire;

if (!wire('modules')->isInstalled('LanguageSupportPageNames')) throw new WireException('Missing module: LanguageSupportPageNames');

// Exclude pages here
function selectors() {
	$selectors = array(
	'', // E.g. template!=variations
	); 
	return implode(',',$selectors);
}

function renderSitemapPage(Page $page) {
	$out = '';
	foreach (wire('languages') as $lg) {
		if (!$lg->isDefault() && !$page->{"status$lg->id"}) continue;
		$out .= "\n<url>" . 
			"\n\t<loc>" . $page->localHttpUrl($lg) . "</loc>" . 
			renderSitemapAlternateLangs($page) .
			"\n\t<lastmod>" . date("Y-m-d", $page->modified) . "</lastmod>" . 
			"\n</url>";
	}
	return $out;
}

function renderSitemapChildren(Page $page) { 

	$out = '';
	$newParents = new PageArray(); 
	$children = $page->children(selectors()); 
	
	foreach($children as $child) {
		$out .= renderSitemapPage($child);
		if($child->numChildren) $newParents->add($child); 
			else wire('pages')->uncache($child); 
	}

	foreach($newParents as $newParent) {
		$out .= renderSitemapChildren($newParent); 
		wire('pages')->uncache($newParent); 
	}

	return $out; 
}

function renderSitemapAlternateLangs(Page $page) {
	$out = '';
	foreach(wire('languages') as $lg) {
		if (!$lg->isDefault() && !$page->{"status$lg->id"}) continue;
		$iso = $lg->language_iso_code; // Custom text field for ISO under system's language template
		$out .=	"\n\t<xhtml:link rel=\"alternate\" hreflang=\"$iso\" href=\"".$page->localHttpUrl($lg)."\"/>";
	}
	return $out;
}

function renderSitemapXML(array $paths = array()) {

	$out = 	'<?xml version="1.0" encoding="UTF-8"?><urlset xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xmlns="https://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="https://www.w3.org/1999/xhtml" xsi:schemaLocation="https://www.sitemaps.org/schemas/sitemap/0.9 https://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd">';

	array_unshift($paths, '/');

	foreach($paths as $path) {
		$page = wire('pages')->get($path); 
		if(!$page->id) continue; 
		$out .= renderSitemapPage($page);
		if($page->numChildren) $out .= renderSitemapChildren($page);
	}

	$out .= "\n</urlset>";

	return $out; 
}
header("Content-Type: text/xml");
echo renderSitemapXML();

Note: The above code if I remember correctly is based on @ryan and @kixe code.

2nd step
Add the new template "sitemap-xml-multilanguage" in PW and under URLs tab on "Should page URLs end with a slash?" select "No".

3rd step
Create a new page under "Home" with the title "sitemap.xml" by using the "sitemap-xml-multilanguage" template.

Final step
Create a text field with the name "language_iso_code" and add it under system's language template under "Setup > Templates > Show system templates? > Yes. Then on each language under "Setup > Languages" you will have to add the iso name. This is useful for the default language.

Here are the results I'm getting on http://localhost/mysite/sitemap.xml with my method:

<urlset xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xmlns="https://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="https://www.w3.org/1999/xhtml" xsi:schemaLocation="https://www.sitemaps.org/schemas/sitemap/0.9 https://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd">
<url>
<loc>http://localhost/mysite/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/"/>
<lastmod>2017-09-02</lastmod>
</url>
<url>
<loc>http://localhost/mysite/en/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/"/>
<lastmod>2017-09-02</lastmod>
</url>
<url>
<loc>http://localhost/mysite/nea/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/nea/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/news/"/>
<lastmod>2017-08-27</lastmod>
</url>
<url>
<loc>http://localhost/mysite/en/news/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/nea/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/news/"/>
<lastmod>2017-08-27</lastmod>
</url>
<url>
<loc>http://localhost/mysite/parastaseis/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/parastaseis/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/shows/"/>
<lastmod>2017-08-07</lastmod>
</url>
<url>
<loc>http://localhost/mysite/en/shows/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/parastaseis/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/shows/"/>
<lastmod>2017-08-07</lastmod>
</url>
<url>
<loc>http://localhost/mysite/omada/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/omada/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/meet-the-team/"/>
<lastmod>2017-08-25</lastmod>
</url>
<url>
<loc>http://localhost/mysite/en/meet-the-team/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/omada/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/meet-the-team/"/>
<lastmod>2017-08-25</lastmod>
</url>
<url>
<loc>http://localhost/mysite/epikoinwnia/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/epikoinwnia/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/contact/"/>
<lastmod>2017-09-02</lastmod>
</url>
<url>
<loc>http://localhost/mysite/en/contact/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/epikoinwnia/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/contact/"/>
<lastmod>2017-09-02</lastmod>
</url>
<url>
<loc>http://localhost/mysite/nea/test-4/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/nea/test-4/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/news/test-4/"/>
<lastmod>2017-08-14</lastmod>
</url>
<url>
<loc>http://localhost/mysite/en/news/test-4/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/nea/test-4/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/news/test-4/"/>
<lastmod>2017-08-14</lastmod>
</url>
<url>
<loc>http://localhost/mysite/nea/test-3/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/nea/test-3/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/news/test-3/"/>
<lastmod>2017-08-11</lastmod>
</url>
<url>
<loc>http://localhost/mysite/en/news/test-3/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/nea/test-3/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/news/test-3/"/>
<lastmod>2017-08-11</lastmod>
</url>
<url>
<loc>http://localhost/mysite/nea/test-2/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/nea/test-2/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/news/test-2/"/>
<lastmod>2017-08-11</lastmod>
</url>
<url>
<loc>http://localhost/mysite/en/news/test-2/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/nea/test-2/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/news/test-2/"/>
<lastmod>2017-08-11</lastmod>
</url>
<url>
<loc>http://localhost/mysite/nea/test-1/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/nea/test-1/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/news/test-1/"/>
<lastmod>2017-08-11</lastmod>
</url>
<url>
<loc>http://localhost/mysite/en/news/test-1/</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/nea/test-1/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/news/test-1/"/>
<lastmod>2017-08-11</lastmod>
</url>
<url>
<loc>
http://localhost/mysite/parastaseis/test/
</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/parastaseis/test/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/shows/test/"/>
<lastmod>2017-09-04</lastmod>
</url>
<url>
<loc>
http://localhost/mysite/en/shows/test/
</loc>
<xhtml:link rel="alternate" hreflang="el" href="http://localhost/mysite/parastaseis/test/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/shows/test/"/>
<lastmod>2017-09-04</lastmod>
</url>
</urlset>

 

and here are the results from your module:

<urlset xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xmlns="https://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="https://www.w3.org/1999/xhtml" xsi:schemaLocation="https://www.sitemaps.org/schemas/sitemap/0.9 https://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd">
<url>
<loc>http://localhost/mysite/</loc>
<lastmod>2017-09-02T05:37:50+03:00</lastmod>
<xhtml:link rel="alternate" hreflang="home" href="http://localhost/mysite/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/"/>
</url>
<url>
<loc>http://localhost/mysite/parastaseis/</loc>
<lastmod>2017-08-07T09:51:08+03:00</lastmod>
<xhtml:link rel="alternate" hreflang="home" href="http://localhost/mysite/parastaseis/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/shows/"/>
</url>
<url>
<loc>
http://localhost/mysite/parastaseis/test/
</loc>
<lastmod>2017-09-04T11:06:36+03:00</lastmod>
<xhtml:link rel="alternate" hreflang="home" href="http://localhost/mysite/parastaseis/test/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/shows/test/"/>
</url>
<url>
<loc>http://localhost/mysite/omada/</loc>
<lastmod>2017-08-25T20:27:09+03:00</lastmod>
<xhtml:link rel="alternate" hreflang="home" href="http://localhost/mysite/omada/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/meet-the-team/"/>
</url>
<url>
<loc>http://localhost/mysite/epikoinwnia/</loc>
<lastmod>2017-09-02T04:10:51+03:00</lastmod>
<xhtml:link rel="alternate" hreflang="home" href="http://localhost/mysite/epikoinwnia/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://localhost/mysite/en/contact/"/>
</url>
</urlset>

 

I hope it can help you fix your module.

[maxf5 133]

Maybe one can make optional module settings for this problem.
Listing every language used on the website and a field for the ISO and a checkbox.
When checked it renders the "custom" ISO for this language.

[Mike Rockett 1,160]

@PWaddict - Thanks! I totally missed out on the actual requirements there. Will work on fixes based on your code soon. Note sure why some of those pages are being ignored as I didn't experience that myself... Will look into it.

@maxf5 - After fixes, the module will create a language ISO field and automatically add it to the language template.

[maxf5 133]

Great idea, looking forward to it :)

[psy 688]

Wishlist/request... would love to see on a per page/template basis a sitemap priority field.

When no priority is specified, Google goes 'top page - 100%', 'sub page - 80%', 'sub-sub-page - 60%'. 

In reality however, the 'sub-page' is often the parent page in the menu. The 'sub-sub-page' has the guts & glory that you want indexed with a high priority.

Google doesn't guarantee but says it will give a sub-sub-page with a priority eg of 80% a higher priority than it's root parent with a priority of 60%.

https://www.sitemaps.org/protocol.html

[Mike Rockett 1,160]

@psy - That's on my wishlist too ;-) Yes, I'll definitely implement this.

[Mike Rockett 1,160]

I'm making progress with all the necessary changes. On my dev server (not git-pushed yet), the following is complete:

1. Languages: Multi-language is sorted. It now renders the sitemap according to Google standards, where a page's <url> is generated for each languages with alternate langs, including current. Method is not the same as above example (page-loop), but expands on the root-first-then-children process - this allows easier flexibility in terms of what we can ignore on a page by page basis.
2. ISOs: Haven't created the separate language ISO field yet, as my my main concern is regarding names vs ISOs. If they are normally identical, then we would only need to declare a proper ISO name for the default language only when its URL matches the default root URL (ie: default URL root = site root). In this case, perhaps it would be better to simply have a module config entry for the default language only? The rationale behind this is that the language name (other than default) should be the same as the iso (for better SEO practice, I'd guess). I haven't yet bumped into a ProcessWire site that doesn't use the ISO for each language name accordingly. If I'm wrong, please point it out, and then I'll use the iso field, per the above example. Or perhaps I should simple allow the flexibility and have the ISO name fall back to the language name if not provided via a field (the default would still need one if it is root)?
3. Image-support: Images are now picked up in all languages, given the process-change. If an image field is not specified in another language, it will fall back to the default language. Support for ImageExtra remains.

Priority: not done yet. I'll follow a basic principal with this: if the priority is set on a specific page, then it will be added. Otherwise, the tag will be left out. If this is not the correct methodology, please let me know. (cc @psy)

Note: I still need to do some testing on this...

[Mike Rockett 1,160]

Quick questions: Should the module -

1. ping sitemap engines (after cache-revocation)?
2. add /sitemap.xml to /robots.text if it exists and is not already added? (This would be done at install-time, and removed at uninstall-time.)

For anyone who may be curious about it, I don't plan on adding XSLT support. End of the day: sitemap.xml is intended for the viewsing of search engines, and not humans. If you want a visual sitemap, best to create your own.

[psy 688]

35 minutes ago, Mike Rockett said:

Priority: not done yet. I'll follow a basic principal with this: if the priority is set on a specific page, then it will be added. Otherwise, the tag will be left out.

The docs say the tag is optional so it sounds right to me. Thanks @Mike Rockett

[Mike Rockett 1,160]

17 hours ago, Mike Rockett said:

ISOs: Haven't created the separate language ISO field yet, as my my main concern is regarding names vs ISOs.

[...]

Just to add to this point, @maxf5's issue can be solved by adding a default-language name to the home page. In LanguageSupportPageNames, the default language URI is the same as it would be without a language, and so it won't show even if the name is set.

Here's the setup I'm using on the built-in language profile:

LanguageSupportPageNames

Home Page default language name

Sitemap url output

    <url>
        <loc>http://pw-modules.local/</loc>
        <lastmod>2017-09-15T15:35:15+02:00</lastmod>
        <xhtml:link rel="alternate" hreflang="en" href="http://pw-modules.local/"/>
        <xhtml:link rel="alternate" hreflang="de" href="http://pw-modules.local/de/"/>
        <xhtml:link rel="alternate" hreflang="fi" href="http://pw-modules.local/fi/"/>
        <...>
    </url>

 

[PWaddict 354]

@Mike Rockett nice "trick"! I don't remember seeing that option on LanguageSupportPageNames. I will definitely gonna use that. I'm currently using the language_iso_code field not only on my sitemap but also on site's navigation (hreflang) and the flag icons but I will delete it

[Mike Rockett 1,160]

1 hour ago, PWaddict said:

@Mike Rockett nice "trick"! I don't remember seeing that option on LanguageSupportPageNames. I will definitely gonna use that. I'm currently using the language_iso_code field not only on my sitemap but also on site's navigation (hreflang) and the flag icons but I will delete it

Indeed it is - and I bumped into how it works by accident. Will be sticking to this method. v0.2 coming out shortly.

[PWaddict 354]

12 minutes ago, Mike Rockett said:

Indeed it is - and I bumped into how it works by accident. Will be sticking to this method. v0.2 coming out shortly.

There is only a small "downside" if the website is already online. All links from the default language (except homepage) will change so the Page Path History module MUST be installed to avoid 404 errors from previously shared pages, google search etc. Make sure to update your README.md with all these important info.

[Mike Rockett 1,160]

5 minutes ago, PWaddict said:

There is only a small "downside" if the website is already online. All links from the default language (except homepage) will change so the Page Path History module MUST be installed to avoid 404 errors from previously shared pages, google search etc. Make sure to update your README.md with all these important info.

Now that I think about it, we do actually need a way to set the iso for sites that do not use a home page name for the default language.