Jump to content
Pete

Module: XML Sitemap

Recommended Posts

I'm on the verge of converting your code back to a template, this is getting silly...

Just want to clarify that the problem is not with Pete's module. There was an obscure problem with the modules installer in the core which has been fixed. I'm not certain this is the problem in your case as you are still getting a different behavior than us, but I suggest grabbing the latest copy of ProcessWire before trying anything else.

Share this post


Link to post
Share on other sites

Just want to clarify that the problem is not with Pete's module.

My apologies for implying any such thing, just getting frustrated. When I saw this module it looked like a great timesaver, and that hasn't worked out.

I'm uploading a fresh copy of PW now, we'll see how it goes.

EDIT: Still no go!

Share this post


Link to post
Share on other sites

Can you confirm that you still get this same error message and that your ProcessWire reports it's version number as 2.2?

Exception: Unable to create path: /MarkupSitemapXML/ (in /home/theseeke/public_html/wire/core/CacheFile.php line 62) This error message was shown because you are logged in as a Superuser. Error has been logged.

thanks,

Ryan

Share this post


Link to post
Share on other sites

I just downloaded a fresh/blank copy of PW 2.2 and installed this module just to make sure it didn't have anything to do with the dev site I had tested on before. But it's working as expected, creating the dir in the right place, etc. So there must be something else that I'm missing. The behavior you are describing definitely indicates a core bug. I can't think of any other possibility. I'm going to do more testing here and hope to find and push a solution shortly.

Share this post


Link to post
Share on other sites

I'm pleased to say that this issue is resolved, and when it came right down to it, Problem Existed Between Keyboard and Chair.

Stupid me forgot to read the instructions, and installed the module to /wire/modules instead of /site/modules.

Now that it's in the right place it's working fine, cheers!

Share this post


Link to post
Share on other sites

Thanks for reporting back, glad that it's working. I was stuck trying to determine how it was doing that, so it's a relief to hear it's resolved. This has still been valuable though as we did solve a bug as a result (top of this page).

Share this post


Link to post
Share on other sites

Hi there,

I've been using this module on a couple of sites in place of the template I previously used and it certainly is a boon in keeping my template folder a lot tidier — great work!

I changed line 53 to check for access when iterating the children so that user-restricted pages do not show up in the sitemap. I think this is a saner default but could be a settting too.

foreach($page->children("check_access=1") as $child) $entry .= $this->sitemapListPage($child);

Thanks,

Stephen

Share this post


Link to post
Share on other sites

Stephen: check_access=1 is default, so you don't have to include that. If you don't want to check access, then you need to use check_access=0.

  • Like 1

Share this post


Link to post
Share on other sites

Very true. I obviously visited the sitemap straight after installing, while still logged in as admin. My mistake!

Thanks, apeisa

Share this post


Link to post
Share on other sites

Want to share a few things I've added for a magazine site I'm building:

First of all, I think "priority" and "changefreq" are fairly important if you're going to have a sitemap at all. This post has some info on Google's guidelines: http://www.eduki.com...-are-important/

What I decided to do was quickly add two global fields to PW so I can set these values manually in each page:

-sitemap_priority

-changefreq

And in the code:

public function sitemapListPage($page) {
$entry = "";
$default_priority = "0.5";
$default_changefreq = "monthly";

include $this->fuel('config')->paths->templates . "sitemap_module_defaults.inc";

if ($page->sitemap_ignore == 0 || $page->path == '/') { // $page->path part added so that it ignores hiding the homepage, else you wouldn't have ANY pages returned
$modified = date ('Y-m-d', $page->modified);
$entry = "\n <url>\n";
$entry .= " <loc>{$page->httpUrl}</loc>\n";
$entry .= " <lastmod>{$modified}</lastmod>\n";

if(!empty($page->sitemap_priority)) {
$entry .= " <priority>{$page->sitemap_priority}</priority>\n";
} else {
$entry .= " <priority>{$default_priority}</priority>\n";
}

if(!empty($page->changefreq)) {
$entry .= " <changefreq>{$page->changefreq}</changefreq>\n";
} else {
$entry .= " <changefreq>{$default_changefreq}</changefreq>\n";
}

 $entry .= " </url>";
if($page->numChildren) {
foreach($page->children as $child) $entry .= $this->sitemapListPage($child);
}
}
return $entry;
}

The sitemap_module_defaults.inc file in the templates dir is so I can set some values on the fly without doing it manually:

<?php
switch($page->template->name) {
case "blog_post":
case "blog_topic":
case "blog_topic_type":
$default_priority = "0.7";
$default_changefreq = "daily";
break;
}
?>

That's done and works fine for me.

Something I found frustrating with the sitemap module was that I couldn't add virtual pages I made. For example, if there's a page with urlSegments with some kind of page manipulation they won't show up on the sitemap, for obvious reasons. So what I did was add this method to the module:

public function sitemapListVirtualPage($httpUrl, $modified = NULL, $sitemap_priority = "0.5", $changefreq = "monthly") {
$entry = "";
$modified = date ('Y-m-d', $modified);
$entry = "\n <url>\n";
$entry .= " <loc>{$httpUrl}</loc>\n";

if($modified) {
$entry .= " <lastmod>{$modified}</lastmod>\n";
}

if($sitemap_priority) {
$entry .= " <priority>" . (float)$sitemap_priority . "</priority>\n";
}

if($changefreq) {
$entry .= " <changefreq>{$changefreq}</changefreq>\n";
}

 $entry .= " </url>";
return $entry;
}

And added this include to the init method before the output is saved to cache:


public function init() {
// Intercept a request for a root URL ending in sitemap.xml and output
if (strpos($_SERVER['REQUEST_URI'], wire('config')->urls->root . 'sitemap.xml') !== FALSE) {
// Check for the cached sitemap, else generate and cache a fresh sitemap
$cache = wire('modules')->get("MarkupCache");
if(!$output = $cache->get("MarkupSitemapXML", 3600)) {
$output = "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n";
$output .= '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">';
$output .= $this->sitemapListPage(wire('pages')->get("/"));

include $this->fuel('config')->paths->templates . "sitemap_module_virtual.inc";

$output .= "\n</urlset>";
$cache->save($output);
}

header("Content-Type: text/xml");
echo $output;
exit;
}
}

And in my sitemap_module_virtual.inc file I did this:

<?php
foreach(wire('pages')->find("template=blog_topic|blog_topic_type") as $real_page) {
$output .= $this->sitemapListVirtualPage("http://" . $this->fuel('config')->httpHost . $real_page->url . "archives/", wire('pages')->get("template=blog_post,sort=-created")->modified, "0.4", "daily");
}
foreach(wire('pages')->find("template=blog_topic|blog_topic_type") as $real_page) {
$output .= $this->sitemapListVirtualPage("http://" . $this->fuel('config')->httpHost . $real_page->url . "rss/", wire('pages')->get("template=blog_post,sort=-created")->modified, "0.4", "daily");
}
?>

In this part, I have manipulated /archive/ and /rss/ sub-pages for each blog_topic and sub-topic(or blog_topic_type) using urlSegments. I wanted these to show up on the sitemap, even though they're probably not super important.

Usually I wouldn't bother doing this if it was just these kinds of pages. But what if you have countless articles with manipulated urls that don't show in the sitemap? This is perfect example of why I did this for the future as well.

I've been trying to understand how to do these things for a while, so I hope it helps someone out.

EDIT: I have a download with all of my changes if anyone's interested in taking a look: http://clintonskakun.com/processwire-docs/posts/the-xml-sitemap-module-with-priority-and-changefreq-and-more/

Edited by ClintonSkakun
  • Like 1

Share this post


Link to post
Share on other sites

Thanks for posting this, these seem like some useful additions and some good insights on sitemap.xml too.

Share this post


Link to post
Share on other sites

jukooz asked me a while back how to stick the sitemaps into a template as he's using the multisite module and currently I guess that this module will pull the sitemap for EVERY site on an install...?

This should work on a per-site basis having it in a separate template file per site as a workaround for now, but pay attention to the comments please as there are things to change per-site - please also note that this is un-tested and largely just pulled from the module and tweaked for pasting into a template for use instead of the module:

EDIT: See attachment as the forum software tries to parse a URL in the code : sitemap.txt

I've also updated the module (see first post) to v1.0.3 to check if the page is viewable before including it in the sitemap - I noticed that it was incorrectly listing pages that had no template file... oops!

  • Like 1

Share this post


Link to post
Share on other sites

Hey,

is it possible to use this module with the LanguageLocalizedURL ?

I wanne make the site multilanguage like this:

www.url.de/de/testindeutsch/

www.url.de/en/testinenglish/

Would like to hear from you... Greets Jens alias DV-JF

Share this post


Link to post
Share on other sites

While I've not tried it, I would guess that this module does not collaborate with or accommodate the LanguageLocalizedURL module in any special way. Though Pete could say for sure. 

Share this post


Link to post
Share on other sites

Hey Ryan,

thx for answering...

Though Pete could say for sure. 

Hope so :)

Greets

Share this post


Link to post
Share on other sites

It doesn't accommodate it at the moment, no, but I may need this myself soon so if you can wait a week or two I may have an update (it's not at the top of my list at the moment but I might surprise myself and do it sooner ;)).

Thinking out loud, it needs:

  • Check if LanguageLocalizedURL is installed
  • Find the root page for each language
  • Generate a sitemap.xml page under each root page (it doesn't really generate a page in the database, it just outputs the sitemap if you request that URL)

I think that's it, but I need to get to grips with the LanguageLocalizedURL first.

Share this post


Link to post
Share on other sites

Pete the LLU module doesn't have separate trees, it uses a gateways page for each language in the root like "/de/" "/en/" with url segments to then get the page and switch user language.  The site structure is still all the same as without, you just use text language fields.

The parsing of the url happend automaticly through these gateway pages and it hooks into Page::path to change the url of the pages system wide. So if you do a echo $page->url you'll get the language url in the language you're currently in. Like /en/about-us/, /de/ueber-uns/. 

Have anyone tried yet if it doesn't already work?

Share this post


Link to post
Share on other sites

Soma just submitted a pull request and I merged it.

The 1.0.4 version should allow you to use this with the LanguageLocalizedURL module - let us know how you get on and thanks to Soma ;)

Share this post


Link to post
Share on other sites

Pete, thanks, but I already pulled another request, forgot to up the cache time.

Also to fully work it would also have to add the "language_published" check to see if language version of the page is really published. Will try to add that later.

Also wanted to add that I never use sitemaps for google and never will again (used to try long time ago, but it doesn't really help at all if you build the site carefully. It just eating time doing it and making sure everthing works still). It's not as easy as it first seems and can even be contra productive if not done carefully. Problem with this module as it is now, it will not find and list pages that may are added through urlSegments and I don't see a way to do it easy. Also it doesn't have weighting etc.

  • Like 1

Share this post


Link to post
Share on other sites

Thanks - merged it :)

The main reason I created it was because it gets a new site listed faster - plain and simple. That was my experience from sitemaps a few years ago at least so I hope it still happens, but it does seem to work from my experience.

I've never bothered with weighting or anything like that. Not bothered about URL segments either as for a start the crawler will find a link to that from a normal page on the site presumably, but my main goal was to get Google to crawl the page sooner than it normally would - and I think it will do this every time it looks at the sitemap and sees a new page (assuming it doesn't find it in the crawl already on its own by following links in the content).

Think of it this way - if you release a brand new site without telling Google, it will take until someone links to your site before Google is aware of its existence. On a small, personal site with no blog or comments that could be a loooong time, but the first thing I usually do is launch a site and submit it so I know it's done.

The argument against this is there are other ways of doing that as search engine companies often have pages where you can just type your domain in and they will search it sometime later, but I like to keep Google informed when I add new pages etc just in case it has a lapse in concentration and misses something :)

Share this post


Link to post
Share on other sites

just uped another pull request... :)

I think google doesn't parse the website if it has a sitemap.xml. Or has this changed? I found myself fixing others website by actually removing the sitemap form google.

Google is so fast nowadays it doesn't matter that much, as it won't be in index before index update. I could be wrong as things change all the time, but thats my personal experience and from readup. Also if you add a sitemap page to your site will help if you really worry about it. As with most seo things you have to take everything with a grain of salt at the end.

It's maybe ok if google doesn't come to your new site if it's new, but tests have shown even only 1 link to your site and google will parse it in 1 day if you have a nice structure.

Share this post


Link to post
Share on other sites

I used to post a link on my website to give my new client sites a poke. Work well. Or even in this forum.

  • Like 1

Share this post


Link to post
Share on other sites

Forgot to mention, I've pulled in the last of Soma's pull requests so this is now v1.0.5 to implement the added functionality he wrote as well as the fixes for the bugs he created :P;)

My fault for not testing :D

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By jploch
      Hey folks,
      for a module (a pagebuilder based on PageTable) I need to save some settings as JSON. The values are saved for each page table item (a pw page). It's working well, but I am looking for ways to improve the structure I have. As I'm not that experienced with JSON, maybe someone more experienced can take a look and tell me if my approach is good practice. 

      My goal is to make all the items accessible by page id, without looping over them (using objects instead of arrays):
      // access from template with pw page var $jsonObject->items->{$page}->cssClass; Her is an example of my JSON structure:
      { "items": { "3252": { "id": "3252", "cssClass": "pgrid-main", "breakpoints": { "base": { "css": { "grid-column-end": "auto", "grid-row-end": "auto", "grid-column-start": "auto", "grid-row-start": "auto", "align-self": "auto", "z-index": "auto", "padding-left": "60px", "padding-right": "60px", "padding-top": "60px", "padding-bottom": "60px", "background-color": "rgb(255, 255, 255)", "color": "rgb(0, 0, 0)" }, "size": "@media (min-width: 576px)", "name": "base" } } }, "3686": { "id": "3686", "cssClass": "test_global", "breakpoints": { "base": { "css": { "grid-column-end": "-1", "grid-row-end": "span 1", "grid-column-start": "1", "grid-row-start": "auto", "align-self": "auto", "z-index": "auto", "padding-left": "0px", "padding-right": "0px", "padding-top": "0px", "padding-bottom": "0px", "background-color": "rgba(0, 0, 0, 0)", "color": "rgb(0, 0, 0)" }, "size": "@media (min-width: 576px)", "name": "base" } } }, "3687": { "id": "3687", "cssClass": "block_editor-3687", "breakpoints": { "base": { "css": { "grid-column-end": "span 2", "grid-row-end": "span 1", "grid-column-start": "auto", "grid-row-start": "auto", "align-self": "auto", "z-index": "auto", "padding-left": "0px", "padding-right": "0px", "padding-top": "0px", "padding-bottom": "0px", "background-color": "rgba(0, 0, 0, 0)", "color": "rgb(0, 0, 0)" }, "size": "@media (min-width: 576px)", "name": "base" } } }, "3696": { "id": "3696", "cssClass": "block_editor-3696", "breakpoints": { "base": { "css": { "grid-column-end": "span 2", "grid-row-end": "span 1", "grid-column-start": "auto", "grid-row-start": "auto", "align-self": "auto", "z-index": "auto", "padding-left": "0px", "padding-right": "0px", "padding-top": "0px", "padding-bottom": "0px", "background-color": "rgba(0, 0, 0, 0)", "color": "rgb(0, 0, 0)" }, "size": "@media (min-width: 576px)", "name": "base" } } } }, "breakpointActive": "base", "breakpointActiveSize": "@media (min-width: 576px)" }  
    • By jploch
      Fieldtype Page Table Grid
      This is a sneak preview of a side project I've been working on for quite some time now. A lot of work and thought has gone into this, so I will most likely release this as a commercial module at some point in the near future. 

      As a designer (and developer) I get the appeal of a WYSIWYG editor. After playing around with some WYSIWYG page builder tools, I always felt something was wrong about them. So I decided to build my own PW version based on PageTable.

      Here is a small demo (using AdminThemeCanvas, but its working with other admin themes as well) :
      There is also a complete website that I built for a friend of mine using this module and some custom blocks.
      Concept
      This fieldtype shares a lot of features with PageTableExtended: it's also an extension of PageTable and renders the block templates in the backend and frontend (native PW templates and fields). You can also add your own css via module settings.
      The difference is, this fieldtype also gives you the ability to rearrange and resize elements in a visual way as well as enable inline editing for text, ckeditor and file fields. Similar (and promising) attempts have been made, but I wanted something based on native CSS grid instead of a CSS framework...so I built my own version. Most CSS frameworks are based on flexbox, which is great for layouting elements horizontally. With CSS grid, you can place elements horizontally and vertically, allowing for layouts that were not previously possible with CSS. Similar to webflow, this fieldtype uses javascript (in the backend) to let you manipulate CSS grid in a visual way to design fully responsive websites (or parts of them). It should still be possible to include a CSS framework if you like (just add the classes to your block markup and include the CSS via module settings).
      The CSS grid layout manipulations are saved in a single field as a JSON array and used to generate a dynamic stylesheet that you simply include in your main template (no inline styles). The styles are saved within the breakpoint you select and cascade down to smaller breakpoints. That means you can specify just the basic breakpoint and adjust other breakpoints if needed. The exception is the mobile breakpoint which will display everything in one column as a default (you can change the layout here too).
      The fieldtype also comes with an optional style panel to manipulate some additional CSS properties directly on the page. You can customize the panel or disable it completely from the module settings (and just use a CSS file that you include via module settings). The style panel is based on inputfields (nothing is saved to the database). This means that you just have to install the module and all fields are available to all blocks automatically (this can be customized). It also has the benefit that your installation is not flooded with fields; this module only installs one field.
      Don't want to give your customer all that power? Design features can be disabled for certain roles. The grid editor role can just edit the content and use the inline editing feature to edit content quickly. You can then also grant access individually to the style panel, resize or drag functionality.
      Features
      Blocks are just pages Blocks are defined by native PW templates and fields Manipulate CSS grid in a visual way to design fully responsive websites (or parts of them) Design features can be disabled for certain roles Inline editing of text, ckeditor and file fields The layout is 100% CSS grid (very small css file size) Simply drag and resize to manipulate grid items directly inside the backend Manipulate grid columns and rows directly on the page (use any number of columns you want) All style manipulations are saved as JSON and used to generate a dynamic stylesheet that you just include in your main template (no inline styles) Nested groups/grids (child pages of nested blocks are created under group parent) Global blocks work with page reference field (changes on one page, changes all blocks on all pages) Manual and auto placement of grid items Define custom icons for your blocks via native template settings (template -> advanced -> icon) Option to load lazysizes in the backend to enable lazy loading of assets with class lazyload Works with all default and ui-kit based admin themes If you have any questions or feedback, let me know.
    • By bernhard
      I built this module because I needed a versatile solution to replace tags and simple if-blocks in some E-Mails and PDF documents.
      If you only need to replace static tags (no if-conditions), then you can use default PW api and need no module:
      $str = "My favourite color is {color}."; $texttools = $sanitizer->getTextTools(); echo $texttools->populatePlaceholders($str, ['color' => 'red']); // output: My favourite color is red. Usage:
      See the two example Files in the folder /replacements

       
      Methods:
      replacementsTable()
      Renders an overview of all available replacements (see the example in the Module's config file:
       
      Create new Replacements:
      Simply copy the sample file and adopt to your needs.
       
      Download:
      https://gitlab.com/baumrock/RockReplacer
    • By bernhard
      DEPRECATED
      I'm using this module in several projects, but it will likely not see any updates in the future. I'm not happy with it and I'm looking for ways to develop better solutions. RockTabulator was my first try, but I'm also not 100% happy with that. The tabulator library is great, but my module implementation is not. I hope to get a good solution soon, but it will be a lot of work...
      ---
      Some of you might have followed the development of this module here: https://processwire.com/talk/topic/15524-previewdiscussion-rockdatatables/ . It is the successor of "RockDataTables" and requires RockFinder to get the data for the grid easily and efficiently. It uses the open source part of agGrid for grid rendering.
       
      WHY?
      ProcessWire is awesome for creating all kinds of custom backend applications, but where it is not so awesome in my opinion is when it comes to listing this data. Of course we have the built in page lister and we have ListerPro, but none of that solutions is capable of properly displaying large amounts of data, for example lists of revenues, aggregations, quick and easy sorts by the user, instant filter and those kind of features. RockGrid to the rescue 😉 
       
      Features/Highlights:
      100k+ rows Instant (client side) filter, search, sort (different sort based on data type, eg "lower/greater than" for numbers, "contains" for strings) extendable via plugins (available plugins at the moment: fullscreen, csv export, reload, batch-processing of data, column sum/statistics, row selection) all the agGrid features (cell renderers, cell styling, pagination, column grouping etc) vanilla javascript, backend and frontend support (though not all plugins are working on the frontend yet and I don't plan to support it as long as I don't need it myself)  
      Limitations:
      While there is an option to retrieve data via AJAX the actual processing of the grid (displaying, filtering, sorting) is done on the client side, meaning that you can get into troubles when handling really large datasets of several thousands of rows. agGrid should be one of the most performant grid options in the world (see the official example page with a 100k row example) and does a lot to prevent problems (such as virtual row rendering), but you should always have this limitation in mind as this is a major difference to the available lister options that do not have this limitation.
      Currently it only supports AdminThemeUikit and I don't plan to support any other admin theme.
       
      Download: https://gitlab.com/baumrock/FieldtypeRockGrid
      Installation: https://gitlab.com/baumrock/RockGrid/wikis/Installation
      Quikckstart: https://gitlab.com/baumrock/RockGrid/wikis/quickstart
      Further instructions: https://gitlab.com/baumrock/RockGrid/wikis/quickstart#further-instructions
      German Translation File: site--modules--fieldtyperockgrid--fieldtyperockgrid-module-php.json
      Changelog: https://gitlab.com/baumrock/FieldtypeRockGrid/raw/master/changelog.md
       
      Module status: alpha, License: MIT
      Note that every installation and uninstallation sends an anonymous google analytics event to my google analytics account. If you don't want that feel free to remove the appropriate lines of code before installation/uninstallation.
       
      Contribute:
      You can contribute to the development of this and other modules or just say thank you by
      testing, reporting issues and making PRs at gitlab liking this post buying me a drink: paypal.me/baumrock/5 liking my facebook page: facebook.com/baumrock hiring me for pw work: baumrock.com  
      Support: Please note that this module might not be as easy and plug&play as many other modules. It needs a good understanding of agGrid (and JavaScript in general) and it likely needs some looks into the code to get all the options. Please understand that I can not provide free support for every request here in the forum. I try to answer all questions that might also help others or that might improve the module but for individual requests I offer paid support (please contact me via PM).
       
      Use Cases / Examples:
      Colored grid cells, Icons, Links etc. The Grid also has a "batcher" feature built in that helps communicating with the server via AJAX and managing resource intensive tasks in batches:

      Filters, PW panel links and instant reload on panel close:

      You can combine the grid with a chart library like I did with the (outdated) RockDataTables module:

    • By Paul Greinke
      Hi there. I wrote a custom module for one of my projects. In fact I maybe want to use my module in other projects too. In order to be variable and customizable  I need to implement some custom hooks into my module. So I can afterwards hook into the my functions in order to modify them to match the needs of the new project.
      I tried simply defining functions with the '__' prefix. But that did not work. I'm imagining something like the following:
      <?php class MyClass { public function ___someFunction() { // Do something } } // ready.php $this->addHookBefore('MyClass::someFunction', function($event) { // some customization }); Is there a way to accomplish that? 
×
×
  • Create New...