Guy Verville

PHP generator class and $pages->find()

Recommended Posts

First of all, I'm not an expert on PHP. I recently read about generators and I understand their usefulness in avoiding loading a set of objects into an array to the point of saturating the memory.

The $pages->find() call is known to be greedy (and slow) when it comes to processing large amounts of pages, because it loads all objects into memory.

Is there a way to use a generator to avoid this problem? Is there a workaround? I know that $pages->findMany() exists, but it is also called greedy.

See https://secure.php.net/manual/en/language.generators.php

Translated with www.DeepL.com/Translator

Share this post


Link to post
Share on other sites

Generators are PHP 5.5+ so it won‘t fit for ProcessWire, which is still running down to PHP 5.3. But the new $pages->findMany() works similarly to how generators work.

  • Like 2

Share this post


Link to post
Share on other sites

I was under the impression that findMany() was loading everything into memory. Maybe I'm wrong...  I will do some tests, thanks.

Share this post


Link to post
Share on other sites

The reason for adding findMany() was to not load all pages into memory like find() does. It does so by harnessing PHP iterators to load batches of pages within a foreach loop and discard already computed batches to free up memory again.

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By John W.
      SYNOPSIS
      A little guide to generating an sitemap.xml using (I believe) a script Ryan originally wrote with the addition of being able to optionally exclude child pages from being output in the sitemap.xml file.
      I was looking back on a small project today where I was using a php script to generate an xml file, I believe the original was written by Ryan. Anyway, I needed a quick fix for the script to allow me to optionally exclude children of pages from being included in the sitemap.xml output.
      OVERVIEW
      A good example of this is a site where if you visit /minutes/ a page displays a list of board meetings which includes a title,  date, description and link to download the .pdf file.
      I have a template called minutes and a template called minutes-document. The first page, minutes, when loaded via /minutes/ simply grabs all of its child pages and outputs the name, description and actual path of an uploaded .pdf file for a visitor to download.
      In my back-end I have the template MINUTES and MINUTES-DOCUMENT. Thus:


      So, basically, their employee can login, hover over minutes, click new, then create a new (child) record and name it the date of the meeting e.g. June 3rd, 2016 :

       
      ---------------------------
      OPTIONALLY EXCLUDING CHILDREN - SETUP
      Outputting the sitemap.xml and optionally excluding children that belong to a template.
      The setup of the original script is as follows:
      1. Save the file to the templates folder as sitemap.xml.php
      2. Create a template called sitemap-xml and use the sitemap.xml.php file.
      3. Create a page called sitemap.xml using the sitemap-xml template
       
      Now, with that done you will need to make only a couple of slight modifications that will allow the script to exclude children of a template from output to the sitemap.xml
      1. Create a new checkbox field and name it:   sitemap_exclude_children
      2. Add the field to a template that you want to control whether the children are included/excluded from the sitemap. In my example I added it to my "minutes" template.
      3. Next, go to a page that uses a template with the field you added above. In my case, "MINUTES"
      4. Enable the checkbox to exclude children, leave it unchecked to include children.
      For example, in my MINUTES page I enabled the checkbox and now when /sitemap.xml is loaded the children for the MINUTES do not appear in the file.

       
      A SIMPLE CONDITIONAL TO CHECK THE "sitemap_exclude_children" VALUE
      This was a pretty easy modification to an existing script, adding only one line. I just figure there may be others out there using this script with the same needs.
      I simply inserted the if condition as the first line in the function:
      function renderSitemapChildren(Page $page) { if($page->sitemap_exclude_children) return ""; ... ... ...  
      THE FULL SCRIPT WITH MODIFICATION
      <?php /** * ProcessWire Template to power a sitemap.xml * * 1. Copy this file to /site/templates/sitemap-xml.php * 2. Add the new template from the admin. * Under the "URLs" section, set it to NOT use trailing slashes. * 3. Create a new page at the root level, use your sitemap-xml template * and name the page "sitemap.xml". * * Note: hidden pages (and their children) are excluded from the sitemap. * If you have hidden pages that you want to be included, you can do so * by specifying the ID or path to them in an array sent to the * renderSiteMapXML() method at the bottom of this file. For instance: * * echo renderSiteMapXML(array('/hidden/page/', '/another/hidden/page/')); * * patch to prevent pages from including children in the sitemap when a field is checked / johnwarrenllc.com * 1. create a checkbox field named sitemap_exclude_children * 2. add the field to the parent template(s) you plan to use * 3. when a new page is create with this template, checking the field will prevent its children from being included in the sitemap.xml output */ function renderSitemapPage(Page $page) { return "\n<url>" . "\n\t<loc>" . $page->httpUrl . "</loc>" . "\n\t<lastmod>" . date("Y-m-d", $page->modified) . "</lastmod>" . "\n</url>"; } function renderSitemapChildren(Page $page) { if($page->sitemap_exclude_children) return ""; /* Aded to exclude CHILDREN if field is checked */ $out = ''; $newParents = new PageArray(); $children = $page->children; foreach($children as $child) { $out .= renderSitemapPage($child); if($child->numChildren) $newParents->add($child); else wire('pages')->uncache($child); } foreach($newParents as $newParent) { $out .= renderSitemapChildren($newParent); wire('pages')->uncache($newParent); } return $out; } function renderSitemapXML(array $paths = array()) { $out = '<?xml version="1.0" encoding="UTF-8"?>' . "\n" . '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'; array_unshift($paths, '/'); // prepend homepage foreach($paths as $path) { $page = wire('pages')->get($path); if(!$page->id) continue; $out .= renderSitemapPage($page); if($page->numChildren) { $out .= renderSitemapChildren($page); } } $out .= "\n</urlset>"; return $out; } header("Content-Type: text/xml"); echo renderSitemapXML(); // Example: echo renderSitemapXML(array('/hidden/page/'));  
      In conclusion, I have used a couple different processwire sitemap generating modules. But for my needs, the above script is fast and easy to setup/modify.
      - Thanks
       
    • By Robin S
      Password Generator
      Adds a password generator to InputfieldPassword.

       
      Usage
      Install the Password Generator module.
      Now any InputfieldPassword has a password generation feature. The settings for the generator are taken automatically from the settings* of the password field.
      *Settings not supported by the generator:
      Complexify: but generated passwords should still satisfy complexify settings in the recommended range. Banned words: but the generated passwords are random strings so actual words are unlikely to occur.  
      https://modules.processwire.com/modules/password-generator/
      https://github.com/Toutouwai/PasswordGenerator
    • By enricob
      Hi,
      I was reading a very interesting post in this forum (https://processwire.com/talk/topic/7166-wp-tavern-article-and-processwire-themes/page-4), about PW and its popularity compared to other cms like Wordpress.
      The key point that has been discussing is: how can PW be more attractive to non developers (ie designers)?
      (ok here another question arises "Do we want that PW reach an audience bigger that developers only?" but this is another topic...)
      Wordpress for example has a theming system that certainly is very attractive and quite easy to use, and I noticed someone in the community is already working on a module that could do some steps in that direction.
      That could be great but there's another thing that in my opinion could help very much: an automatic template file generator.
      Let me explain...
      I think that one of the things that scares a person which is not a developer the first time he uses PW, is the fact that after creating some fields and a template that contains these fields from the administration panel, he has to create a new empty file named as the template and put it inside the templates folder.
      Obviously from a developer point of view this is not a big problem, I go searching the docs and try to find what to put in this file, but from the point of view of a person that has no the developer mental scheme that thing could be scary at the beginning, and I imagine that if this person is trying out a lot of cms to find out which is better for him, he could easily think "ah dawn this is too complicated for me!".
      Ok, so what if I can, after creating a template, press a button and let the system create the template files for me? This could be great!
      I can now go to PW tree page, add a page with the template I created before and when I navigate the site I have already a page that is working, and it also outputs the fields that I filled!
      Then, I go editing the template file that has been created automatically and change the html as I prefer, but I don't have to bother about php code at all...
      The next step is thinking about a small wizard that create the file when I press the generator button: for example I have to choose if I am creating a simple detail template or a list template. In case of the list maybe we need the user to enter some filter parameters to generate the array list (template type, parent, etc.) and so on...
      So, this is not a trivial task but maybe not even impossible.
      In the future you can also think about a way to make this module useful even for developers, for example creating some template for outputting the html of the template files.... But maybe I am going too far
      So basically I think it could be very interesting to create a module that adds this functionality to PW...
      What do you think?
    • By gebeer
      Hi all,
      today I stumbled upon yeoman and went searching for a PW generator.
      Larry Botha mentioned in a side not in his thread about MVC template approach that he has put a yeoman generator for PW together. I test installed it and it is working great.
      Pretty fascinating how quick we can setup a PW project with yeoman. This definitely can be a great timesaver.
      Larry's generator sets up a pretty opinionated PW install with his MVC structured fixate template and KSS etc.
      It would be great if we could have an unopinionated version that gives us a clean PW install. We could add options to the generator for installing a choice of boilerplate templates like the blank template, Foundation, Bootstrap, Blog etc.
      Just wanted to get your opinion on this. If any of you are interested, we could pull something together.
      Cheers
      Gerhard