Leaderboard

We encountered a performance problem with a custom search today, finding out that one particular database query lasted seconds. Digging deeper revealed it has to do with repeaters when there are enough of them. The site has pages for companies (about 1500 of them) each of which has repeater items attached. Most of them have one single item, total amount of repeater items being under 2000. This leads to about 3500 extra pages in the tree as a penalty for using repeaters (yes, there would have been other possible approaches as well). So, the amount of pages isn't anything massive. Now, when a find("parent=/companies/, title%=keyword") is executed, the repeater fieldtype catches the use of title field (used in the repeaters as well) and tries to ensure no repeater pages get returned by adding !has_parent=<repeaters-page-id> to the selector string. While this is logically right, the resulting SQL query becomes slow due to the extra join used to filter out repeater pages. With those 1500 + 1500 + 2000 = 5000 pages there are about 9000 rows in the pages_parents table (which is used for implementing has_parent selector). This isn't that much either, but it becomes a problem when there's a join between pages and pages_parents. I'm not saying it wouldn't be possible to optimize the query, but as I'm not certainly skilled enough to do it, I went down another road. And the solution is dead simple - actually it's possible to catch it by just looking at those bolded selector strings above . Ready? Ok, if we're searching for pages whose parent is /companies/ why should pages under /processwire/repeaters/ be explicitly filtered out? There's no way they'll match, so we'll just skip the extra repeater filter. And if you we're actually trying to find pages under /processwire/repeaters/ you didn't want the filter to kick in either (well, don't do that anyways, they're internals). Here's a little diff showing the change I made to fix this thing for us (there's a pull request on the way Ryan): $ git diff diff --git a/wire/modules/Fieldtype/FieldtypeRepeater/FieldtypeRepeater.module b/wire/modules/Fieldtype/FieldtypeRepeater/FieldtypeRepeater.module index 20bb52c..ab9db28 100644 --- a/wire/modules/Fieldtype/FieldtypeRepeater/FieldtypeRepeater.module +++ b/wire/modules/Fieldtype/FieldtypeRepeater/FieldtypeRepeater.module @@ -138,6 +138,10 @@ class FieldtypeRepeater extends Fieldtype implements ConfigurableModule { // ensure that the repeaters own queries work since they specify a templates_id if($name == 'templates_id' && in_array($selector->value, $templatesUsedByRepeaters)) $includeAll = true; + // optimization: if parent (or parent_id) is given, there's no need to explicitly exclude repeaters + // TODO: has_parent with values other than parents of repeaters-page could be catched as well + if($name == 'parent' || $name == 'parent_id') $includeAll = true; + if($includeAll) break; } } Another performance issue solved. Not a very common one I think, but still worth a fix. Edit: Ryan, no pull request this time. Didn't manage to do it right. Next time there'll be a branch before any changes, maybe that'll do it.

This looks like a great tool to create graphs with jQuery and Raphaël http://oesmith.github.com/morris.js/

Ok, one more Just used this to build a rather complex expression http://gskinner.com/RegExr/ Easy peasy!

Nik, this has been pushed to the dev branch: https://github.com/ryancramerdesign/ProcessWire/commit/eecd862a6ecf90f2c6b1c173c4d129c5152a31e7 In addition adding in the isGuest() check and the has_parent check, I changed the line you added to be this: // optimization: if parent, parent_id, template, or templates_id is given, and an equals '=' operator is used, // there's no need to explicitly exclude repeaters since the parent and/or template is specific if(in_array($name, array('parent', 'parent_id', 'template', 'templates_id')) && $selector->operator == '=') $includeAll = true; Basically added in 'template' and 'templates_id' since those are also specific enough so as to warrant exclusion of the filter. And added an extra check that the operator was "=", since if they used something like not equals (!=), or greater than/less than (>< <= >=), that would still warrant inclusion of the filter. But just in case I got any of this wrong, I've kept it in the dev branch for further testing.

What about using InputfieldForm. ... $form = $this->modules->get("InputfieldForm"); $form->attr('action', $url); $form->attr('method', 'post'); $form->attr('id', 'myform'); ... $form->attr('value', $table->render());

We're probably only a few days from bringing in the new jQuery into the master branch. So far I've only come across the one issue I mentioned before and nothing else has turned up yet. If that's the only issue, I'd consider that fairly minor (easy to fix). So no reason to delay further on bringing in newer versions of jQuery and jQuery UI.

Namespacing not supported by PHP 5.2.x which currently represents a large base of ProcessWire users. Probably by ProcessWire 2.4 we'll require PHP 5.3 at which time we can namespace.

Thanks Nik, this is a good find and fix. I am updating the code in the dev branch to include the line you added (no need for a pull request here). Using the same logic, I've also added a couple more things: 1. If wire('user')->isGuest(); then there's no need to exclude repeater pages because the user already doesn't have access to them. Meaning, they aren't going to show up in the results anyway, so no need to have the extra filter. 2. if 'has_parent' is specified with a value other than homepage, we don't need to have the filter either. Yes, they could still be included if you did a $pages->find('has_parent=/processwire/, title*=something'); but I think that's okay and maybe still an expected behavior. Though I think this is a rare query anyway. Another related idea is that I could bypass the "has_parent!=n" filter entirely and just have it exclude by template. For example, rather than adding "has_parent!=n", it could add "templates_id!=1|2|3" (where 1, 2, 3 are templates used internally by repeaters), which would probably be more efficient than the join that comes from has_parent. Last thing I want to mention is that the "%=" uses a MySQL "LIKE" which is non-indexed and inherently slow. You can get better performance by using "*=", which uses a fulltext index. Though the difference in speed isn't noticeable on smaller sites, but might make a difference in your case.

I know that I pushed this update yesterday right when I wrote my message. But look what was waiting for me in my shell window when I just now checked: I typed in the push and just assumed it went through yesterday, as it always does. Then moved onto the next thing. For some reason GitHub is prompting me for a username (didn't expect that). Anyway, I just typed it in now so hopefully it should be online finally.

You've already got my two common patterns: the basic profile and the blog profile. processwire.com and modules.processwire.com are both built in a manner very similar to the basic profile (seeing as they essentially are the basic profile). Though I did modify them a bit: rather than using separate head/foot includes, they include a "main.inc" which has the entire markup template in it. main.inc also includes sidebar.inc to cover the navigation and widgets that get displayed in the sidebar on any page.

It's possible to use <repeater-name>.<field-in-repeater> as a field name. For example: $matches = $pages->find("title|my_repeater.my_field~=$q");

That's good to hear, thanks Ryan!

One way would be to use a separate file for the view. I tried not to use any html in my .module-file and instead ended up with a TemplateFile instance. The view is created during init() and different variables are populated depending on the given input. Then view's render() method is called to output it all. Hmm, the explanation isn't very clear, but if this was what you were trying to achieve then you might want to take a look at my Selector Test module's implementation at https://github.com/n...essSelectorTest. The lines with "$this->view" in the module-file and the view itself (view.php) should give an idea of what I was trying to say above. If you're building a module, even a process one, then this could be helpful. If this came up somewhere else, you'll probably get much better suggestions later on from others.

Thanks, great alternative to jqPlot.

I just got this error too, but found that it persisted even after I double-checked my assets and config.php permissions. I had installed the site using the ProcessWire Blank Profile, so figured I'd try it without that. Not sure why, but it did the trick. Removing the current install and reinsalling while sticking with the default site cleared up the issue.

Hey Guys - this is awesome! I've learned so much and really simplified things. I'm totally blown away with this approach and just how powerful and flexible ProcessWire is! Thanks for all your help and advice!

My suggestion was simply to create a "projects" page, and have all the projects as it's children: -projects (template "projects") --projectA (template "project") --projectB (template "project") --projectC (template "project") Template "projects" doesn't need any field, or even a template file associated. It exists only for holding the children. (in the "family tab" of template editing, choose "project" as the only allowed template for children) Template "project" has the fields "title", "budget" and "description". Doesn't need a template file if you don't want them to have their own URL. (in the "family tab" check "no" for "can this template be used for new pages?") On your "construction projects" page template do something very similar to the above: <? foreach($pages->get("/projects/")->children as $project): ?> <h3>project name</h3> <p><?=$project->title ?></p> <h3>project budget</h3> <p><?=$project->budjet ?></p> <h3>project description</h3> <p><?=$project->description ?></p> <? endforeach; ?> The editor just have to create a new page under "projects" for each project. This is the most common way of doing this kind of thing on PW.

You would create a new field using type 'Repeater', not a template. Though technically you are correct in that a repeater field does create a template to group them, behind-the-scenes. But this is not something you usually need to think about when dealing with repeater fields.

Teppo, the dev branch now now has switched from extension detection to type detection per your suggestion and code. https://github.com/ryancramerdesign/ProcessWire/blob/dev/wire/core/ImageSizer.php Thanks, Ryan