nik

That's more or less what I though you were trying to do - so I'm thinking it was me who wasn't clear enough then.

Because of memory limitations you can't foreach through the whole 3000 page resultset in one go, just like you said in the first place. Instead you can loop through the very same 3000 pages, but in 500 page pieces.

In a hurry, again. Not sure about the syntax and definitely not tested, but you'll get the idea:

$start = 0;
$limit = 500;
do {
 // replace "..." with your actual selector
 $results = $pages->find("..., start=$start, limit=$limit");

 foreach ($results as $resultPage) {
   // do you magic here, collect the results that match to another PageArray maybe?
 }

 // free some memory
 $pages->uncacheAll();

 // advance to next set
 $start = $start + $limit;

} while (count($results) > $limit);

Hope this helps. There could be another variable to make sure the do-while doesn't get crazy, but I left that out for now.

I'm not good with images, but let's try.

So, in your main content, you're trying to display a profile page for the user whose name is given in the first urlSegment, am I correct? This assumption is based on the code above.

Assuming I was right so far, how about these?

1) You've got a page, say /user-profile/ that's using a template with urlSegments in use? The name of the page doesn't matter, but there should be one.

2) And the code you've pasted above is in that templates template file?

3) And you've got /user-profile/thomas in your browser when you want to see profile for user "thomas"?

You should add a check to see if the profile was actually found before accessing any of its properties, like this:

if ($profile->id) {
// profile with the given user name was found, we're good to go

// your code here

} else {
// profile could not be found, display en error or something
}

Can't help much with the image stuff itself, but just trying to dig out some more information for the ones who can. And possibly the problem is somewhere else and you'll get on track by trying to answer my questions.

ProcessWire removes duplicates grouping results by page id thus giving only distinct pages as a result. So no need for a distinct there.

That 3000 records shouldn't include any duplicates assuming it's a result from a single $pages->find() call. Solution would be to add more restricting selectors to the find itself.

If that's not possible then pagination is the way to go. Add start and limit selectors to get records from 0-999 (start=0, limit=1000) and loop that increasing start by 1000 on every iteration until you've got less than 1000 result rows returned.

This way you'll end up with less Page objects in memory at the same time. Something like 500-1000 rows at a time should be fine, but that depends on how heavy data you've got there. (Actually, only autojoin fields and the fields you're accessing do count here.) You may need to call $pages->uncacheAll() after every iteration to flush the previous iteration's Page objects from memory.

No example at the moment, sorry, got to go for now.

Actually.. If you'd hook Session::authenticate() (as I already suggested) before calling Session::login(), you'd be able to do whatever check necessary for authenticating the user - or no check at all if it's all covered already. This way all the magic in login() would be executed and a real session with everything in order would exists. You'd probably want to call $users->setCurrentUser() on top of that as well to make the login effective on this very request.

And you didn't have to make any more permanent hook to Session::authenticate(), just a one-timer before your own login() call. A before hook with $hookEvent->replace set to true.

Forgive me everyone if I'm just hallucinating here, still on painkillers...

I had this exact problem last week when fiddling around with login stuff. Didn't have the time to dig any deeper then, but now I do.

Could it be because of sessionChallenge-setting? Looks like a challenge-cookie is being set only during the normal login() method. Challenge is checked by isValidSession() during Session::construct() and if it doesn't match user will be immediately logged out. And as challenge MD5 generation uses logged in user's id, it just won't match when $session->_user_id has been tampered with. So sessionChallenge does offer extra security like it was supposed to.

Thus, only setting $session->_user_id isn't enough when sessionChallenge is set to true in site/config.php. @apeisa, try it out after setting sessionChallenge to false.

And there's also sessionFingerprint with similar effects, it seems. Hmm, no, this wouldn't actually be a problem I think. There is a session for guest user as well, so fingerprint does exists and is valid. Anyways, I didn't try this out yet, but I think this shouldn't be a problem after all.

It wouldn't be wise to duplicate these things to own code, so looks like there's a need for some addition somewhere to make it possible to log another user in via API without knowing the password. More permanently than for the current request only that is. But is there a security concern lurking somewhere I'm not able to see from here?

Thanks Ryan! Seems to be working here as well. Just had a quick test with those selectors mentioned above, nothing more than that though. Will let you know if anything comes up after trying this out in the actual scenario where there are a couple of more selectors involved too.

That "LEFT JOIN ... WHERE xx IS NULL" -thingie is something I never seem to get right at once. But I'm glad you did.

For the access rights bit, can you tell me which line in my review you are referring to so I can change it? Thanks!

Text says:

If I click on the ‘Settings’ link at the top of the post, I can further modify things such as access rights for the post.

The screenshot below the text is "ProcessWire-15.jpg" (see the text under section "Who can access this page?" in the picture).

Just read it quickly through and it's looking good!

I'm on a mobile here, but still a couple of small things on the review: access rights are defined with the page's template (says so in your screenshot too ) and can't be modified editing the page itself, unless the template is changed to another.

And then the modules are only checked from your local installation by pressing that button. There is a Module Manager module by Soma though that has the function you we're describing (and much more).

Still, great work you've done here!

We've got pages and tags. Pages use template "company" and the template has a Page-field called "tags" (input method is checkboxes, template restricted to "tag"). Like this:

c1 (tags: "t1")

c2 (tags: "t1", "t2")

c3 (tags: "t2", "t3")

c4 (tags: none)

Finding companies having (at least) one of the tags t2 or t3 would be:

$pages->find("template=company, tags.title=t2|t3");

This returns pages c2 and c3 as expected. So far so good.

But when we're trying to find pages without certain tag(s) selected, results aren't what we expected them to be. Like this:

$pages->find("template=company, tags.title!=t1");

This gives again pages c2 and c3. There's obviously at least two things wrong with the resultset:

- page c2 does have tag t1 --> should not be included

- page c4 does not have tag t1 --> should be included

And finally, what we were actually trying to achieve was finding pages with tag t3 but without tag t2:

$pages->find("template=company, tags.title=t3, tags.title!=t2");

And this gives page c3, which is more or less understandable given the previous example of != not working the way it's supposed to be. Still it's wrong.

It seems to me that != operator does not work correctly with fields having more (or less!) than one row per page in their database table. This just isn't covered with the way JOINs are used at the moment (and once again I'm not skilled enough to say how it should be done).

I'm going to be away from the forums for a few days but I'm sure teppo will take over the conversation if there's any details missing regarding the way we're trying to do this. And yes, we're aware using filter() for the != -part would probably solve this, partly. Still the selectors don't work as expected and pagination would have to be handled somehow.

(Edit: fixed a typo 'page t3' -> 'page c3' so that it wouldn't be misleading)

@ryan: On autojoin being a requirement for sorting you said:

This used to be the case but isn't anymore.

I guess the help text in the advanced section of field edit should be corrected then? Currently it's misleading.

If checked, the data for this field will be loaded with every instance of the page, regardless of whether it's used at the time. If unchecked, the data will be loaded on-demand, and only when the field is specifically accessed. Enabling autojoin also allows the field to be used as a key for sorting pages.

Makes sense, must agree on everything you're saying.

In ProcessWire, any page field can be represented in a more basic type like this. This is what enables a high amount of portability with the data for export/import and web service situations. But that doesn't mean it's recommended API usage. For API usage, you should treat it as the type you specified in the field settings, be that a PageArray or a Page.

ProcessWire really is great with export/import like you've intended. It's kind of a side effect that some a bit unusual things work. My habit of reading the source code doesn't always (usually even) lead to right conclusions and it would probably be better not to say everything aloud. Maybe I should just start using Soma's cheatsheet more instead of that.

I was just surprised to see that assign really works that way there as I never would've tried that out myself.

I'm not aware of anywhere else that this type of overloading would be possible. I also don't want to recommend these abstractions for regular API usage. This is purely for the benefit of behind-the-scenes portability and automation. I don't think it's so applicable outside of the context of a $page->set() situation.

Couldn't agree more on this now that I understand it wasn't for convenience but for something totally different. Still, it would be great if there was a way to check which features aren't supposed to be used via API but are reserved for internal use. Then it would be 'at your own risk' to use something like that.

Or is it actually so that the cheatsheet tries to cover more or less everything that is safe and recommended to use? As I said before, I haven't used that enough, clearly. I'm trying to follow recommendations myself whenever possible, but it isn't always clear where to find the recommendation (other than asking here, but somehow I'm not so used to doing that every time, yet).

Documenting internal methods and usages in the comments has been also discussed before and that's a good way to achieve what I'm after here. It doesn't make things clear to someone who just happens to try some syntax that's not meant to be used - but what does, apart from disabling things somehow (no, I'm not suggesting that at all).

This is enough for now, I think. ProcessWire works great and is great, and will be surprising me every now and then for a very long time. And I'm very happy with the way things are even if I may whine a little sometimes.

You wouldn't happen to have something like this executed on those 3rd level pages before the foreach?

$page->parent->name = 'home';

After that line everything works exactly the way you described. Don't know why you'd do that on purpose, but for example a missing second equals sign in an if statement would explain this.

Looks like you're right martind, sorry for the confusion.

At least I got confused with this, so let's make things clear in case someone else stumbled across this sometime later:

When dealing with FieldtypePage (a field of type Page in Setup -> Fields), it's ok to assign a single Page object to the field and things just work. If the field is set to be dereferenced as PageArray (edit the field -> details), the assigned page is added to the array. Otherwise the given Page object is naturally assigned to the field (field is set to be dereferenced as Page).

On the other hand (this is what got me confused in the first place), when dealing with plain PageArray simple assign of a Page does not work, but add() method has to be used. It's the FieldtypePage that adds the magic.

While it's great to have such convenience added here and there, one would expect things to work the same way where applicable. At least I would. This is actually why I was so keen on making sort, start and limit with WireArrays work more like they work when using selectors. I think there are some other cases as well where this kind of alignment could be a good thing.

..is there a more elegant possibility to check against existing entries in the page-field?

Yes there is, method has(). Assuming you're trying to remove a page from a page field if it is there already and otherwise add it there:

if($user->abos->has($p)) {
   $user->abos->remove($p);
} else {
   $user->abos->add($p);
}

Notice also that $user->abos is a PageArray while $p is a Page, so plain assign wont work. You'd have to use add() method as in my example above.

Antti: at the moment, yes. It's just that I've got to find a way to optimize repeater performance or create a module before I can post anything and keep the ratio up high .

On the topic (or a bit closer to it): I'd like to see a hot-air balloon with ProcessWire logo up in the sky. Eye-catching. Any balloon hobbyists around?

There's also Session::authenticate(User $user, $pass) available for hooking. Session::login calls that to check the password, so overriding that is probably exactly what you're looking for.

(Edit: This just got deprecated , but I'll still leave it here - see "Part 2").

Go and see Setup -> Fields -> body. What's the type of the field?

I'm guessing it's "Textarea" as there doesn't seem to be fields for different languages in the database. That kind of selector would work only for fields of type *Language, "TextareaLanguage" in this case most likely.

You can change the type safely, but still after that for the selector to work in a reasonable way you'd have to input some content in the very language you're using in your search. But that was actually what you must have been trying to do anyway.

(..a few hours later..)

Part 2

Had another idea a bit later and while the earlier explanation could have been right, there's another possibility as well. And I think this is much more probable than the first one as I can't understand why you'd be using a selector like that if you didn't already have actual multilingual data in there - stupid me.

So now I'm thinking 1029 is the id of you default language (see Setup -> Languages -> default - what's the value of id parameter?). That special kind of selector would only work for languages other than the default one as its field is named just "data", not "data1029" for instance. So when the language is set to default ($user->language->name == 'default'), you'd have to use a regular selector without the ".data{$user->language}" part (or just ".data" without the id, I think).

Next part will follow in the morning - hope not.

So you get whatever is in markup/layouts/{$page->layout}.php twice in your final html? If so, then there has to be another inclusion of markup/index.php somewhere. Or a loop in page-listing.php so that the include line gets executed twice. Or a loop in markup/index.php so that the layout gets included twice.

But I'm a bit confused: you're saying you wouldn't have found the issue without the breakpoint. So does that mean the html output is ok, and nothing is duplicated there? Then you wouldn't have an actual problem, other than a crazy debugger maybe . If the layout template was actually executed twice you'd definitely have the output twice in your html. Just checking I've understood you right.

Hmm, there's "500 (Internal Server Error)" from the server side (Ajax POST in your first screenshot). That's where the first problem must be. Anything in the error logs?

It's defined in php.ini (or .htaccess) using "post_max_size". And "upload_max_filesize" must be enough also.

Hmm, I wonder why is it "post_max_size" that's used here anyway? Maybe smaller of those two should be reported out?

See: http://processwire.c...fly/#entry16017

Now you just need to get that "arrayToCSV => false"-option down to MarkupPagerNav =).

If you're calling $somePageArray->renderPager() yourself, you can just pass options array as an argument. But if you're using ProcessPageType::renderList() (reading minds here...) then it would be another thing to ask Ryan to modify. So maybe not renderList($selector) but renderList($selector, $pagerOptions=null) and $pagerOptions as an argument to render() here: https://github.com/ryancramerdesign/ProcessWire/blob/master/wire/modules/Process/ProcessPageType/ProcessPageType.module#L119

Or something like that, no time to try that out at the moment.

Looks like ProcessPageType derived modules are a bit special. In ProcessPageType::init() $this->pages is initialized to whatever is found from the fuel with the page name. Page name being 'users' or 'roles' or 'permissions' works nicely as those keys exist in the fuel. There is a fallback to plain Pages object on the next line, but that works only so far, as you've noted. ProcessPageType::renderList() doesn't check if $this->pages is of type PagesType as is done in init(), but just tries to call getTemplate (which is defined in PagesType class). And as plain Pages object isn't of PagesType, this fails.

But even though I found what happens, I'm not sure why is it so. Ryan may be the only one to say whether it's something to be fixed in ProcessPageType::renderList() for instance or is this just The Wrong Way To Go in the first place. I'd say there's just a little fix needed in the beginning of ProcessPageType::renderList() but it could be something else as well.

If you want to take this road before someone's able to give you more definitive answers, you can proceed by renaming your page to 'users' (obviously not under access then, but setup for example) - that'll do the trick. And I mean *name*, title doesn't matter.

No comments on the code itself at the moment (looking good though!), but the texts could be translatable right from the beginning don't you think? And of course that hard coded 'paakayttaja' into a setting, but that one you already mentioned earlier.

I think we'll have to take a detailed look into this someday soon at work, will be used here as well for sure. Nice module!

You sure are fast. And it was expected you came up with other similar optimizations as well.

Changes are looking good and performance gain is what I hoped for: using the dev branch cut the time on that single query from about 3 seconds down to less than 20 milliseconds. And the resulting data is the same.

Another related idea is that I could bypass the "has_parent!=n" filter entirely and just have it exclude by template. For example, rather than adding "has_parent!=n", it could add "templates_id!=1|2|3" (where 1, 2, 3 are templates used internally by repeaters), which would probably be more efficient than the join that comes from has_parent.

+1 for this! As template_id is a native field in pages, I'm sure that would perform much better just because of that. In addition that change would boost this kind of queries even more as MySQL would be able to drop more rows based on where conditions. So go for it.

Actually all of this has revealed a performance issue with "has_parent!=n". That still performs poorly when there are lots of pages, especially in a deep structure resulting in lots of rows in pages_parents and thus large join between pages and pages_parents. Luckily that particular type of selector isn't one of the most used. Still possibly worth thinking if there was something that could be done. (Edit: also without the negation - I'll have to look into this more!)

Last thing I want to mention is that the "%=" uses a MySQL "LIKE" which is non-indexed and inherently slow. You can get better performance by using "*=", which uses a fulltext index. Though the difference in speed isn't noticeable on smaller sites, but might make a difference in your case.

This is what we suspected ourselves at first also as the query in question had several fields using "%=". But it turned out that dropping all of those from the query didn't help at all. And those tests I mentioned above had these "LIKE"-operators in place again, so no problem there. I guess we need more data to hit performance issues with "LIKE".

And added an extra check that the operator was "=", since if they used something like not equals (!=), or greater than/less than (>< <= >=), that would still warrant inclusion of the filter. But just in case I got any of this wrong, I've kept it in the dev branch for further testing.

Yes, this operator thing popped into my mind also when driving home from work. Didn't see that coming at first, good thing you did.

By the way, testing these kind of things needs MySQL query cache to be turned off as it screws up results when same exact queries are repeated. I did a little addition like this to wire/core/DatabaseQuerySelect.php during my tests:

$ git diff
diff --git a/wire/core/DatabaseQuerySelect.php b/wire/core/DatabaseQuerySelect.php
index b9cbcee..4f82d2d 100644
--- a/wire/core/DatabaseQuerySelect.php
+++ b/wire/core/DatabaseQuerySelect.php
@@ -98,6 +98,8 @@ class DatabaseQuerySelect extends DatabaseQuery {
}
if(!$sql) $sql = "SELECT ";

+ $sql .= "SQL_NO_CACHE ";
+
foreach($select as $s) $sql .= "$s,";
$sql = rtrim($sql, ",") . " ";
return $sql;

This doesn't catch every single query but vast majority of them. I was wondering if a config variable could be added so that this SQL_NO_CACHE addition could be controlled from site/config.php?

It was also helpful to add "Queries"-part of wire/templates-admin/debug.inc to the very template that had the problem. Maybe a little module should be written to display that debug data on every page when debug flag is on. Actually there are other things I'd also like to see in the bottom of every page when developing things - so yet another entry on my growing modules-to-do-list.

It's possible to use <repeater-name>.<field-in-repeater> as a field name. For example:

$matches = $pages->find("title|my_repeater.my_field~=$q");