BillH

Ah, sorry, you probably want: $traders = $this->pages->find($selector); Regarding the sorting, very good points from @Jan Romero. I'd add that it's not necessarily the case that sorting by relevance would produce results that are particularly useful for users. It'd depend on the nature of the data and so on. If you want to read how it's done, it's described on https://dev.mysql.com/doc/internals/en/full-text-search.html, but it'd be a major project in itself figuring it all out! And one suggestion: if you're not anticipating too many results, it can take an acceptable amount of time to produce a results array, loop through the array adding a sort criterion (such as a relevance score), and then sorting by that criterion.

Some good suggestions from @Jan Romero. If you're still interested in exploring the ProFields Table approach, there is actually a possible way of handling the files. Make a template (without a file) to hold files, and load the files, one per page. Then use a Page type in the Table to hold a link to the relevant page. The added complexity of this approach might make it unsuitable, but you may be able to think of a good way of arranging things. And on the plus side, you can easily add extra data fields to the files (dates, source, etc) if that's useful.

I've no definitive answer, but my thoughts are these. The data appears to have a clear hierarchy, so if there'll be no complications you haven't mentioned, using either standard pages or repeaters or tables would all be OK as data structures. When it comes to building templates, all three approaches are more or less as easy as each other. The approach you choose should thus probably be governed mainly by making things as easy as possible for users. For the pages on which items are displayed in tables (Netti in your example), using a repeater would get everything onto one page, thus matching the site and making navigation in the backend significantly easier. Using a ProFields Table would have the same benefits as a repeater, and additionally would give a really clear, neat, compact result. This might well be the nicest option (so long as there's a small amount of budget available to cover the licence).

Just my personal view, but PW isn't an MVC framework, so trying to make it behave like one doesn't really sound like a good plan. My guess is that you'd end up making things very difficult for little benefit. I'd suggest going about projects in a PW-like way unless there's a particular reason not to. For testing, there are lots of tools out there, e.g. https://www.cypress.io/, and you might find this module useful: https://modules.processwire.com/modules/process-nette-tester/ And I'd strongly recommend this module: https://modules.processwire.com/modules/tracy-debugger/

I'd start in findTraders() by changing the fourth-from-last line to: $traders = $pages->find($selector); Then it'd be worth reviewing https://processwire.com/docs/selectors/#sort. In particular note the advice in the section about what happens if you don't specify a sort: "it is generally a good idea to include a sort=property when using $pages->find()". At least it'd be worth trying with sorting in all cases, which could narrow down the issue. Another possible place where things could go wrong is the loop that renders the results, represented by "..." in your last post. But if it's a simple foreach loop, it should be OK. By the way, when debugging searches I find that printing the result array, which is a PW PageArray, can help with understanding what's going on. For example: echo "<pre>".print_r($traders, true)."</pre>";

Have you looked in the errors and exceptions logs? They might give you a clue about what's happening. And you've probably thought of this, but it might be worth turning the modules off to see what happens. This thread might give you some other ideas, particularly if it turns out the issue is that the sessions time out: Two things to note. The first is that, from memory (though I might be remembering wrongly), the issue was worse with Chrome. Second, as the last post in the thread says, I never worked out what the problem really was, but a new PW installation did the trick in the end.

Hi @Patrik97 It might be worth you saying what you're trying to achieve. The kind of approach you describe would only rarely be necessary, though you may have particular needs. I'm wondering if there might be a much easier way of doing things.

Does it help to add a sort term to the selector (e.g. sort=title)? What happens if you don't paginate the results? If the non-paginated results are as expected, then the issue is probably in the pagination. But what method are you using to paginate the results? Is it as described at https://processwire.com/docs/front-end/markup-pager-nav/?

Have you thought of writing a Textformatter module? They're really very straightforward - you don't even need to use hooks. There are good examples to copy in wire > modules > Textformatter (for instance, TextformatterNewlineBR.module is a nice starting point). Once you've put your module in the site > modules and installed, apply the Textformatter on the Details tab of the fields where it's needed.

There's no need at all to understand the PW architecture or database structure and relations - or even to think about these things. Stick to using PW selectors (https://processwire.com/docs/selectors/) and all the hard work is done for you. Selectors are one of the really great features of PW! (Note that if you later decide to use RockFinder for the extra speed, it also understands the PW database structure so that you don't have to.)

RockFinder can massively reduce search times. I've used it to reduce complex searches from several seconds to fractions of a second. However, if your searches aren't too complex, even on a fairly large number of pages the difference could be between part of a second using PW selectors and a very small part of a second using RockFinder. This isn't really going to solve your problem. So I'd suggest sorting out the rendering first, as suggested by others, with RockFinder as a possible later refinement.

The basic method is to find the top level pages using a selector, then loop through those pages and their children, and children of children, etc. I recommend starting here: https://processwire.com/talk/topic/12953-simple-navigation/?do=findComment&comment=117536 Note particularly the two really useful links in the response from kongodo. It can be easiest to make a special case for the home page, and start your looping at the next level down.

It feels like this should be obvious or it should be easy to find a solution in the forums, but I've got stuck! I'm trying to set the value of a field (based on other field values) when an existing page loads for editing. Here's a reduced test version of the code that doesn't work, from ready.php: wire()->addHookAfter('Page::loaded', function($event) { $page = $event->object; if ($page->template == 'article') { // Get field values and do calculations here // but for testing: $calculatedValue = "Testing 123"; $page->of(false); $page->set('test_field', $calculatedValue); } }); Note that if I start the code like this, it works fine when the page is saved: wire()->addHookAfter('Pages::saveReady', function($event) { $page = $event->arguments[0]; if ($page->template... So I presume there's something wrong with the hook or getting the page. But what?

Just to round this topic off, in the end I got everything working fine, and it has been running without problems for several days now. I achieved this simply by moving the site to a new cloud server. I set up a clean PW install, and then copied across all files and database content. Straightforward enough, but I've still no idea what caused the original problem!

Sadly, I spoke (or wrote) too soon! The problem continues, but is highly intermittent: sometimes people can log in, and sometimes they can't. As before, when the systems freezes (or whatever it's doing), eventually a server error appears, and it's necessary to reset things in the browser to get any access to pages in the site at all. It seems after all that it's (probably) nothing to do with +FollowSymLinks, which is set in the vhosts file anyway. It may well have been chance that the problem went away (aparently for over a day) when htaccess was changed. Support at the server company looked through their logs, and it seems that the 500 errors are caused by timeouts on the server, but the cause of these remains a mystery. They sent me this: [Tue Oct 15 12:43:01.590548 2019] [proxy_fcgi:error] [pid 14565] [client 127.0.0.1:57438] AH01068: Got bogus version 34, referer: http://phpstack-273028-1014063.cloudwaysapps.com/admin/ [Tue Oct 15 12:43:01.590608 2019] [proxy_fcgi:error] [pid 14565] (22)Invalid argument: [client 127.0.0.1:57438] AH01075: Error dispatching request to : (passing brigade to output filters), referer: http://phpstack-273028-1014063.cloudwaysapps.com/admin/ [Tue Oct 15 12:43:02.804616 2019] [proxy_fcgi:error] [pid 14651] [client 127.0.0.1:57461] AH01068: Got bogus version 97 [Tue Oct 15 12:43:02.804649 2019] [proxy_fcgi:error] [pid 14651] (22)Invalid argument: [client 127.0.0.1:57461] AH01075: Error dispatching request to : (passing brigade to output filters) [Tue Oct 15 15:15:09.753656 2019] [proxy_fcgi:error] [pid 18607] (70007)The timeout specified has expired: [client 127.0.0.1:58757] AH01075: Error dispatching request to : (polling), referer: http://phpstack-273028-1014063.cloudwaysapps.com/admin/ [Tue Oct 15 15:15:39.741697 2019] [proxy_fcgi:error] [pid 18611] (70007)The timeout specified has expired: [client 127.0.0.1:58763] AH01075: Error dispatching request to : (polling), referer: http://phpstack-273028-1014063.cloudwaysapps.com/admin/access/users/edit/?id=1027&s=1&c=1 [Tue Oct 15 15:16:03.569705 2019] [proxy_fcgi:error] [pid 18635] (70007)The timeout specified has expired: [client 127.0.0.1:58771] AH01075: Error dispatching request to : (polling) [Tue Oct 15 15:17:26.509702 2019] [proxy_fcgi:error] [pid 18684] (70007)The timeout specified has expired: [client 127.0.0.1:58785] AH01075: Error dispatching request to : (polling) [Tue Oct 15 15:18:00.381772 2019] [proxy_fcgi:error] [pid 18704] (70007)The timeout specified has expired: [client 127.0.0.1:58807] AH01075: Error dispatching request to : (polling), referer: http://phpstack-273028-1014063.cloudwaysapps.com/admin/ [Tue Oct 15 15:22:03.217694 2019] [proxy_fcgi:error] [pid 18758] (70007)The timeout specified has expired: [client 127.0.0.1:58835] AH01075: Error dispatching request to : (polling), referer: http://phpstack-273028-1014063.cloudwaysapps.com/admin/ [Tue Oct 15 15:31:09.990353 2019] [proxy_fcgi:error] [pid 2405] (70007)The timeout specified has expired: [client 127.0.0.1:49876] AH01075: Error dispatching request to : (polling), referer: http://phpstack-273028-1014063.cloudwaysapps.com/admin/ [Tue Oct 15 15:33:32.793776 2019] [proxy_fcgi:error] [pid 2495] (70007)The timeout specified has expired: [client 127.0.0.1:49902] AH01075: Error dispatching request to : (polling) [Tue Oct 15 15:37:36.761771 2019] [proxy_fcgi:error] [pid 2593] (70007)The timeout specified has expired: [client 127.0.0.1:49939] AH01075: Error dispatching request to : (polling), referer: http://phpstack-273028-1014063.cloudwaysapps.com/admin/ [Tue Oct 15 15:44:45.069830 2019] [proxy_fcgi:error] [pid 2859] (70007)The timeout specified has expired: [client 127.0.0.1:49982] AH01075: Error dispatching request to : (polling) [Tue Oct 15 15:51:48.169792 2019] [proxy_fcgi:error] [pid 2969] (70007)The timeout specified has expired: [client 127.0.0.1:50024] AH01075: Error dispatching request to : (polling), referer: http://phpstack-273028-1014063.cloudwaysapps.com/admin/ [Tue Oct 15 15:52:43.337813 2019] [proxy_fcgi:error] [pid 3013] (70007)The timeout specified has expired: [client 127.0.0.1:50041] AH01075: Error dispatching request to : (polling), referer: http://phpstack-273028-1014063.cloudwaysapps.com/admin/ At the moment I can think of only three things changing on the server that might account for the intermittent nature of the problem, though it could be nothing to do with these: Activity by the current user (though the freeze happens even if the user goes straight to log in). Other users logging in, and their activity in the system. Events in two other PW applications on the same server (separate installations using different databases). It would be nice to avoid having to rebuild the installation on a new server (it's large and complex), and not knowing the cause of the problem, there'd be no guarantee that it'd work. Any ideas or suggestions gratefully received!

THE SOLUTION IN THIS POST TURNS OUT NOT TO BE CORRECT - SEE THE NEXT POST I finally found a solution. I realised that, after a very long wait, apparently frozen browser windows were, at least sometimes, returning internal server errors. So I looked at the suggestions for error 500 in the PW readme and in the .htaccess file. The one that fixed it is in item 1 of .htaccess: uncomment Options +SymLinksifOwnerMatch and comment Options +FollowSymLinks. There are various things I'm not clear about, such as why this issue seemed to caused problems with database connections on log in, why it hasn't affected two other PW sites on the same cloud server (presumably there is some difference, such as this site being the Apache default), and so on, but the solution might be useful to others.

Many thanks for the replies. @Rudy The credentials do very occasionally work, so I think they must be correct. @dragan That seems like a good analysis. I was beginning to think it might be something to do with the server set up rather than on the PW end of things. I'll get on to hosting support and will report back if this fixes it.

I have moved a site to a new server and can't access the admin. Accessing the front end of the site seems to be working as normal. And I can get to the log-in page for the admin. But then when I enter username and password, everything freezes while waiting for the next page to load. Then, after this has happened, I can no longer access any page on the site (including the front end). If I restart the browser, however, we go back to square one - that is, I can access the site until I try to log in, when everything freezes again. There seems to be some instability in this behaviour - occasionally I or others can log in, for example - but mostly it is as just described. The error logs seem to indicate it's something to do with accessing MySQL (see below), but the fact that re-starting the browser seems to re-set things makes me think that perhaps it's primarily an issue with sessions or something like that. From the PW errors log: 2019-10-09 15:59:13 ? http://phpstack-273028-1014063.cloudwaysapps.com/ Error: Exception: SQLSTATE[HY000] [2002] Connection refused (in /home/273028.cloudwaysapps.com/ebmtwcxxzm/public_html/wire/core/ProcessWire.php line 382) 2019-10-09 15:59:13 ? http://phpstack-273028-1014063.cloudwaysapps.com/admin/ Error: Exception: SQLSTATE[HY000] [2002] Connection refused (in /home/273028.cloudwaysapps.com/ebmtwcxxzm/public_html/wire/core/ProcessWire.php line 382) 2019-10-09 15:59:15 ? http://phpstack-273028-1014063.cloudwaysapps.com/admin/ Error: Exception: SQLSTATE[HY000] [2002] Connection refused (in /home/273028.cloudwaysapps.com/ebmtwcxxzm/public_html/wire/core/ProcessWire.php line 382) And the entries from the corresponding time in the PW exceptions log: 2019-10-09 15:59:10 admin http://phpstack-273028-1014063.cloudwaysapps.com/admin/ SQLSTATE[HY000]: General error: 2006 MySQL server has gone away (in /site/assets/cache/FileCompiler/site/modules/ProDrafts/ProDrafts.module line 423) 2019-10-09 15:59:10 admin http://phpstack-273028-1014063.cloudwaysapps.com/admin/ SQLSTATE[HY000]: General error: 2006 MySQL server has gone away (in /site/assets/cache/FileCompiler/site/modules/ProDrafts/ProDrafts.module line 423) 2019-10-09 15:59:10 admin http://phpstack-273028-1014063.cloudwaysapps.com/admin/?login=1 DB connect error 2002 - Connection refused (in /wire/core/Database.php line 79) 2019-10-09 15:59:10 admin http://phpstack-273028-1014063.cloudwaysapps.com/admin/?login=1 DB connect error 2002 - Connection refused (in /wire/core/Database.php line 79) 2019-10-09 15:59:10 ? ? SQLSTATE[HY000]: General error: 2006 MySQL server has gone away (in /wire/core/WireDatabasePDO.php line 480) 2019-10-09 15:59:10 admin http://phpstack-273028-1014063.cloudwaysapps.com/about/ SQLSTATE[HY000] [2002] Connection refused (in /wire/core/WireDatabasePDO.php line 217) 2019-10-09 15:59:10 admin http://phpstack-273028-1014063.cloudwaysapps.com/admin/?login=1 SQLSTATE[HY000]: General error: 2006 MySQL server has gone away (in /site/assets/cache/FileCompiler/site/modules/ProDrafts/ProDrafts.module line 423) 2019-10-09 15:59:10 admin http://phpstack-273028-1014063.cloudwaysapps.com/admin/?login=1 SQLSTATE[HY000]: General error: 2006 MySQL server has gone away (in /site/assets/cache/FileCompiler/site/modules/ProDrafts/ProDrafts.module line 423) 2019-10-09 15:59:10 ? ? SQLSTATE[HY000]: General error: 2006 MySQL server has gone away (in /wire/core/WireDatabasePDO.php line 480) 2019-10-09 15:59:10 admin http://phpstack-273028-1014063.cloudwaysapps.com/ SQLSTATE[HY000] [2002] Connection refused (in /wire/core/WireDatabasePDO.php line 217) 2019-10-09 15:59:10 ? ? SQLSTATE[HY000]: General error: 2006 MySQL server has gone away (in /wire/core/WireDatabasePDO.php line 480) 2019-10-09 15:59:10 admin http://phpstack-273028-1014063.cloudwaysapps.com/ SQLSTATE[HY000] [2002] Connection refused (in /wire/core/WireDatabasePDO.php line 217) 2019-10-09 15:59:10 admin http://phpstack-273028-1014063.cloudwaysapps.com/about/ SQLSTATE[HY000] [2002] Connection refused (in /wire/core/WireDatabasePDO.php line 217) 2019-10-09 15:59:10 admin http://phpstack-273028-1014063.cloudwaysapps.com/ SQLSTATE[HY000] [2002] Connection refused (in /wire/core/WireDatabasePDO.php line 217) 2019-10-09 15:59:10 admin http://phpstack-273028-1014063.cloudwaysapps.com/ SQLSTATE[HY000] [2002] Connection refused (in /wire/core/WireDatabasePDO.php line 217) 2019-10-09 15:59:10 ? ? SQLSTATE[HY000]: General error: 2006 MySQL server has gone away (in /wire/core/WireDatabasePDO.php line 480) 2019-10-09 15:59:10 admin http://phpstack-273028-1014063.cloudwaysapps.com/ SQLSTATE[HY000] [2002] Connection refused (in /wire/core/WireDatabasePDO.php line 217) 2019-10-09 15:59:10 admin http://phpstack-273028-1014063.cloudwaysapps.com/ SQLSTATE[HY000] [2002] Connection refused (in /wire/core/WireDatabasePDO.php line 217) 2019-10-09 15:59:13 ? ? Unable to load WireDatabasePDO - SQLSTATE[HY000] [2002] Connection refused (in /wire/core/WireDatabasePDO.php line 217) 2019-10-09 15:59:13 ? ? SQLSTATE[HY000] [2002] Connection refused (in /wire/core/ProcessWire.php line 382) 2019-10-09 15:59:13 ? ? Unable to load WireDatabasePDO - SQLSTATE[HY000] [2002] Connection refused (in /wire/core/WireDatabasePDO.php line 217) 2019-10-09 15:59:13 ? ? SQLSTATE[HY000] [2002] Connection refused (in /wire/core/ProcessWire.php line 382) 2019-10-09 15:59:15 ? ? Unable to load WireDatabasePDO - SQLSTATE[HY000] [2002] Connection refused (in /wire/core/WireDatabasePDO.php line 217) 2019-10-09 15:59:15 ? ? SQLSTATE[HY000] [2002] Connection refused (in /wire/core/ProcessWire.php line 382) It might also be worth noting that I have another PW installation running on the same cloud server (using a different MySQL database) in AFAIK the exactly the same environment with no problems at all. I am entirely stumped on where to go next, and I haven't managed to find any post in the forums regarding exactly this issue. Does anyone have any ideas?

I've come up with some seriously fast ways of doing this – following the suggestion from @Zeka and taking inspiration from the code in RockFinder. I'm posting them here in case anyone finds them useful. The RockFinder approach in my previous post takes about 0.95 seconds (across about 5,000 pages with about 1,500 associated images, resulting in about 30 pages in the final array), which is fine. But the following, using direct SQL access, produces exactly the same result in around 0.0055 seconds (much of which is making the final page array, so if there were more pages it would be somewhat slower). startDate = strtotime("-2 months"); // Run query on database $startDateString = date('Y-m-d H:i:s', $startDate); $queryString = "SELECT pages_id FROM field_article_images WHERE created >= '{$startDateString}'"; $queryResult = $this->db->query($queryString); // Get array of unique page IDs $pageIdArray = $queryResult->fetch_all(MYSQLI_NUM); $pageIdArray = array_unique($pageIdArray, SORT_REGULAR); // Make PW page array $pagesWithRecentImages = new PageArray(); foreach($pageIdArray as $pageIdSubArray) { $pwPage = $pages->get($pageIdSubArray[0]); $pagesWithRecentImages->add($pwPage); } And if an array of image data from the database table would be sufficient, the following is even faster, at around 0.0014 seconds. $startDate = strtotime("-2 months"); // Run query on database $startDateString = date('Y-m-d H:i:s', $startDate); $queryString = "SELECT * FROM field_article_images WHERE created >= '{$startDateString}'"; $queryResult = $this->db->query($queryString); // Make array of image data $imageData = $queryResult->fetch_all(MYSQLI_ASSOC); (Adding "ORDER BY created DESC" to the SQL query above could be a useful addition.) It's worth noting that the SQL queries above are quite straightforward because they are on a single table, and the image field in question appears only on relevant pages. For more complex situations, using RockFinder would be a great deal easier, and I'd guess the speed difference would be less.

After some experimentation, here's what I'm now using to get the relevant pages: $startDate = strtotime("-2 months"); // Use RockFinder to get recently saved pages with images $recentPagesWithImagesSelector = "has_parent=/articles/,article_images.count>0,modified>{$startDate},include=unpublished"; $finderRPWI = new RockFinder($recentPagesWithImagesSelector,['title']); $recentPagesWithImages = $finderRPWI->getPages(); // Use standard PW find to reduce to pages with at least one recent image $pagesWithRecentImages = $recentPagesWithImages->find("article_images.created>{$startDate}"); No doubt this could be improved (e.g. by maintaining an index of recent images), but it reduces the page-loading time to something acceptable. Note that I tried a one-step process using RockFinder only with "article_images.created>{$startDate}" in the selector, but this produced a few unexpected results, and anyway was slower than the above. I didn't investigate what was happening in any depth. I have also used RockFinder in another place on the page where a search was running slowly, and the page is now loading more quickly than before I added the list of images!

Thanks @Zeka, that makes a big difference, reducing the time taken to less than half of what is was previously. And thanks also @Autofahrn, that sounds a really interesting approach to improving performance further. The system is already quite well established, so I probably can't switch to having each image on a page (or at least not easily), but some variation on your idea might work well. For example, hooking on image load and storing relevant data (such as the time of the most recent image load) in a field on the page or some similar approach might work. And @Zeka, the SQL and RockFinder ideas seem interesting too. I now need to close down for the day, but I'll look into all these further over the next day or so.

I'm using the following code to get an array of recently added images with various bits of information about them, sorted in reverse order of when they were added. $startDate = strtotime("-2 months"); $pagesWithRecentImages = $pages->find("article_images.created>{$startDate},include=unpublished"); $recentImages = array(); foreach($pagesWithRecentImages as $pwri) { foreach($pwri->article_images as $image) { if ($image->created >= $startDate) { $sortKey = $image->created; $recentImages[$sortKey] = array($image->page->title, $image->page->url, $image->basename, $image->created); } } } krsort($recentImages); It works fine, and I'm displaying the contents of the array on a dashboard page on the admin side. However, it runs rather slowly, delaying the loading of the dashboard page by a few seconds (around six or seven seconds when getting images from about 5,000 pages). The page already takes 3 or 4 seconds to load (it shows other stuff too), and a total of ten seconds or so is a bit frustrating for users. I've been trying to work out a faster way of getting an array of images (perhaps using different selector), but so far with no success. Can anyone think of a way to get it to run more quickly? Thanks!

The issue that I raised on May 8 has now gone away. It was very probably caused by an entirely separate problem I was having with permissions, and thus is nothing to do with this module! I posted about the permissions issue at https://processwire.com/talk/topic/19266-page-edit-permission-not-working/.

Strangely, the problem has gone away all by itself! Obviously this is a good thing for the project, but it means that I don't know the cause of the trouble and can't offer a solution. The only thing I can think of that I changed and that might have affected permissions in some way - though this is a total guess - is that I discovered an image field which, for no reason I'm aware of, had access control enabled (I may have accidentally clicked something or done something through the API). It's possible, though I can't think why, that disabling this access control triggered the resolution of the issue. If anyone else comes across this problem, it'd be interesting to know how they fix it.

Thanks, Robin S, these look like really useful suggestions. I've certainly tried suggestion 3 already (many times!), but I'll get on with 1, 2 and 4. I'll report back.