Lutz

get() should return a NullPage in both cases, if $categories returns a PageArray. Since find() could return a PageArray or an array: what does $categories return?

@teppo Thank you very much. Yes, I plan to go with ProCache and to avoid template caching for most of the templates. I think I misunderstood the comment of Soma regarding the template cache problem, thanks for the hint.

Ok, I read the issue 432 thread again, so yes, the need to "revisit" (Ryan) could refer to the implementation of the template cache only. However, it would be good to know if pagefileExtendedPaths is a reliable option and I'm glad that you suspect that this is true.

@teppo Maybe I'm wrong, but no, the config option pagefileExtendedPaths is a result of that discussion and the comments of @Soma and @ryan refer to the concept of extended paths.

@teppo I hope so. There's an open ticket "Feature-Request: More options how to store files under site/assets/files" from 2014, where @Soma warned regarding problems with pagefileExtendedPaths when using template cache and @ryan answered in his last comment: "That's a good point, as the structure is flat. Will have to revisit that with a similar solution to the files at some point in the future." https://github.com/ryancramerdesign/ProcessWire/issues/432

We still have this warning in wire/config.php: "The extended file mapping feature is not yet widely tested, so consider it beta." Does anyone have $config->pagefileExtendedPaths in use on a production server?

@horst Great, thanks for the explanation. Despite such rare errors I personally prefer those loose connections between Dev (local), Staging and Production, compared to the stricter ones you usually need with Craft.

@horst There's another thread, SVG files do not seem to be involved there.

@dragan Many thanks. I think the timings strongly depend on the conditions. Sometimes a large number of pages could be uncached, and I know from experience that peaks (e.g. 60+ page views per second) can change the meaning of overhead enormously. Maybe it's interesting to have a closer look on all those queries, thanks for the example. I too see queries that I don't understand so far. For example 37 SELECT id, templates_id, parent_id FROM pages WHERE (name=:name) LIMIT 2 38 SELECT id, templates_id, parent_id FROM pages WHERE (name=:name) LIMIT 2 39 SELECT id, templates_id, parent_id FROM pages WHERE (name=:name) LIMIT 2 40 SELECT id, templates_id, parent_id FROM pages WHERE (name=:name) LIMIT 2 or SELECT false AS isLoaded, pages.templates_id AS templates_id, pages.*, pages_sortfields.sortfield, (SELECT COUNT(*) FROM pages AS children WHERE children.parent_id=pages.id) AS numChildren,field_title.data AS `title__data` FROM `pages` LEFT JOIN pages_sortfields ON pages_sortfields.pages_id=pages.id LEFT JOIN field_title ON field_title.pages_id=pages.id WHERE pages.parent_id=1018 AND pages.templates_id=54 AND pages.id IN(1061) GROUP BY pages.id I will have a look at ProfilerPro, maybe it could help to get a better understanding of the boot process.

@Robin S Those results are some with auto-prepend (_init.php) and auto-appended _main.php (both empty), but they were nearly identical without. Tracy: yes, this especially was my hope, that those additional queries etc. could be a factor. I will continue testing.

@Zeka Wow, 502 PDO queries, thank you. Well, the numbers of database queries shouldn't be the decisive factor, but under certain conditions those higher numbers maybe could become a problem, I fear.

Thank you LostKobrakai and dragan, I will have a look on a different server. (There's an older ExpressionEngine installation with an earlier version of the site on the same server, showing far less db queries, lower CPU usage and slightly less memory usage on pages with all the EE tags placed, with several loops from different channels and entries relations. So I was surprised to see this results for completely empty templates.) dragan, how many PDO Queries ($database) did Tracy count?

Hi LostKobrakai, another example. PW 3.0.131, 26 templates, 16 fields, on the front-end with Tracy, Category page with just one field, template file empty: Execution time 341.5 ms CPU usage user + system 9 % + 1 % Peak of allocated memory 14.70 MB Included files 149 PDO Queries ($database) 57 Selector Queries 7 Article page with 11 fields, template file empty: Execution time 338.3 ms CPU usage user + system 11 % + 0 % Peak of allocated memory 14.40 MB Included files 149 PDO Queries ($database) 85 Selector Queries 7 (On a dedicated server with 8 GB RAM, PHP 7.2, MySQL 5.7, Apache.) I have no idea what could be done to reduce the number of queries and the CPU and memory usage needed to boot PW. Maybe we could reduce the number of fields, but 16 isn't a big number and it's just a set to start, and it would be difficult to reduce the number of templates. Of course, that's not a problem with small to medium sized and low traffic websites, but it seems to limit the scalability. And yes, with low traffic and a few million pages this shouldn't be a problem either, or with 1000 pages and high traffic. But I'm not sure how we could use PW with a few million URLs and e.g. 100 page views per second (peak). And I really would be very glad to become more optimistic in this regard. Apart from the number of fields and templates, are there any other vectors to optimize regarding the server load on boot? Maybe it would be helpful to collect some best practice techniques to improve the scalability of PW besides optimizations of templates or caching.

In your .htaccess file, have a look at "11. OPTIONAL: Set a rewrite base if rewrites aren't working properly on your server." Uncomment the line "RewriteBase /".

Yes, but it seems to become a lot easier to trigger a network error with the dev versions. I tested with 3.0.84 right now: I was logged in, clicked the edit button on an empty test page (on the front-end), the editor was loading (I thought it loaded complete), I clicked on Pages (main nav) – network error, the pages list disappeared. cmd+shift+R, the page reloaded, the problem persists. You have to open the pages list in a new window/tab to get the pages list back. I have to setup a new dev server this weekend, will continue testing with 3.0.90.

I couldn't reproduce the issue exactly, especially I couldn't found an issue regarding the session data so far. What I found was something similar: When you click around in the admin area (UIKit), it's relatively easy to provoke a network error. If one occur, chances are high that the page list disappears and is not to refresh with a reload of the page (in Firefox: cmd+shift+R [macOS] or ctrl + F5 [Windows]). Maybe this similar issue is an UIKit one, but it' just a guess: Usually there's a div container .PageList and style attr "display: block;" around the page list. If the list disappears like described above, there are additional styles applied: "overflow: hidden; height: 1px; padding-top: 0px; margin-top: 0px; padding-bottom: 0px; margin-bottom: 0px;". The list is still there, so in this case there's definitely no issue with session data involved. Strange thing is that in this case of a similar issue you are able to refresh and to get the page list displayed again (and e.g. to get the link to the Debug Mode Tools to work again) throuph opening the page list in a new window. Maybe anyone have an idea what's causing THIS issue.

It's not a browser caching issue, that was the first thing I checked. I guess we have to look at something that happens when sessions get updated. Maybe the values for the keys ProcessPageAdd and AdminThemeDefault get lost. However, I don't understand why deleting cookies is not sufficient to bring the nav back, why it's necessary to restart the browser, too.

First I thought it could be UIKit-related, but I tested this admin theme with 3.0.86 only, i.e. not with 3.0.62. So, no... I guess it's a problem with sessions, but had no time to take a closer look.

Same here (Firefox 58.0.1 and 59.0b6). I saw it first when testing 3.0.86, but since then twice with 3.0.62. Navigation stops working, you have to use an url to log out (/{processwire admin}/login/logout/). Workaround: delete cookies and restart browser. A rather unsuitable procedure for customers...

Thanks Soma, good to know. Template cache on or off? 400'000+ pages: on Apache or NGINX?

A lot of interesting ideas here. Nevertheless, I'm a bit nervous, because I hoped to see a focus on critical features missing when building larger websites, with full-fledged editorial teams (larger teams, with copy editors and editors, designers, ...). Think about editorial flows: Of course there are already helpful parts available, e.g. ProDraft. But it would be a big help to have page versioning in core, and collision detection! And think about teams with specialists for different work areas: Photographers and photo editors, who upload files into a central media repository and categorize them, so that editors have a logical place to search for article images, headers. To find an unique image language, sometimes with repeated use of some of the images. To publish in this way is a difficult process. Not something that could easily get replaced by AI. It's a special kind of work of human beeings, who like to be creative and want to get technical support in a way they understand and they need. They want to have access to large galleries that they can filter. To look at their selections in a preview, that shows teasers next to the others in the area concerned. To get an impression how it works, how images works with headlines, and teasers works beside their neighbors. And publishers like to see reasons to have confidence in the scalability and further development of key features for their workflows, their cms. Therefore it seems to be important to have a, in this regard fundamental, set of key features for larger publishing teams available, in a quality and with support, that creates trust in pw as a solid and future-proof platform for larger projects. I understand the difficulties and I know some of the arguments regarding versioning, or for one-page-one-image workarounds (against a central media management). But I'm not convinced by these arguments. More in detail, regarding the media management question. I think the maybe best way to achieve a pw kind of a media solution wouldn't be a central, fixed media library, but to build something like a central media archive out of pages – but with multiple images per page, selectable from an image field, like images of other pages are selectable from ckeditor fields. And yes, I know that there are modules that try to offer such a solution. So maybe it's not too difficult to come to a solution in technical terms. However, it makes a difference to see such a module and such a workflow as a well known and documented way to build a reliable media management, to know it as a part of trying to establish pw as a solid alternative for building larger websites with larger teams. Similarly regarding scalability questions. We have $config->pagefileExtendedPaths. But is that thoroughly enough tested that we can say we can definitely count on it? Does template cache work with it or not? (I wasn't able to test it out by myself.) Ryan wrote 2014: "Will have to revisit that with a similar solution to the files at some point in the future". (https://github.com/ryancramerdesign/ProcessWire/issues/432) If we could clarify this question, that would be very good. I mean, a good thing for the ProcessWire 2018 Roadmap. And last but not least: official NGINX support. To be clear: I'm not concerned to criticize any contribution to the roadmap, or have anything to complain about pw. On the contrary, I hope it will be possible to advance pw. But I think it would be a good idea to focus on critical areas regarding the use of pw for larger projects, for larger editorial teams. Decisions regarding a publishing platform depends on features we can offer, but not in a simple way, that we just could show this or that functionality. More important, I think, is to give a perspective that creates confidence in serious steps to build reliable tools, to enable reliable workflows for demanding tasks of teams, i.e. to establish pw, even for more complex teams of editors and for larger websites. This is a lesson to learn from the development of Craft. Of course, it doesn't make much sense to compare the projects, and to compare them is not my point. Success in one way or another not always depends on resources. Sometimes it's more a question of clarity and organization – sometimes success depends primarily on focusing on key issues. Although the basic requirements are very different, in many cases pw is a valuable solution right now. But it would be good to find the critical questions that the project should face, and to leave less important ones aside. EDIT: A short note regarding the NGINX question, apart from posts in the forum there's an older, very simple attempt on howtoforge, https://www.howtoforge.com/running-processwire-on-nginx-lemp-on-debian-wheezy-ubuntu-13.04. And an article about installing pw 3.0.62 on Ubuntu 17.04/17.10 with NGINX, https://websiteforstudents.com/install-processwire-on-ubuntu-17-04-17-10-with-nginx-mariadb-and-php-support/. Just as an example what a useful approach could look like: https://github.com/nystudio107/nginx-craft Would be great to achieve something like that for pw, with special consideration of ProCache.

@adrian: I tested 2.0.5, everything worked as expected. Another improvement, thanks! Just for the records, I only tested importing TSV by clipboard, not to import a CSV file. All possible variants: TSV with two columns, four columns (no empty cells and rows with two empty columns), TSV with five columns--into the mentioned field with four columns.

@adrian, good idea, yes, I will have a look...

Good to know, and yep: "this approach seemed the simplest without messing with anything else". So, I'm happy with that solution. This last problem wasn't a deal-breaker, of course. However, it's great to have this new flexibility. You know, it's good to keep things simple for the ones who use those featuers to import just by copy and paste from Excel--even though they lack any experience in using CSV/TSV... Thanks again.

Adrian, just finished some tests, everything seems to work as expected now! I thought for a moment we could loop with a while instead of the foreach($data as $subfieldKey => $fieldValue), using $i (number of subfieldNames). But the array_push does it. Great response time, thanks a lot!