thetuningspoon

Eyup, good point. Which is why this can come in handy.

What I meant was whether PW was using the same basic flow: Converting a selector to an SQL select statement, getting the IDs of the matching pages, and then calling getById() to load the actual pages into a PageArray. I spent this morning doing a deep dive into the core and have confirmed that this is how it's working. I was also able to simplify the example by @LostKobrakai to the following: $pf = $this->pages->getPageFinder(); $query = $pf->find($selector, ['returnQuery' => true]); # Show sql //$query->getQuery(); # Modify query //$query->where($sql); $statement = $query->execute(); $statement->execute(); # Load the pages $ids = array(); while($row = $statement->fetch()) $ids[] = $row[0]; $myPages = $this->pages->getById($ids); I haven't solved the pagination side of things yet. Unfortunately PagesLoader::find() is doing quite a bit of work that we're not able to take advantage of due to the fact that we have to bypass it completely and go straight to PagesLoader::find() in order to get the DatabaseQuerySelect() object. I'm not sure if this problem can be solved without modifying the core or duplicating a lot of its code. For future reference, this is the basic flow of a Pages::find() call (sans various options and some intermediary methods): Pages::find() Does nothing on its own. Delegates to PagesLoader::find() PagesLoader::find() Checks for page in runtime cache, returns cached page if it's in memory Creates a selector object from your selector string (PageFinder::find() can also do this, as I discovered) PageFinder::find() Turns selector object/string into a ProcessWire DatabaseQuerySelect object (via PageFinder::getQuery()) Turns DatabaseQuerySelect into an SQL select statement and queries the database Returns a multidimensional array with the ID, parent ID, and template ID of each matching page (OR a ProcessWire DatabaseQuerySelect object if the $returnQuery option is true) PagesLoader::getById() Takes an array of IDs Creates a page object for each ID and populates it with fields from the database (an additional database query). This is where any autojoin fields are pulled from the database. PagesLoader::find() Sorts pages by template (?) Sets up pagination Returns final PageArray

This seems to do it: $pf = $this->pages->getPageFinder(); $selector = new Selectors($selector); $query = $pf->find($selector, ['returnVerbose' => true, 'returnQuery' => true]); $statement = $query->execute(); $statement->execute(); $ids = array(); while($row = $statement->fetch()) $ids[] = $row[0]; $myPages = $this->pages->getById($ids); Is this how PW constructs the PageArray during a regular $pages->find()? So even if you were autojoining all your fields, it is still doing one query to find the matching pages and then another separate query for each page to load the desired fields?

Yes, I understand. I meant how does it work under the hood. I guess path is a dynamic page property, so it requires constructing the page object to get it? You mean with a $pages->get()? Wouldn't that mean going back to the database again to build each page? Anyway, thank you for explaining further. At this point I am wondering if you can provide any insight to my original question, which was how I can modify the sql of a regular $pages->find() and then return the results as a PageArray.

@bernhard How does the closure thing work? Is it creating a page object for each result? Can you give an example of when the closure would be required? Thanks

@bernhard This looks very cool, but I am not totally clear on what it does and how it does it. Can you give me a semi-technical explanation of how your module works (inputs and outputs) and where it plugs into the core? What exactly does a call to RockFinder return? Is there a way to get page objects from the results if I don't need the scalability features of the module and just want a normal page array in the end? Can I still use pagination with the results? Also, as cool as your module looks (and it looks very cool), if what I'm wanting to do can be achieved easily with just the core, I'd prefer to keep it simple.

@LostKobrakai I've been playing around with your example here and it's really neat. I've been looking for a way to add or modify a selector with custom SQL. It looks like your onto something here. However, I can't figure out what to do after running the query. How to I convert the results into a PageArray? Also, I noticed in studying the core a bit that this technique bypasses some of what would normally occur in a $pages->find(), including (I think) setting up the pagination and the loadOptions. In my particular use case I have a selector that does exactly what I want, but need to add a custom ORDER_BY for a more complex sort than PW allows.

I get that, but if everyone is using their own helper libraries and different frameworks, you lose the great jQuery ecosystem and all the plugins and libraries built on top of it. You could end up having to include a bunch of code from different libraries that are doing the same thing in order to use the existing solutions you want. That feels like a step backward to me. I don't see why jQuery can't just be retooled so that it's using newer, faster technology under the hood, with the same compact, standardized api on the front end. Maybe have a legacy jQuery and a new, slimmer jQuery for those that don't need old browser support. Regarding jQuery and more advanced JS applications, I don't think jQuery should be thought of as competing against vue or react. It's at a lower level in the stack. It doesn't impose structure on your application, but that doesn't mean it prevents you from creating your own structure. Disclaimer: These are just my feelings as a developer who is still trying to build my applications largely server-side, where my ajax calls usually return blocks of html encoded within JSON instead of pure data structures. This keeps all rendering and business logic server side in order to keep the application code simple and DRY. Not sure if I really know enough about what I'm talking about when it comes to the new client-side development paradigm. I feel like both are reasonable approaches to development (depending on the application's requirements). It's when you don't choose one or the other that things really get messy and WET.

Sorry to say, whenever I see the vanilla JS alternatives to jQuery, it reminds me why I still use jQuery

@mel47 Thanks. I just pushed another release so the module will only load in the admin, as it was before. But it is still no longer limited to specific admin processes.

Just released an update on GitHub: -Added support for repeaters using the new InputfieldPage::renderReadyHook method (Will fall back to hooking into render if using on an older version of PW) -Specifying processes is no longer required (I'm not sure why I limited the modules use to just specific admin pages before, but there may have been a reason. Let me know if you see any problems with this) I looked into the possibility of making the modal optional, but it's not such a simple task with the way the JS is currently structured. I would like to do some refactoring, but not sure if/when I'll have the time.

@Kiwi Chris Did you try this? https://processwire.com/talk/topic/9857-module-page-field-edit-links/

Hey guys, for some reason the forum has not been notifying me of new replies in this thread, so I apologize for my absence. @Robin S Thanks for the info on the new InputfieldPage::renderReadyHook method. I will update the module asap. I think that as it stands right now, the module does work in a repeater as long as the main page also has a page field that is using the module. The problem is that the js and css assets are not included by the repeater. @Ipa it looks like the javascript for these two modules is conflicting. I am not familiar with HelperFieldLinks. Can you explain its purpose? Also, can you get me a larger screen shot of this in context of the rest of the field? I don't think this would be too difficult to add. Good idea.

Not a question, but just something I discovered that I wanted to share on the forum since I haven't seen it discussed anywhere. If you are familiar with the "autojoin" option on individual ProcessWire fields, enabling this feature means that every time a page with that field on it is loaded into memory, the field will be included with it immediately. (Normally just the page's name and meta data is included, and getting a specific field from the page requires a separate trip to the database. This is to conserve memory) Normally this is fine and it keeps things simple, but occasionally you may be loading a lot of data in a consistent fashion (for, example, to populate a table) and know exactly which fields you need. With the autojoin option on fields, you would have to enable autojoin for all pages at all times, which you probably don't want. With the following code, you can do a $pages->find() and specify exactly which fields you want to autojoin with it: $pages->find("template=whatever", ['loadOptions' => ['joinFields' => ['filed1','field2','field3']]] ); I tested out in debug mode on one of my projects, and it seemed to significantly reduce the number of SQL queries required for an html data table I was building. Hope that helps someone!

Why did I not know about this module until now?

Mystery solved: https://github.com/processwire/processwire-issues/issues/430

The resolution to this: https://github.com/processwire/processwire-issues/issues/430

@ryan Could you provide any insight on this? Thanks

@dragan Yes. Unfortunately, findMany() has the same issue in that the queries are still separate and the initial queries still have to return all of the matching page IDs. Pages sum doesn't fit this use case, I don't think.

We are putting PW through its paces on one of our projects. Currently we have 3.5 million pages and counting. Everything has been scaling well and is running great, except for some of our selectors that use subfields. Our project involves physical units (represented by PW pages) that transmit their status to our application each day, resulting in a new page for each transmission. We are using a page field on the unit to store a reference to its latest transmission. We then need to filter the units by their last transmission date, so we are using the following selector: $pages->find('template=unit, last_transmission.date_field<-1 day, last_transmission.date_field>=-7 day, sort=created, limit=10'); Once we had several thousand unit pages in the system, this selector began to fail. After doing some debugging to see what SQL PW was actually producing, we discovered that the problem was that the selector was generating three separate sql queries. One returned the ids of all of the units that had a last_transmission date greater than a given timestamp, one returned the ids of all the units that had a date less than a given timestamp, and the last took the results of the other two queries and applied any remaining selectors. Since there were several thousand units, the first two queries caused a memory error. We resolved this issue by using the created date instead of a custom date field on the last transmission, so the selector changed to: $pages->find('template=unit, last_transmission.created<-1 day, last_transmission.created>=-7 day, sort=created, limit=10'); For some reason PW was able to combine this into a single SQL query, whereas it was unable to do so with the custom field. Not sure if this is the intended behavior or not, but it looks like this is an area of PW core that might be improved upon. Has anyone else come across this issue and are there any other workarounds I might not be aware of?

@kongondo Correct, I only want Page and not Child 1. Though it doesn't matter if Child 1 is matched too, because it has a different template from Page and I can exclude it by specifying the template in the selector. @tpr I tried your suggestion yesterday after seeing your post. The example I gave in my initial post was oversimplified. My selector is actually more like children.children.pageField=123 So that didn't work. Any ideas? I restructured the page tree to get things working for now (the pages I'm referencing in the selector are now direct children), but this may not be ideal in the long run.

Just ran into a requirement on one of my sites which I don't think PW supports yet. I'm trying to match all pages that have a certain child, but the child is not an immediate child of the page. It is two levels down: Page -- Child 1 ---- Sub-child 1 ---- Sub-child 2 < I only want to select the parent Page if this one exists -- Child 2 This would be similar to the has_parent selector, but in the opposite direction. I searched the forums but didn't find any discussion of this feature/request. I seem to recall seeing it brought up at some point in the past, though.

https://github.com/processwire/processwire-issues/issues/379

@bernhard This looks amazing! I developed a CRM for a customer using ListerPro with some custom enhancements, but this is another level up! Must have been a LOT of work. Hope you were paid well

@Robin S No, I am using ID. It works fine when editing the user in ProcessPageEdit (or from Access/Users), but not ProcessProfile.