searching multiple fieldtypes

[evan]

Hi all,

I have a website that has Pages with text and Page fieldtypes. Is there an elegant way to search both of them (in this case, the Page titles of the referenced pages) at the same time for a search string?

Doing a search of the Page titles and bodies is easy enough:

$my_textual_results = $pages->find("body|title%=$query, sort=-posted, limit=10");

And of the referenced Pages, too:

$my_referenced_pages = $pages->get('/referenced-pages/')->find("title%=$query");
$my_referenced_results = $pages->find("referenced_pages=$my_referenced_pages, sort=-posted, limit=10");

Now how do I combine these two search queries into one? I tried an 'or' operator between the two different search parameters:

$my_referenced_pages = $pages->get("/referenced-pages/")->find("title%=$query");
$total_results = $pages->find("body|title%=$query | referenced_pages=$my_referenced_pages, sort=-posted, limit=10");

...but that didn't work.

Any ideas? Thanks!

-evan

[slkwrm]

You can safely leave your request separate, but merge them afterwards:

$total_results->import($my_textual_results);

[diogo]

You are aware that "get" will only give you the first referenced page? In the search wouldn't work an array anyway, but because the variable is in plural, i get the feeling that you want something different.

edit: I was sleeping when I wrote that... you can use a page array on the selector, and you are using a "find" right after the "get"

No more reading PW forum on the mobile! Still you don't really need to do it in two steps, because this would be enough:

$my_referenced_pages = $pages->find("title%=$query, parent=/referenced-pages/");

As for your question: I don't know how to combine them with one query, but with slkwrm's solution you wouldn't have the pages sorted exactly as you want.

[ryan]

Just for clarification, the intended search would be for:

(body OR title contains $query) OR (referenced_pages contains $my_referenced_pages)

and not this?

(body OR title contains $query) AND (referenced_pages contains $my_referenced_pages)

In ProcessWire you can't do the first one in 1 find(), at least not yet. Though the second one you can do and is pretty common:

body|title%=$query, referenced_pages=$my_referenced_pages

I'm hoping to make this possible pretty soon (but it isn't yet):

body|title|referenced_pages.title%=$query

[diogo]

Ryan, I think he wants the first, and THAT would be a great addition to pw

It would also work with repeaters?

[ryan]

body|title|referenced_pages.title%=$query

Attached is an update to /wire/modules/Fieldtype/FieldtypePage.module that lets you do this, once you replace your existing file with the one below. You should be able to specify just about any native or custom page field after the dot, like above (where we're using 'title'). If anyone can test it out to confirm I'd appreciate it. I'll commit to the source after testing a little more locally too.

FieldtypePage.module

It would also work with repeaters?

This type of syntax is already used by repeaters. But if you mean something like: "repeater_field.referenced_pages.title%=$query", I don't think the attached update would let you do that, though havent' tried. If it doesn't, it's probably not much of a stretch to support it though.

[diogo]

Actually I meant the normal use for repeaters... disregard that.

I tried the module. it seems to work fine

[evan]

Just for clarification, the intended search would be for:

(body OR title contains $query) OR (referenced_pages contains $my_referenced_pages)

Yep, that's what I wanted to do.

Attached is an update to /wire/modules/Fieldtype/FieldtypePage.module that lets you do this, once you replace your existing file with the one below. You should be able to specify just about any native or custom page field after the dot, like above (where we're using 'title'). If anyone can test it out to confirm I'd appreciate it. I'll commit to the source after testing a little more locally too.

FieldtypePage.module

Wow, totally amazing! Thanks so much Ryan! It works perfectly on my end.

[thomas]

I just used this for my search question (http://processwire.com/talk/topic/1561-search-fieldtype-pages/) and it works great. Thanks!

[slkwrm]

Tried it out too, works great!

Experimenting with selectors I found something that makes me now ask the following question:

$results1 = $pages->find("$selector1");
$results2 = $pages->find("$selector2");  
$allResults = $results1->import($results2)->sort("-modified")->find("limit=5");

This code works. But if I use the following code, it ignores the second parameter (limit):

$allResults = $results1->import($results2)->find("sort=-modified, limit=5");

And if I switch them to find("limit=5, sort=-modified") then sort parameter is ignored. Is it ok?

[Soma]

$allResults = $results1->import($results2)->find("sort=-modified")->limit(5);

[slkwrm]

Soma, your example doesn't work for me. I guess limit() function doesn't exist, am I right? My question here is: Why is the second selector in my example ignored? I'm sure Ryan can help here

[Soma]

Sorry wrong brain.

$allResults = $results1->import($results2)->sort('-modified')->find("limit=5");

Ryan can't help much here.

[ryan]

This code works. But if I use the following code, it ignores the second parameter (limit):

This sounds like a bug to me. I'll take a closer look here, thanks for pointing it out.

[nik]

body|title|referenced_pages.title%=$query

I've got an almost similar issue as evan did - except for that I'd need this applied to parent page as well (parent.title instead of referenced_pages.title). So, Ryan, do you think there was a way to achieve this?

I've got a page structure of this kind: /countries/<country>[/<state>]/<city>/<data> (yes, the same again =) and would like to list data pages having search terms in their title OR in their parents (city page) title, sorted by the titles of the found data pages. Having thousands of data pages a lightweight pagination support is a must (one implemented at the database level that is) and making two different finds and joining the results afterwards would leave me without such.

I think the proper way around this would be to use either-or selectors (as evan had tried in the first place) but they're only mentioned in the roadmap for PW versions 2.4+.

My current solution is a module with a horrible hack and plenty of copied core code letting me do a find operation using plain SQL (UNION of the two find statements). It sure does work at the moment but as the site evolves, my mostly hardcoded SQL statements will most probably fail sooner or later. So, what I'm working on now is a more generic module implementing one variation of either-or selectors. It seems to be possible to get working, but has even more copied core code.

But now this recent development on Page fields left me thinking there may be some other solution to my problem, an easier one I hope . And at least it's time to get a second opinion on how to try to solve this. I'm eager to make this into a module (or even core code) if there happens to be a suitable solution available - one that I'm able to implement with a little guidance to the right direction.

So what do you think, Ryan and others as well?

[Soma]

This sounds like a bug to me. I'll take a closer look here, thanks for pointing it out.

But this really works, there's no bug here:

$allResults = $results1->import($results2)->sort('-modified')->find("limit=5");

Do you mean with making it one selector? This has never worked I think, also MadeMyDay has found about it some time ago.

[WillyC]

It is

bug

[Soma]

I've got an almost similar issue as evan did - except for that I'd need this applied to parent page as well (parent.title instead of referenced_pages.title). So, Ryan, do you think there was a way to achieve this?

I've got a page structure of this kind: /countries/<country>[/<state>]/<city>/<data> (yes, the same again =) and would like to list data pages having search terms in their title OR in their parents (city page) title, sorted by the titles of the found data pages. Having thousands of data pages a lightweight pagination support is a must (one implemented at the database level that is) and making two different finds and joining the results afterwards would leave me without such.

I think the proper way around this would be to use either-or selectors (as evan had tried in the first place) but they're only mentioned in the roadmap for PW versions 2.4+.

My current solution is a module with a horrible hack and plenty of copied core code letting me do a find operation using plain SQL (UNION of the two find statements). It sure does work at the moment but as the site evolves, my mostly hardcoded SQL statements will most probably fail sooner or later. So, what I'm working on now is a more generic module implementing one variation of either-or selectors. It seems to be possible to get working, but has even more copied core code.

But now this recent development on Page fields left me thinking there may be some other solution to my problem, an easier one I hope . And at least it's time to get a second opinion on how to try to solve this. I'm eager to make this into a module (or even core code) if there happens to be a suitable solution available - one that I'm able to implement with a little guidance to the right direction.

So what do you think, Ryan and others as well?

I think it would be possible merging 2 results, but it may not as optimized loading the page array ( though not whole page object ) for thousands of pages. But if it's not 100'000 I think it should work. Also it would be possible to create a pagination by hand.

On the other hand building sql queries aren't really save maybe but can be most optimized choice.

Though what you could do which would be as optimized is use a hidden field on template so store parent title. Create a autojoin field "parent_title". Then populate that field using a hook module. Then you can do:

$results = $pages->find("parent_title|title%=searchtext");

So you could write and autoload that stores the parent title on page save or move and parent save to update all children pages. Also a bootstrap script that you can update after changes or a lazy cronjob or cronjob that update's all in one call (all parents children)

[ryan]

The subfields on page references update is now committed in the core so that you can do these types of queries (as mentioned above):

referenced_pages.title*=$query

Also want to mention a limitation: You can't do this with in-memory selection (at least not yet). So the find(), filter() and not() in WireArray/PageArray won't recognize this… just the PageFinder ($pages API var). I'm guessing it'll be awhile before someone even tries it, but just wanted to mention it.

I've got an almost similar issue as evan did - except for that I'd need this applied to parent page as well (parent.title instead of referenced_pages.title). So, Ryan, do you think there was a way to achieve this?

I think so. It was a pretty simple matter supporting the subfields of page references, so I think the same would go for parent. I will look into it here more and hopefully get this added soon. Seems like a very nice thing to have.

This code works. But if I use the following code, it ignores the second parameter (limit):

$allResults = $results1->import($results2)->find("sort=-modified, limit=5");

And if I switch them to find("limit=5, sort=-modified") then sort parameter is ignored. Is it ok?

This bug should now be fixed and committed to the core. It sounds like MadeMyDay mentioned this issue too (?), but somehow I missed that message. Thanks to both of you for tracking it down and sorry I missed the message about it the first time around.

[Soma]

Thanks Ryan! Well he mentioned it on IRC, so I remembered to split it up so it works from a forum post.. not sure it really was about the exactly same, but order of selector fields matters. Some things he recognized he make a post you even liked.

[nik]

I've got an almost similar issue as evan did - except for that I'd need this applied to parent page as well (parent.title instead of referenced_pages.title). So, Ryan, do you think there was a way to achieve this?

I think so. It was a pretty simple matter supporting the subfields of page references, so I think the same would go for parent. I will look into it here more and hopefully get this added soon. Seems like a very nice thing to have.

This isn't probably at the top of your todo list, but you wouldn't happen to have any news on this, would you?

I'm still trying to avoid replicating fields from the parent page as there are multiple languages involved as well. Still that may be a better solution than the almost-fixed-sql-statement-thingie-module I currently have...

[ryan]

This is definitely still on the to-do list. Schedule's been tough here lately (been out of town on the weekends), but should be able to give this a closer look hopefully this week.

[ryan]

Per @nik's request (from awhile ago), I've expanded support for the parent field selectors that you can use with $pages->find() and related functions. You can now pass those functions selectors like this:

parent.name=about 
parent.name=about|contact
parent.title*=Something
parent.body*=Something|Something Else
parent.title|parent.body*=Something|Something Else
parent.template=basic-page
parent.template|template=basic-page|fancy-page

// and I think this is the one that @nik requested: 
parent.title|title*=Something

Pretty much any combination of stuff can be used with the parent selector. Previously, you could only use id or path.

As an added bonus, you can now also perform partial matches on a page name:

name%=something
name^=some
name$=thing
parent.name%=something
parent.name|name%=something
parent.name|parent.title|title%=something|something else

These new features are available via the latest commit to the dev branch (upcoming 2.3).

[MatthewSchenker]

Hi Ryan,

Terrific!

It's excellent the way you continue extending the powers of ProcessWire while still leaving the clean stucture in place and not making any assumptions about how the API must be used.

Looking forward to 2.3!

Thanks again,

Matthew

[ryan]

Thanks Matthew, these API improvements are one of the most fun parts for me. Though I think the next step will be to get comprehensive unit test library specific to PW selectors, so that I can go in and add or tweak stuff, and be able to make sure I'm not breaking anything.