PW 3.0.160 – New search abilities

[ryan]

In ProcessWire 3.0.160 we’ve got some major upgrades and additions to our text-matching selectors and operators. This brings a whole new level of power to $pages->find() and similar API calls, especially when it comes to search engine type queries. This blog post also includes a demo search engine that lets you test everything out live—

https://processwire.com/blog/posts/pw-3.0.160/

[Robin S]

Wow, that's a motherload of new search operators. Awesome!

I usually want to include as many results as possible to so I tend use the LIKE operator for text searches. To bring in even more results it would be cool to use query expansion too, but I noticed that there isn't an operator that combines those two. Is that combination not possible for technical reasons?

[ryan]

@Robin S That's correct that the combination is not possible for technical reasons. The query expansion is a feature of MySQL fulltext indexes in a standard natural language match/against query, so it's not something that can be used with LIKE (which does not use fulltext index), and also can't be used with match/against boolean mode (which is how partial word matches can use fulltext). So there are only certain operators we can use query expansion with.

My experience so far is that query expansion is pretty inconsistent as to when it's useful. But since it's something that MySQL supports I thought PW should provide the option. If a search returns no results, query expansion isn't going to help, because it needs there to be matches in order to expand upon them. I'm currently thinking I might use it on one or two word searches that might otherwise only return a small number of results. If your goal is to bring in as many results as possible I would start experimenting with the "Contains match and expand" **+= operator, which uses the standard MySQL match/against logic, and doesn't require all search words to be present. Though depending on the search, once you go past one or two words with query expansion, it tends to add noise to the results. So I've been thinking a few of these new operators might be made configurable to automatically substitute another operator under certain conditions, like to a non-expand version depending on word count, or automatically substitute another operator when one produces no results. 

[szabesz]

7 hours ago, ryan said:

So I've been thinking a few of these new operators might be made configurable to automatically substitute another operator under certain conditions, like to a non-expand version depending on word count, or automatically substitute another operator when one produces no results. 

That sounds like a powerful feature for sure. Thanks four these nice additions anyway.

[adrian]

@ryan - just playing around with some of the new options here and they are really awesome, thank you!

I was initially confused about how to use this example:

• +(foo bar) or +"foo bar" indicates the phrase "foo bar" MUST appear

My experiments showed this. Unless I am missing something, to use the double quote option, you need to have two sets, with the inner set escaped. This is OK, but I feel like perhaps it could be better documented.

 

The other thing I have noted is a feature request to add support for excluding stopwords when doing partial matches. For example, "University of Queensland" is great when you're searching for the phrase, but if you are using multiple operators at a time, eg: *=~=~|= you really don't want that last operator (~|=) to match the "of" if the other words don't exist. Is there a way to do this already, or is it something that you could support fairly easily? Or, would you recommend that we do this at the server level using MySQL's "ft_stopword_file" setting?

[adrian]

@ryan - also wondering what you think about supporting multiple operator options in the ProcessPageSearch settings? I am quite liking the *=~*= combination.