module SearchEngine

[adrian]

Thanks for the explanation @teppo - I started to realize how it works as I played around with things.

I like your initial thinking, although I do think it's slightly confusing, but I also don't think it's worth your time to reconfigure.

I have another feature request for you ?  I know I could do this if I went with the json rendering of results and then manipulating and rendering myself, but I think it would be great if the "Indexed Templates" config setting was an ASM Select field so we could choose template order. I'd love to see this template order used to group and order the rendered search results. I think it would be nice to have the option for using the template label to add group headings to the results. It would also be good to have an option for presenting these template grouped results into different tabs - a bit like the PW website's search results.

Actually, speaking of the PW website search functionality, maybe adding an automatic option for AJAX rendering of results below the search box would also be nice.

I know all of these are possible with manual rendering, but I think we could all benefit from an inbuilt implementation.

BTW, this module really is awesome - thank you!

[teppo]

13 minutes ago, adrian said:

I have another feature request for you ?  I know I could do this if I went with the json rendering of results and then manipulating and rendering myself, but I think it would be great if the "Indexed Templates" config setting was an ASM Select field so we could choose template order. I'd love to see this template order used to group and order the rendered search results. I think it would be nice to have the option for using the template label to add group headings to the results. It would also be good to have an option for presenting these template grouped results into different tabs - a bit like the PW website's search results.

Interesting idea — added to the backlog ?

13 minutes ago, adrian said:

Actually, speaking of the PW website search functionality, maybe adding an automatic option for AJAX rendering of results below the search box would also be nice.

I know all of these are possible with manual rendering, but I think we could all benefit from an inbuilt implementation.

I have a "mostly functional" proof of concept of this on one of my sites. This will likely be the next feature I work on after I get current dev branch merged ?

13 minutes ago, adrian said:

BTW, this module really is awesome - thank you!

?‍♂️

[adrian]

1 minute ago, teppo said:

Interesting idea — added to the backlog

Just to give you a simple example of why I think this is a must-have feature. I am working on a site where the staff are mentioned in blog posts, but obviously if a user types in the person's name, the first result they want to see is the staff member's info page, rather than the blog posts that mention them.

4 minutes ago, teppo said:

I have a "mostly functional" proof of concept of this on one of my sites.

Awesome - I'll hold off on doing this myself.

[adrian]

Sorry, one more thought on the template search priority order. I think it would be great to have an option so that the template of the current page (and it's child page's template) were automatically given higher priority over the order set in the config settings. Or some other way to specify the order of priority depending on the page being searched from.

[teppo]

Just released a new version of the module — here's the full changelog:

### Added
- Support for Indexer actions, the first one of which adds support for rendering FormBuilder forms as part of the search index. This feature is currently considered experimental.
- Option to specify custom path for front-end themes via the Advanced settings section in module config.
- EditorConfig (.editorconfig) and ESLint (.eslintrc.json) config files for defining coding style for IDEs.
- Support for collapsible content blocks in Debugger.

### Changed
- Indexed templates option in module config is now AsmSelect, which makes it possible to organize indexed templates by preferred priority.
- Query class converts lesser than and greater than to HTML entities to allow matching said entities, as well as encoded HTML markup.
- All Debugger CSS classes refactored to follow `.pwse-` format.

 

On 2/9/2021 at 10:00 PM, adrian said:

Just to give you a simple example of why I think this is a must-have feature. I am working on a site where the staff are mentioned in blog posts, but obviously if a user types in the person's name, the first result they want to see is the staff member's info page, rather than the blog posts that mention them.

The first part of this is now built in, i.e. the indexed templates setting uses AsmSelect and the order of indexed templates gets stored. The module doesn't yet actually do anything with this information, so there's more work to be done. That'll be the focus of next release ?

On 2/9/2021 at 10:13 PM, adrian said:

I think it would be great to have an option so that the template of the current page (and it's child page's template) were automatically given higher priority over the order set in the config settings. Or some other way to specify the order of priority depending on the page being searched from.

I'll give this more thought once I (hopefully) figure out the initial template order thing.

[adrian]

Thanks @teppo - looking forward to seeing all these new features.

I have another one for you ?

Whenever I build a search engine I emulate Google's "search phrase" use of double quotes to effectively switch from ~= to *=

Would you be willing to support something like that also?

[teppo]

6 minutes ago, adrian said:

Whenever I build a search engine I emulate Google's "search phrase" use of double quotes to effectively switch from ~= to *=

Would you be willing to support something like that also?

Just to make sure: do you mean that the chosen operator depends on whether the search query is wrapped in quotes, or that a single query can contain (multiple of) both? ?

First one would be relatively simple; this would likely mean extending module config with an optional "literal match selector" setting, which would then be used in case there are quotes around the term. If it's the latter, I'd be interested in hearing how this works.

[adrian]

@teppo - your first description is all I am really looking for. Obviously the second would be great, but I expect impossible with PW selectors.

 

 

[adrian]

Hi @teppo - I've noticed a few issues with apostrophes when searching.

The first issue is perhaps not something to can easily fix if you rely on PW selectors to do the search, but it is a shame that omitting an apostrophe from the search term results in no matches.

 

 

The second is more fixable - notice how the results don't include the _auto_desc when you enter a search term with an apostrophe. The second screenshot shows the same returned results, but this time I searched for the phrase without the word with the apostrophe and the _auto_desc is included as expected.

 

The final thing I noticed is that when the search term has an apostrophe, input->get->q ends up being empty. I was using it to populate the search input field so the user gets the results page with the search populated with the phrase they just searched for.

Thanks.

 

[adrian]

Hi again @teppo - I am looking to include an image for each result. The key thing is that I want to show a thumbnail version so I want to be able to call PW's size() method on the image object, rather than simply using {item.images.first.url}

I think there are probably a few different ways to accomplish this - maybe by doing a str/preg_replace on the rendered results, or maybe hooking into one of the render() methods within your module, or perhaps doing a completely custom rendering of results, although I am currently not sure how to show the _auto_desc when doing a custom render. Is there a trick for this?

Otherwise, I wonder if this is something you think might be a good feature to include so it's easier to do this via the standard renderResults() method?

Thanks for your thoughts / ideas.

[teppo]

1 hour ago, adrian said:

Hi again @teppo - I am looking to include an image for each result. The key thing is that I want to show a thumbnail version so I want to be able to call PW's size() method on the image object, rather than simply using {item.images.first.url}

I think there are probably a few different ways to accomplish this - maybe by doing a str/preg_replace on the rendered results, or maybe hooking into one of the render() methods within your module, or perhaps doing a completely custom rendering of results, although I am currently not sure how to show the _auto_desc when doing a custom render. Is there a trick for this?

I'd recommend hooking into Renderer::renderResult or Renderer::renderResultDesc, depending on your needs. Both are related to rendering a single result and have access to the Page object in question, as well as the Data object containing predefined template strings etc.

Let me know if you run into any trouble. As for whether this might be a good feature addition — perhaps, though I've only had need for this once so far, so I'm not certain yet. If it's easy to add via hook and a relatively rare need, then it's probably best to leave it at that ?

I'll take a closer look at the apostrophe issues later!

[adrian]

Thanks @teppo - hooking into Renderer::renderResult worked great!

Gotta say, this is a really well thought out, and well coded module.

Cheers!

[adrian]

Just a tip - I replaced the "Page fields to search" setting in the PW core Page Search (ProcessPageSearch) module to use the search_index field. It seems like a great alternative when you want to have the admin live search find pages based on the content of lots of fields (not just title etc).

[adrian]

@teppo - it would be great when using the %= operator if the auto description could also show the separate words where they appear, rather than showing no text at all. Does that make sense?

Thanks.

[teppo]

Took a lot longer than anticipated to get this out of the oven, but version 0.29.0 of SearchEngine was just released. Word of warning: this is the biggest new release for the module so far (in terms of features as well as changed lines of code) so please test carefully!

Main features included in this release:

• New group_by option in find_args. For now only supported values are null (default) and "template", the latter of which (obviously) groups results by template. Here grouping means two things: first of all results are rendered in tabs on the front-end, and second of all instead of a single Query object the module instead returns a QuerySet object, which in turn contains multiple Query objects.
• New results_grouped_by option in render_args. This accepts any field name, page property, etc. and essentially divides found results into groups by said field value / property. For an instance if you specify "template.label", you'll get a list of results with subheadings for templates.

While both of these options are related to grouping results, they work on an entirely different level: group_by performs separate SQL queries and splits results into their own Query objects, while results_grouped_by is something that only affects the render phase (front-end). It's also possible to combine the two, in case that makes sense ?

There's also a new custom sort setting "_indexed_templates" (results are sorted by the order of templates in the "indexed templates" config setting) and new argument "pinned_templates" for the find() method that will show results with provided templates at the top of the results list.

[adrian]

Hi @teppo - I am excited for these new features - thanks for you hard work. Unfortunately, when I install it I get this error:

Deprecated: Required parameter $query follows optional parameter $args in /Users/ajones/Sites/ecoreportcard/site/modules/SearchEngine/lib/Renderer.php on line 615 Deprecated: Required parameter $type follows optional parameter $args in /Users/ajones/Sites/ecoreportcard/site/modules/SearchEngine/lib/Renderer.php on line 718 Deprecated: Required parameter $type follows optional parameter $args in /Users/ajones/Sites/ecoreportcard/site/modules/SearchEngine/lib/Renderer.php on line 755

and then when I try to do a search, I get:

SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ':pf19s0X IN BOOLEAN MODE) ) )) \n
LEFT JOIN templates AS pwse_t ON pwse_t.id=pa...' at line 4

Reverting to the last version instantly fixes these.

Note that for my testing I am using the simple:

<?= $modules->get('SearchEngine')->render() ?>

approach to handle everything.

Let me know if you need me to dig a little deeper.

[teppo]

Thanks @adrian! First one was a PHP8 issue, while the second (and bigger) problem was related to fulltext index queries. Both should now be fixed in version 0.29.1, though please let me know if it doesn't work for you.

Also: I had accidentally grouping by template enabled by default, this is now disabled by default.

[adrian]

Thanks @teppo - all those errors are now gone.

I am trying out the group by template options but not seeing any delineation in the rendered results, with them all being in the same <ul> block. Perhaps I am missing something.

$searchEngine = $modules->get('SearchEngine');

echo $searchEngine->renderStyles();
echo $searchEngine->renderScripts();

$findOptions = [
    'group_by option' => 'template'
];

$renderOptions = [
    'results_grouped' => 'template.label'
];

$query = $searchEngine->find($input->get->q, $findOptions);

echo $searchEngine->renderResults($renderOptions, $query);

I am also curious how you have handled pagination of results in this tabbed results scenario (since I am not getting it working, I am not sure yet whether this works or not).

What I actually ended up doing for my project that just went live is to return all results, but then have buttons that look like tabs, but each one is for a different template and I use them to add a "type" (template) parameter to the query string, eg: https://ian.umces.edu/search/page2?type=media&q=seagrass - this ensures that pagination works within the results for that template. 

OT, but I am also seeing errors with the Debugger.

1) The Debug Index, Debug Page and Reindex Page buttons always gives me this error: array_map(): Argument #2 ($array) must be of type array, null given in /modules/SearchEngine/lib/Debugger.php:628

2) The Debug Query always give me: PHP Warning: Undefined array key "active" in .../modules/SearchEngine/lib/Renderer.php:371

 

 

 

[3fingers]

Quote

...and I use them to add a "type" (template) parameter to the query string

How, if I can ask? :)

[adrian]

Spoiler

<?php namespace ProcessWire;

$searchEngine = $modules->get('SearchEngine');

$searchEngine->addHookAfter('Renderer::renderResult', function($event) {
	$p = $event->arguments[0];
	$image = $p->getunFormatted('pdf_images|image|images|video_images')->first();
	if($image) {
		$thumb = $image->size(0, 260, array('upscaling' => false));
		$event->return = '
		<div class="row">
			<div class="small-12 medium-4 large-5 xlarge-4 xxlarge-3 columns teaser__image">
				<a href="'.$p->url.'">
					<img style="max-height: 300px" src = "'.$thumb->url.'" />
				</a>
			</div>
			<div class="small-12 medium-8 large-7 xlarge-8 xxlarge-9 columns teaser__content">
			' . $event->return . '
		</div>';
	}
});

$types = array(
	'all' => 'All',
	'staff' => 'People',
	'publication' => 'Publications',
	'media' => 'Media Library',
	'blog_post' => 'Blog Posts',
	'enewsletter_article' => 'Enewsletter Articles',
	'project' => 'Projects'
);

$findOptions = [
	'limit' => 25,
	'sort' => '-date, -published, title',
	'operator' => '~=',
	'query_param' => 'q'
];

if($input->get->type && $input->get->type != 'all') {
	if(array_key_exists($input->get->type, $types)) {
		$type = $input->get->selectorValue('type');
		$input->whitelist('type', $type);
		$findOptions['selector_extra'] = 'template=' . $type;
	}
}

$renderOptions = [
	'result_summary_field' => '_auto_desc',
	'templates' => [
		'result_path' => '<div class="{classes.result_path}">{item.template.label}</div>',
	],
	'pager_args' => [
		'center' => true,
		'nextItemLabel' => "Next <span></span>",
		'nextItemClass' => 'pager-item',
		'previousItemLabel' => "<span></span> Prev",
		'previousItemClass' => 'pager-item',
		'lastItemClass' => '',
		'currentItemClass' => 'current',
		'separatorItemLabel' => '<span>&hellip;</span>',
		'separatorItemClass' => 'uk-disabled',
		'listMarkup' => "<ul class='pagination text-center'>{out}</ul>",
		'itemMarkup' => "<li class='{class}'>{out}</li>",
		'linkMarkup' => "<a href='{url}'>{out}</a>",
		'currentLinkMarkup' => "{out}"
	],
	'strings' => [
		'results_heading' => (isset($type) ? $types[$type] : 'All results'),
	],
];

$query = $searchEngine->find($input->get->q, $findOptions);

?>

<div id="page-content" class="content" pw-replace>
    <div class="panel--default-layout">
		<div class="row">
			<div class="small-12 medium-12 large-10 large-offset-1 columns main">
				<div class="panel-pane pane-node-content">
					<div class="pane-content">
						<div class="content">
							<?php
							foreach($types as $name => $label):
								if($name != 'all' && $pages->has('search_index'.$findOptions['operator'].$query->query.', template='.$name) === 0) continue;
								?>
								<a class="button small<?=((!isset($type) && $name == 'all') || (isset($type) && $name === $type) ? ' buttonactive' : '')?>" href="<?=$page->url?>?q=<?=$input->get->q?>&type=<?=$name?>"><?=$label?></a>
							<?php endforeach;
								echo  $searchEngine->renderResults($renderOptions, $query);
							?>
						</div>
					</div>
				</div>
			</div>
        </div>
    </div>
</div>

 

Please ignore the awful zurb foundation markup - this was an ASAP conversion of a site from an unusable Drupal setup so I rebuilt using the old site's markup / css. At some point this will be redone as well, but at least it all works now ?

You can see though that each button adds the type and then I use that like this:

$findOptions['selector_extra'] = 'template=' . $type;

You can ignore the hook on Renderer::renderResult at the top - it's just for adding the thumbnails to the results.

[3fingers]

Thanks adrian, I had to put my karate headband on and take some time to study your code (eg. right now I don't get where the code below came from).

$args = $query->original_args;

I'll do some research ?

...by the way using tailwindcss lately I'm not negatively impressed by the markup you mentioned ?

Thanks!

[teppo]

2 hours ago, adrian said:

 

$searchEngine = $modules->get('SearchEngine');

echo $searchEngine->renderStyles();
echo $searchEngine->renderScripts();

$findOptions = [
    'group_by option' => 'template'
];

$renderOptions = [
    'results_grouped' => 'template.label'
];

$query = $searchEngine->find($input->get->q, $findOptions);

echo $searchEngine->renderResults($renderOptions, $query);

 

So I just had a two-hour nap by accident (...) and perhaps I'm not fully awake yet, but if you are literally using "group_by option" as the option name, it should be just "group_by".

2 hours ago, adrian said:

I am also curious how you have handled pagination of results in this tabbed results scenario (since I am not getting it working, I am not sure yet whether this works or not).

What I actually ended up doing for my project that just went live is to return all results, but then have buttons that look like tabs, but each one is for a different template and I use them to add a "type" (template) parameter to the query string, eg: https://ian.umces.edu/search/page2?type=media&q=seagrass - this ensures that pagination works within the results for that template. 

Pretty much the same approach as you've got there.

Unless the (somewhat experimental and likely not fully functional) "autoload_result_groups" render option is enabled, SearchEngine also adds a query param (find_args group_param, default name for this is "t"), so that the final URL is something like ?t=media&q=seagrass. The autoload option loads all groups right away, and requires some JS trickstery to get things working; I'm pretty sure that paging won't work there as expected, this i mostly something I added for the Debugger ?

2 hours ago, adrian said:

OT, but I am also seeing errors with the Debugger.

1) The Debug Index, Debug Page and Reindex Page buttons always gives me this error: array_map(): Argument #2 ($array) must be of type array, null given in /modules/SearchEngine/lib/Debugger.php:628

2) The Debug Query always give me: PHP Warning: Undefined array key "active" in .../modules/SearchEngine/lib/Renderer.php:371

Thanks for reporting, I'll take a closer look at these next ?

[adrian]

I'm actually not sure why I used that and then

$args['operator']

instead of simply:

$findOptions['operator']

I can't see any reason at the moment not to change to the simpler version.

[adrian]

@teppo - I think I am the one who was asleep - not sure how I ended up with 'group_by option' but if I change it to 'group_by' then I get this error (maybe also PHP8 related)

SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ':pf19s0X IN BOOLEAN MODE)) ) )) \n
LEFT JOIN templates AS pwse_t ON pwse_t.id=p...' at line 4

[teppo]

2 hours ago, adrian said:

@teppo - I think I am the one who was asleep - not sure how I ended up with 'group_by option' but if I change it to 'group_by' then I get this error (maybe also PHP8 related)

SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ':pf19s0X IN BOOLEAN MODE)) ) )) \n
LEFT JOIN templates AS pwse_t ON pwse_t.id=p...' at line 4

This should be fixed now in 0.29.2. The module was relying on DatabaseQuerySelect::getQuery(), which no longer returns usable queries as of 3.0.158 (or something along those lines) ?