module SearchEngine

[GhostRider]

@teppo Perhaps my comment from February got lost in the mix.

This search function is such a great addition to the ProcessWire ecosystem.

I am wondering if you would consider adding FieldsetPage fields to the list of searchable fields?

I use FieldsetPage fields often to create components. From there it is a mater of adding that one field to a template or a repeater to add the component.

Thank you for your time.

[teppo]

1 hour ago, GhostRider said:

I am wondering if you would consider adding FieldsetPage fields to the list of searchable fields?

I use FieldsetPage fields often to create components. From there it is a mater of adding that one field to a template or a repeater to add the component.

Sorry for taking so long to reply — this is now the top item on my todo list. I'll try to get it included sometime soon!

[GhostRider]

Amazing! @teppo Really appreciate all your effort.

[teppo]

@GhostRider, FieldsetPage support is now included with module version 0.20.0. I didn't have much to test with, but it did seem to work as expected on my limited use case — let me know if you run into any issues and I'll be happy to take a closer look ?

[GhostRider]

Hi @teppo. Thanks for addressing this. I'll give it a test drive later today or tomorrow on a current project.

[GhostRider]

@teppo Thanks for adding FieldsetPage support. It's working as expected!

[CalleRosa40]

@teppo Thanks for your help! Sorry for the delay (and sorry I have to get back to this). Yes, you're right, I use Hanna Code on my site. My code works fine though except when called by your indexer. I noticed another thing: When trying to save the page which calls my HC, PW gives the same error as above.

This is the content of my HC causing the trouble:

if($page->images_gallery->count()) { // images_gallery is a repeater field
	$gallery = "<div class='gallery'>";
	foreach($page->images_gallery as $img) {
	    if($img->image_caption) {
		    $titleAttr = "title='" . strip_tags($img->image_caption) . "'";
		    $imgCaption = "<div class='lightbox-caption'>$img->image_caption</div>";
	    } else {
	        $titleAttr = "";
	        $imgCaption = "";
	    }
		$gallery .= "
			<a href='{$img->image->url}' $titleAttr>
				<img src='" . $img->image->size(250, 250)->url . "' alt='" . $img->image->description . "'>
				$imgCaption
			</a>
		";
	}
	$gallery .= "</div>";
	echo $gallery;
}

This is tailored to output formatting being on, as is usually the case when HC is called. How does your indexer handle OF when it pulls content from fields? I'm still confused by the fact that triggering your indexer via API does NOT lead to any error. The error seems to appear only when the backend is "involved". Any ideas? Thanks a lot!

[999design]

Hi @teppo, thanks very much for creating this module.

We are looking to include a site search on a client's website. The website uses sub pages as on-page blocks that are rendered out as part of the parent page (allows for some nice flexible layouts). Do you know if this module would be suitable for also searching through these blocks and displaying the relative parent link in the search results?

[antpre]

Hy, Thanks for the module.

Just wanted to try it today. An upon uploading the module on my processwire installation. I got this parse error message (syntax error, unexpected '?' line 478).
I am running it on the latest stable processwire version and php 7.x.

Thanks in advance

 

[teppo]

On 4/16/2020 at 12:04 PM, CalleRosa40 said:

@teppo Thanks for your help! Sorry for the delay (and sorry I have to get back to this). Yes, you're right, I use Hanna Code on my site. My code works fine though except when called by your indexer. I noticed another thing: When trying to save the page which calls my HC, PW gives the same error as above.

...

This is tailored to output formatting being on, as is usually the case when HC is called. How does your indexer handle OF when it pulls content from fields? I'm still confused by the fact that triggering your indexer via API does NOT lead to any error. The error seems to appear only when the backend is "involved". Any ideas? Thanks a lot!

Sorry for the delay from my part as well... ?

Indexer grabs the value using $page->getFormatted($field_name). This means that output formatting is temporarily enabled for the page, value is read, and then output formatting is disabled (assuming it was disabled in the first place).

I'm not able to make this work via the API either — getting the same error there, so I'm also somewhat confused about this. Could mean that there's some difference in the way we're testing this. Anyway, both $page->save() and direct call to SearchEngine::indexPage($page) lead to the exact same error in my tests using your Hanne Code snippet ?

In this case the problem seems to be that your Hanna Code snippet is accessing Repeater Pages that were loaded with output formatting disabled (FieldtypeRepeater::___wakeupValue()). I've been trying to work around this for a while now, but to be honest I have no idea how to do that without causing potential issues and/or slowdowns elsewhere. I do have a couple of ideas I'd like to try, but it may take a while to figure this out, and even then the conclusion could be that this is simply too much of a hassle to automatically handle in SearchEngine.

The TL;DR here is that at least for the time being a slight modification is required for the Hanna Code snippet:

	foreach($page->images_gallery as $img) {
		$img->of(true);

(Or, alternatively, you could get $img->image with $img->getFormatted('image').)

I'll keep trying, but I can't give you any promises regarding a possible future Hanna Code compatibility enhancement yet ?

[teppo]

On 5/4/2020 at 11:52 AM, 999design said:

We are looking to include a site search on a client's website. The website uses sub pages as on-page blocks that are rendered out as part of the parent page (allows for some nice flexible layouts). Do you know if this module would be suitable for also searching through these blocks and displaying the relative parent link in the search results?

Hey @999design!

SearchEngine can handle pages that are stored in PageTable or Repeater fields automatically, but if you're literally using subpages and there's no backend / field level connection between the parent and the children, then this would (at least for now) require a custom hook. In other words you can use SearchEngine, but you'll have to add a bit of extra code to populate the index.

Something like this should do it:

$wire->addHookAfter('SearchEngine::savedPageIndex', function(HookEvent $event) {
	$page = $event->arguments[0];
	if ($page->template == 'ContainerPage' && $page->children->count()) {
		$searchEngine = $event->modules->get('SearchEngine');
		foreach ($page->children as $child) {
			$child_index = $searchEngine->indexPage($child, false, [
				'return' => 'index',
			]);
			$page->search_index .= "\n" . $child_index[0];
		}
		$page->save('search_index', [
			'quiet' => true,
			'noHooks' => true,
		]);
	}
});

Note that you need to use SearchEngine 0.21.0 for this to work; I just released a new version that made Indexer::indexPage() a bit more flexible. Also note that if you're building a multi-lingual site, indexPage() will return an array where the index is language ID, and you'll need to use setLanguageValue() to store it for each language one by one ?

[teppo]

9 hours ago, antpre said:

Hy, Thanks for the module.

Just wanted to try it today. An upon uploading the module on my processwire installation. I got this parse error message (syntax error, unexpected '?' line 478).
I am running it on the latest stable processwire version and php 7.x.

Hey @antpre!

Could you check what's on that line and paste it here, and also double check your PHP version? If I'm looking at the correct line of code, 478 should be this one:

    public function createIndexField(string $index_field_name, string $redirect_url = null): ?Field {

... and if that's the case, the only matching issue I can think of would be the use of nullable return type ("?Field"). Support for nullable return types was added in PHP 7.1, so this would suggest that either you're running an earlier version of PHP, or there's something else wrong with the setup (or perhaps I've got the wrong line) ?

[999design]

13 hours ago, teppo said:

Hey @999design!

SearchEngine can handle pages that are stored in PageTable or Repeater fields automatically, but if you're literally using subpages and there's no backend / field level connection between the parent and the children, then this would (at least for now) require a custom hook. In other words you can use SearchEngine, but you'll have to add a bit of extra code to populate the index.

Something like this should do it:

$wire->addHookAfter('SearchEngine::savedPageIndex', function(HookEvent $event) {
	$page = $event->arguments[0];
	if ($page->template == 'ContainerPage' && $page->children->count()) {
		$searchEngine = $event->modules->get('SearchEngine');
		foreach ($page->children as $child) {
			$child_index = $searchEngine->indexPage($child, false, [
				'return' => 'index',
			]);
			$page->search_index .= "\n" . $child_index[0];
		}
		$page->save('search_index', [
			'quiet' => true,
			'noHooks' => true,
		]);
	}
});

Note that you need to use SearchEngine 0.21.0 for this to work; I just released a new version that made Indexer::indexPage() a bit more flexible. Also note that if you're building a multi-lingual site, indexPage() will return an array where the index is language ID, and you'll need to use setLanguageValue() to store it for each language one by one ?

That's amazing, thank you very much for the detailed reply @teppo . We now have a few good options to suggest to our client.

Much appreciated ?

[antpre]

Thanks for your answer... and indeed i am runing php 7.0.33.

Regards

[Fuzen]

It’s been a while since I developed a new project with PW, but I have an upcoming client/project that could certainly use SearchEngine. I have a question about search results:

Is SearchEngine able to return results into groups? For instance, the client is a home builder/seller. If a user performs a search, it would be nice to return the results as sets. In their case, single-floor homes; two-story homes; etc. That’s just an example. The bottom line is that their homes are divided into categories, so it would be nice if users could see results grouped as such.

Is this possible?

[teppo]

17 hours ago, fuzenco said:

Is SearchEngine able to return results into groups? For instance, the client is a home builder/seller. If a user performs a search, it would be nice to return the results as sets. In their case, single-floor homes; two-story homes; etc. That’s just an example. The bottom line is that their homes are divided into categories, so it would be nice if users could see results grouped as such.

Is this possible?

At the moment SearchEngine doesn't have built-in support for this.

That being said: you could create a custom search page and use SearchEngine to index and (optionally) find items, which you then output manually. Or you could hook into Renderer::renderResults() and customize the output there. So yeah, there are ways to handle this, but it will require a bit of extra work ?

Out of interest: is each group going to contain items of a single template, i.e. in your example would "single-floor home" and "two-story home" be templates, or would this be some sort of option on a shared template (integer, options field, or something along those lines)? Grouping results by template might actually be a good idea as a built-in feature, just wondering if that'd work in your case.

[2hoch11]

On 7/16/2019 at 9:41 AM, teppo said:

…

Anyway, currently the ("full", i.e. a form and a results list) rendered state looks like this:

…

 

#1 "Full" result
How can I get this "full" result? The "standard" rendering outputs just the title and the URL.

#2 Languages
I assume that it needs "manual" rendering to output a language specific output … instead of english ("Search results", "x results for:", and so on) …

#3 Select (product) field
Taking consideration of #1 and #2 … I have a repeater matrix in the basic-page(es). I use manual indexing in the SearchEngine Module. RepeaterMatrix > Text fields are working well with this setup. I set the "search_index" field to "visible in editor" and this is what it stores (for example in french):

Vente au détail Solutions pour votre application de vente au détail Si vous êtes propriétaire d'un appartement, responsable d'un datacentre ou directeur d'hôtel, nous avons des exemples et des solutions pour chaque besoin d'application. Array Exemples & Solutions : Array Array Array Array 
{}

(…there's a lot of "Array" here… hmm…)

One of the repeater matrix items is just an option field (to select a product category [green products, yellow products, brown products…]). This is just an information what to render in frontend, but the product texts are stored in different pages. So there is no text/product-infos to store in the "search_index" field.

How can I manage to insert the selected products into the "search_index" field?

My first thought was to add a new field to the template… let's call it "search_dump". On every "page save" I populate this text area according to the RepeaterMatrix -> product_selection field (I guess this can be done by a hook). Then I have to recreate the search index manually in the backend GIU or by API:

$modules->get('SearchEngine')->indexPage($page);

[Fuzen]

On 5/23/2020 at 11:05 AM, teppo said:

Out of interest: is each group going to contain items of a single template, i.e. in your example would "single-floor home" and "two-story home" be templates, or would this be some sort of option on a shared template (integer, options field, or something along those lines)? Grouping results by template might actually be a good idea as a built-in feature, just wondering if that'd work in your case.

Generally, this would be a shared template as the homes all contain the same content blocks (title, description, number of bedrooms, number of bathrooms, picture gallery, etc.). The only difference really is the classification of homes. Years ago, I used to work with the CMS Expression Engine which would allow for this type of grouping. I can’t remember how it did it, but it was was useful.

[teppo]

2 hours ago, fuzenco said:

Generally, this would be a shared template as the homes all contain the same content blocks (title, description, number of bedrooms, number of bathrooms, picture gallery, etc.). The only difference really is the classification of homes. Years ago, I used to work with the CMS Expression Engine which would allow for this type of grouping. I can’t remember how it did it, but it was was useful.

Thanks for the clarification ?

Grouping can be done by pretty much any imaginable way with ProcessWire data types as well, but I was wondering if this would be solvable with something that's "generic enough" to be added to the module as a feature. Technically I could add settings for grouping with custom fields etc. as well, but that sounds like it would be a lot more complex — as a module setting, meaning that it would have to work with a lot of different configurations — than just by template.

[teppo]

On 5/25/2020 at 6:36 PM, 2hoch11 said:

#1 "Full" result
How can I get this "full" result? The "standard" rendering outputs just the title and the URL.

The view in the screenshot is what you should get by calling  <?= $modules->get('SearchEngine')->render() ?>.

If you mean that you get the search form and a list of results but the results don't have a summary field, then it's possible that you need to define the summary field — this is assuming that your pages don't actually have a filled-in summary field on them:

$config->SearchEngine = [
    'render_args' => [
        // Summary of each result (in the search results list) is the value of this field.
        'result_summary_field' => 'summary',
    ],
];

On 5/25/2020 at 6:36 PM, 2hoch11 said:

#2 Languages
I assume that it needs "manual" rendering to output a language specific output … instead of english ("Search results", "x results for:", and so on) …

These are all translatable strings — so no, manual rendering is not needed, you just need to translate the module files to your language ?

On 5/25/2020 at 6:36 PM, 2hoch11 said:

#3 Select (product) field
Taking consideration of #1 and #2 … I have a repeater matrix in the basic-page(es). I use manual indexing in the SearchEngine Module. RepeaterMatrix > Text fields are working well with this setup. I set the "search_index" field to "visible in editor" and this is what it stores (for example in french):

Vente au détail Solutions pour votre application de vente au détail Si vous êtes propriétaire d'un appartement, responsable d'un datacentre ou directeur d'hôtel, nous avons des exemples et des solutions pour chaque besoin d'application. Array Exemples & Solutions : Array Array Array Array 
{}

(…there's a lot of "Array" here… hmm…)

The array part is interesting. perhaps options field (if I understood the next part correctly and this is what you're using here) isn't getting indexed quite right; I'll take a closer look at that soon.

On 5/25/2020 at 6:36 PM, 2hoch11 said:

One of the repeater matrix items is just an option field (to select a product category [green products, yellow products, brown products…]). This is just an information what to render in frontend, but the product texts are stored in different pages. So there is no text/product-infos to store in the "search_index" field.

How can I manage to insert the selected products into the "search_index" field?

My first thought was to add a new field to the template… let's call it "search_dump". On every "page save" I populate this text area according to the RepeaterMatrix -> product_selection field (I guess this can be done by a hook). Then I have to recreate the search index manually in the backend GIU or by API:

$modules->get('SearchEngine')->indexPage($page);

The code I posted a few replies ago on this thread was actually almost directly applicable to your case as well. Here's a rough idea — you still need to fill in the blanks, most importantly the way you're fetching those products:

$wire->addHookAfter('SearchEngine::savedPageIndex', function(HookEvent $event) {
	$page = $event->arguments[0];
	if ($page->template == 'TemplateWithProducts') {
		$searchEngine = $event->modules->get('SearchEngine');

        // I don't know exactly how you're fetching those products; include your logic here :)
        $products = $pages->find('template=product, some_rule=' . $page->some_field);

        foreach ($products as $product) {
            $product_index = $searchEngine->indexPage($product, false, [
				'return' => 'index',
			]);
			$page->search_index .= "\n" . $product_index[0];
        }
		$page->save('search_index', [
			'quiet' => true,
			'noHooks' => true,
		]);
	}
});

Note that if this is a multi-lingual site, that will require some additional tweaks (indexPage() will return an index value per language, and you need to use setLanguageValue() to store it).

Let me know if any of this doesn't seem quite right, and I'll be happy to take a closer look ?

[montero4]

Hi,

I installed Search Engine and it looks like it's working. Thank you! I just noticed I now have this error every time I save a blog entry.

"Method Pageimages::width does not exist or is not callable in this context"

Based on a previous reply, it appears it's a hanna code issue, but this code does output an image on the front end... Is there something wrong with it?

 

$page=$pages->get($id); //gets the page
    
$image = $page->image->width(1240, ["suffix" => "srcset"]); //create image that is 1200px wide

Thank you for any clues on how to fix it.

Regards,

 

[teppo]

Hey @montero4,

First of all, just wanted to say that I really need to figure out some way to handle these conflicts with Hanna Code — it's becoming a common issue. I'll see what I can do about that soon ?

Technically there's nothing wrong in your code. It will work just fine on the front-end, where it's really intended to run. The problem is that in order to get useful index out of field content, SearchEngine actually gets it via $page->getFormatted(), which means that TextFormatters (such as Hanna Code) will also run in the admin side.

In this case you're using $pages->get() to get another Page, and since the default in the admin is that output formatting is off, this means that this new Page will have output formatting disabled — and that's why $page->image actually returns an instance of Pageimages (instead of a single Pageimage).

... anyway, just wanted to explain what's going on in here ?

Right now the easy fix would be checking for the type in the Hanna Code snippet. (Again, I'd like to automatically handle this in SearchEngine, but I can't say for sure when I'll get to that.)

$page=$pages->get($id); //gets the page
    
$image = $page->image instanceof Pageimages ? $page->image->eq(0) : $page->image;
$image = $image->width(1240, ["suffix" => "srcset"]); //create image that is 1200px wide

Alternatively you could make sure that output formatting state is on — it's mostly a matter of taste, really:

$page=$pages->get($id); //gets the page

$of = $page->of() // store initial output formatting state for later use
$page->of(true); // make sure that output formatting is on
    
$image = $page->image->width(1240, ["suffix" => "srcset"]); //create image that is 1200px wide

$page->of($of); // restore initial output formatting state

 

[dynweb]

Using the module on a multi-language site. When saving a page, I get the following error: "Call to a member function setLanguageValue() on string", File: .../modules/SearchEngine/lib/Indexer.php:99

 98:                        if ($index_field_exists) {
 99:                            $page->get($index_field)->setLanguageValue($language, $index[$language->id]);
101:                        }

I suppose we need a $page->of(false) here before calling setLanguageValue() ?

[Fuzen]

On 5/26/2020 at 10:41 AM, teppo said:

Thanks for the clarification ?

Grouping can be done by pretty much any imaginable way with ProcessWire data types as well, but I was wondering if this would be solvable with something that's "generic enough" to be added to the module as a feature. Technically I could add settings for grouping with custom fields etc. as well, but that sounds like it would be a lot more complex — as a module setting, meaning that it would have to work with a lot of different configurations — than just by template.

Yes, I can imagine it would be complex. I think ExpressionEngine did it by means of an inline group-by. For instance:

<li class="whatever" groupby="datacolumn">

And then there was a sort-by also:

<li class="whatever" groupby="datacolumn" sortby="whatever">

 

[xportde]

 

Hi @teppo,

thanks for the great module, we already use it with several customers.

I have a question concerning the savePageIndex-Hook: It works perfectly on saving a single page, but not, when I use the "Index pages now"-function in module settings page. How could I achieve this?

Best regards,

Thomas.