module SearchEngine

[markus-th]

On 10/14/2021 at 9:47 PM, tires said:

I chose the Operator:
~=
(i.E. All given words appear in compared value, in any order. Matches whole words. Uses “fulltext” index.)

But if i enter "everything 2021" i am not finding the page with the titel "everything is great in 2021".

Try this as Operator:

~%= or ~*=

I use this on all my sites with the AjaxSearch module, but it should work here as well.

Here you can find Ryan's blogpost: https://processwire.com/blog/posts/pw-3.0.160/

[sebr]

Hi

I get a fatal error when I click on "Debugger" in module settings page.

Return value of SearchEngine\Debugger::getWords() must be of the type array, null returned

File: .../modules/SearchEngine/lib/Debugger.php:599

Sorry I just update the module and I resolved it with the 0.30.3 version.

have a nice day

[ErikMH]

I’ve just set up searching with SearchEngine with basically all of the default options, and I’m really impressed! I‘ve turned on group-by-template, and it’s almost exactly what I want!

The template-named filtering buttons are terrific and intuitive.... I find that I’d like to render them as *plurals*, though, where the template names are *singular*. For example, I have a template *Person* that I would like to see as *People*, and another template *Journal* that I’d like to see as *Journals*.

Is there a built-in way to do this? If not, does anyone have any suggestions as to how I might go about implementing this.... Just guidance as to where I might put a switch/case statement would be acceptable, though I wouldn’t be surprised if there was a more elegant solution.

Thanks!

[ErikMH]

Well, it looks like I should do something with `tabs_tab` — but I’m really not sure what. Is this something I can change in site->config.php? I don’t see how....

[adrian]

 @gornycreative - if I understand correctly, this should work.

$searchEngine->addHookAfter('Renderer::renderTabLabel', function($event) use($types) {
    $event->return = $event->return . 's';
});

I was modifying the tabs a little more via a $types associative array, which you can see here. Just for fun I also included the hook I used to add images to the results.

Spoiler

$types = array(
    'All' => 'All',
    'Staff' => 'People',
    'Publication' => 'Publications',
    'Media' => 'Media Library',
    'Blog Post' => 'Blog Posts',
    'Enewsletter Article' => 'Enewsletter Articles',
    'Project' => 'Projects',
    'Education' => 'Education',
    'Education Program' => 'Programs',
    'Education Course' => 'Courses',
    'Page' => 'Pages'
);

$searchEngine = $modules->get('SearchEngine');

// cleanup for weird links like
// https://mysite.com/search/?q=test&t=blog_post>previous blog posts<%2Fa>).<%2Fp> <p><img title%3D
$t = $input->get('t');
$templateLabel = '';

if($t != '') {
    $template = $templates->get($t);
    if($template) {
        $templateLabel = $template->label;
    }
    else {
        // remove everything after (and including) first non-alphanumeric character
        preg_match("/[a-zA-Z0-9-_]+/", $t, $match);
        $t = $match[0];
        $template = $templates->get($t);
        if($template) {
            $templateLabel = $template->label;
        }
    }
}

if($t != '' && $t != $input->get('t')) {
    $session->redirect($page->url.'?q='.$input->get('q').'&t='.$t);
}
elseif($t != '' && !isset($types[$templateLabel])) {
    $session->redirect($page->url.'?q='.$input->get('q'));
}

$searchEngine->addHookAfter('Renderer::renderTabLabel', function($event) use($types) {
    $event->return = $types[$event->return];
});

$searchEngine->addHookAfter('Renderer::renderResult', function($event) {
    $p = $event->arguments[0];
    $image = $p->getunFormatted('pdf_images|image|images|video_images')->first();
    if($image) {
        $thumb = $image->size(0, 260, array('upscaling' => false));
        $event->return = '
        <div class="row">
            <div class="small-12 medium-4 large-5 xlarge-4 xxlarge-3 columns teaser__image">
                <a href="'.$p->url.'">
                    <img style="max-height: 300px" src = "'.$thumb->url.'" />
                </a>
            </div>
            <div class="small-12 medium-8 large-7 xlarge-8 xxlarge-9 columns teaser__content">
            ' . $event->return . '
        </div>';
    }
});

In the end I actually stopped using the built in tabs feature because even though it's really awesome, it doesn't work with multiple operators like: *=~= so I built up the tabs manually and now I can even change the operator if the user has wrapped something in double quotes to make it an exact match.

[gornycreative]

@adrian you answered @ErikMH tab question...

I was asking about the auto_desc being blank for partial word matches on the terms - but I can also see how this might get sticky unless you limited the generation to a single/first appearance. There could be dozens of places where a term partially matches on a page.

[teppo]

On 7/19/2021 at 5:24 PM, gornycreative said:

This has been great. I just wanted to confirm one thing I noticed which is that _auto_desc doesn't seem to highlight/load a summary on partial word matches, although the search itself picks them up.

So for example if I search for 'business' using a partial match operator, any article search_index that includes the whole word 'business' will create a summary with the word 'business' marked... and results will show up where businesses and businessmen are in the index, but no summary with a highlighted partial word appears. The summary for these entries is blank.

Is this the intended behavior?

Sorry for missing this question — and no, it doesn't seem like intended behaviour.

I've made some adjustments to the latest version, 0.30.4, that should make things better in this regard. At least it helped with these exact terms (business/businesses/businessmen) in my limited tests. Please let me know if this doesn't work, or if it makes things somehow worse, though ?

[olafgleba]

First, teppo, thanks  a lot for this module! It is really a great timesaver.

SearchEngine v. 0.30.5, PW 3.0.192 dev.

I struggle a bit with text translation feature (Labels, Placeholder a.s.o.). As i need to translate those text in several languages, initializing SearchEngine with $user->$language conditions statically is rather not an option.

I see the getDefaultStrings() function, but have no clue how to benefit from it. Maybe i overlook some docs. It would be very kind, if you or someone else could give me a brief overview how to set this up.

Currently i simply render SE straight out of the box (with some $config->SearchEngine = [...] tweaks before it).

Many thanks in advance, Olaf

[olafgleba]

Meanwhile i found the way to deal with my approach for translations. Possibly it helps others, so i post it here.

Tl;dr: I have a function to enable translation of static template strings. This allows me to reference a _strings.php file as site translation file in the PW Backend. To apply this to the SearchEngine Module, you have to edit some configuration settings before rendering.

$searchEngine = $modules->get('SearchEngine');

$config->SearchEngine = [
  'render_args' => [
    'strings' => [
      'form_label' => _t('Suche', 'search'),
      'form_input_placeholder' => _t('Ihr Suchbegriff...', 'search'),
      'form_submit' => _t('Suche', 'search'),
      'results_heading' => _t('Suchergebnisse', 'search'),
      'results_summary_one' => _t('Ein Ergebnis für "%s":', 'search'),
      'results_summary_many' => _t('%2$d Ergebnisse für "%1$s":', 'search'),
      'results_summary_none' => _t('Keine Ergebnisse für "%s".', 'search'),
      'errors_heading' => _t('Hm, ihre Suchanfrage konnte nicht ausgeführt werden', 'search'),
      'error_query_missing' => _t('Bitte tragen Sie einen Suchbegriff in das Eingabefeld ein', 'search'),
      'error_query_too_short' => _t('Bitte verwenden sie mindestens %d Zeichen für den Suchbegriff.', 'search'),
    ]
  ]
]

<?= $searchEngine->renderForm(); ?>
<?= $searchEngine->renderResults(); ?>

===

Approach to translate:

_func.php


/**
 * This function enables the translation of static template strings.
 * All translations are done in `_strings.php`. More info within the
 * comments of `_strings.php`.
 *
 * @param  string $text       text to translate
 * @param  string $context    Context to allow doubles
 * @param  string $textdomain Point static textdomain
 * @return function           Function for gettext parser
 */
function _t($text, $context = 'Generic', $textdomain = '/site/templates/_strings.php') {
  return _x($text, $context, $textdomain);
}

 

_strings.php


/**
 * Static translated template strings to use project wide
 * in any templates. This spares us defining page fields
 * for every piece of code that doesn't have to be editable
 * within the admin interface. The function doesn't need to
 * execute with PHP, so we wrap it in a comment leaving
 * the code only visible to the language parser.
 *
 * Relies on function `_t` (s. `_func.php`)
 *
 * Usage:
 * 1. Define string in markup, e.g. "_t('<string>', '<context>')"
 * 2. Insert string in `_strings.php`, e.g. "_x('<string>', '<context>')";
 *    both without the surrounding double quotes. This is just to avoid 
 *    this comment is parsed.
 * 3. Choose the language you like to translate the default language strings
 * 4. Under (possibly still empty) "Site Translation Files" click on "Translate File" to get
 *    a list of translatable files. Choose the `_strings.php` and click send/save. You'll
 *    see the translatable string phrases.
 */

/*!

_x('Suche', 'search')
_x('Ihr Suchbegriff...', 'search')
_x('Suchergebnisse', 'search')
_x('Ein Ergebnis für "%s":', 'search')
_x('%2$d Ergebnisse für "%1$s":', 'search')
_x('Keine Ergebnisse für "%s".', 'search')
_x('Bitte tragen Sie einen Suchbegriff in das Eingabefeld ein.', 'search')
_x('Bitte verwenden sie mindestens %d Zeichen für den Suchbegriff.', 'search')

*/

[Torsten Baldes]

@teppo

Great module! I just found it after years building my own indexes (with FieldtypeConcatenate).

I have a special case where I use a Repeater Matrix inside a Repeater Matrix (building columns). I added all the fields inside the inner repeater (including the repeater itself) to the indexed fields list. But the contents won't get indexed. For simple repeaters (just one level) it works as expected.

Is there a way to add the contents of the inner Repeater Matrix to the index?

Thanks!

[olafgleba]

Just as an addition to my post above. Maybe someone benefits from it...

In a multilanguage environment the HTML5 validation prevents controll over the output on search input errors (example: when the search string is too short). It skips translated strings and always gets populated depending on the browser language settings.

To disable the HTML5 validation, you must enhance the template render args for the form with the attribute novalidate.

$config->SearchEngine = [
  'render_args' => [
    'templates' => [
      'form' => '<form id="{form_id}" class="{classes.form}" action="{form_action}" role="search" novalidate>%s</form>',
    ],
  ],
]

With that, errors are displayed within the regular HTML Elements (which respects translated strings). I don't see any drawbacks so far.

See also the comment from @teppo below!

[teppo]

On 4/22/2022 at 12:23 PM, olafgleba said:

In a multilanguage environment the HTML5 validation prevents controll over the output on search input errors (example: when the search string is to short). It skips translated strings and always gets populated depending on the browser language settings.

To disable the HTML5 validation, you must enhance the template render args for the form with the attribute novalidate.

$config->SearchEngine = [
  'render_args' => [
    'templates' => [
      'form' => '<form id="{form_id}" class="{classes.form}" action="{form_action}" role="search" novalidate>%s</form>',
    ],
  ],
]

With that, errors are displayed within the regular HTML Elements (which respects translated strings). I don't see any drawbacks so far.

Quick note on this: I can definitely see how this could be an issue, and if you require full control over error messages then shutting down browser errors and relying on PHP defined ones instead is a good option.

That being said, personally I tend to keep browser level validation enabled due to a few reasons:

• Although these errors are "defaults" and it's difficult to alter them (constraint validation API is an option, but at least in my experience it's a bit heavy to use), these messages should be in the language chosen by the user. Not the language they're browsing the site in, but the language they're using at OS / browser level. Thus translating to the language of the site may, in some cases, have negative consequences.
• Personally I like to get feedback early, and thus it feels more intuitive when errors can be displayed before sending and processing the form.
• Finally I prefer to rely on HTML validation if/when possible for accessibility. PHP generated errors can be added in a way that makes them properly accessible, but it takes a bit of extra work, and typically requires some JS as well. At least in theory HTML level validation features should have the widest possible support ?

Anyway, just saying that there are valid reasons for both approaches — combining front-end and backend validation, vs. backend validation only.

[teppo]

On 4/13/2022 at 4:50 PM, Torsten Baldes said:

Great module! I just found it after years building my own indexes (with FieldtypeConcatenate).

Thanks, glad to hear you like it ?

On 4/13/2022 at 4:50 PM, Torsten Baldes said:

I have a special case where I use a Repeater Matrix inside a Repeater Matrix (building columns). I added all the fields inside the inner repeater (including the repeater itself) to the indexed fields list. But the contents won't get indexed. For simple repeaters (just one level) it works as expected.

Is there a way to add the contents of the inner Repeater Matrix to the index?

Did you add the "outer" Repeater Matrix field to indexed fields as well?

SearchEngine goes through all indexable fields (supported and configured to be indexed), and it should work fine with nested Repeater or Repeater Matrix fields, as long as every containing field is also set to be indexed. Thus if you have field rm_1 with another matrix field rm_2, you need to add both rm_1 and rm_2 to indexed fields.

Anyway, let me know if it still doesn't work and I can take a closer look!

[olafgleba]

On 4/29/2022 at 10:49 AM, teppo said:

Quick note on this: I can definitely see how this could be an issue, and if you require full control over error messages then shutting down browser errors and relying on PHP defined ones instead is a good option. That being said, personally I tend to keep browser level validation enabled due to a few reasons: [...]

@teppo I completely agree. Thank you for your important addition.

In situations where the search section is not a essential part of the UX of a site/project (with only low budget for this part), it is possibly o.k. to use just one approach. Still combining client- and server validation is to prefer in most cases.

[cst989]

Hello, this is a great module! 

Is it possible to make subfields of Profields: Combo selectable for indexing?

[xportde]

Hi @teppo, 

thank you for the great module! We use it for an internal project where the search function is very important.

Actually we have replaced some fields of type FieldtypeTextareas by FieldtypeCombo (both pro fields). Unfortunately FieldtypeCombo is not yet supported by SearchEngine, we have done this temporarily by the follwing lines in Indexer::___getIndexValue() at line 243:

            } else if ($field->type instanceof \ProcessWire\FieldtypeCombo) {
                return implode(
                    ' ... ',
                    array_filter(
                        array_values($page->get($field->name)->getArray()),
                        'strlen'
                    )
                );

Could you consider this in your next release of SearchEngine, please?

Best regards,

Thomas from XPORT.

[teppo]

On 5/16/2022 at 5:18 PM, cst989 said:

Is it possible to make subfields of Profields: Combo selectable for indexing?

 

On 5/17/2022 at 3:00 PM, xportde said:

Actually we have replaced some fields of type FieldtypeTextareas by FieldtypeCombo (both pro fields). Unfortunately FieldtypeCombo is not yet supported by SearchEngine, we have done this temporarily by the follwing lines in Indexer::___getIndexValue() at line 243:

            } else if ($field->type instanceof \ProcessWire\FieldtypeCombo) {
                return implode(
                    ' ... ',
                    array_filter(
                        array_values($page->get($field->name)->getArray()),
                        'strlen'
                    )
                );

Could you consider this in your next release of SearchEngine, please?

Sorry for the delay — Combo field is now supported in SearchEngine 0.31.0.

The end result should be very similar to that generated by @xportde modification to Indexer, but I decided to implement it in a slightly different way. Let me know if you run into any issues with this solution, though. Just trying to figure out an approach that would/could work for other, yet unforeseen fieldtypes in the future ?

[xportde]

@teppoWorks great, thank you very much!

[teppo]

Quick heads-up:

As of version 0.35.0 SearchEngine supports indexing file/image custom fields. In order to avoid surprises, I ended up adding these as separately selectable indexed fields. For an example: in order to have custom field "author_email" for image field "photo" indexed, you'll have to select "photo.author_email" from the "indexed fields" list in module config.

There's a "file_field.*" setting available as well, in case you want to index all existing custom fields for a specific file field.

Note that description and tags also need to be specifically selected. In earlier versions description text was always indexed, but now if you select the file field name and nothing more, only file name(s) (and content, if you have a suitable file indexing module installed and enabled) are indexed.

--

In order to index file/image custom fields I had to refactor parts of the Indexer class. I hope I didn't break too many hooks, but there's a possibility that something goes wrong. Please test carefully, and let me know if you run into any issues!

[ngrmm]

@teppo I get this error when I add a repaterMatrix field to indexed fields

 

TracyDebugger:

TypeError

SearchEngine\IndexValue::__construct(): Argument #1 ($value) must be of type string, null given, called in /…/htdocs/domain.com/_NEW/site/modules/SearchEngine/lib/Indexer.php on line 297 search►

––––

File: .../domain.com/_NEW/site/modules/SearchEngine/lib/IndexValue.php:33

23:         *
24:         * @var array
25:         */
26:        protected $meta = [];
27:    
28:        /**
29:         * Constructor
30:         *
31:         * @param string $value
32:         */
33:        public function __construct(string $value = '') {
34:            $this->value = $value;
35:        }
36:    
37:        /**

using pw 3.0.200 © 2022, search engine 0.35.0  and php 8

[teppo]

Thanks for reporting the issue, @ngrmm. This should be fixed in SE 0.35.1.

I wasn't able to reproduce this issue myself, so please let me know if the problem still persists and I'll be happy to take a closer look ?

[ngrmm]

On 8/4/2022 at 6:01 PM, teppo said:

Thanks for reporting the issue, @ngrmm. This should be fixed in SE 0.35.1.

I wasn't able to reproduce this issue myself, so please let me know if the problem still persists and I'll be happy to take a closer look ?

yes, your update fixes the issue, thanks!

[BrendonKoz]

I'd like to say that although it might've been easier on the developer (and more like PW field template rendering) to allow a single HTML-esque template file for overriding the default templating, I very much appreciate the ability to create our own custom arguments within the config's `render_args` array. It made it easier to customize my own template (which used additional elements and attributes) while still offering me to create, additionally named, custom keys in the multidimensional array! Because of that, looking back at my custom template file config it should make more sense what property is controlling what area of the template strings within the form.

The only thing that I felt silly about was not realizing the `minified_resources` was a boolean setting in the config and not an automatic check via the filesystem. I prefer how it is configured as opposed to what I thought it was doing, but I didn't realize it at first. Oops! An excellent module - thanks, teppo!

Questions:

I'm going to continue investigating, but...is it possible to access a search result page object? I'd like to render the breadcrumb per search result, but I'm not (yet?) seeing how to access the returned $page properties from the find operation within the templates. I think I'll likely need to use a hook for this but haven't yet dived that far into things. [EDIT: Okay, looks like I hook into renderResult.]

Can the `limit` setting value be made to allow the theme config to override? It seems I currently have to set this on the site config level. The below snippet was not working for me.
 

$theme_args = [
    'find_args' => [
        'limit' => 10,
    ],
    'theme_styles' => [],
    'render_args' => [...]
];

EDIT:

Thanks to adrian I now see I can manually pass in the limit value by calling `find()` and passing in a query object. It might be easier to simply offer `limit` within the theme config, but at least for now, this is also just as easy. ? In fact it might be easier since one doesn't have to worry about the SearchEngine theme and can just handle it in their template file.

$searchEngine = $modules->get('SearchEngine');
$findOptions = ['limit' => 25];
$query = $searchEngine->find($sanitizer->selectorValue($input->get->q), $findOptions);
$searchEngine->renderResults($renderOptions, $query);

[BrendonKoz]

After adding my larger RepeaterMatrix field to the indexed_fields, on my first page save test, I seem to be running into a warning at line 315 of /lib/Indexer.php, "foreach() argument must be of type array|object, null given". I'm not sure why it's detecting the value as null... I'm not sure what I'm doing wrong here. Any thoughts? For reference, the snippet of code for this method is below:

<?php
protected function ___getRepeatableIndexValue(\Processwire\Page $page, \ProcessWire\Field $field, array $indexed_fields = [], string $prefix = ''): array {
    $index = [];
    $index_num = 0;
    $prefixes = $this->getOptions()['prefixes'];
    foreach ($page->get($field->name) as $child) {

At this moment within the loop, the value of the referenced parameters are:

• $page = \ProcessWire\RepeaterMatrixPage ... it's forPage attribute is null (perhaps that's why?) but its forField value is my primary repeater field: $content_body
• $field = \ProcessWire\RepeaterField ... this field, `faqs` is itself a repeater (not matrix) field, which contains a combo field (faq), which has attribute names "question" (text), and "answer" (CKEditor)

I believe I have all relevant fields added to the indexed fields selection for SearchEngine within this context: content_body, faqs, and faq. Does SearchEngine support RepeaterMatrix (and/or Repeater) fields with multiple levels of depth?

ADDENDUM: I just noticed that on this particular page I'm saving (as a test with these fields added), the RepeaterMatrix doesn't allow the use of faqs (it's a template-based override), nor are there any faqs in the content of the page.

[Clarity]

Hello, @teppo and everyone!
Can you please tell me how to modify the query during search?