Jump to content
teppo

SearchEngine

Recommended Posts

SearchEngine 0.10.0 was just released. Not a whole lot of stuff in this release – basically just one new feature, one minor addition to the default theme CSS, and some PHPDoc improvements.

### Added
- New Renderer::___renderResultsJSON() method for rendering search results as a JSON string.
- Additional CSS rules to make sure that visited links appear correctly in the default output.

While building an AJAX suggest search feature it occurred to me that it would be nice if SearchEngine could return search results as JSON out of the box. Newly added renderResultsJSON() method provides this capability, and new settings results_json_fields and results_json_options allow customising what gets returned, and how.

More details (and an example of using this feature) in the README: https://github.com/teppokoivula/SearchEngine#json-output.

  • Like 4

Share this post


Link to post
Share on other sites

Hello - Thanks for your great module!

I have a special need for the side, I'm working on right now:  Is there a possibility not to show the "result_summary_field" in the rendered results, but the 'near' content of the search-match in the search field itself - with the match highlited?

So lets say there is this content in the search_index field: ' Hey - this is just a simple little test for the search engine!'
I am searching for the phrase 'little'.
The rendered result now should show me lets say: '... just a simple little  test for the s...'

To be honest: My php-skills are not that good to understand all the files of your module -  but as far as I can see, I am afraid that it will not be possible in an easy way to change the code for my needs!? 

Thanks for any reply! 

  • Like 1

Share this post


Link to post
Share on other sites
On 9/24/2019 at 11:02 PM, planmacher said:

I have a special need for the side, I'm working on right now:  Is there a possibility not to show the "result_summary_field" in the rendered results, but the 'near' content of the search-match in the search field itself - with the match highlited?

Assuming that I correctly understood your need, this would require dynamically generating the descriptions. Technically it should be doable by hooking into Renderer::RenderResultDesc(), but you'd have to provide the entire description generation and highlighting logic yourself, so it might be a bit difficult to achieve in practice.

This is actually something that I've got planned, but I'm not really sure when I'll get to it. Anyway, it's good to know that there's demand for it.

Share this post


Link to post
Share on other sites

Couple of updates during the past week, 0.11.0 and 0.11.1. I'll attach the changelog below, but here's a summary of changes:

  • Saving just a single field (via API or perhaps some async feature) should now correctly regenerate the search index.
  • Page Reference fields are supported: the title and name values of referenced pages are stored in the index. Page names can also be search with the field name; if, say, you have a Page Reference field called "tags", you can find specific matches by searching for "tags:[tag-name]" ("tags:tailwind" etc.)
  • One relatively big change behind the scenes is that now the search index is generated after page has been saved, by hooking into Pages::savedPageOrField and triggering a new save for just the index field, quietly and with hooks disabled.

Originally the module hooked into Pages::saveReady and generated the index so that it got saved along with any user-provided changes, but it turns out that this could result in some rather obscure bugs due to output formatting being enabled on the fly (which is intentional, and required for the most representative search index) which in turn was triggering interesting side effects under some specific conditions. So far the new method hasn't resulted in any unexpected side effects as far as I can tell, but it's worth pointing out this is a pretty big update.

## [0.11.1] - 2019-09-26

### Changed
- Index value gets saved in Pages::savedPageOrField instead of Pages::saved.

## [0.11.0] - 2019-09-26

### Added
- Support for indexing Page Reference fields.
- Support for indexing non-field Page properties (id, name).
- New hookable method Indexer::___getPageReferenceIndexValue().

### Changed
- Index value gets saved in Pages::saved instead of Pages::saveReady so that we can avoid messing with the regular save process.

### Fixed
- Fixed the "save" behaviour of the Indexer::indexPage() method.

 

Share this post


Link to post
Share on other sites

Hi Teppo, is there a way to make the search language aware? - I mean getting the right context from mutilanguage fields per active language? Probably this would mean an indexing per language?

If not, I wonder if at least all language text entries from a multi language field get indexed? - so a search in the non default language might find maybe too much, but not too little?

  • Like 1

Share this post


Link to post
Share on other sites
On 9/30/2019 at 11:01 AM, ceberlin said:

Hi Teppo, is there a way to make the search language aware? - I mean getting the right context from mutilanguage fields per active language? Probably this would mean an indexing per language?

Sorry for the late reply. This is definitely something I'll have to figure out, one way or another, just haven't had the time (or need for that matter) yet.

I'll have to do some testing first and get back to this later 🙂

  • Like 1

Share this post


Link to post
Share on other sites

Hi Teppo, I'm close to build my first-time-ever search engine in processwire and wonder if you have made progress into the multilanguage side of your cool module 🙂
Not a big deal anyway, otherwise I will try another approach 🙂

Thanks!

  • Like 1

Share this post


Link to post
Share on other sites

Just a FYI @teppo I'll definitely need a site search on my next project so I'll try your module soon. I've found https://markjs.io/ today and it seems to be great (and MIT). Maybe this could be implemented into your module for highlighting the results? Or did you have any other solutions in mind?

Would you be interested in me bringing markjs into your module or would you prefer if I build something seperate?

Share this post


Link to post
Share on other sites
1 hour ago, bernhard said:

Just a FYI @teppo I'll definitely need a site search on my next project so I'll try your module soon. I've found https://markjs.io/ today and it seems to be great (and MIT). Maybe this could be implemented into your module for highlighting the results? Or did you have any other solutions in mind?

Would you be interested in me bringing markjs into your module or would you prefer if I build something seperate?

Hey Bernhard!

SearchEngine already provides a way to highlight hits found from the generated summary, but since there's currently no built-in way to show dynamic summaries on the search page, this doesn't apply to the entire index. See here for an example: https://wireframe-framework.com/search/?q=wireframe.

Please let me know if I've completely misunderstood what markjs does – I'm interested, but not quite sure if it would be beneficial, and/or how it would differ from current situation 🙂

 

Share this post


Link to post
Share on other sites
1 hour ago, 3fingers said:

Hi Teppo, I'm close to build my first-time-ever search engine in processwire and wonder if you have made progress into the multilanguage side of your cool module 🙂
Not a big deal anyway, otherwise I will try another approach 🙂

This is on the top of my to-do list now 🙂

  • Thanks 1

Share this post


Link to post
Share on other sites
37 minutes ago, teppo said:

Hey Bernhard!

SearchEngine already provides a way to highlight hits found from the generated summary, but since there's currently no built-in way to show dynamic summaries on the search page, this doesn't apply to the entire index. See here for an example: https://wireframe-framework.com/search/?q=wireframe.

Please let me know if I've completely misunderstood what markjs does – I'm interested, but not quite sure if it would be beneficial, and/or how it would differ from current situation 🙂

Thx, seems that I missed that 🙂 

  • Like 1

Share this post


Link to post
Share on other sites

@ceberlin @3fingers

As of 0.12.0 SearchEngine now supports multi-language indexing and searching. This is based on the native language features, so the results you see depend on the language of current user etc. While I don't have a good test case at hand right now, I did some quick testing on one of my own sites and it seemed to work pretty much as expected – though please let me know if there are problems with the latest version 🙂

What you need to do to enable this is convert the index field (which is by default called search_index) from FieldtypeTextarea to FieldtypeTextareaLanguage.

  • Like 3

Share this post


Link to post
Share on other sites

Thanks @teppo ! You rock! 🙂

I will report back my experience with your module as soon as my client give me the ok with the implementation.

Once again, thanks!

Edit:

I found couple typos in the README inside the "manual approach" section (a double echo and a $searchEngine declaration referenced as $searchengine w/o camel casing), I paste the correction below as hidden:

Spoiler

<?php namespace ProcessWire;
$searchEngine = $modules->get('SearchEngine');
?>
<head>
    <?= $searchEngine->renderStyles(); ?>
    <?= $searchEngine->renderScripts(); ?>
</head>

<body>
    <?php
    // Note: results are rendered before form because this way the form instantly
    // has access to whitelisted query string (if a search was already performed).
    $results = $searchEngine->renderResults();
    $form = $searchEngine->renderForm();
    echo $form . $results;
    ?>
</body>

 

Also, in the "Advanced Use" :

Spoiler

if ($q = $sanitizer->selectorValue($input->get->q)) {
    // This finds pages matching the query string and returns them as a PageArray:
    $results = $pages->find('search_index%=' . $query_string . ', limit=25');

    // Render results and pager with PageArray::render() and PageArray::renderPager():
    echo $results->render(); // PageArray::render()
    echo $results->renderPager(); // PageArray::renderPager()

    // ... or you iterate over the results and render them manually:
    echo "<ul>";
    foreach ($results as $result) {
        echo "<li><a href='{$result->url}'>{$result->title}</a></li>";
    }
    echo "</ul>";
}

 

the "$query_string" variable declaration...shouldn't be "$q" or I am missing something?

  • Thanks 1

Share this post


Link to post
Share on other sites

I'm making progresses!

Everything is working fine except for the renderPager, which loose the query string on subsequential pages:

$searchEngine = $modules->get('SearchEngine');
$form = $searchEngine->renderForm();
echo $form;

$out= '';
if ($q = $sanitizer->selectorValue($input->get->q)) {
    $results = $pages->find('search_index%=' . $q . ', limit=25');
	foreach ($results as $result) {
	$out.= // my code here, everything is great and populated, even repeater matrix elements whaoo! :)
	}
}
$out.= $results->renderPager();
echo $out;

On page http://localhost:7880/my_url/root/it/search/?q=my_query everything is ok.
As soon as I go to page2 the url is http://localhost:7880/my_url/root/it/search/page2 (the query is gone and nothing more is shown).

Btw my "search.php" template has "Allow page numbers" in the backend set to ON.

Any clue? 🙂

Share this post


Link to post
Share on other sites
18 hours ago, 3fingers said:

On page http://localhost:7880/my_url/root/it/search/?q=my_query everything is ok.
As soon as I go to page2 the url is http://localhost:7880/my_url/root/it/search/page2 (the query is gone and nothing more is shown).

Btw my "search.php" template has "Allow page numbers" in the backend set to ON.

Any clue? 🙂

The trick is to whitelist "q", after which MarkupPagerNav automatically sticks with it. Add this after you've validated the $q variable:

$input->whitelist('q', $q);

Note also that if this is your literal code, you should move the renderPager call in the if statement – if $q isn't set or validated, $results will be null, and this will cause an error 🙂

Share this post


Link to post
Share on other sites

Just released SearchEngine 0.13.0. This version adds more validation regarding the search index field: there's a warning if the field is of wrong type, an option to automatically create it if it doesn't exist (i.e. it has been removed after the module was installed), and there's a notice in the module settings screen if the index field exists and is valid but hasn't been added to any templates yet.

Additionally there's a fix for an issue where FieldtypeTextareaLanguage wasn't recognised as a valid index field type in module settings. This could've resulted in the index field setting getting unintentionally cleared if/when module settings were saved.

  • Like 1
  • Thanks 1

Share this post


Link to post
Share on other sites

Hi @teppo, another gentle question for you 🙂

I'm trying to style the pagination, using this code:

$config->SearchEngine = [

    // This are the settings I already used, successfully, elsewhere in the site to style my pagination.

    'pager_args' => [
        'nextItemLabel' => "Next",
        'previousItemLabel' => "Prev",
        'listMarkup' => "<ul class='pagination'>{out}</ul>",
        'itemMarkup' => "<li class='pagelink {class}'>{out}</li>",
        'linkMarkup' => "<a href='{url}'>{out}</a>",
        'currentLinkMarkup' => "{out}",
        'currentItemClass' => "current",
    ],

	// other settings here stripped out in this example.
];

Those settings seem to be ignored though. I'm surely missing something, can you point me in the right direction? 🙂

Share this post


Link to post
Share on other sites
4 minutes ago, 3fingers said:

Those settings seem to be ignored though. I'm surely missing something, can you point me in the right direction? 🙂

If you're using the code you posted earlier, i.e. "$results = $pages->find(...)" and "$out.= $results->renderPager()", SearchEngine settings won't kick in at all. This is plain old native ProcessWire stuff, so you'd have to provide pager settings directly to the renderPager() method 🙂

SearchEngine pager_args only apply if you're using it's native rendering features. Though please let me know if I'm missing the point here!

Share this post


Link to post
Share on other sites

You right! Passing them directly to the renderPager() method now works. Just to clarify, SearchEngine settings kick in, but not those dedicated to pagination 🙂

Thanks @teppo, glad you're on the same time-zone as mine, you're always super fast 🙂

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Gadgetto
      SnipWire - Snipcart integration for ProcessWire
      Snipcart is a powerful, developer-first 3rd party HTML/JavaScript shopping cart platform. SnipWire is the missing link between Snipcart and the content management framework ProcessWire.
      With SnipWire, you can quickly turn any ProcessWire site into a Snipcart online shop. The SnipWire plugin helps you to get your store up and running in no time. Detailed knowledge of the Snipcart system is not required.
      SnipWire is free and open source! A lot of work and effort has gone into development. It would be nice if you could donate an amount to support further development:

      Status update links (inside this thread) for SnipWire development:
      2020-01-19 --> integrated taxes provider finished (+ very flexible shipping taxes handling) 2020-01-14 --> new date range picker, discount editor, order notifiactions, order statuses, and more ... 2019-11-15 --> orders filter, order details, download + resend invoices, refunds 2019-10-18 --> list filters, REST API improvements, new docs platform, and more ... 2019-08-08 --> dashboard interface, currency selector, managing Orders, Customers and Products, Added a WireTabs, refinded caching behavior 2019-06-15 --> taxes provider, shop templates update, multiCURL implementation, and more ... 2019-06-02 --> FieldtypeSnipWireTaxSelector 2019-05-25 --> SnipWire will be free and open source If you are interested, you can test the current state of development:
      https://github.com/gadgetto/SnipWire
      If you like, you can also submit feature requests and suggestions for improvement. I also accept pull requests.
      ---- INITIAL POST FROM 2019-05-25 ----
      I wanted to let you know that I am currently working on a new ProcessWire module that fully integrates the Snipcart Shopping Cart System into ProcessWire. (this is a customer project, so I had to postpone the development of my other module GroupMailer).
      The new module SnipWire offers full integration of the Snipcart Shopping Cart System into ProcessWire.
      Here are some highlights:
      simple setup with (optional) pre-installed templates, product fields, sample products (quasi a complete shop system to get started immediately) store dashboard with all data from the snipcart system (no change to the snipcart dashboard itself required) Integrated REST API for controlling and querying snipcart data webhooks to trigger events from Snipcart (new order, new customer, etc.) multi currency support self-defined/configurable tax rates etc. Development is already well advanced and I plan to release the module in the next 2-3 months.
      I'm not sure yet if this will be a "Pro" module or if it will be made available for free.
      I would be grateful for suggestions and hints!
      Please have a look at the screenshots to get an idea what I'm talking about (open spoiler):
      (Please note: these screenshots are from an early development state of SnipWire. To see actual screens please have a look at later posts below!)
       
    • By d'Hinnisdaël
      Happy new year, everybody 🥬
      I've been sitting on this Dashboard module I made for a client and finally came around to cleaning it up and releasing it to the wider public. This is how it looks.
      ProcessWire Dashboard

      If anyone is interested in trying this out, please go ahead! I'd love to get some feedback on it. If this proves useful and survives some real-world testing, I'll add this to the module directory.
      Download
      You can find the latest release on Github.
      Documentation
      Check out the documentation to get started. This is where you'll find information about included panel types and configuration options.
      Custom Panels
      My goal was to make it really simple to create custom panels. The easiest way to do that is to use the panel type template and have it render a file in your templates folder. This might be enough for 80% of all use cases. For anything more complex (FormBuilder submissions? Comments? Live chat?), you can add new panel types by creating modules that extend the DashboardPanel base class. Check out the documentation on custom panels or take a look at the HelloWorld panel to get started. I'm happy to merge any user-created modules into the main repo if they might be useful to more than a few people.
       Disclaimer
      This is a pre-release version. Please treat it as such — don't install it on production sites. Just making sure 🍇
      Roadmap
      These are the things I'm looking to implement myself at some point. The wishlist is a lot longer, but those are the 80/20 items that I probably won't regret spending time on.
      Improve documentation & add examples ⚙️ Panel types Google Analytics ⚙️ Add new page  🔥 Drafts 🔥 At a glance / Page counter 404s  Layout options Render multiple tabs per panel panel groups with heading and spacing between ✅ panel wrappers as grid item (e.g. stacked notices) ✅ Admin themes support AdminThemeReno and AdminThemeDefault ✅ Shortcuts panel add a table layout with icon, title & summary ✅ Chart panel add default styles for common chart types ✅ load chart data from JS file (currently passed as PHP array) Collection panel support image columns ✅ add buttons: view all & add new ✅
    • By Robin S
      This module is inspired by and similar to the Template Stubs module. The author of that module has not been active in the PW community for several years now and parts of the code for that module didn't make sense to me, so I decided to create my own module. Auto Template Stubs has only been tested with PhpStorm because that is the IDE that I use.
      Auto Template Stubs
      Automatically creates stub files for templates when fields or fieldgroups are saved.
      Stub files are useful if you are using an IDE (e.g. PhpStorm) that provides code assistance - the stub files let the IDE know what fields exist in each template and what data type each field returns. Depending on your IDE's features you get benefits such as code completion for field names as you type, type inference, inspection, documentation, etc.
      Installation
      Install the Auto Template Stubs module.
      Configuration
      You can change the class name prefix setting in the module config if you like. It's good to use a class name prefix because it reduces the chance that the class name will clash with an existing class name.
      The directory path used to store the stub files is configurable.
      There is a checkbox to manually trigger the regeneration of all stub files if needed.
      Usage
      Add a line near the top of each of your template files to tell your IDE what stub class name to associate with the $page variable within the template file. For example, with the default class name prefix you would add the following line at the top of the home.php template file:
      /** @var tpl_home $page */ Now enjoy code completion, etc, in your IDE.

      Adding data types for non-core Fieldtype modules
      The module includes the data types returned by all the core Fieldtype modules. If you want to add data types returned by one or more non-core Fieldtype modules then you can hook the AutoTemplateStubs::getReturnTypes() method. For example, in /site/ready.php:
      // Add data types for some non-core Fieldtype modules $wire->addHookAfter('AutoTemplateStubs::getReturnTypes', function(HookEvent $event) { $extra_types = [ 'FieldtypeDecimal' => 'string', 'FieldtypeLeafletMapMarker' => 'LeafletMapMarker', 'FieldtypeRepeaterMatrix' => 'RepeaterMatrixPageArray', 'FieldtypeTable' => 'TableRows', ]; $event->return = $event->return + $extra_types; }); Credits
      Inspired by and much credit to the Template Stubs module by mindplay.dk.
       
      https://github.com/Toutouwai/AutoTemplateStubs
      https://modules.processwire.com/modules/auto-template-stubs/
    • By Mike Rockett
      Jumplinks for ProcessWire
      Release: 1.5.60
      Composer: rockett/jumplinks
      ⚠️ NOTICE: 1.5.60 is an important security patch-release for an XSS vulnerability discovered by @phlp. It's HIGHLY RECOMMENDED that all Jumplinks users update to the latest version as soon as possible.
      Jumplinks is an enhanced version of the original ProcessRedirects by Antti Peisa.
      The Process module manages your permanent and temporary redirects (we'll call these "jumplinks" from now on, unless in reference to redirects from another module), useful for when you're migrating over to ProcessWire from another system/platform. Each jumplink supports wildcards, shortening the time needed to create them.
      Unlike similar modules for other platforms, wildcards in Jumplinks are much easier to work with, as Regular Expressions are not fully exposed. Instead, parameters wrapped in curly braces are used - these are described in the documentation.
      Under Development: 2.0, to be powered by FastRoute
      As of version 1.5.0, Jumplinks requires at least ProcessWire 2.6.1 to run.
      View on GitLab
      Download via the Modules Directory
      Read the docs
      Features
      The most prominent features include:
      Basic jumplinks (from one fixed route to another) Parameter-based wildcards with "Smart" equivalents Mapping Collections (for converting ID-based routes to their named-equivalents without the need to create multiple jumplinks) Destination Selectors (for finding and redirecting to pages containing legacy location information) Timed Activation (activate and/or deactivate jumplinks at specific times) 404-Monitor (for creating jumplinks based on 404 hits) Additionally, the following features may come in handy:
      Stale jumplink management Legacy domain support for slow migrations An importer (from CSV or ProcessRedirects) Feedback & Feature Requests
      I’d love to know what you think of this module. Please provide some feedback on the module as a whole, or even regarding smaller things that make it whole. Also, please feel free to submit feature requests and their use-cases.
      Note: Features requested so far have been added to the to-do list, and will be added to 2.0, and not the current dev/master branches.
      Open Source

      Jumplinks is an open-source project, and is free to use. In fact, Jumplinks will always be open-source, and will always remain free to use. Forever. If you would like to support the development of Jumplinks, please consider making a small donation via PayPal.
      Enjoy! 🙂
    • By Robin S
      Add Image URLs
      Allows images/files to be added to Image/File fields by pasting URLs.

      Usage
      Install the Add Image URLs module.
      A "Paste URLs" button will be added to all image and file fields. Use the button to show a textarea where URLs may be pasted, one per line. Images/files are added when the page is saved.
       
      https://github.com/Toutouwai/AddImageUrls
      https://modules.processwire.com/modules/add-image-urls/
×
×
  • Create New...