Autocomplete and Other Search Enhancements

[Michael Murphy]

I'm trying to work out the best way to implement an autocomplete search using this jqueryui plugin :

http://jqueryui.com/demos/autocomplete

I have over 1500 street names (sepearte pages in PW) which I'd like users to find via an autocomplete search.

My first idea is to create a seperate search page which will return the results in json format (if called via ajax). Similar to this FAQ which explains how to work with ajax :

http://processwire.c...__fromsearch__1

I'm currently using the %= like style selector which works well for the type of results I want. Just a little worried about performance hit when using this selector, especially if I have an autocomplete search - the number of documents I want to search should not be more than 1500.

Is this a good approach? Any other ideas or suggestions?

Also related to search, does anyone have any tips for improving the default search functionality? How about more advanced wildcard type searches? or maybe a google style "did you mean ______?"

Thanks,

Michael

[ryan]

Michael, I just replied to a PM from someone else about this, so it's quite a coincidence for the next message I read to be on the same topic. I'd already started an autocomplete Inputfield using jQuery UI some time ago, and just never got around to finishing it. Since it's come up again, perhaps now is a good time for me to finish it.

I'm currently using the %= like style selector which works well for the type of results I want.

If you are only dealing with ~1500 street names there won't be much if any noticeable performance hit from %= vs *=. Granted *= is using an index and %= isn't, but 1500 items is chump change to MySQL, especially with something short like street names.

How about more advanced wildcard type searches?

When you use *= or %= (as opposed to ~=), note that those are wildcard searches when used with single words. They will find all words matching or beginning with the specified term. Meaning a search for "start" will also match "started", "starts", and "starter" (as examples). Also take a look at the ^= and $= operators, which will match at the beginning or end of a string. We are pushing the limits of what a fulltext index will let us do, but you can get more wildcard type options using a MySQL LIKE query directly to have something like "st_rt" match "start" or "%start" match "restart". I've not found these to be useful in my own web development projects so haven't implemented them into PW operators. But we may bring more MySQL LIKE and REGEXP commands into PW selectors in the future. My feeling is that I don't really want to encourage people to use them because they aren't scalable, relying upon full non-indexed table scans. But when dealing with items at the scale we were talking about (1500) it doesn't matter much.

or maybe a google style "did you mean ______?"

I don't how to implement such a thing at present (with MySQL), but will keep an eye on such possibilities in the future. I believe that Google finds those 'did you mean' options by monitoring repeated search behavior at a large scale over time.

[Michael Murphy]

Thanks again for the detailed explanation - always very helpful in understanding.

You're right, the %= does work like a wildcard search. My main concern is that people might spell a street name differently to what we've stored in the database. I imagine a "search spelling suggestions" functionality would be difficult to implement - however the autocomplete search should get around this problem.

[ryan]

Michael, I just added the PageAutocomplete to the core (and got your tweet that you already saw it). I wanted to mention that after you save your field settings with 'PageAutocomplete' as the input field, you'll see some more configuration options appear on the 'Input' tab. This includes what operator you want to use (like %= vs *=, etc.), as well as what fields you want it to search.

[Michael Murphy]

Thanks for mentioning as I didn't notice those extra options. This is a really great addition for users managing lots of page references.

Is there a simple way to integrate this field on the front-end of a website?

[ryan]

Is there a simple way to integrate this field on the front-end of a website?

This inputfield uses the ProcessPageSearch AJAX API, which is accessible to logged-in users only that have administrative access. Meaning, it can't be used on the front-end of a site, unless for an administrative user.

However, this Inputfield is mainly just a front-end to the jQuery UI autocomplete, with some tweaks to make it consistent with PW's admin style and offer some configurable options. So one could very easily implement the same sort of functionality on the front-end of a site by creating a page with it's own template to handle the ajax calls, translate them to a $pages->find() operation, and return the matches. It doesn't need to do any more than this:

/site/templates/search.php

<?php
$resultsArray = array();
if($input->get->q) {
   $q = $sanitizer->selectorValue($input->get->q); 
   $results = $pages->find("title*=$q, limit=50"); 
   foreach($results as $p) $resultsArray[$p->id] = $p->title;
}
echo json_encode($resultsArray); 

From there, the rest is just jQuery UI autocomplete:

http://jqueryui.com/demos/autocomplete/#multiple-remote

[Soma]

Just tried tweetsy to Ryan, but it's better to write here again : Well I thought having a path auto complete would be cool, so typing "/cat.." would "/category/..." I'm not talking about the label. But maybe that works as good, as long as the title/name's are consistent, otherwise it could be confusing.

Anyway. Great thanks for the awesome new module!

[Michael Murphy]

Thanks for the example Ryan, I got it working! Had to tweak the json output, but was much simpler than I thought. This method for creating basic json apis or any type of api could be very useful.

One other thing I did was to use the

if($config->ajax)

method, so I could return a normal results page if not using ajax.

[ryan]

Just tried tweetsy to Ryan, but it's better to write here again : Well I thought having a path auto complete would be cool, so typing "/cat.." would "/category/..." I'm not talking about the label. But maybe that works as good, as long as the title/name's are consistent, otherwise it could be confusing.

It'll work with any text field (or combination of text fields). But url or path is not actually a DB field that exists anywhere in PW. Instead it's determined at runtime for each page... so it's not one that can be queried. Title is probably the best way to go in most cases.

if($config->ajax) method, so I could return a normal results page if not using ajax.

Excellent point -- this is a great way to go so that you can use the same template for regular and ajax results.

[Soma]

Ryan, I know that path can't be queried, but it would be possible to have path autocompletion in some way. Do you think it would be good to have a new inputfieldtype for this?

I found in certain circumstances using the page reference with the label "title" isn't always clear what page you're selecting, as the "where is this located" question arises. I have ie. a page field to select from 100's of download's (which are pages), so the most practicable and sure option was to use "path" as the label. Problem was since there could be different branches with pages using same or similar names, but have a different parent, defining another category. So instead of selecting from a long list, using the asm select input type, it would be nice to have autocomplete for paths.

Some time ago, would have liked to use the PageListSelectMultiple for this, but it would let me select all pages instead of only from the defined "parent". It won't get saved but you can still select it and it gets added to the list, but after a save it's gone, as it's not allowed to be selected, without a message. I thought maybe it would be best to have the pages not allowed to be selected "disabled/locked". And if a parent page is selected only render page list below it. I remember you thought that it's not easy because of how it works now and would require some rewrite. What are you're thoughts on this subject now?

[ryan]

I found in certain circumstances using the page reference with the label "title" isn't always clear what page you're selecting, as the "where is this located" question arises. I have ie. a page field to select from 100's of download's (which are pages), so the most practicable and sure option was to use "path" as the label. Problem was since there could be different branches with pages using same or similar names, but have a different parent, defining another category. So instead of selecting from a long list, using the asm select input type, it would be nice to have autocomplete for paths.

I think I might need to see an example, because the way you describe it, it seems to me like you'd want to setup the searchField as 'title', and the label as 'path'. That's something you can already do. But if you literally want to be dealing with paths for the search portion, you can search with the components of a path (page names). You can set it to use 'name' as the searchField. But because 'name' is just a word and not technically a fulltext field in ProcessWire, it's not in a fulltext index. That means you have to type the name, like 'about' (rather than 'abo') before autocomplete will return matching pages. But if you want to give it a try, edit your Page field using the autocomplete, and do this:

1. Leave all the selection fields blank so that it's searching your whole site (assuming that's what you want)

2. Choose 'path' for the 'label field'.

3. Under 'Autocomplete advanced options' choose '= Equals [exact]' for the operator and enter 'name' for 'Fields to query for autocomplete'. Note that '=' is the only operator that will work with the name field, so don't bother trying the others.

Another route you can take is to create a new text field, call it 'mypath' or something like that, and have a module hooked to Pages::save that auto-populates mypath with $page->path() when the page is saved. Then tell the autocomplete to use mypath as the searchField. Just note that if one of the parent pages is moved or renamed, that mypath will be out of date. This may not be an issue on site where the structure is already concrete. But on an existing site, you'd have to make a script that goes and saves every page the first time to initially populate that mypath fields.

Some time ago, would have liked to use the PageListSelectMultiple for this, but it would let me select all pages instead of only from the defined "parent". It won't get saved but you can still select it and it gets added to the list, but after a save it's gone, as it's not allowed to be selected, without a message. I thought maybe it would be best to have the pages not allowed to be selected "disabled/locked". And if a parent page is selected only render page list below it. I remember you thought that it's not easy because of how it works now and would require some rewrite. What are you're thoughts on this subject now?

Currently if you choose a parent and use a PageListSelectMultiple (for example) it'll let you select (and it will keep) anything within that parent's decedents. I just tested it here, and think that's the way it's always been? If I recall, you were asking about the 'template' selection, which PageListSelect Inputfields don't work with at present.

When it comes to the Autocomplete combined with a parent selection, then it's only going to search the parent you specify (which may or may not be the behavior you want). But if you want it to search the entire structure contained by that parent, then you'll want to use the Parent in addition to the 'Custom selector to find selectable pages' option (just added it yesterday). Put anything in that selector, like "id>0" or "sort=name" or whatever you want. When you use that selector with parent selected, then it'll do a $parent->find($selector) rather than a $pages->find($selector)... meaning it'll return results for that parent and anything below. Another way you can do the same thing is to have no parent selected, and then enter "has_parent=123" for the selector, where 123 is the ID of the parent.

[Soma]

Thanks Ryan, but I think you kinda missunderstood what I was trying to say. But that's my fault, because I struggle to explain it clearly.

I was just trying to explain where I'm coming from that I think autocomplete for paths could be useful. But this gets complicated with all those terms we use and lots of options... apart from that I struggle to get the point clear.

But your explanations may help understand a little more. I didn't knew for example that the "name" field only works with "=" and not "*=". I think something along your second route would be possible, but I guess a special inputfield would be required to search for paths using strpos and finds and the like.

Anyway.

For the first part: Yes that's the setup I ended with, when trying your new AC module, "path" as label and searching on "title". I think this work also nicely.

Secon part: BUT my point was, that in the one project I had to chose the asm multiple select along with the "path" as label and then using a custom code to pull pages from the download section pages. (Pretty nice option there!), but when setting it up, I would have really liked to use the "PageListSelectMultiple" input type for that. So they can browse for the download pages. Don't get me wrong this "works", but my point was that it isn't hiding pages from the tree that aren't allowed to be selected. If I for example define a template or a "parent", it still will let me select a page that isn't allowed in the first place (as reported some time ago already). It just isn't well to allow page to be selected from the tree, when it isn't allowed, but it gets added to the list and stays there until I save page, then it is removed from the selected pages. If you still don't know what I'm talking about, I'll record you a video showing it.

[ryan]

If I for example define a template or a "parent", it still will let me select a page that isn't allowed in the first place (as reported some time ago already). It just isn't well to allow page to be selected from the tree, when it isn't allowed, but it gets added to the list and stays there until I save page, then it is removed from the selected pages.

This is the case with template, but not parent. If you set a parent, then PageListSelect assumes that to be the starting point in the tree. The pages don't have to have that parent directly. They just have to have that parent as an ancestor. This is the way it's supposed to work, and the way it worked here just testing it. Are you getting a different behavior?

But you are right with regard to template. PageList doesn't take template into account because PageList is designed to list hierarchy not type. It's feasible in the future that template could be combined with parent as a constraint within the hierarchy defined by parent, but that ability isn't there right now. As a result, template constraints shouldn't be used with PageListSelect inputs.

I guess a special inputfield would be required to search for paths using strpos and finds and the like.

It's not just a matter of an Inputfield here. The paths for pages don't actually exist anywhere that can be searched. Paths are constructed and determined at runtime, and only for pages that are loaded. There's nothing to strpos() or find(). So we'd need to find a place for them to be cached before such a solution could search them (using that 'mypath' field, for example). I'll keep thinking if there's other ways.

[Soma]

You're right Ryan, sorry, using the "parent" does only show the part. I think I was mixing up something. Yes it was when using template restriction. As mentioned it would be cool to have pages "disabled" that aren't allowed to be selected to avoid the problem.

As for the autocomplete with a new inputfield type, I was thinking of a way to extract the "name" information using a site map list as a string, to then compare. Or maybe something with using get/find on the "name" and then successively complete one url segment after another, just a shame that it doesn't work with fulltext index searches, as it would be much easier. But maybe I'm going to far, and it isn't really something needed that.

[DaveP]

I imagine a "search spelling suggestions" functionality would be difficult to implement

This is a particular interest of mine - you can do something quite useful with mySQL's 'sounds like' comparison and then maybe sort your results using PHP's levenshtein function. It's an approach I've used in the past and works quite well.

[Michael Murphy]

Thanks Dave, looks very interesting and maybe not as difficult as I first thought.

Cheers,

Michael

[landitus]

I'm following the code above to make an autocomplete enhancement to a search form using jquery ui 1.8. I made the search template using the same code but I'm having trouble making it to work. I think I'm making a mistake in the js code:

$("#search_query" ).autocomplete({
	      source: function( q, response ) {
	        $.ajax({
	          type: "GET",	
	          url: "/busqueda",
	          dataType: "JSON",
	          success: function( data ) {
	           	console.log(data); 
	          }
	        });
	      }
});

I'm reading the jquery UI docs but I can't extrapolate a single example to my needs. 

[ryan]

It doesn't look to me like you are sending your query (q variable) through the ajax request? I'm guessing the code above would just retrieve the URL /busqueda/ with no query.

[Soma]

Maybe trailing slash?

Ryan. I think source is for the params.

[landitus]

Mmm, I tried to add a data option:

$("#search_query" ).autocomplete({
	      source: function( q, response ) {
	        $.ajax({
	          type: "GET",	
	          url: "/busqueda/",
	          data: q, 
	          dataType: "JSON",
	          success: function( data ) {
	           	console.log(data); 
	          }
	        });
	      }
	      
	    });

But I still get "[]" in the console. I've noticed that "?term=ferr" shows up in the network tab. I guess there's no q for jquery, but uses "term". I haven't found a way to modify this, yet. Does this mean I have to replace "term" for "q" in the search function?

[Soma]

Try

Data: { 'q' : q}

[landitus]

Thanks for the suggestions, but I still get a [] in the console with this code:

<form id='top-search-form' action='<?php echo $pages->get('/busqueda')->url ;?>' method='get'>
    <ul>
	<li>
	<label><?= __('Buscar') ;?></label>
	<input type='text' name='q' id='search_query' placeholder='<?= __('Buscar') ;?>...' value='<?php echo $input->whitelist('q'); ?>' />
	<button type='submit' id='search_submit'><?= __('Buscar') ;?></button>
	</li>
  </ul>
</form>

$("#search_query" ).autocomplete({
	      source: function( q, response ) {
	        $.ajax({
	          type: "GET",	
	          url: "/busqueda",
	          Data: { 'q' : q},
	          dataType: "JSON",
	          success: function( data ) {
	           	console.log(data); 
	          }
	        });
	      }
	      
	    });
	

 
/**
 * Search template
 *
 */
if($config->ajax) {
	
	$resultsArray = array();
	if($input->get->q) {
	    $q = $sanitizer->selectorValue($input->get->q); 
	    $results = $pages->find("title*=$q, limit=50"); 
	    foreach($results as $p) $resultsArray[$p->id] = $p->title;
	}
	echo json_encode($resultsArray);
}
else {
  ... working search code 
}

[Soma]

Have you tried 'data' instead of Data.

sorry mobile autocorrect..

[landitus]

yep, both. Actually, "Data" outputs [] and "data" outputs nothing

[Soma]

According to autocomplete documentation you'd have to do something like this (works).

$('#search_query').autocomplete({
    source: function(request, response){
        $.ajax({
            url: '/search/',
            type: 'get',
            dataType: 'json',
            data: { 'q': request.term },
            success: function(result){
                response(result);
            }
        });
    },
    minLength: 2
});