Splitting search results by page

[Dean]

I have an issue with ordering and repeated results when limiting search results and splitting into pages. Selector is shown just above letters.

This is what I get when all results fit on one page:

If I limit to 5 per page, I get a different order, missing results, and a repeated result on page 2:

Can anyone shed any light please?

[BillH]

Does it help to add a sort term to the selector (e.g. sort=title)?

What happens if you don't paginate the results?

If the non-paginated results are as expected, then the issue is probably in the pagination. But what method are you using to paginate the results? Is it as described  at https://processwire.com/docs/front-end/markup-pager-nav/?

 

 

 

[Dean]

Thanks @BillH, I'm using the code from the Skyscraper demo:

// check if there are any keyword searches in the selector by looking for the presence of 
// ~= operator. if present, then omit the 'sort' param, since ProcessWire sorts by 
// relevance when no sort specified.
if(strpos($selector, "~=") === false) $selector .= "sort=name, ";

I'm using renderPager():

$traders = findTraders($selector, 5);
$pager = $traders->renderPager();
...
echo $pager;

findTraders() is a slightly modified version of findSkyscrapers()

function findTraders($selector, $limit=10) {
	// check if there are any keyword searches in the selector by looking for the presence of 
	// ~= operator. if present, then omit the 'sort' param, since ProcessWire sorts by 
	// relevance when no sort specified.
	if(strpos($selector, "~=") === false) $selector .= "sort=title, ";
	
	$selector .= "limit=$limit";

	echo '<small>'.$selector.'</small><br />';
	
	// now call upon ProcessWire to find the skyscrapers for us
	$traders = pages($selector);

	return $traders;
}

 

[BillH]

I'd start in findTraders() by changing the fourth-from-last line to:

$traders = $pages->find($selector);

Then it'd be worth reviewing https://processwire.com/docs/selectors/#sort. In particular note the advice in the section about what happens if you don't specify a sort: "it is generally a good idea to include a sort=property when using $pages->find()". At least it'd be worth trying with sorting in all cases, which could narrow down the issue.

Another possible place where things could go wrong is the loop that renders the results, represented by "..." in your last post. But if it's a simple foreach loop, it should be OK.

By the way, when debugging searches I find that printing the result array, which is a PW PageArray, can help with understanding what's going on. For example:

echo "<pre>".print_r($traders, true)."</pre>";

 

[Dean]

Sorry @BillH, I had to work on something else for a few days. So, back to this. If I change

$traders = pages($selector);

to

$traders = $pages->find($selector);

I get

Notice: Undefined variable: pages
Fatal Error: Uncaught Error: Call to a member function find() on null

Regarding sorting, I'm sorting by title if I haven't specified a search string

if(strpos($selector, "~=") === false) $selector .= "sort=title, ";

If there is a search string, are the results not sorted by ProcessWire according to relevance?

[Jan Romero]

I think this is a bug in ProcessWire. When you use the ~= operator, ProcessWire translates it into MySQL’s

select /*[…]*/ match(field_title.data) against('+New York' in boolean mode) as _score_field_title_data1
/*[…]*/
order by _score_field_title_data1 desc

The score it uses to sort is generated by the database according to the number of times the string appears in the text (probably more complicated than that, but whatever).

The problem is that several results will have the same score, so the order among them is not deterministic. The database just does whatever it wants at that point, so the order may change inbetween queries, even if the queries are the same, but especially if they differ, even just in LIMIT and OFFSET.

If you add "sort" to your selector string, ProcessWire will still use the score to sort, but only after your specified fields.

What ProcessWire should do is ALWAYS sort by page id at the end, to keep the order deterministic. It might also be nice to make the relevance scores accessible in selector strings, so you could do something like

"title|trader~='New York', sort=-title_relevance, sort=-trader_relevance, sort=-modified, sort=-id"

Now you could have results in order of textual relevance, and if the relevance is equal, get the most current ones first, or whatever you desire. Obviously the id is a pretty stupid way to sort, but since it’s strictly unique, it’s a stable way to break ties, so I’d always put it last.

[BillH]

2 hours ago, Dean said:

Notice: Undefined variable: pages
Fatal Error: Uncaught Error: Call to a member function find() on null

Ah, sorry, you probably want:

$traders = $this->pages->find($selector);

Regarding the sorting, very good points from @Jan Romero.

I'd add that it's not necessarily the case that sorting by relevance would produce results that are particularly useful for users. It'd depend on the nature of the data and so on. If you want to read how it's done, it's described on https://dev.mysql.com/doc/internals/en/full-text-search.html, but it'd be a major project in itself figuring it all out!

And one suggestion: if you're not anticipating too many results, it can take an acceptable amount of time to produce a results array, loop through the array adding a sort criterion (such as a relevance score), and then sorting by that criterion.