Jump to content
bernhard

[deprecated] RockFinder1

Recommended Posts

1 hour ago, dragan said:

it may have to do with how mySQL handles case-sensitivity on different OS

I think you're right.  I still wonder why the SQL is altered (RF or OS or combi OS/SQL), but I also understand now that I simply made the mistake by using uppercase for fields either way.  Good to know for future projects...

Share this post


Link to post
Share on other sites

@KarlvonKarton I had that excact problem with uppercase fields on a Windows server also. Then on the live server which is a Linux machine, the queries did not work, becaus lowercase was used. Make sure to use all lowercase fields in your database.

Share this post


Link to post
Share on other sites

RockFinder now supports array-syntax, which makes the code a lot cleaner:

AxdiArx.png4dvtm4D.png

  • Like 8

Share this post


Link to post
Share on other sites

Thanks for the update - Would you have an idea why my queries are running so much faster on mysql5.7 than mysql5.6 ?  What is your version on your dev platform ? Did you tested the two server version on your side ? 


Check. I switched my MySQL server version on MAMP from 5.6 to 5.7 and when I clicked a button which execute quite big query, it was, oh wow, instant. Switched back to 5.6, the query took like 8 to 12  52 seconds 😳 

Still, I didn't investigated what's going on, but result... and the same happened on a Windows Server with MySQL 5.6/5.7 upgrade.

 

611595178_Capturedecran2019-01-23a10_10_18.png.774b36454624f6614a9b1375c018d1ca.png

 

Edit: Another one looking through millions of pages...

2054576922_Capturedecran2019-01-23a11_45_33.png.b6918465764978dec8891934bc6090bd.png

Edited by flydev
The big query
  • Like 1

Share this post


Link to post
Share on other sites

I'm using 5.7 both on dev and live. 

3 hours ago, flydev said:

Would you have an idea why my queries are running so much faster on mysql5.7 than mysql5.6 ?

Sorry, no idea 😐 http://smalldatum.blogspot.com/2017/06/sysbench-for-mysql-50-51-55-56-57-and-8.html , while http://www.oaktable.net/content/mysql-56-vs-57 says they perform quite similar. Maybe 5.6 has a problem with those subqueries?

  • Thanks 1

Share this post


Link to post
Share on other sites

I am going to ask on StackExchange but I got another (magic) find, which give incredible performance on InnoDB schema.

 

Adjusting `innodb_buffer_pool_size` (it need to be calculated) give better result than those in the above screenshots... 

The query to calculate the size to set based on the current data set :

SELECT CONCAT(ROUND(KBS/POWER(1024,
IF(PowerOf1024<0,0,IF(PowerOf1024>3,0,PowerOf1024)))+0.49999),
SUBSTR(' KMG',IF(PowerOf1024<0,0,
IF(PowerOf1024>3,0,PowerOf1024))+1,1)) recommended_innodb_buffer_pool_size
FROM (SELECT SUM(data_length+index_length) KBS FROM information_schema.tables
WHERE engine='InnoDB') A,
(SELECT 2 PowerOf1024) B;

 

So for my DB I needed to set the var to 2GB (still not 100% sure) but the result are there - lower than in my previous screenshot : 

 

From 18s to 4s :

203214322_Capturedecran2019-01-23a15_20_33.png.b395054a1ee51080f4aad4f511ab826c.png

 

From 96ms to ˜6ms :

1190313192_Capturedecran2019-01-23a15_21_01.png.f4d139e1934473f84adc0aa8c8672ea1.png

 

💪 😋

  • Like 2

Share this post


Link to post
Share on other sites

Nice! 🙂 Please let us now about the results/answers.

Share this post


Link to post
Share on other sites

New version online: 1.1.2

  • The selector for the finder is now always stored as string, but can be retrieved as string or array - getSelectorStr() or getSelectorArr()

yNdQaD5.png

This can be useful to share code for grids that are similar but need little adjustments, like one grid shows all available trainings, one grid shows only trainings that where booked by the current user:

include(__DIR__.'/trainings.php');

$finder->setSelectorValue('client', $this->user);
$finder->addField('done');

$this->setData($finder);

 

  • Like 2

Share this post


Link to post
Share on other sites
4 hours ago, flydev said:

1190313192_Capturedecran2019-01-23a15_21_01.png.f4d139e1934473f84adc0aa8c8672ea1.png

How do you generate these nice logs/speed tests?

Share this post


Link to post
Share on other sites

I got the following notice in TracyDebugger after upgrading to 1.1.2:

PHP Notice: Array to string conversion in ...\bewerbertool-talents\wire\core\PageFinder.php:195

and the grid does not load if TracyDebugger is active because of this notice, I think.

The notice seem to come from my query which has an OR condition in it:

$stellen = new RockFinder("template=stelle,status=1|2049", ['title', 'stellekosten', 'kostenproclick', 'mitarbeiterid', 'created', 'bookmark']);

If i disable Tracy it works fine, and also if I click the reload grid button. But on a browser reload the error happens.

It worked before even with Tracy active.

Share this post


Link to post
Share on other sites

KfPqOYn.png

OR selectors should not be a problem. Could you please try to refresh all cached files? Maybe there's something else that causes this? Please try to narrow it down a little more... What's the array that is converted to string on line 195 of PageFinder.php ?

Share this post


Link to post
Share on other sites

Seems to have to do with the status field, which is for pageStatus. The line 195 in PageFinder.php returns this:

image.png.a1479394400af0f8aee8ae930dac6241.png

I solved it with using a different query. I only used status=1|2049 before, because "include=all" did not work, and I even had unpublished pages in my result.

And that was what I wanted. Show only published pages, or show them all. Don't know why it did not work before, but now it does again.

  • Like 1

Share this post


Link to post
Share on other sites

Hi there,

first of all thanks for this great module! 
Im working on a search wich will need a lot of filtering.
For example i want to get all persons with a height between 160cm & 170cm and a weight between 50kg & 60kg

For the moment i came up with this approche.
But im wondering if there is a more elegant "build in" functionality to solve this.

The Array $searchFields could contain more or less fields as in this example .

$searchFields = [ [ 'promo_koerpergroesse', 160, 170 ], [ 'promo_koerpergewicht', 50 ,60 ] ];
foreach ( $searchFields as $searchField => $searchFieldValue ) 
{
	$filterField 	= $searchFieldValue[0];
	$filterValMin 	= $searchFieldValue[1];
	$filterValMax 	= $searchFieldValue[2];

	$finder->filter( function( $row ) use ( $filterField, $filterValMin, $filterValMax ) {
		return $row->$filterField > $filterValMin & $row->$filterField < $filterValMax; 
	});
}

Thanks / ciao Simon

Share this post


Link to post
Share on other sites

Just merged a PR that takes care of camelCase fieldnames: https://github.com/BernhardBaumrock/RockFinder/pull/4/commits/c3bef03c5fc00b9439c0b6bba0997a5843f5868e

@simonGG the filter might work, but it's for sure more efficient to do this via SQL. But if you don't have lots of entries it might be easier to stay with your solution.

  • Like 1

Share this post


Link to post
Share on other sites
3 hours ago, bernhard said:

Just merged a PR that takes care of camelCase fieldnames: https://github.com/BernhardBaumrock/RockFinder/pull/4/commits/c3bef03c5fc00b9439c0b6bba0997a5843f5868e

@simonGG the filter might work, but it's for sure more efficient to do this via SQL. But if you don't have lots of entries it might be easier to stay with your solution.

Hey, PR author here, just wanted to thank you for this great module @bernhard. I wrote a module which was loading more than a thousand pages and it was taking almost a minute to load... now it's down to a second or less ! 🙏

  • Like 2
  • Thanks 1

Share this post


Link to post
Share on other sites
7 minutes ago, monollonom said:

Hey, PR author here, just wanted to thank you for this great module @bernhard. I wrote a module which was loading more than a thousand pages and it was taking almost a minute to load... now it's down to a second or less ! 🙏

Great to hear that 🙂 Thank you for your PR 🙂 

  • Like 1

Share this post


Link to post
Share on other sites

Hi,

(disclaimer : I'm a noob at sql)

I am facing a situation where I would need to get the titles of pages linked in a page reference within a repeater. I tried a few combinations but each time I get an empty array. Maybe somebody can point me to the right direction ?

Context : I have an event page with sub-events (repeater), each of which are held by different persons (page reference).

Thanks !

Edit : in the end I created a page reference field in my template that is populated with the persons linked in each sub-events, using a hook. Simpler...

Share this post


Link to post
Share on other sites

Hi @bernhard,

I've not had a chance to test this yet.  Quick question, in multilingual setups, is it language aware? If not, can it be made to be language aware?

Many thanks for all the hard work!

Share this post


Link to post
Share on other sites

Hi @kongondo,

it is language aware in general, yes. There might be some special cases though like repeater fields, where some additional tweaks would be necessary. I've updated the docs with some words about that: https://github.com/BernhardBaumrock/RockFinder/blob/master/readme.md#multilanguage

Hope that helps. Easy finders should just be fine and you can always use RockFinder as a start and join custom SQL as you need 🙂 

  • Like 1

Share this post


Link to post
Share on other sites
1 hour ago, bernhard said:

Hope that helps. Easy finders should just be fine

It does, thanks! An easy finder like page title in 'my language' is what I was after. I'd also want this for page reference fields, pointing to the found pages titles.  I'll check the docs.

Cheers.

Share this post


Link to post
Share on other sites

The docs are far from complete. For page reference fields you might have a look at https://github.com/BernhardBaumrock/RockFinder/blob/master/readme.md#joins

Joins are great, because you can split the queries into easy finders, see how it works and then join everything together in one result. Complex fields like page reference fields or repeaters might mess up some queries or might need some extra work on the module.

Share this post


Link to post
Share on other sites

I am getting ERR_RESPONSE_HEADERS_TOO_BIG in Chrome with the RockFinder Tester. After googling, I tried to add this line at the top:

header_remove('Set-Cookie');

which doesn't change anything. Did you ever encounter this as well?

Problem is, this even happens for really small datasets like limit=10 :-|

In Firefox there are no such problems, even with huge datasets.

Edited by dragan
Firefox

Share this post


Link to post
Share on other sites

I get this error on one site using tracy debugger. Are you using tracy?

Share this post


Link to post
Share on other sites

This happens when I run setup/rockfindertester/, but Tracy is enabled. When I use RF code inside Tracy console in Chrome, everything's fine.

 

Share this post


Link to post
Share on other sites

Two small questions (and I guess I know the answers already):

a) I can't get any RF output (arrays, objects) to play nice with PW's pagination / renderPager(). I guess pagination expects a PageArray, not just any kind of array. I know RF was never meant to be used in the frontend, but I tried anyway (and of course, I know building my own pagination with something like Tabulator is quite easy). I just wanted to compare an existing page (how much speed I would gain using RF instead of native PF find).

b) I tried queries like 

$finder = new \ProcessWire\RockFinder('parent=1041, include=all, template=project, sort=-year', ['title', 'year', 'modified', 'url']);

url is not recognized. I guess because a PW page URL is not stored inside the DB as-is, but is created when calling the $page->url() method. Am I correct? (PW has to assemble parents(s), page-names etc.)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Gadgetto
      SnipWire - Snipcart integration for ProcessWire
      Snipcart is a powerful 3rd party, developer-first HTML/JavaScript shopping cart platform. SnipWire is the missing link between Snipcart and the content management framework ProcessWire.
      With SnipWire, you can quickly turn any ProcessWire site into a Snipcart online shop. The SnipWire plugin helps you to get your store up and running in no time. Detailed knowledge of the Snipcart system is not required.
      SnipWire is free and open source licensed under Mozilla Public License 2.0! A lot of work and effort has gone into development. It would be nice if you could donate an amount to support further development:

      Status update links (inside this thread) for SnipWire development
      2020-07-03 -- SnipWire 0.8.7 (beta) released! Fixes some small bugs and adds an indicator for TEST mode 2020-04-06 -- SnipWire 0.8.6 (beta) released! Adds support for Snipcart subscriptions and also fixes some problems 2020-03-21 -- SnipWire 0.8.5 (beta) released! Improves SnipWires webhooks interface and provides some other fixes and additions 2020-03-03 -- SnipWire 0.8.4 (beta) released! Improves compatibility for Windows based Systems. 2020-03-01 -- SnipWire 0.8.3 (beta) released! The installation and uninstallation process has been heavily revised. 2020-02-08 -- SnipWire 0.8.2 (beta) released! Added a feature to change the cart and catalogue currency by GET, POST or SESSION param 2020-02-03 -- SnipWire 0.8.1 (beta) released! All custom classes moved into their own namespaces. 2020-02-01 -- SnipWire is now available via ProcessWire's module directory! 2020-01-30 -- SnipWire 0.8.0 (beta) first public release! (module just submitted to the PW modules directory) 2020-01-28 -- added Custom Order Fields feature (first SnipWire release version is near!) 2020-01-21 -- Snipcart v3 - when will the new cart system be implemented? 2020-01-19 -- integrated taxes provider finished (+ very flexible shipping taxes handling) 2020-01-14 -- new date range picker, discount editor, order notifiactions, order statuses, and more ... 2019-11-15 -- orders filter, order details, download + resend invoices, refunds 2019-10-18 -- list filters, REST API improvements, new docs platform, and more ... 2019-08-08 -- dashboard interface, currency selector, managing Orders, Customers and Products, Added a WireTabs, refinded caching behavior 2019-06-15 -- taxes provider, shop templates update, multiCURL implementation, and more ... 2019-06-02 -- FieldtypeSnipWireTaxSelector 2019-05-25 -- SnipWire will be free and open source Plugin Key Features
      Fast and simple store setup Full integration of the Snipcart dashboard into the ProcessWire backend (no need to leave the ProcessWire admin area) Browse and manage orders, customers, discounts, abandoned carts, and more Multi currency support Custom order and cart fields Process refunds and send customer notifications from within the ProcessWire backend Process Abandoned Carts + sending messages to customers from within the ProcessWire backend Complete Snipcart webhooks integration (all events are hookable via ProcessWire hooks) Integrated taxes provider (which is more flexible then Snipcart own provider) Useful Links
      SnipWire in PW modules directory SnipWire Docs (please note that the documentation is a work in progress) SnipWire @GitHub (feature requests and suggestions for improvement are welcome - I also accept pull requests) Snipcart Website  

       
      ---- INITIAL POST FROM 2019-05-25 ----
       
    • By Sten
      Hello
      Till now I hacked something with the twig template but it works no more with new PW versions so I look forward to create a module. I am working on a site in multiple languages : French, English, Italian, German, Spanish, Portuguese, Hebrew, Russian. The new posts are entered in any language with a field for language. Till now, I got twig files to get the translations with constants defined for each part of the pages.
      So I'd like to create a module to include theses files added according to the url /fr/en/...
      Have you some observations to do before I begin about the direction to take ?
      Thank you
    • By ukyo
      Mystique Module for ProcessWire CMS/CMF
      Github repo : https://github.com/trk/Mystique
      Mystique module allow you to create dynamic fields and store dynamic fields data on database by using a config file.
      Requirements
      ProcessWire 3.0 or newer PHP 7.0 or newer FieldtypeMystique InputfieldMystique Installation
      Install the module from the modules directory:
      Via Composer:
      composer require trk/mystique Via git clone:
      cd your-processwire-project-folder/ cd site/modules/ git clone https://github.com/trk/Mystique.git Module in live reaction with your Mystique config file
      This mean if you remove a field from your config file, field will be removed from edit screen. As you see on youtube video.
      Using Mystique with your module or use different configs path, autoload need to be true for modules
      Default configs path is site/templates/configs/, and your config file name need to start with Mystique. and need to end with .php extension.
      Adding custom path not supporting anymore !
      // Add your custom path inside your module class`init` function, didn't tested outside public function init() { $path = __DIR__ . DIRECTORY_SEPARATOR . 'configs' . DIRECTORY_SEPARATOR; Mystique::add($path); } Mystique module will search site/modules/**/configs/Mystique.*.php and site/templates/Mystique.*.php paths for Mystique config files.
      All config files need to return a PHP ARRAY like examples.
      Usage almost same with ProcessWire Inputfield Api, only difference is set and showIf usage like on example.
      <?php namespace ProcessWire; /** * Resource : testing-mystique */ return [ 'title' => __('Testing Mystique'), 'fields' => [ 'text_field' => [ 'label' => __('You can use short named types'), 'description' => __('In file showIf working like example'), 'notes' => __('Also you can use $input->set() method'), 'type' => 'text', 'showIf' => [ 'another_text' => "=''" ], 'set' => [ 'showCount' => InputfieldText::showCountChars, 'maxlength' => 255 ], 'attr' => [ 'attr-foo' => 'bar', 'attr-bar' => 'foo' ] ], 'another_text' => [ 'label' => __('Another text field (default type is text)') ] ] ]; Example:
      site/templates/configs/Mystique.seo-fields.php <?php namespace ProcessWire; /** * Resource : seo-fields */ return [ 'title' => __('Seo fields'), 'fields' => [ 'window_title' => [ 'label' => __('Window title'), 'type' => Mystique::TEXT, // or InputfieldText 'useLanguages' => true, 'attr' => [ 'placeholder' => __('Enter a window title') ] ], 'navigation_title' => [ 'label' => __('Navigation title'), 'type' => Mystique::TEXT, // or InputfieldText 'useLanguages' => true, 'showIf' => [ 'window_title' => "!=''" ], 'attr' => [ 'placeholder' => __('Enter a navigation title') ] ], 'description' => [ 'label' => __('Description for search engines'), 'type' => Mystique::TEXTAREA, 'useLanguages' => true ], 'page_tpye' => [ 'label' => __('Type'), 'type' => Mystique::SELECT, 'options' => [ 'basic' => __('Basic page'), 'gallery' => __('Gallery'), 'blog' => __('Blog') ] ], 'show_on_nav' => [ 'label' => __('Display this page on navigation'), 'type' => Mystique::CHECKBOX ] ] ]; Searching data on Mystique field is limited. Because, Mystique saving data to database in json format. When you make search for Mystique field, operator not important. Operator will be changed with %= operator.
      Search example
      $navigationPages = pages()->find('my_mystique_field.show_on_nav=1'); $navigationPages = pages()->find('my_mystique_field.page_tpye=gallery');
    • By Robin S
      This is a module I made as an experiment a while ago and never got around to releasing publicly. At the time it was prompted by discussions around using Repeater fields for "page builder" purposes, where the depth feature could possibly be used for elements that would be nested inside other elements. I thought it would be useful to enforce some depth rules and translate the depth data into a multi-dimensional array structure.
      I'm not using this module anywhere myself but maybe it's useful to somebody.
      Repeater Depth Helper
      This module does two things relating to Repeater fields that have the "Item depth" option enabled:
      It enforces some depth rules for Repeater fields on save. Those rules are:
      The first item must have a depth of zero. Each item depth must not be more than one greater than previous item depth. It provides a RepeaterPageArray::getDepthStructure helper method that returns a nested depth structure for a Repeater field value.
      Helper method
      The module adds a RepeaterPageArray::getDepthStructure method that returns a multi-dimensional array where the key is the page ID and the value is an array of nested "child" items, or null if there are no nested children.
      Example

      The module doesn't make any assumptions about how you might want to use the depth structure array, but here is a way you might use it to output a nested unordered list.
      // Output a nested unordered list from a depth structure array function outputNestedList($depth_structure, $repeater_items) { $out = "<ul>"; foreach($depth_structure as $page_id => $nested_children) { $out .= "<li>" . $repeater_items->get("id=$page_id")->title; // Go recursive if there are nested children if(is_array($nested_children)) $out .= outputNestedList($nested_children, $repeater_items); $out .= "</li>"; } $out .= "</ul>"; return $out; } $repeater_items = $page->my_repeater; $depth_structure = $repeater_items->getDepthStructure(); echo outputNestedList($depth_structure, $repeater_items);
       
      https://github.com/Toutouwai/RepeaterDepthHelper
      https://modules.processwire.com/modules/repeater-depth-helper/
    • By MoritzLost
      Cacheable Placeholders
      This module allows you to have pieces of dynamic content inside cached output. This aims to solve the common problem of having a mostly cacheable site, but with pieces of dynamic output here and there.  Consider this simple example, where you want to output a custom greeting to the current user:
      <h1>Good morning, <?= ucfirst($user->name) ?></h1> This snippet means you can't use the template cache (at least for logged-in users), because each user has a different name. Even if 99% of your output is static, you can only cache the pieces that you know won't include this personal greeting. A more common example would be CSRF tokens for HTML forms - those need to be unique by definition, so you can't cache the form wholesale.
      This module solves this problem by introducing cacheable placeholders - small placeholder tokens that get replaced during every request. The replacement is done inside a Page::render hook so it runs during every request, even if the response is served from the template cache. So you can use something like this:
      <h1>Good morning, {{{greeting}}}</h1> Replacement tokens are defined with a callback function that produces the appropriate output and added to the module through a simple hook:
      // site/ready.php wire()->addHookAfter('CachePlaceholders::getTokens', function (HookEvent $e) { $tokens = $e->return; $tokens['greeting'] = [ 'callback' => function (array $tokenData) { return ucfirst(wire('user')->name); } ]; $e->return = $tokens; }); Tokens can also include parameters that are parsed and passed to the callback function. There are more fully annotated examples and step-by-step instructions in the README on Github!
      Features
      A simple and fast token parser that calls the appropriate callback and runs automatically. Tokens may include multiple named or positional parameters, as well as multi-value parameters. A manual mode that allows you to replace tokens in custom pieces of cached content (useful if you're using the $cache API). Some built-in tokens for common use-cases: CSRF-Tokens, replacing values from superglobals and producing random hexadecimal strings. The token format is completely customizable, all delimiters can be changed to avoid collisions with existing tag parsers or template languages. Links
      Github Repository & documentation Module directory If you are interested in learning more, the README is very extensive, with more usage examples, code samples and usage instructions!
×
×
  • Create New...