bernhard

RockFinder - Highly Efficient and Flexible SQL Finder Module

Recommended Posts

WHY?

This module was built to fill the gap between simple $pages->find() operations and complex SQL queries.

The problem with $pages->find() is that it loads all pages into memory and that can be a problem when querying multiple thousands of pages. Even $pages->findMany() loads all pages into memory and therefore is a lot slower than regular SQL.

The problem with SQL on the other hand is, that the queries are quite complex to build. All fields are separate tables, some repeatable fields use multiple rows for their content that belong to only one single page, you always need to check for the page status (which is not necessary on regular find() operations and therefore nobody is used to that).

In short: It is far too much work to efficiently and easily get an array of data based on PW pages and fields and I need that a lot for my RockGrid module to build all kinds of tabular data.


Basic Usage

getObjects.png.33c56b6030f7c49cfbd88591c016463d.png

 

Docs & Download

https://modules.processwire.com/modules/rock-finder/

https://gitlab.com/baumrock/RockFinder/tree/master

 

Changelog

  • Like 11
  • Thanks 1

Share this post


Link to post
Share on other sites

If anybody is already using RockFinder make sure you have the latest version (1.0.0). Just pushed an update because the previous version didn't work with checkbox selectors because I only used the WHERE part of the original query. I switched now to a join, that should work in 100% of the cases:

screencapture-360-test-admin-setup-rockfindertester-2018-05-16-17_22_57.thumb.png.c7b37ee8d866377947bfe6e414a05999.png

  • Like 1

Share this post


Link to post
Share on other sites

Please submit this gold module to the modules directory so we can update it in one click 🤩

 

Edit:  Just re-installed the module from scratch, I got an error when I try to access the RockFinder tester, see the attached screenshot :

rockfindertrace.thumb.png.4b147201aef348e4bbb224ca5da2c227.png

 

Share this post


Link to post
Share on other sites

Hi @flydev sorry but I cannot reproduce this bug. I need step by step instructions, pw version etc...

Quote

Please submit this gold module to the modules directory so we can update it in one click 🤩

It is already there: https://modules.processwire.com/modules/rock-finder/ not sure when it updates, though (it's still version 7 instead of 1.0.0). 

Share this post


Link to post
Share on other sites
  • ProcessWire 3.0.98
  • PHP 7.0.10

I just installed the module from your Gitlab repo then I clicked Setup > RockFinder tester and the error happen.

I think you can't reproduce this error because you have InputfieldRockGrid (called at line 50 in ProcessRockFinder.module.php) but its not shipped with module. That explain the warning.

 

wow those smileys are big lol

 

Share this post


Link to post
Share on other sites

please try to install the regular module before the process module

1 minute ago, flydev said:

I think you can't reproduce this error because you have InputfieldRockGrid (called at line 50 in ProcessRockFinder.module.php) but its not shipped with module.

No, I have a clean installation only with AOS + Tracy and it worked

Share this post


Link to post
Share on other sites
  1. Installed a stock ProcessWire 3.0.102
  2. I Downloaded the module RockFinder 1.0.0
  3. I installed RockFinder
  4. I installed ProcessRockFinder
  5. I clicked Setup > RockFinder tester

I have the same warning / error.

Just for testing I replaced InputfieldRockGrid on line 50 by InputfieldTextarea, I can access the process module.

  • Like 1

Share this post


Link to post
Share on other sites

Ah, sorry @flydev I missed that this happens when clicking on the module's page and not during installation.

I pushed an update that does the check if RockGrid is available, otherwise it does not add this field (and shows only SQL). The process module is not really helpful without RockGrid, I know, but the day will come where I can release RockGrid 😉

  • Like 1

Share this post


Link to post
Share on other sites

okay 🙂

 

PS: to update the modules directory, just re-submit the module and the module will be updated automatically in a second.

  • Like 2

Share this post


Link to post
Share on other sites
On 5/16/2018 at 6:56 PM, flydev said:

PS: to update the modules directory, just re-submit the module and the module will be updated automatically in a second.

this does not seem to work. current module version is 1.0.2 and https://modules.processwire.com/modules/rock-finder/ still shows 1.0.0

Share this post


Link to post
Share on other sites
1 hour ago, kongondo said:

Maybe it only works with GitHub hosted modules? 

That what I was thinking.

Share this post


Link to post
Share on other sites

Yeah, coz it's not even picking up the README.

Share this post


Link to post
Share on other sites

just added the possibility to add custom sql statements easily. that way you can easily do "reverse queries", for example show all projects that have the current page selected in a page-reference-field:2018-05-28--11-00-32.png.435e4e37d618f136e6ea6e276dd939a4.png

 

Aggregations like sum(), min(), max() should also be easily possible like this, though they might not be as efficient as doing a group_by manually on the resulting sql.

  • Like 2

Share this post


Link to post
Share on other sites

Just released RockGrid, so RockFinder Tester is now usable in this module 🙂 

  • Like 1

Share this post


Link to post
Share on other sites

Pushed a little Update for basic support of options fields (using the same sql queries as file fields).

But I want to share something I needed today to keep my requests simple and my code clean: You can quite easily join two finders where manipulating the SQL would maybe quite complex:

$this->rg->assets->add($this->config->paths->siteModules . 'FieldtypeRockGrid/lib/moment.min.js');
$this->rg->assets->add($this->config->paths->siteModules . 'FieldtypeRockGrid/lib/currency.min.js');

$efforts = new RockFinder('template=effort', [
  'title',
  // fields
]);

$projects = new RockFinder('template=project', [
  'title',
  // fields
]);

$sql = 'SELECT
    efforts.*,
    projects.id AS projectid,
    projects.title AS projectname,
    projects.project_status AS projectstatus
  FROM (' . $efforts->getSQL() . ') AS efforts
  LEFT JOIN field_project_efforts AS ef ON ef.data = efforts.id
  LEFT JOIN (' . $projects->getSQL() . ') AS projects ON projects.id = ef.pages_id /* projectid */
  ';

$this->setData($sql);

Result (with RockGrid and some JavaScript, of course):

2018-08-10--23-36-10.thumb.png.ea0078fd11b2d93526a8122fe249217d.png

  • Like 2

Share this post


Link to post
Share on other sites

Added support for joining multiple finders. You can even join finders on columns of previously joined finders 🙂

In this example I'm joining projects (finder1) and their related clients (single page reference field of project) and then I join the referral contact of the project's client based on the "camefrom" id of finder2:

join.png 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   1 member

  • Similar Content

    • By OLSA
      Hello for all,
      ConfigurationForm fieldtype module is one my experiment from 2016.
      Main target to build this module was to store multiple setup and configuration values in just 1 field and avoid to use 1 db table to store just single "number of items on page", or another db table to store "layout type" etc. Thanks to JSON formatted storage this module can help you to reduce number of PW native fields in project, save DB space, and reduce number of queries at front-end.
      Install and setup:
      Download (at the bottom ), unzip and install like any other PW module (site/modules/...). Create some filed using this type of field (ConfigurationForm Fieldtype) Go to field setup Input tab and drag some subfields to container area (demo). Set "Name" and other params for subfields Save and place field to templates ("Action tab") How to use it:
      In my case, I use it to store setup and configurations values, but also for contact details, small content blocks... (eg. "widgets").
      Basic usage example:
      ConfigForm fieldtype "setup" has subfields:
      "limit", type select, option values: 5, 10, 15, 20
      "sort", type select, option values: "-date", "date",  "-sort", "sort"
      // get page children (items) $limit = isset($page->setup->limit) ? $page->setup->limit : 10; $sort = isset($page->setup->sort) ? $page->setup->sort : '-sort'; $items = $page->children("limit=$limit, sort=$sort");  
      Screenshots:


       
      Notes:
      Provide option to search inside subfields Provide multilanguage inputs for text and textarea field types Provide option for different field layout per-template basis Do not place/use field type "Button" or "File input" because it won't works. Please read README file for more details and examples Module use JSON format to store values. Text and textarea field types are multilanguage compatible, but please note that main target for this module was to store setup values and small content blocks and save DB space. Search part inside JSON is still a relatively new in MySQL (>=5.77) and that's on you how and for what to use this module.
      Thanks:
      Initial point for this fieldtype was jQuery plugin FormBuiled and thanks to Kevin Chappel for this plugin.
      In field type "link" I use javascript part from @marcostoll module and thanks to him for that part.
      Download:
      FieldtypeConfigForm.zip
      Edit: 14. August 2018. please delete/uninstall previously downloaded zip
      Regards.
         
    • By bernhard
      @Sergio asked about the pdf creation process in the showcase thread about my 360° feedback/survey tool and so I went ahead and set my little pdf helper module to public.
      Description from PW Weekly:
       
      Modules Directory: https://modules.processwire.com/modules/rock-pdf/
      Download & Docs: https://gitlab.com/baumrock/RockPdf
       
      You can combine it easily with RockReplacer: 
      See also a little showcase of the RockPdf module in this thread:
       
    • By Thomas Diroll
      Hi guys I'm relatively new to PW and just finished developing a page for a client. I was able to include all necessary functionality using the core fieldtypes but now I it seems that I need to extend them with a custom one. What I need is a simple button, that copies the absolute url (frontend not PW-backend) of the page which is currently edited to the clipboard. As this feature is only needed inside a specific template, I tend to use a custom fieldtype which provides this feature. I've been looking inside the core modules code (eg. FieldtypeCheckbox.module) but I don't really get the structure of it and how its rendered to the admin page. I also didn't find a lot of tutorials covering custom fieldtypes.
      Maybe some of you could give me some tips on how to write a basic custom fieldtype that renders a button which copies the value of
      page->httpUrl() to the clipboard using JS. Thanks!
    • By bernhard
      Some of you might have followed the development of this module here: https://processwire.com/talk/topic/15524-previewdiscussion-rockdatatables/ . It is the successor of "RockDataTables" and requires RockFinder to get the data for the grid easily and efficiently. It uses the open source part of agGrid for grid rendering.
       
      WHY?
      ProcessWire is awesome for creating all kinds of custom backend applications, but where it is not so awesome in my opinion is when it comes to listing this data. Of course we have the built in page lister and we have ListerPro, but none of that solutions is capable of properly displaying large amounts of data, for example lists of revenues, aggregations, quick and easy sorts by the user, instant filter and those kind of features. RockGrid to the rescue 😉 
       
      Features/Highlights:
      100k+ rows Instant (client side) filter, search, sort (different sort based on data type, eg "lower/greater than" for numbers, "contains" for strings) extendable via plugins (available plugins at the moment: fullscreen, csv export, reload, batch-processing of data, column sum/statistics, row selection) all the agGrid features (cell renderers, cell styling, pagination, column grouping etc) vanilla javascript, backend and frontend support (though not all plugins are working on the frontend yet and I don't plan to support it as long as I don't need it myself)  
      Limitations:
      While there is an option to retrieve data via AJAX the actual processing of the grid (displaying, filtering, sorting) is done on the client side, meaning that you can get into troubles when handling really large datasets of several thousands of rows. agGrid should be one of the most performant grid options in the world (see the official example page with a 100k row example) and does a lot to prevent problems (such as virtual row rendering), but you should always have this limitation in mind as this is a major difference to the available lister options that do not have this limitation.
      Currently it only supports AdminThemeUikit and I don't plan to support any other admin theme.
       
      Download: https://gitlab.com/baumrock/RockGrid
      Installation: https://gitlab.com/baumrock/RockGrid/wikis/Installation
      Quikckstart: https://gitlab.com/baumrock/RockGrid/wikis/quickstart
      Further instructions: https://gitlab.com/baumrock/RockGrid/wikis/quickstart#further-instructions
       
      Module status: alpha, License: MIT
      Note that every installation and uninstallation sends an anonymous google analytics event to my google analytics account. If you don't want that feel free to remove the appropriate lines of code before installation/uninstallation.
       
      Contribute:
      You can contribute to the development of this and other modules or just say thank you by
      testing, reporting issues and making PRs at gitlab liking this post buying me a drink: paypal.me/baumrock/5 liking my facebook page: facebook.com/baumrock hiring me for pw work: baumrock.com  
      Support: Please note that this module might not be as easy and plug&play as many other modules. It needs a good understanding of agGrid (and JavaScript in general) and it likely needs some looks into the code to get all the options. Please understand that I can not provide free support for every request here in the forum. I try to answer all questions that might also help others or that might improve the module but for individual requests I offer paid support for 60€ per hour.
       
      Changelog
      180730 support subdir installations 180711 bugfix (naming issue) 180630 alpha realease  
      Use Cases / Examples:
      Colored grid cells, Icons, Links etc. The Grid also has a "batcher" feature built in that helps communicating with the server via AJAX and managing resource intensive tasks in batches:

      Filters, PW panel links and instant reload on panel close:

      You can combine the grid with a chart library like I did with the (outdated) RockDataTables module:

    • By tpr
      ProcessNetteTester
      Run Nette Tester tests within ProcessWire admin.
      (continued from here)

      Features
      AJAX interface for running Nette Tester tests, in bulk or manually display counter, error message and execution time in a table run all tests at once or launch single tests show formatted test error messages and report PHP syntax errors stop on first failed test (optional) hide passed tests (optional) display failed/total instead passed/total (optional) re-run failed tests only (optional) auto scroll (optional) include or exclude tests based on query parameters start/stop all tests with the spacebar reset one test or all tests (ctrl+click) TracyDebugger File Editor integration https://modules.processwire.com/modules/process-nette-tester/
      https://github.com/rolandtoth/ProcessNetteTester