RockFinder3 - Combine the power of ProcessWire selectors and SQL

[bernhard]

Combine the power of ProcessWire selectors and SQL

Differences to previous RockFinder modules

• RockFinder3 comes with extensive docs ? 
• RF3 supports chaining: $RockFinder3->find("template=foo")->addColumns(['foo']).
• RF3 fully supports multi-language.
• RF3 makes it super-easy to add custom columnTypes.
• RF3 makes it easier to use custom SQL statements.
• No bloat! The module does just do one thing: Finding data.

Differences to findRaw

• Background: RockFinder has been there before findRaw existed
• findRaw makes it a little easier to query page fields, but RockFinder is more flexible in that regard
• RockFinder might be faster than findRaw: see here

DOCS & DOWNLOAD: https://github.com/baumrock/rockfinder3

 

Thanks for using RockFinder3. If you find RockFinder3 helpful consider giving it a star on github or saying thank you. I'm also always happy to get feedback in the PW forum!

Happy finding

[adrian]

Hey @bernhard - looking awesome - I think it's time I took this for a proper test spin ?

Just tested my first query and got this error:

[bernhard]

Thx @adrian does the error disappear when you remove the "created" column?

[adrian]

1 minute ago, bernhard said:

Thx @adrian does the error disappear when you remove the "created" column?

Well, it's not just the "created" field - the same error shows with the "name" field as well.

Loving some of the options this has BTW - having those docs has really shown me what this can do!

Just a thought about the Process module and the Tracy Console - I am not sure it's clear that the Process module is only needed to load Tabulator - otherwise loading the Console panel from any other page works for dumping the objects in the normal Tracy way.

I do think it would be great to figure out a way to load the Tablulator script on demand in Tracy if a RF dump() is called.

[bernhard]

Please try the latest commit ? 

[bernhard]

2 minutes ago, adrian said:

Just a thought about the Process module and the Tracy Console - I am not sure it's clear that the Process module is only needed to load Tabulator - otherwise loading the Console panel from any other page works for dumping the objects in the normal Tracy way.

True. Do I state somewhere that you need to use the processmodule for dumping?

[adrian]

2 minutes ago, bernhard said:

Please try the latest commit ? 

That fixed it - thanks!

[adrian]

@bernhard - you might be interested in this: https://github.com/processwire/processwire-issues/issues/1173#issuecomment-632868896 and Ryan's reply. Once you upgrade to the latest PW dev and use your dumpSQL() method you'll see what I mean ?

[adrian]

@bernhard - test out the attached version which avoids the need for the Process module by loading the tabulator files on demand.

 

RockFinder3.module.php

[bernhard]

Thx @adrian works a treat ? I have slightly modified it so that it checks if Tabulator exists before loading it.

Seems you have some code prettifier active: https://github.com/baumrock/RockFinder3/commit/2863b76e9109673d7095e08d5d0d9b6ad5193f3c What are you using?

1 hour ago, adrian said:

@bernhard - you might be interested in this: https://github.com/processwire/processwire-issues/issues/1173#issuecomment-632868896 and Ryan's reply. Once you upgrade to the latest PW dev and use your dumpSQL() method you'll see what I mean ?

I've followed this issue and was a little afraid something would break anything in rockfinder, but I did the upgrade just before posting this topic and everything seemed to work. If you are referring to the screenshots containing "old" values instead of bind variables that's just because I made them before upgrading the core. Or is there anything I missed out or need to check again?

[adrian]

2 minutes ago, bernhard said:

Seems you have some code prettifier active: https://github.com/baumrock/RockFinder3/commit/2863b76e9109673d7095e08d5d0d9b6ad5193f3c What are you using?

Just have VSCode set up to automatically remove trailing spaces at the end of lines. It's a pet peeve of mine and something I wish Ryan would do with the PW core.

3 minutes ago, bernhard said:

Or is there anything I missed out or need to check again?

Nothing new - it's just a shame that we can no longer see the values in query - makes it impossible to easily copy / paste a query into Adminer etc to test directly.

Are you going to completely remove the Process module now? Does it serve some other purpose?

[bernhard]

4 minutes ago, adrian said:

Are you going to completely remove the Process module now? Does it serve some other purpose?

That just happened with the latest commit ? Thanks!

4 minutes ago, adrian said:

makes it impossible to easily copy / paste a query into Adminer etc to test directly.

Didn't think of that and that would have really made the development difficult! I'll post a comment in the issue ? 

5 minutes ago, adrian said:

Just have VSCode set up to automatically remove trailing spaces at the end of lines. It's a pet peeve of mine and something I wish Ryan would do with the PW core.

Thx for clarifying. I also don't like them, but it seems that Ryan likes them, because we have this setting in /wire/.editorconfig

trim_trailing_whitespace = false

Thx, just enabled the setting in VSCode as well ? 

[adrian]

13 minutes ago, bernhard said:

I also don't like them, but it seems that Ryan likes them, because we have this setting in /wire/.editorconfig

Assume you have read these:

https://github.com/ryancramerdesign/ProcessWire/issues/710
https://github.com/ryancramerdesign/ProcessWire/issues/1757

He says that "whitespace is preferred in many cases", but I just don't get it - not sure how trailing whitespace has any advantage, but maybe I am missing something ?

[adrian]

4 minutes ago, bernhard said:

That just happened with the latest commit ? Thanks!

Remember you have some wording in the docs that needs changing, eg "The ProcessModule of RockFinder3 does use Tracy for dumping the results, so TracyDebugger is required for the ProcessModule to run."

[bernhard]

Thx, is removed ? 

[adrian]

18 minutes ago, bernhard said:

Didn't think of that and that would have really made the development difficult! I'll post a comment in the issue

Yeah, please do - I think it's going to prove quite painful if we can't generate queries that work independently. I make use of the "Selector Queries" feature on the RequestInfo panel in Tracy a LOT and it's no longer as useful. Sounds like Ryan is keen to come up with a fix, but an extra voice might help motivate him ?

[bernhard]

Done ? Just realized that this update actually breaks a lot of the functionality provided by RockFinder3 - so it's critical to get an update for this! ? 

[adrian]

Just now, bernhard said:

this update actually breaks a lot of the functionality provided by RockFinder3

Yeah, I just saw your post - that's too bad - hopefully there is a solution!

[adrian]

Hey @bernhard - attached is a new version that loads Tabulator if someone wants to make a dump() call directly to the page within their template file.

Basically it just duplicates the JSloader functionality within this module.

I was also thinking about maybe making a barDump() version that would emulate bd() but with Tabulator output, but didn't get there yet.

I decided to take a look at adding a barDump() option. To test you'll need the new Rockfinder and TD.php files attached.

Now you can call the new barDump() method from anywhere (including template files, and the Console panel) and Tracy Dumps panel will receive the tabulated output, or the dumped object if you pass "true" for the title.

Please test thoroughly and feel free to tweak behavior as needed.

I should note that I changed the return for the dump method from $this to $out. I guess this will break chaining if someone wants to add another method after this. Maybe there is a better way to do this but was in a hurry.

 

RockFinder3.module.php TD.php

[psy]

Hi @bernhard love your work.

Is RF2 upgradeable to RF3 or is RF3 a completely different module?

[bernhard]

Thx @psy

It depends on how you have used it I guess. But you can definitely run both beside each other. That's why they have separate names so that I can use them together in my projects without the need of refactoring existing code. In the long run RockFinder3 should be the only module to stay. Meaning that if you build a new project you should definitely use RF3 only and if you find anything that does not work yet with RF3 just give me a ping and I'll implement it. Most of that should be easy and maybe just copy&pasting methods from RF2 to RF3 and doing some refactoring.

So I guess the answer is: It is a completely different module ? 

[bernhard]

9 hours ago, adrian said:

Now you can call the new barDump() method from anywhere (including template files, and the Console panel) and Tracy Dumps panel will receive the tabulated output, or the dumped object if you pass "true" for the title.

Please test thoroughly and feel free to tweak behavior as needed.

I should note that I changed the return for the dump method from $this to $out. I guess this will break chaining if someone wants to add another method after this. Maybe there is a better way to do this but was in a hurry.

Great idea ? I've implemented your changes on the bardump branch: https://github.com/baumrock/RockFinder3/commit/436257b8822fd89e76951db1521c8f2619ef880c

Once you have your code in Tracy please tell me and I can merge to master ? 

The new section of the readme:

 

Dumping data

For small finders Tracy's dump() feature is enough, but if you have more complex finders or you have thousands of pages this might get really inconvenient. That's why RockFinder3 ships with a custom dump() method that works in the tracy console and turns the result of the finder into a paginated table (using tabulator.info).

For all the dumping methods you can provide two parameters:

1. The title of the dump*
2. The settings for the rendered tabulator table

*Note that if you set the title to TRUE the method will not only dump the tabulator but also the current RockFinder3 object (see the next example).

dump() or d()

$RockFinder3
  ->find("template=cat")
  ->addColumns(['title', 'created'])
  ->dump(true);

barDump() or bd()

For situations where you are not working in the console but maybe in a template file or a module the barDump() method might be useful.

Dumping the SQL of the finder

To understand what is going on it is important to know the SQL that is executed. You can easily dump the SQL query via the dumpSQL() or barDumpSQL() methods. This even supports chaining:

$RockFinder3
  ->find("template=cat")
  ->addColumns(['title'])
  ->dumpSQL()
  ->addColumns(['owner'])
  ->dumpSQL()
  ->dump();

[bernhard]

v1.0.3 adds two new features:

• addPath()
• each()

Callbacks

RockFinder3 supports row callbacks that are executed on each row of the result. Usage is simple:

each()

$RockFinder3
  ->find("template=cat")
  ->addColumns(['title', 'weight'])
  ->each(function($row) { $row->myTitle = "{$row->title} ({$row->weight} kg)"; })
  ->dump();

These callbacks can be a great option, but keep in mind that they can also be very resource intensive! That applies even more when you request page objects from within your callback (meaning there will be no benefit at all in using RockFinder compared to a regular $pages->find() call).

addPath()

A special implementation of the each() method is the addPath() method that will add a path column to your result showing the path of every page. This will not load all pages into memory though, because it uses the $pages->getPath() method internally.

$RockFinder3
  ->find("template=cat")
  ->addColumns(['title', 'weight'])
  ->addPath("de")
  ->dump();

If you need the path for linking/redirecting from your data to the pages it might be better to build a custom redirect page that works with the page id, so you don't need the overhead of getting all page paths:

<a href='/your/redirect/url/?id=123'>Open Page 123</a>

If you really need to access page objects you can get them via the $finder parameter of the callback:

$finder->each(function($row, $finder) {
  $row->foo = $finder->pages->get($row->id)->foo;
}

[flydev]

I have the same question than @psy but about RockFinder 1 ?

I think I have already my answer..

 

Anyway, thanks version 3 @bernhard ??

Edited May 25, 2020 by flydev ??
wrong ppl tag

[bernhard]

7 minutes ago, flydev ?? said:

but about RockFinder 1 ?

FockFinder1 had a totally different implementation (not based on DatabaseQuerySelect) and also a very different API, so there will be lots of differences. I think the concept of joins and relations should be very flexible though. And it's now also a lot easier to create custom column types that basically let you join whatever data you need. So I think RF3 should really be an improvement in all areas.

If you find any use cases that RF3 does not support, please let me know ?