Working with virtual pages and virtual pageimage objects

[gebeer]

Now that the major update for my ImagePicker module is out, I wanted to share how I managed to make images from a folder in site/templates/  behave like a Pageimage with full resizing/cropping capabilities. This was probably the biggest challenge in developing the new features.

I first thought to create a hidden page in the page tree that will hold all these images. But this would not have been a clean solution. So I took a closer look on how the Pageimage and Pageimages objects are constructed. Finally I decided to use a virtual (or fake) page to hold the images. It required some method overriding of the Page class and the PagefilesManager class to adjust the paths. Normally images for a page can only live in a folder in site/assets/files/pageid. By manipulating the path of Pagefilesmanager I was able to create a custom ImagePickerVirtualpage object and assign the images from the folder to that page. The virtual page is created at runtime and the DB never sees it. It took quite some trial and error to accomplish this. But finally, you can now manipulate these images just like any other Pageimage object (resizing/cropping). The resulting image variations will be stored in the same folder.
There is one little stumbling block. I use an existing images field from the site my field is installed on and assign it to a virtual template of my virtual page. This field then holds all the folder images. I figured that every site that will use my module will already have an images field installed. Very rarely this will not be the case. So for the time being I am leaving the logic as it is. If I find people running into problems, I will adjust this. Code can be found here.

One more thing that offered some challenge was storing the data as JSON in the DB and have it fully searchable through the API. Luckily I found an old (and abandoned?) module JsonNativeField from @BitPoet that does just that: store values as JSON and make them searchable. So I borrowed the DB schema and  most of the getMatchQuery logic from that module. I still have 2 extra columns for pageid and filename in the DB table. These make sure that in the Pagefinder you can use subfields. Kind of redundant info in these columns but I couldn't figure out another way of making the subfield search work in Pagefinder. If you happen to know how to accomplish this, please enlighten me.
Main advantage of JSON storage is that my image field can be used with the marcrura's SettingsFactory module.
Also storing values of multiple images this way will be easy and will come in handy once I extend my module to support multiple images.

[Macrura]

@gebeer Thanks for your work on this!

I was going to propose that the module installs a page for use with images, but you have found a possibly better solution with the virtual page.
And in terms of this new Fieldtype module using json, and the fact that it supports the SettingsFactory is great, and I hope it didn't cause any issues, but thanks again for going to the nth degree and making this work!

Lastly, I only worry about the JsonNativeField in that it requires InnoDB.. guess it might be time to make the switch on sites using MyISAM..

[szabesz]

4 hours ago, Macrura said:

JsonNativeField in that it requires InnoDB.

Ooops. I have no site using InnoDB and I am not planning to use it. So no ImagePicker module for me I guess ?

Edited December 18, 2019 by szabesz
typo

[Robin S]

@gebeer, I have similar concerns as @szabesz, not just about the InnoDB requirement but the MySQL >= 5.7.8 also. I think a lot of hosts and existing sites out there are not going to meet that requirement. For instance all my sites use MariaDB and as far as I know there isn't full JSON support in MariaDB yet.

Could you please clarify if InnoDB / MySQL >= 5.7.8 are strict requirements for the module? And if the module will partially work with MyISAM / MariaDB / MySQL < 5.7.8 which features would not be available?

[adrian]

@gebeer - no time to see what you've done vs what I have done in the past, but I just wanted to quickly link to: 

 in case it's at all useful.

Maybe your approach is the same or better?

[gebeer]

5 hours ago, Robin S said:

Could you please clarify if InnoDB / MySQL >= 5.7.8 are strict requirements for the module? And if the module will partially work with MyISAM / MariaDB / MySQL < 5.7.8 which features would not be available?

At the moment these are strict requirements. To be honest, I wasn't aware of the fact that JSON data type in MySQL requires InnoDB. The module, as it is now, will not work on MySQL < 5.7.8
I added that info to the module requirements.

Almost all of the sites that I am working on run on MySQL >= 5.7.8. InnoDB. But I understand that not every site can fulfill these requirements.

After reading your comments here about this possibly being an issue for some people, I am thinking about redesigning the storage part. ATM I already have separate columns for storing pageid and filename plus that extra JSON column. Will contemplate some more and see if I can come up with a good solution. Maybe only using json if the hosting environment supports this. I'll see.

 

[gebeer]

4 hours ago, adrian said:

Maybe your approach is the same or better?

Thanks for that link. I first was taking a similar approach, passing a fake page to construct the Pageimages object. But every time I added an image from the folder to the Pageimages instance with $pageimages->add($pathtofile) it would add this file to the folder again. So with every add() call I created redundant extra files. BUT I didn't pass a Field instance like you did. So will give this a try.

[kongondo]

19 hours ago, gebeer said:

extra JSON column

@gebeer

What exactly are you storing in this JSON column? I've just had a quick look and it seems you only store pageid and the filename (which you also store separately in the their respective columns). Am I missing something?

[gebeer]

4 hours ago, kongondo said:

What exactly are you storing in this JSON column? I've just had a quick look and it seems you only store pageid and the filename (which you also store separately in the their respective columns). Am I missing something?

Yes, I have a 'double storage' at the moment. Because I had not finally decided how to handle this. So atm there is redundant data. Main reason for adding that JSON column was so I can support SettingsFactory which only can store text values.
Also, after switching to JSON storage, I left pageid and filename columns so that in the Pagefinder you can choose these as subfields. Do you know a better way to handle this?

Anyways, I will be going  to set data column as a text in the next version to avoid the MySQL >= 5.7.8 requirement and amend the query logic to query pageid and filename columns instead of doing the JSON_SEARCH() on the data column.

[gebeer]

On 12/19/2019 at 2:11 AM, szabesz said:

Ooops. I have no site using InnoDB and I am not planning to use it

May I ask for the reason behind this? Ryan mentions some advantages of InnoDB over MyIsam in this blogpost.

The good news for you: next version (v1.1.2) of the module will revert back to not using MySQL native JSON data type so it will be compatible MyIsam.

[gebeer]

On 12/19/2019 at 4:53 AM, adrian said:

Maybe your approach is the same or better?

Just wanted to follow up on this. In the current master (v1.1.1) I changed the logic loosely following adrian's approach. This is working fine now. Also it removed an issue that occured with my previous approach. I still need the virtual page, though, to instantiate the Pageimages object. But now my virtual page doesn't need a real images field anymore ?. It is working with a virtual Field now. Code is here.

 

[kongondo]

8 hours ago, gebeer said:

Main reason for adding that JSON column was so I can support SettingsFactory which only can store text values.
Also, after switching to JSON storage, I left pageid and filename columns so that in the Pagefinder you can choose these as subfields. Do you know a better way to handle this?

Thanks for the clarification.

I don't know how SettingsFactory works. However, I doubt it accesses your field's database schema. I would imagine it will only be interested in the data you return in ___wakeupValue(). In addition, I don't know how SettingsFactory fetches the data, i.e. automatically or via an API called by the developer. Is it not possible for devs to convert what is returned in FIP to JSON/text for use in SettingsFactory? Alternatively, if you wish to return JSON values, you can do that in ___wakeupValue() but store your respective data sets (pageid and filename) in separate columns without the need for that extra column. It's not a deal breaker in FIP but you really want to avoid data redundancy in your database.

Maybe if I understood better how SettingsFactory works I'd be able to give a better-informed response.

[gebeer]

43 minutes ago, kongondo said:

Alternatively, if you wish to return JSON values, you can do that in ___wakeupValue() but store your respective data sets (pageid and filename) in separate columns without the need for that extra column

Thank you for the info. That is exactly what I am going to do now. Up to the point where I started developing this module, I was not really familiar with how field values travel through a live cycle. But meanwhile I realized, that the return value of wakeupValue() is what gets passed to the inputfield. I am returning json here already. And now that I decided to kick the JSON data column altogether, the redundancy will be gone ?

[kongondo]

16 minutes ago, gebeer said:

But meanwhile I realized, that the return value of wakeupValue() is what gets passed to the inputfield.

Yeah. You can do a lot of fancy stuff in wakeupValue(). You can even create run time values and add them as properties in your $field object, or even fetch data elsewhere and add these as runtime properties, etc.

[gebeer]

@kongondo is the data column mandatory? I get strange results for searches.

[kongondo]

13 minutes ago, gebeer said:

@kongondo is the data column mandatory? I get strange results for searches.

Yes it is, unless something changed that I am not aware of. In __sleepValue() and __wakeupValue() convert it from/to your property and in getMatchQuery() make sure to pass the correct $subfield (as you are currently doing in both respects). What are you currently using data for? pageid? Please show us an example search. 

[gebeer]

2 minutes ago, kongondo said:

What are you currently using data for? pageid? Please show us an example search

I had kicked data alltogether until I realized that a search for none empty returnes SQL error. Now I added it back in and will use it for the filename. Then everything should work, I guess...

[kongondo]

6 minutes ago, gebeer said:

Then everything should work, I guess...

Yes it should (but test ?). Just remember that in your getDatabaseSchema() for data, you will need to set correct column type (text or varchar, etc) and the correct index type (fulltext most likely if using text).

[gebeer]

12 minutes ago, kongondo said:

Yes it should (but test ?)

Testing right now and it is working. One thing boggles me. In Pagefinder, when selecting Subfield, I cannot select filename as subfield, only pageid. I had this with another custom fieldtype and always wondered how to have the alias for data show up here.

[kongondo]

53 minutes ago, gebeer said:

custom fieldtype and always wondered how to have the alias for data show up here.

Try ___getSelectorInfo(). Grep wire folder for full examples. Here's some example code. @see the label index in the code.

<?php

	/**
	 * Get information used for InputfieldSelector interactive selector builder
	 *
	 * This is for Lister purposes.
	 * We want nice labels for our lister selects (i.e. not raw db ones, i.e. 'some_column').
	 *
	 * @param Field $field The field we are working with.
	 * @param array $data Array of extra data, when/if needed.
	 * @return array
	 *
	 */
	public function ___getSelectorInfo(Field $field, array $data = array()) {
		$info = parent::___getSelectorInfo($field, $data);
		
		## filterable subfields for this field ##

		// we get rid of the subfield 'data' instead, we'll use 'custom one' => IF APPLICABLE TO YOU!
		if(isset($info['subfields']['data'])) unset($info['subfields']['data']);
		// unset misc_data (example column we don't need in filter)
		if(isset($info['subfields']['misc_data'])) unset($info['subfields']['misc_data']);
		$subfields = array(
			// @note: @see  getMatchQuery! 
			
			// text column example
			'some_text_column' => array(
				'name' => 'some_text_column',
				'input' => 'text',
				'label' => $this->_('My Text Column'),// NICE LABEL FOR LISTER
				// @note: comment out those that don't make sense!
				'operators' => array('%=', '!%=', '*=', '!*=', '~=', '!~=', '^=', '!^=', '$=', '!$=', '=', '!=', '=""', '!=""'),
				'options' => array(),
			),
			// number column example
			'some_number_column' => array(
				'name' => 'some_number_column',
				'input' => 'number',
				'label' => $this->_('My Number Column'),
				// @note: commented out those that don't make sense
				'operators' => array('=', '!=', /*'<', '>', '<=', '>=',*/ '=""', '!=""'),
				'options' => array(),
			),
		);

		$info['subfields'] = array_merge($info['subfields'], $subfields);

		return $info;

	}

If it doesn't work, I'll need to dig a litter deeper...

Edited December 20, 2019 by kongondo

[kongondo]

You might also need something like this in your getMatchQuery()

// if normal sql characters, do 'normal' query, else do fulltext search
if($this->wire('database')->isOperator($operator)) {
	return parent::getMatchQuery($query, $table, $subfield, $operator, $value);
}

else {
  $ft = new DatabaseQuerySelectFulltext($query);
  $ft->match($table, $subfield, $operator, $value);
  return $query;
}

 

[gebeer]

15 minutes ago, kongondo said:

Try ___getSelectorInfo(). Grep wire folder for full examples. Here's some example code. @see the label index in the code.

Big thanks for this one. I'll give it a shot.

10 minutes ago, kongondo said:

You might also need something like this in your getMatchQuery()

This I already covered. Got it from the Map Marker example module ?

EDIT: New v1.1.2 went out just before I read this. Will update soon.

[kongondo]

1 hour ago, kongondo said:

In __sleepValue() and __wakeupValue() convert it from/to your property and in getMatchQuery() make sure to pass the correct $subfield (as you are currently doing in both respects).

Oops. I was looking at the wrong file, so not sure if you are currently doing this? If not, here are examples:

public function ___wakeupValue(Page $page, Field $field, $value) {

    // some code...

    // start a blank value to be populated
    $myClassWireArray = $this->getBlankValue($page, $field); 

    // if we were given a blank value, then we've got nothing to do: just return a blank MyClassWireArray
    if(empty($value) || !is_array($value)) return $myClassWireArray; 

    // create new myClassObj objects from each item in the array
    foreach($value as $v) {
        $myClassObj = new MyClass();
      
        
        // @note we're converting 'data' to 'property1' (this is the filename)
        $myClassObj->property1 = $v['data']; 
        $myClassObj->property2 = $v['column_2']; 
        $myClassObj->property3 = $v['column_3'];
        $myClassObj->setTrackChanges(true); 
        $myClassWireArray->add($myClassObj); 
    }

    $myClassWireArray->resetTrackChanges(); 

    return $myClassWireArray;

}

public function ___sleepValue(Page $page, Field $field, $value) {

		$sleepValue = array();

        // some code....

		// convert each MyClassObj to an array within sleepValue
		foreach($value as $myClassObj) {
			$sleepValue[] = array(
				'data' => $myClassObj->property1, // @note: property1 is becoming data (this is the filename)
				'column_2' => (int) $myClassObj->property2,
				'column_3' => (int) $myClassObj->property3
				); 
		}

		return $sleepValue;

    }

 

Edited December 20, 2019 by kongondo

[gebeer]

@kongondo all is good! That  ___getSelectorInfo() method was exactly what I needed. It is already live on github

[kongondo]

8 minutes ago, gebeer said:

This I already covered. Got it from the Map Marker example module ?

EDIT: New v1.1.2 went out just before I read this. Will update soon.

Great!

Btw..

You don't need to set inbuilt ProcessWire columns like 'pages_id in your getDatabaseSchema() :-). ProcessWire will do it automatically. Same with extra, unless you really are adding something extra.

1 minute ago, gebeer said:

@kongondo all is good! That  ___getSelectorInfo() method was exactly what I needed. It is already live on github

Thanks for confirming! ??