[SOLVED] Connect local images, add variations, other strategies ?

[happywire]

[solution]

=========================================================

Trying to come up with a strategy to handle images. variations and webp versions thereof.

From what I understand from the docs and posts here this can be done in the following ways.

In a template on first page load of that template.
In the admin panel on upload of an image with a hook intercepting the process.

Given both a try and am not sure if this works for me or if I am doing something wrong.
Things time out or the versions expected are not created.

This more so with ImageMagick doing the encoding rather than the native image resizer.
At this stage I would also do not know how to visually output verbose log of the encoding process.

Now I guess there are following options..

1. Local variations and webp versions are connected with pages and db
Given I use ImageMagick locally to be able to pass it the options similar to here.
https://www.smashingmagazine.com/2015/06/efficient-image-resizing-with-imagemagick/
I then have all the variations for a single or multiple images.

How do I "sync" those with the db and the page I want those images and their variations to reside?
How can I import prepared variations of an image into a page while not importing all variations as a single image?
By giving the webp versions a certain filename suffix, could those also be made variations of the original image?

I found this topic but that approach creates pages and possibly loads all images as a single image?
I need to have the 1 original image and then other files as variations of that original image.

Best would be if I could drop the original and all variations and webp versions into the assets folder for that page.
Then simply do a refresh of that page in the admin panel and have a script connect the original, variations and webp versions with that page and the db.
This so that I can then work with it in the page template with the images object.
That should hold an array of let's say, all urls to all the images, all image filenames and ideally width and height values.
The description would be given manually on the single original image for example.

2. Local files are uploaded with a module
I have found http://modules.processwire.com/modules/jquery-file-upload/
Am wondering if that supports to upload 1 master image and read the variations/webp versions as such, perhaps by checking the image filename?

3.Local files are uploaded and managed with a module
I have also found https://mediamanager.kongondo.com/.
Same thing here, not sure if that supports what I am trying to get done.
Manage one original image per page and assign that variations of it before I upload it.
Ideally I like to see something like upload - image - variations and then be able to pick all variations from the corresponding assets folder - save, done.
I can now use the images object for the page in the template and work with the images array.

4.Local files and all variations are uploaded as single images for the page in question
This is simply done with the native upload for images.
All images are singular and have no variations.

Somehow getting option 1 working would be awesome.
However if I am missing a concept here or misunderstand how files and images are supposed to be handled with ProcessWire then please let me know.

At this stage I am not looking for output generating modules or solutions.
But rather something that can perhaps easily add existing images to certain ProcessWire pages.

[evan]

Probably the easiest way to add existing images and variations to PW pages would be via the API – take a look at the PageImage class docs:

<?php namespace ProcessWire;

// Bootstrap ProcessWire: https://processwire.com/docs/front-end/include/
include './index.php';

$results = $pages->find('id=1'); // Your page query

$myImage = '/myImage.jpg'; // The image

foreach ($results as $result) {

	// Add initial image
	$result->of(false);
	$result->imageField->add($myImage);
	$result->save();

	// Retrieve added image, add description
	$lastImage = $result->imageField->last();
  	$lastImage->description = "My description";
  	
	// Create variation of last image
	$variation = $lastImage->size(500, 500);
	$result->imageField->add($variation);

	// Add additional variations here, and format conversion

	$result->save();
  
}

You'll need to augment this code to accommodate an existing list of images, or multiple images per page, to match against the PW pages.

WebP (or any image format conversions, it seems) are not natively supported by PW, but it looks like some folks have figured out some solutions (inline, and more integrated) :

 

 

[happywire]

@evan Thank you for your help.

Let's assume this case.
Using the Intermediate profile.
I made a Gallery page, made a gallery-page.php file and connected it with a template file that is called "gallery-page".
Now inside gallery-page.php I simply put var_dump($page) and that returns me the "gallery-page" object.
There I can see the page object and its arrays.
For example I can see that under the ['data'] key there is a ['path'] key that has a value of "/shared/httpd/intermediatewire/htdocs/site/assets/files/1016/".

Spoiler

object(ProcessWire\Page)#188 (19) {
  ["instanceID"]=>
  int(10)
  ["id"]=>
  int(1016)
  ["name"]=>
  string(7) "gallery"
  ["path"]=>
  string(9) "/gallery/"
  ["status"]=>
  string(0) ""
  ["template"]=>
  string(12) "gallery-page"
  ["parent"]=>
  string(1) "/"
  ["numChildren"]=>
  int(0)
  ["sort"]=>
  int(0)
  ["sortfield"]=>
  string(4) "sort"
  ["created"]=>
  string(32) "2019-03-14 11:20:11 (1 week ago)"
  ["modified"]=>
  string(36) "2019-03-24 13:18:46 (23 minutes ago)"
  ["published"]=>
  string(32) "2019-03-14 11:20:11 (1 week ago)"
  ["createdUser"]=>
  string(5) "admin"
  ["modifiedUser"]=>
  string(5) "admin"
  ["isLoaded"]=>
  int(1)
  ["outputFormatting"]=>
  int(1)
  ["hooks"]=>
  array(15) {
    ["Page::addUrl"]=>
    string(59) "PagePathHistory->hookPageAddUrl() in PagePathHistory.module"
    ["Page::removeUrl"]=>
    string(62) "PagePathHistory->hookPageRemoveUrl() in PagePathHistory.module"
    ["Page::editable"]=>
    string(53) "PagePermissions->editable() in PagePermissions.module"
    ["Page::publishable"]=>
    string(56) "PagePermissions->publishable() in PagePermissions.module"
    ["Page::viewable"]=>
    string(53) "PagePermissions->viewable() in PagePermissions.module"
    ["Page::listable"]=>
    string(53) "PagePermissions->listable() in PagePermissions.module"
    ["Page::deleteable"]=>
    string(55) "PagePermissions->deleteable() in PagePermissions.module"
    ["Page::deletable"]=>
    string(55) "PagePermissions->deleteable() in PagePermissions.module"
    ["Page::trashable"]=>
    string(54) "PagePermissions->trashable() in PagePermissions.module"
    ["Page::restorable"]=>
    string(55) "PagePermissions->restorable() in PagePermissions.module"
    ["Page::addable"]=>
    string(52) "PagePermissions->addable() in PagePermissions.module"
    ["Page::moveable"]=>
    string(53) "PagePermissions->moveable() in PagePermissions.module"
    ["Page::sortable"]=>
    string(53) "PagePermissions->sortable() in PagePermissions.module"
    ["before Page::render"]=>
    string(51) "PageRender->beforeRenderPage() in PageRender.module"
    ["Page::render"]=>
    string(45) "PageRender->renderPage() in PageRender.module"
  }
  ["data"]=>
  array(2) {
    ["title"]=>
    string(7) "Gallery"
    ["images"]=>
    object(ProcessWire\Pageimages)#108 (6) {
      ["count"]=>
      int(1)
      ["page"]=>
      string(9) "/gallery/"
      ["field"]=>
      string(6) "images"
      ["url"]=>
      string(24) "/site/assets/files/1016/"
      ["path"]=>
      string(61) "/shared/httpd/intermediatewire/htdocs/site/assets/files/1016/"
      ["items"]=>
      array(1) {
        ["pia13316.jpg"]=>
        array(12) {
          ["url"]=>
          string(36) "/site/assets/files/1016/pia13316.jpg"
          ["filename"]=>
          string(73) "/shared/httpd/intermediatewire/htdocs/site/assets/files/1016/pia13316.jpg"
          ["filesize"]=>
          int(755905)
          ["description"]=>
          string(0) ""
          ["tags"]=>
          string(0) ""
          ["created"]=>
          string(19) "2019-03-24 12:39:37"
          ["modified"]=>
          string(19) "2019-03-24 12:39:37"
          ["filemtime"]=>
          string(19) "2019-03-24 12:39:37"
          ["width"]=>
          int(3545)
          ["height"]=>
          int(3600)
          ["suffix"]=>
          string(0) ""
          ["variations"]=>
          array(2) {
            [0]=>
            string(18) "pia13316.260x0.jpg"
            [1]=> // ********** THIS PART HERE SHOULD BE ADDED VIA THE API AND NOT ON PAGE LOAD IN THE ADMIN PANEL **********
            string(20) "pia13316.300x200.jpg"
          }
        }
      }
    }
  }
}

So far so good.

In the admin panel, on the Gallery page I only upload the original image (hi-res) by hand.
Hit Save and the default variation, 0x260 or 260x0, is being created, done.
Not sue why sometimes 260x0 and other times 0x260 default image variation is being created.
I don't mind since I don't want to work with these default variations.
Let's assume I/user/client does this for all the images they like to have on that page, just upload original his-res image via admin panel.

Now, external from ProcessWire, so not using image->size(), I run several image resizing/encoding/compression scripts/libs on that folder "/shared/httpd/intermediatewire/htdocs/site/assets/files/1016/".
That creates exactly the versions (variations) of the original image I like to have, with the exact commands for the image resizing/encoding/compressions scripts that I like to give.
These new versions (variations) of the original image also adhere to the ProcessWire naming scheme.
Meaning original filename DOT width x height DOT file extension (jpg, png, webp, etc), i.e. sunshine.300x200.jpg will be created of original sunshine.jpg.

Back to your example.
Let's say I want to add this image. https://www.jpl.nasa.gov/spaceimages/details.php?id=PIA13316
The original hi-res image is here original image
In gallery-page.php I simply put this given this post I found https://www.jpl.nasa.gov/spaceimages/details.php?id=PIA13316
 

<?php namespace ProcessWire;

// inside gallery-page.php

$page->of(false);
$page->images->add('https://photojournal.jpl.nasa.gov/jpeg/PIA13316.jpg');
$page->save();

That indeed uploads the original image to the page (and also creates the default (260x0 or 0x260) variation).
So this works but it is not really what I am after, albeit we are close already.

Now, given I have manually, through the admin panel, uploaded the original (hi-res) images to the pages I want and ran all image resizing/encoding/compression scripts/libs on those folder folder I am left with "variations" of the original image.
How can I, for example in ready.php now tell ProcessWire to do the following.
a) Scan the assets folders for pages with images.
b) If you find "variations" of an already uploaded original image then add those as "variations" to the page.
c) ?

What have I tried?
Back to gallery-page.php I have tried this.
Trying to add an already existing variation of an already uploaded image.

<?php namespace ProcessWire;

// gallery-page.php

$imageVariation = '/pia13316.300x200.jpg';

$page->of(false);
$page->images->add($imageVariation);
$page->save();

This results in Error: Exception: Unable to read: /pia13316.300x200.jpg (in /shared/httpd/intermediatewire/htdocs/wire/core/Pagefile.php line 162).
So ProcessWire cannot find the file.
How can I tell ProcessWire where the image for that page is please? ?
I tried putting it directly in the corresponding assets folder of the Gallery page, that did not work.
I assume, being in gallery-page.php or ready.php I do not need to bootstrap ProcessWire through include './index.php'; any more, tried and it gave me an error, so leaving that out.

Now one thing that happens with ProcessWire is that if there is an image with the variations naming scheme, filename DOT width x height DOT file extension inside any of the assets folders, ProcessWire will add that file as "variations" if that page is reloaded in the admin panel. So that is a somewhat good thing. ?

But now let's say there are quite a few pages, each have images, the external image resizing/encoding/compressions scripts/libs ran on all those assets folders and created variations.
Now in order for all those to be "linked" to ProcessWire I would have to re-visit all those pages in the admin panel, reload them and save them.
Something I do not want to do. ?

Therefore, ideally I am looking for something, ideally in ready.php, as I assume that runs on every page load, admin panel or frontend, that does scan the corresponding assets folders and adds the externally created variations.
So, in ready.php (please tell me if another location is better) I would do something like this.

https://processwire.com/talk/topic/4530-how-to-get-a-list-of-all-pages-recursiv-without-admin-pages/
https://processwire.com/talk/topic/1216-how-do-i-test-for-images/?tab=comments#comment-10877

<?php namespace ProcessWire;

// ready.php

$pageWithImages = $pages->find("images.count>0");

var_dump($pageWithImages);

// Trying to get the array 'items' returns bool(false)
var_dump($pageWithImages['items']);

// Trying to get the array 'items' as object property returns NULL
var_dump($pageWithImages->items);

So I am unfortunately stuck at trying to access the object property "items".

Spoiler

object(ProcessWire\PageArray)#196 (7) {
  ["hooks"]=>
  array(2) {
    ["PageArray::render"]=>
    string(60) "MarkupPageArray->renderPageArray() in MarkupPageArray.module"
    ["PaginatedArray::renderPager"]=>
    string(56) "MarkupPageArray->renderPager() in MarkupPageArray.module"
  }
  ["count"]=>
  int(3)
  ["items"]=>
  array(3) {
    [0]=>
    string(1) "/"
    [1]=>
    string(12) "/about/what/"
    [2]=>
    string(9) "/gallery/"
  }
  ["total"]=>
  int(3)
  ["start"]=>
  int(0)
  ["limit"]=>
  int(0)
  ["selectors"]=>
  string(27) "images.count>0, status<1024"
}

Spoiler

bool(false)

Spoiler

NULL

Given I can somehow access 'items' I would do..

foreach items
- get assets path
- get original images
- get all variations
- add variations to assets path
- save page
end foreach

Do you have an idea why I cannot var_dump 'items' ? No clue. ?

To conclude, I would be super happy about either you having "an idea" or code samples or links to tuts/forum posts/docs that would help me get this done somehow.
New to ProcessWire and trying to wrap my head round how I can integrate this image workflow with ProcessWire.
Any help much appreciated. ?
 

[Autofahrn]

@happywire just a little hint about image variations: Image variations (those files with DOT <width>x<height> DOT <ext> nomenclature) only live in filesystem not in the database. If you externally (outside of PW) remove or create additional variants in the proper location (your /site/assets/files/<ImageID>) folder, these changes are instantly reflected to any image enumeration (for example getVariations() method). There is no need to visit the admin or do any manual update after each change.

If you do a $image->size() that method only looks into the assets path of that particular image to ensure the requested variation (namely file with specified width and height nomenclature) exists. If that's not the case, it will be created on the fly. There is no big magic happening behind the curtains.

I guess its much simpler to accomplish what you intend to do: Just instruct your external script to create the variant next to the original file.

[happywire]

@Autofahrn

2 hours ago, Autofahrn said:

- these changes are instantly reflected to any image enumeration

- there is no need to visit the admin or do any manual update after each change

- just instruct your external script to create the variant next to the original file

Excellent!

I figured that once the variants have the PW variations nomenclature they are added.
However I was only ever in the admin panel and checked the "Variations" tab after making the variants.
There they would only show up after hitting Save for that page. To be expected.
So this lead me to think that the same thing goes for the frontend. My bad!!
That is why I thought I have to Save each page in order for PW to a) put those into its db and b) make them show up as "Variations".
Interesting that PW does not put the variations into its db.
I thought they have to be there to be able to be shown.

Just did a test, made variants of an image in the assets folder, var_dumped the variations for that page in the frontend and they are all there, without having to save them into the db or make PW save the page in the admin panel.
Uff!! Great to know. This should be documented somewhere. Super nice.

[Autofahrn]

25 minutes ago, happywire said:

There they would only show up after hitting Save for that page.

or simply reload... ?

25 minutes ago, happywire said:

Interesting that PW does not put the variations into its db.

Why should it? This only would add redundancy of the same information which may go out of sync for some reason.

On the downside for this filename-only implementation is, that there only may be one variant with a particular dimension. So you can't have a 300x300 variant with low and another one with high quality, for example. And if you later decide to use different scaling parameters, you'll need to delete the old variant (file) first for the new conversion to happen (see Pageimage::size() for a more detailed description).

[happywire]

Another very important find I just made that might be helpful for others as well.
There is no need to upload a webp version of an image in the admin panel if you have already uploaded the jpg version of it to that page.
As long as the webp versions of that image adhere to the variations naming, meaning filename DOT width x height DOT file extension, they will be served by ProcessWire!
That is scary but also awesome. Not sure yet.

For what I am trying to do though, that is great news.
Now I can just pull the jpg variations array, swap the file extension for webp, have my srcset array for webp, done.

Well one thing though.
With this approach there will be no "official variations" of webp versions in ProcessWire.
Meaning one will not be able to work with "variations" in the code or dump them.

Ideally it would be great if given the original filename one could add variations of different file types to the same variations array.
Something like this if you dump an $image.

$image
    info
    path
    url
    array "variations"
        jpg
            image.0x260.jpg
            image.2400x360.jpg
            ...
        webp
            image.0x260.webp
            image.2400x360.webp
            ...
        png
            image.0x260.png
            image.2400x360.png
            ...

In the admin panel, when clicking on Variations for an image, it would kind of be helpful to include an "add variation" button where then users can add variations of different file types to the original image.Perhaps with this approach also different versions of the same image, i.e. a 300x300 version with low quality and a 300x300 version with high quality could be included.

But I think this is already possible by just including one more DOT and simply have hq (high quality) and lq (low quality) in front of it.
Meaning filename DOT lq DOT width x height DOT file extension and then filename DOT hq DOT width x height DOT file extension.

Just me thinking out loud.. and more testing to be done.
?

[szabesz]

6 hours ago, happywire said:

...image adhere to the variations naming, meaning filename DOT width x height DOT file extension, they will be served by ProcessWire!
That is scary but also awesome. Not sure yet.

I have a site where I also take advantage of this. I implemented a custom image process algorithm, whereby I check the extension of the uploaded image, rename it and after that create predefined variations (using Imagick) based on the aspect ratio and/or the dimensions. I put all this into a custom module which hooks into InputfieldFile::fileAdded, something like:

$this->addHookBefore('InputfieldFile::fileAdded', $this, 'hookRenamePng', array('priority' => 10));
$this->addHookAfter('InputfieldFile::fileAdded', $this, 'hookCreateJpgClones', array('priority' => 10));

BTW, have you read this Github issue? https://github.com/processwire/processwire-issues/issues/703 it is somewhat related.

[happywire]

@szabesz Nice one, thank you.

Yes, would I have known this from the beginning only!
Getting a proper workflow done and will share here once results are somewhat solid.
Missing this in the docs.

Yes read through the issue, thx.
I agree, would opt for all image manipulation to be external in a way.
How I see it, ProcessWire is more for devs than ready to go for end users/clients.
So let devs handle images the way they like and need.
Sure ProcessWire is already doing that in a way, though it needs to be documented.
Could not agree less with this https://github.com/processwire/processwire-issues/issues/703#issuecomment-470406082
And please put somewhere in the docs "If you have files in your assets folder for a page that follow a certain naming convention ProcessWire will serve them, regardless of having a db connection or not".

https://github.com/Toutouwai/UniqueImageVariations looks interesting.
What I am after really would be a module where devs can put in the command line for image encoding/conversion/dimensions/naming and have the pure libs, like cwebp and/or ImageMagick installed/integrated  by default or at least as a module.
I like to have control over what I do with the images and no rely on a 90 or 100 setting and guess what it going on exactly in the background.
https://developers.google.com/speed/webp/docs/cwebp
https://imagemagick.org/script/command-line-tools.php
https://www.digitalocean.com/community/tutorials/how-to-create-and-serve-webp-images-to-speed-up-your-website

Who knows, getting to know ProcessWire more every day and learning PHP alongside I might be able to write such a module myself..
?

[szabesz]

@happywire I think in the foreseeable future we cannot expect Ryan to change the way ProcessWire manages images so it is up to us to implement it the way we want to. As it is an "advance topic" I do not think it will be documented in details, except if we do it ourselves. One good way of doing it is how  @bernhard wrote a great tutorial, which was happily published by Ryan in the blog: https://processwire.com/blog/posts/building-custom-admin-pages-with-process-modules/ If you have the time to do your own research regarding this issue and ask others for more info then you might want to sum it up for us to in a similar tutorial. Most of us have their own way of managing images but this should not bother you, just do it your own way and help others by publishing it you have the time.

ProcessWire pages under processwire.com/docs/ cover most what is needed for developers to get started but the blog articles often highlight even more, it's just that they are buried in there. Forum and Githubs issues also have a lot of information, but those are even more buried. However, I do not think that it will be possible to organize it all, so doing our own research is a must. Most of us perform custom goggle searches in order to dig into all these sources. The goos thing is that under the hood of ProcessWire things rarely change, so most things you are going to read in these sources are still relevant and work. However, the there have been major upgrade to images field, just search like: https://www.google.hu/search?q=image+site%3Aprocesswire.com

For example:

"In the new images field, they are now 260 pixels in the smaller dimension, scaled to 130 pixels (for HiDPI/retina support). The actual size and HiDPI support can be adjusted in config.php settings in $config->adminThumbOptions, though we think most will likely want to keep the default settings." https://processwire.com/blog/posts/major-images-field-upgrade/

So using gridSize it is possible to set the dimensions: https://github.com/processwire/processwire/blob/649d2569abc10bac43e98ca98db474dd3d6603ca/wire/config.php#L636

I do not exactly know how it should work though. When I set 'gridSize' => 100 and 'scale' => 1.0 then "1.0=force non-hidpi" should kick in, and in my understanding with this setting I should get images no bigger than 100x100 pixels. However, with this setting I get 200x200 pixels or less. Maybe it's me who does not understand the comment in the code. Also, there is this part in the comment: "1=allow hidpi, 0.5=always hidpi" which does not make any difference at all. In my case not matter what is set to scale, the admin thumb will always be twice the value of gridSize.

3 hours ago, happywire said:

Who knows, getting to know ProcessWire more every day and learning PHP alongside I might be able to write such a module myself.

It's not that hard, just sart by reading https://processwire.com/docs/modules/ and Bernhard's article.