-
Posts
4,077 -
Joined
-
Last visited
-
Days Won
87
Everything posted by horst
-
Thanks Testing with the plain text storage is ok in that context. But to use it for production, I would suggest you wait for the SQlite storage, because with it you will be able to store different states of an email sending job: new/added to the queue pulled by a worker for execution successfully sent by a worker, or not successfully sent At least it should store sent (yes / no) and track (max) tries.
-
Crowdfunded Tinypng Integration Module
horst replied to OllieMackJames's topic in Module/Plugin Development
Yes, it also is an API, and it is not free, only the first 500 image (variations!) per month. So, for me that's exactly the point why I hasitated to say I'm interested in working on this. As it is no problem for me to name the exact and best points where and how to hook into PW's image procesing chain for this, I'm not comfortable with the part that needs to be build that, for example, observes the amount of total submissions in a given paid or free plan per month, or should be there to detect if something went wrong, (e.g. a script goes wild) you can reach a 500 limit in under one hour or shorter. (500.000 variations = $1065.50 and max 5.000.000 variations = $10065.50) For me, the integration in the image processing chain looks easy, but I assume that this will take only 20%-30% of the whole module. 70%-80% will be taken by the required "overhead". @teppo: Regarding exec() AFAIK for imageprocessing without exec(), there is the bundled GD-lib and the Imagick PHP extension. And than a lot of other solutions that all rely on CLI processing. So, it doesn't to be not uncommon to use it for image processing . It needs to be integrated into the right server configuration, I believe. On the servers I've spoken from, exec is disabled if php is running as www-run or something that like, what it is as apache-module-handler. When switching the PHP handler to a fast-cgi version, PHP is only allowed to be invoked and running in the context of the user / owner of the account, what also is restricted in filesystem to only his own directory, etc. Are there more aspects to put attention to? -
Crowdfunded Tinypng Integration Module
horst replied to OllieMackJames's topic in Module/Plugin Development
@matjazp: Thanks for clarifying. I didn't know that there is no further maintainance of the minimize service. I also know that we would need CLI processing via PHPs exec(), and that this isn't supported on (some) shared hosts. But also the bullet proofed method would be to process variations and minifying locally. A less bullet proofed method would be to use "a sort of" remote service via http on the own host, if possible, because you also have full control and the http protocoll overhead is small and results in appropriate fast speed. The last one in the hirarchy is to use a foreign remote service, as you have no control and a lot of overhead. I'm interested in how many shared hosts have the ability to choose php fastcgi handler for subdirectories (via htaccess), where you than have support for exec(). I'm on this sort of shared servers myself and use PHP as apache_module with disabled exec() as the (main) PHP handler for the whole site. But I can define fastcgi handler(s) for A) a specific subdirectory or B) for a specific defined file extension. This way I'm able to run my PW site within a fast apache_module PHP handler and pass the final step of optimization via http to a cgihandler script. This is really fast, as I do not have to sent and retrieve files, I also do not need to resolve a DNS for the connection, as it is on the same (local) host. If there is the possibilty for others too, I'm happy to share my solution. -
Crowdfunded Tinypng Integration Module
horst replied to OllieMackJames's topic in Module/Plugin Development
@OllieMackJames: interesting thing. There are some thoughts that comes to my mind directly in regard for an understanding how imageprocessing works in PW: there is already a service integrated for minifying images (jpeg and png): ProcessImageMinimize.pw it is not possible to further manipulate minimized images, as I assume you meant with other modules (thumbnails etc) should work with the minimized images. This will lead in very very bad quality and sometimes corrupt looking images. It is mandatory to create every variation from the original, un-minified image, or at least from an image that was not compressed and has 100% quality, what is not the same like the original image. So, it is mandatory to send every image variation as last step of the processing chain to the minimize service (or a local program) Compression of the original image on upload is not possible if you want to create any variations from it. (see above!) Hope this helps a bit. PS: There was also a module that handles local optimizing of jpegs (OptimJpeg). Maybe we should update this to also support png, by asking the author or by forking it? -
Hi and welcome @lokomotivan. Regarding the security question in the register process of the IP.board software, I will point our forums admin to that. I'm not aware that these questions are belong to the registration process, maybe this was sometimes added after I have registered here. But one thing is sure, I personally would have entered it also in the correct spelling multiple times!
-
It's on the way: https://github.com/ryancramerdesign/ProcessWire/pull/1646
-
@Lostkobrakai: Good point! I need to bookmark this! Settings in max dimensions break the chain, we lost all EXIF and other markers in the "original" image. Would be good if we could write it back in this special case.
-
@HannaP: yes, that's the normal behave with autoRotated enabled. When disabled, every image would be generated and displayed with uncorrected / wrong orientation. If you have autoRotation on, all derived image variations gets corrected in imagesizer, if: the original image itself contains the right entry for the Orientation in Exif. the setting for auto rotation gets not altered somewhere on the way until it reaches the imagesizer.
-
I have a test version of imagesizer now where the chosen algorithm depends on available memory. If there is enough memory, it uses USM, if there is less memory, it levels down to use the other algorithm, and if there also isn't enough memory available for this, sharpening is completly skipped. After some testing I will commit this to github.
-
@ottogal: I have found out that I have tricked me myself when changing settings for imageSizerOptions in site/config.php for testing. I also have installed Pia, Pageimage Assistant module, what overrides some settings of site/config.php. So, those settings should be applied in the modules config page in that case. As a result, I decided to removed this config settings from the module. I will upload a new version to github after I can reach there servers again. (currently unreachable from here). Maybe you have installed Pia too?
-
So true!
-
Is this a bit nitpicking ? I meant what imagesizer can do when completly acting automated. Having a way to correct those wrong original images would be good. I'm not in this RTE usage, but I saw in imagesizer that there is already a setting for rotate that has nothing to do with autoRotation. Also, when I remember correct, wasn't there within the 2.6-dev branch already the ability to rotate images? Will this come back?
-
@HannaP: already done. You sent me two images, whereas the original dog.jpg is displayed correct, and the original tree.jpg is displayed 90 degrees rotated. Checking the settings in exif-Orientation outputs for both files 8, what indicates that they both need to be rotated by 90 degrees, what the imagesizer does. As result, you afterwards has one image displayed correct (the tree.jpg) and the dog.jpg is displayed wrong. But this depends on the wrong stored value in the file. This image looks like it was manipulated after it came from the camera, because it contains a logo. I assume the software that merged the logo into it has corrected the orientation but hasn't updated the exif settings. That's simply it. There is nothing that imagesizer can do here.
-
@Rick: If an author modifies a page, i would assume that he inspect the result on the frontend, before publishing a page. Exactly with this inspection, he raises the first page load and all imagevariations get created. And as an author or admin user, he has / should take this time to wait. After that, the first regular visitor gets presented the cached variations. Hhm, or not?
-
It output is 6. Thats fine and indicates that the image should be rotated by 270 degrees. The imagesizer does handle this correct. WHat needs to be inspected is the way along through config settings merging by other modules / classes / instances that read / modify and merge those settings before they finally reach the imagesizer. (e.g. Pageimage, ProcessPageEditImageSelect, etc) Thanks, i will look into this, maybe tomorrow, and report back.
-
Thanks for testing Spica! All what you have found out belongs to sharpening, and indeed there are two instances where currently no memory-check is invoked. Those both (not several!) instances belong to the unsharpMask function, what is enabled by default to create the best possible result for your images. So, as we have to find a balance between fast image processing and to avoid going out of memory, every user should take a bit care with his sites. If you are low on memory, you definetly should limit the max dimensions for upload images. Besides that, you uncovered that we need to describe more features we have under the hood with image processing. With a slide change in the $config settings we can disable the usage of the UnsharpMask for sharpening and switch to another algorithm, what needs much much lesser memory. This unsharp algorythm was there earlier, but we decided to switch to the visually better working USM by default in times where 256M seems to be a common value for memory usage. Conclusion: we will implement memory checking for those two instances currently not covered too, and we will provide flags within the $config settings that enables users to simply switch between sharpening algorythms in systems with low memory. Once again, thanks for your tests and reports, it is much appreciated.
-
Oh, sorry there was a bug in the function. I have corrected it above. But maybe we should use a more thouroughful checking function: function checkOrientation($filename) { $out = array('file_exists' => file_exists($filename)); $out['exif_read_data'] = function_exists('exif_read_data') ? 'TRUE' : 'FALSE'; $exif = function_exists('exif_read_data') ? @exif_read_data($filename, 'IFD0') : null; $out['exif'] = (is_array($exif) ? 'TRUE' : 'FALSE'); $out['exif["Orientation"]'] = (isset($exif['Orientation']) ? 'TRUE' : 'FALSE'); $out['Orientation'] = htmlentities(strval($exif['Orientation'])); $ret = ''; foreach($out as $k => $v) $ret .= "<li>$k = $v</li>"; return "<ul>$ret</ul>"; } $image = $page->images->get('name=problematic.jpg'); $result = checkOrientation($image->filename); // flush the $result to the browser, e.g. with echo, // or add it to your delayed output buffer, if you uses one
-
Indeed! What you describe here is, that in the upload process the settings seems to be used toggled to its opposite. I will check this by myself in the next days and report back here. Thank you for your help.
-
@HannaP: Hi, this is not logical to me. I need to clarify what you mean with "They are ok in large version": Do you mean the original image or a resized variation? --- Regarding exif-tag values: the PHP function exif_read_data() can extract the IFD0 marker from Exif where it looks for the value in key 'Orientation'. Orientation only holds an integer in the range of 1 - 8, what not only represents the orientation but also if it is flipped vertical or horizontal. But neither 'bottom-left' nor 'Rotate 270 CW' will be stored there I assume. This sounds like the interpreted integer by two different softwares. Can you confirm this? If you want to check the direct return from the function, use this to test, and if it is not an integer in the range 1-8, please report back and sent me such an image. function checkOrientation($filename) { if(!function_exists('exif_read_data')) return false; $exif = @exif_read_data($filename, 'IFD0'); if(!is_array($exif) || !isset($exif['Orientation'])) return false; return $exif['Orientation']; } var_dump(checkOrientation($page->images->get('name=problematic.jpg')->filename));
-
I fully agree. From a logical point of view, it is simply not possible, that the imagesizer does handle variation creations different. The imagesizer doesn't know about the disered target. It must have to do with something other. As I have posted above, the ONLY difference between admin-thumbnails and other rezised variations is the default configuration. Therefore it is important that you doublecheck that all settings mentioned above ($config->imageSizerOptions => 'autoRotation' AND $config->adminThumbOptions=> 'autoRotation') are set identical. (and without typos!) It would be really helpful if you can do a test as follows Copy both of these config settings into the site/config.php, if not already there. Set both autoRotation settings to false Somewhere in a template file get you one of the problematic images, $page->images->get("name=problematic.jpg") and run an image call like this: $image = $page->images->get("name=problematic.jpg"); if($image) { $image->removeVariations(); // remove all variations, including AdminThumb $image = $image->width(intval($image->width / 2)); // create a new, halfsized variation echo "<img src='{$image->url}' />"; } You now should see a halfsized image, not corrected in rotation. Now go into the backend edit page, where PW will create a new adminthumb. The adminthumb also should be displayed without autocorrection. If you click onto it to view the original image, this also isn't autocorrected. All three images should have the identical orientation. Ok, step 2: set autoRotation for both settings to true call the page with the code from above with the same problematic image and report back which image variation has what orientation:frontend, halfsized is in right orientation: yes [ ] - no [ ] backend Original is in right orientation: yes [ ] - no [ ] backend Thumbnal is in right orientation: yes [ ] - no [ ] And please doublecheck the config settings before testing. If this will not show matching results, we have to look into Pageimage, what is the instance just before imagesizer. But before that, I want to be sure that there is no misconfiguration.
-
As sharpening is the last manipulation step, we can try to optimize the imagesizer in that regard that we try to free memory for one image instance before we pass the $thumb-instance to the sharpening method. If you like, you can first backup the imagesizer.php and then rearrange this part // optionally apply sharpening to the final thumb if($this->sharpening && $this->sharpening != 'none') { // @horst if(IMAGETYPE_PNG != $this->imageType || ! $this->hasAlphaChannel()) { // is needed for the USM sharpening function to calculate the best sharpening params $this->usmValue = $this->calculateUSMfactor($targetWidth, $targetHeight); $thumb = $this->imSharpen($thumb, $this->sharpening); } } // write to file $result = false; switch($this->imageType) { case IMAGETYPE_GIF: // correct gamma from linearized 1.0 back to 2.0 $this->gammaCorrection($thumb, false); $result = imagegif($thumb, $dest); break; case IMAGETYPE_PNG: if(!$this->hasAlphaChannel()) $this->gammaCorrection($thumb, false); // always use highest compression level for PNG (9) per @horst $result = imagepng($thumb, $dest, 9); break; case IMAGETYPE_JPEG: // correct gamma from linearized 1.0 back to 2.0 $this->gammaCorrection($thumb, false); $result = imagejpeg($thumb, $dest, $this->quality); break; } if(isset($image) && is_resource($image)) @imagedestroy($image); // @horst if(isset($thumb) && is_resource($thumb)) @imagedestroy($thumb); if(isset($thumb2) && is_resource($thumb2)) @imagedestroy($thumb2); if(isset($image)) $image = null; if(isset($thumb)) $thumb = null; if(isset($thumb2)) $thumb2 = null; to become this one: if(isset($image) && is_resource($image)) @imagedestroy($image); if(isset($thumb2) && is_resource($thumb2)) @imagedestroy($thumb2); if(isset($image)) $image = null; if(isset($thumb2)) $thumb2 = null; // optionally apply sharpening to the final thumb if($this->sharpening && $this->sharpening != 'none') { // @horst if(IMAGETYPE_PNG != $this->imageType || ! $this->hasAlphaChannel()) { // is needed for the USM sharpening function to calculate the best sharpening params $this->usmValue = $this->calculateUSMfactor($targetWidth, $targetHeight); $thumb = $this->imSharpen($thumb, $this->sharpening); } } // write to file $result = false; switch($this->imageType) { case IMAGETYPE_GIF: // correct gamma from linearized 1.0 back to 2.0 $this->gammaCorrection($thumb, false); $result = imagegif($thumb, $dest); break; case IMAGETYPE_PNG: if(!$this->hasAlphaChannel()) $this->gammaCorrection($thumb, false); // always use highest compression level for PNG (9) per @horst $result = imagepng($thumb, $dest, 9); break; case IMAGETYPE_JPEG: // correct gamma from linearized 1.0 back to 2.0 $this->gammaCorrection($thumb, false); $result = imagejpeg($thumb, $dest, $this->quality); break; } if(isset($thumb) && is_resource($thumb)) @imagedestroy($thumb); if(isset($thumb)) $thumb = null; We only have to move 4 lines that try to release gd image handles above the call to the sharpening method.
-
Hi Spica, thanks for the report. With todays large images from smartphones, 128MB is to small for them to be processed in a good manner. The line 1655 is image sharpening! There is no extra checking for memorylimit in this function. Please can you test the same page / image with the error and than add this additional option to the options array of the api call: $image->size($width, $height, array('sharpening' => 'none')); or alternatively, set the sharpening to 'none' in the site/config.php $config->imageSizerOptions = array( 'upscaling' => true, // upscale if necessary to reach target size? 'cropping' => true, // crop if necessary to reach target size? 'autoRotation' => true, // automatically correct orientation? 'sharpening' => 'none', // sharpening: none | soft | medium | strong 'quality' => 90, // quality: 1-100 where higher is better but bigger 'hidpiQuality' => 60, // Same as above quality setting, but specific to hidpi images 'defaultGamma' => 2.0, // defaultGamma: 0.5 to 4.0 or -1 to disable gamma correction (default=2.0) );
-
HermodBB - Basic BB/Forum/Comments module
horst replied to GuruMeditation's topic in Module/Plugin Development
For me, this looks good. If this is a module and it creates all the required fields and templates and handles data / pages, what is missing? This is a lot! If you have the possibility to upload this into a GitHub repo, this would be a good start to test, if someone will laugh, I don't think so. But maybe others start to use it and contribute a bit here and a piece there. Who knows? -
Hi @Spica, please have a look into ImageSizer. PW already calculates dimensions and compare the needed amount of ram with available ram _before_ opening an image or _before_ creating another instance of an already loaded image: https://github.com/ryancramerdesign/ProcessWire/blob/master/wire/core/ImageSizer.php#L412 https://github.com/ryancramerdesign/ProcessWire/blob/master/wire/core/ImageSizer.php#L1913 Only thing what comes to my mind is to increase memory, use smaller images or upload only one after the other and not a lot in parallel. Have you tried to upload multiple images as zip-archive? ------------------------------------------------------------------------------------- You can test and call the method from the outsite of imagesizer too, if you like: $info = getimagesize($image->filename); $sourceDimensions = array($info[0], $info[1], $info['channels']); $result = ImageSizer::checkMemoryForImage($sourceDimensions); var_dump($result); EDIT: the only thing what maybe happen in your system is that PHP isn't able to read a setting for max_mem, please have a look here to this line: https://github.com/ryancramerdesign/ProcessWire/blob/master/wire/core/ImageSizer.php#L1934 If you use the above code for a test, and var_dump($result); will show null, PHP isn't able to read a seting for max_mem or there isn't one set. In this case, please also refer to output from phpinfo().
-
The Imagesizer detects the settings of the Exif orientation tag and autocorrects this where needed, if it is enabled. The setting to enable / disable this for all variations can be set in $config->imageSizerOptions "autoRotation". The default is true: $config->imageSizerOptions = array( 'upscaling' => true, // upscale if necessary to reach target size? 'cropping' => true, // crop if necessary to reach target size? 'autoRotation' => true, // automatically correct orientation? 'sharpening' => 'soft', // sharpening: none | soft | medium | strong 'quality' => 90, // quality: 1-100 where higher is better but bigger 'hidpiQuality' => 60, // Same as above quality setting, but specific to hidpi images 'defaultGamma' => 2.0, // defaultGamma: 0.5 to 4.0 or -1 to disable gamma correction (default=2.0) ); There is an extra setting for this for AdminThumbs, default is true, too: $config->adminThumbOptions = array( 'width' => 0, // max width of admin thumbnail or 0 for proportional to height 'height' => 100, // max height of admin thumbnail or 0 for proportional to width 'scale' => 1, // admin thumb scale (1=auto detect, 0.5=always hidpi, 1.0=force non-hidpi) 'upscaling' => false, 'cropping' => true, 'autoRotation' => true, // automatically correct orientation? 'sharpening' => 'soft', // sharpening: none | soft | medium | strong 'quality' => 90, 'suffix' => '', ); Please ensure that you have autoRotation set to true for both use cases.