horst

Successfully tested PIA with PW 3.0 and set the version to 1.0.0 stable now.

Just to clarify: the further processing does not have to be done from within Processwire, nor does it to be done by PHP. You can use PW to only collect the data, maybe in the example Sqlite storage handler. To further process the data records, you can use every software that can communicates to a Sqlite DBfile. If you need different fields in a Sqlite table, just copy the WireQueueSqlite3 module, - change the file- and class name to be no duplicates of the distributed storage handler, - and, at least, change the fielddefinitions for the DB table. But you can change every thing you want. If you want or need to use other storage types and setup a new WireQueueSomething module, - please feel free to submit / commit it.

Wire Queue Wire Queue is a module that allows easy creation and usage of Queues in ProcessWire. It is based upon a basic parent module (WireQueue) that should have one or multiple StorageHandler modules installed too. This beta release contains a simple plain text storage module, WireQueueTextfile, and a simple Sqlite3-DB storage module, WireQueueSqlite3. The base module creates the needed: FIELDS (wire_queue_type, wire_queue_state, wire_queue_storage_options) TEMPLATES (wire-queue-container, wire-queue-storage, wire-queue-tools) PAGES (wire-queues = container for all Queuepages, wire-queue-storages = container for StoragetypePages) ROLE (wire-queue-admin) Each storage module creates one page under wire-queue-storages. New Queues can be created in the backend by adding a new page under "Wire Queues". After creation one have to select a Storage type from availables list and publish the page. After that, there are some buttons available to start / pause / and close the queue. Putting and getting data to and from the queue is done via API calls. First you have to get the page that holds the queue object. // get and validate the queue handle if($queue = $pages->get(SELECTOR_TO_DESIRED_PAGE)->wireQueue()) { // wrap your data into an array and pass it into the queue $success = $queue->addItem($data); ... } // get and validate the queue handle if($queue = $pages->get(SELECTOR_TO_DESIRED_PAGE)->wireQueue()) { $data = $queue->getItem(); ... } Thats basically all what you want from a queue. Following are a few conveniences: $queue->getPage()->title gives you the title of the queue, ($queue->getPage() returns the page object) $queue->className() shows the StorageType of the queue $queue->getState() and $queue->getStateStr() returns the current state of a queue: 1 = new / empty 2 = enabled / running 3 = paused 4 = closed / archived $queue->itemCount() gives the total number of all pending items in the queue Here is code that gives an overview of all Queues in the system: $container = $pages->get('template=wire-queue-container'); $bodyContent = "<h1>$container->title</h1>"; $bodyContent .= "<p>There are currently {$container->numChildren} Queues defined:</p>"; $bodyContent .= "<ul>"; foreach($container->children() as $p) { if(! ($queue = $p->wireQueue())) continue; $bodyContent .= "<li>{$queue->getPage()->title}<ul>"; if(!$queue->ready2use()) { $bodyContent .= "<li>{$queue->className}</li>"; $bodyContent .= "<li>This Storagetype is not ready to use! The System seems not to provide all requirements.</li>"; $bodyContent .= "</ul></li>"; continue; } $bodyContent .= "<li>{$queue->className}</li>"; $bodyContent .= "<li>{$queue->getStateStr()} ({$queue->getState()})</li>"; $bodyContent .= "<li>Currently are {$queue->itemCount()} items pending!</li>"; $bodyContent .= "</ul></li>"; } $bodyContent .= "</ul>"; Following is a screenshot of the backend The module is available in the modules directory: http://modules.processwire.com/modules/wire-queue/ Or you get it from Github: https://github.com/horst-n/WireQueue. . . . The Sqlite3 storage handler not only let you push and pull data items to and from it, it also can keep track of the current state of a record. If you use multiple / different workers for pulling and processing the data, you can store an ID for them too. This is how the DB Table looks like: The Wire Queue Sqlite3 storage handler provides the methods addItem($arrayData) // same as WireQueueTextfile getItem($worker = null) // same as WireQueueTextfile, (but the textfile storage cannot support $worker!) updateItemState($id, $state, $worker = null) // this you can use for further processing, if you want . addItem($arrayData) $arrayData is a, maybe associative, array containing all your data for one record. The method stores it under the next free id under data in the Sqlite-DB-file and sets the state to 0. The field worker is empty for new added records. Following is a basic working example for pushing data records into a queue: // you have created a queue in PW, the ID of the page is 1420 for example // here is some skeleton code for an importer that uses this queue // get and validate the queue handle if(! ($queue = $pages->get('id=1420')->wireQueue())) exit(); // we could not get the Queuepage // now start to scan / read data for your imports, wrap each record into an array and put it into the queue foreach($pages->find(YOURSELECTOR) as $p) { $data = array($p->field1, $p->field2); $queue->addItem($data); } . . getItem($worker = null) $worker is an integer that you can define to suite your needs. If you don't use or don't want identify multiple workers, just ommit it. The method pulls one pending record from the queue, changes the state from 0 to 1, and returns an associative array with the keys id and data. array('id' => integer, 'data' => array) You will need the id if you further want to use the queue to keep track of processing steps. You must pull your stored $data from $array['data'] and use the id for further storing the state. . updateItemState($id, $state, $worker = null) $id identifys the record a worker has processed, for $state you can define and use every integer you like, but not 0 or 1. If you also want to store altered data, and not only different states, you can use updateItem($id, $state, $worker = null, $data = null) instead of updateItemState(). . Here is a working example with a bit pseudo code how workers can get a pending record to process it further and store back the result of the process: // you have created a queue in PW, the ID of the page is 1420 for example // here is some skeleton code for an importer that uses this queue // get and validate the queue handle if(! ($queue = $pages->get('id=1420')->wireQueue())) exit(); // we could not get the Queuepage // we do not use different workers here in that example, so we do not pass an worker id here $tmp = $queue->getItem(); // get a record from the queue $record_id = $tmp['id']; // store the record id $data = $tmp['data']; // get the data array // process the $data ... // and get the result of the process, in this example true or false $result = processMyRecord($data); // as new records have a state = 0, fetched records have a state = 1, // you may define a state of 2 for successful processed records, and higher ones for failed records $state = true === $result ? 2 : 3; $queue->updateItemState($record_id, $state); . . getItem4FurtherProcessing($state, $worker = null) The $state you pass to the method is the state you want get the record for. If there is one pending, its state will be set +1 and the id and data is passed back to you in an associative array: array('id' => integer, 'data' => array). --------- Here is a pseudo code example how (multiple) worker scripts may batch process queue records with the sqlite storage handler // on the server in this example, everyscript will timeout / die after 180 seconds // we start a timer $time = Debug::timer(); // we use different instances of workers that pull and process records from the queue, // so additionally to the processings states, we also want to store the worker ids $worker = 2000; // now start to process items by pulling one after the other from the queue while(150 > Debug::timer($time)) { $tmp = $queue->getItem($worker); // get a record from the queue if(!$tmp) continue; // maybe currently are no pending records available $record_id = $tmp['id']; // store the record id $data = $tmp['data']; // get the data array $result = processMyRecord($data); // process the data and get back the result $state = true === $result ? 2 : 3; // define an integer for the new state, according to the result of processing the data $queue->updateItemState($record_id, $state, $worker); } // we are close to the timeout for the script, so finish it with calling it itself to get a fresh run $session->redirect('./');

It is an awesome video, so, also with music. Have had a nice looking. Thanks man! Will also try the module in the next time.

I assume this will not work with a simple proxy as it is not that easy to trick google / youtube.

I'm on mobile and not answering directly to your question, but want mention that instead of the forums search box, you should use google with site:processwire.com/talk as additional search tag.

I'm really happy with NuSpheres PhpED on Windows! If you create an project it also knows more of it as you or I ever will, It has all sorts of Tooltips and Popuplists with classes, properties, functions, that also contain links to the files wheere they live, etc. It has a pane called Code Navigator which shows this list(s) of the current opened file with synchronized focus, means: it highlights that method where your cursor in the editor pane currently is. It has an own live debugger that can be embedded as a php extension in apaches webserver, also on remote machines, that can be controlled with little browser extensions or else. It supports remote editing via sftp or ftpes, ssh, terminals, it has a DB client that fastly displays information of mysql or sqlite databases and it has much more what I currently have not tested. Its memory consumption is not low but around 0.5 GB when I have opened 1-3 projects simultaneous. Downside is, it is not free. But it is worth its costs.

What do you mean with "this"?

Very nice! - ähm but unfortunately not my colors. Ough. Please can we have a less colorful theme or a theme switcher with one less colorful, once all other is running? please, please, please, ...

Thanks Testing with the plain text storage is ok in that context. But to use it for production, I would suggest you wait for the SQlite storage, because with it you will be able to store different states of an email sending job: new/added to the queue pulled by a worker for execution successfully sent by a worker, or not successfully sent At least it should store sent (yes / no) and track (max) tries.

Yes, it also is an API, and it is not free, only the first 500 image (variations!) per month. So, for me that's exactly the point why I hasitated to say I'm interested in working on this. As it is no problem for me to name the exact and best points where and how to hook into PW's image procesing chain for this, I'm not comfortable with the part that needs to be build that, for example, observes the amount of total submissions in a given paid or free plan per month, or should be there to detect if something went wrong, (e.g. a script goes wild) you can reach a 500 limit in under one hour or shorter. (500.000 variations = $1065.50 and max 5.000.000 variations = $10065.50) For me, the integration in the image processing chain looks easy, but I assume that this will take only 20%-30% of the whole module. 70%-80% will be taken by the required "overhead". @teppo: Regarding exec() AFAIK for imageprocessing without exec(), there is the bundled GD-lib and the Imagick PHP extension. And than a lot of other solutions that all rely on CLI processing. So, it doesn't to be not uncommon to use it for image processing . It needs to be integrated into the right server configuration, I believe. On the servers I've spoken from, exec is disabled if php is running as www-run or something that like, what it is as apache-module-handler. When switching the PHP handler to a fast-cgi version, PHP is only allowed to be invoked and running in the context of the user / owner of the account, what also is restricted in filesystem to only his own directory, etc. Are there more aspects to put attention to?

@matjazp: Thanks for clarifying. I didn't know that there is no further maintainance of the minimize service. I also know that we would need CLI processing via PHPs exec(), and that this isn't supported on (some) shared hosts. But also the bullet proofed method would be to process variations and minifying locally. A less bullet proofed method would be to use "a sort of" remote service via http on the own host, if possible, because you also have full control and the http protocoll overhead is small and results in appropriate fast speed. The last one in the hirarchy is to use a foreign remote service, as you have no control and a lot of overhead. I'm interested in how many shared hosts have the ability to choose php fastcgi handler for subdirectories (via htaccess), where you than have support for exec(). I'm on this sort of shared servers myself and use PHP as apache_module with disabled exec() as the (main) PHP handler for the whole site. But I can define fastcgi handler(s) for A) a specific subdirectory or B) for a specific defined file extension. This way I'm able to run my PW site within a fast apache_module PHP handler and pass the final step of optimization via http to a cgihandler script. This is really fast, as I do not have to sent and retrieve files, I also do not need to resolve a DNS for the connection, as it is on the same (local) host. If there is the possibilty for others too, I'm happy to share my solution.

@OllieMackJames: interesting thing. There are some thoughts that comes to my mind directly in regard for an understanding how imageprocessing works in PW: there is already a service integrated for minifying images (jpeg and png): ProcessImageMinimize.pw it is not possible to further manipulate minimized images, as I assume you meant with other modules (thumbnails etc) should work with the minimized images. This will lead in very very bad quality and sometimes corrupt looking images. It is mandatory to create every variation from the original, un-minified image, or at least from an image that was not compressed and has 100% quality, what is not the same like the original image. So, it is mandatory to send every image variation as last step of the processing chain to the minimize service (or a local program) Compression of the original image on upload is not possible if you want to create any variations from it. (see above!) Hope this helps a bit. PS: There was also a module that handles local optimizing of jpegs (OptimJpeg). Maybe we should update this to also support png, by asking the author or by forking it?

Hi and welcome @lokomotivan. Regarding the security question in the register process of the IP.board software, I will point our forums admin to that. I'm not aware that these questions are belong to the registration process, maybe this was sometimes added after I have registered here. But one thing is sure, I personally would have entered it also in the correct spelling multiple times!

It's on the way: https://github.com/ryancramerdesign/ProcessWire/pull/1646

@Lostkobrakai: Good point! I need to bookmark this! Settings in max dimensions break the chain, we lost all EXIF and other markers in the "original" image. Would be good if we could write it back in this special case.

@HannaP: yes, that's the normal behave with autoRotated enabled. When disabled, every image would be generated and displayed with uncorrected / wrong orientation. If you have autoRotation on, all derived image variations gets corrected in imagesizer, if: the original image itself contains the right entry for the Orientation in Exif. the setting for auto rotation gets not altered somewhere on the way until it reaches the imagesizer.

I have a test version of imagesizer now where the chosen algorithm depends on available memory. If there is enough memory, it uses USM, if there is less memory, it levels down to use the other algorithm, and if there also isn't enough memory available for this, sharpening is completly skipped. After some testing I will commit this to github.

@ottogal: I have found out that I have tricked me myself when changing settings for imageSizerOptions in site/config.php for testing. I also have installed Pia, Pageimage Assistant module, what overrides some settings of site/config.php. So, those settings should be applied in the modules config page in that case. As a result, I decided to removed this config settings from the module. I will upload a new version to github after I can reach there servers again. (currently unreachable from here). Maybe you have installed Pia too?

So true!

Is this a bit nitpicking ? I meant what imagesizer can do when completly acting automated. Having a way to correct those wrong original images would be good. I'm not in this RTE usage, but I saw in imagesizer that there is already a setting for rotate that has nothing to do with autoRotation. Also, when I remember correct, wasn't there within the 2.6-dev branch already the ability to rotate images? Will this come back?

@HannaP: already done. You sent me two images, whereas the original dog.jpg is displayed correct, and the original tree.jpg is displayed 90 degrees rotated. Checking the settings in exif-Orientation outputs for both files 8, what indicates that they both need to be rotated by 90 degrees, what the imagesizer does. As result, you afterwards has one image displayed correct (the tree.jpg) and the dog.jpg is displayed wrong. But this depends on the wrong stored value in the file. This image looks like it was manipulated after it came from the camera, because it contains a logo. I assume the software that merged the logo into it has corrected the orientation but hasn't updated the exif settings. That's simply it. There is nothing that imagesizer can do here.

@Rick: If an author modifies a page, i would assume that he inspect the result on the frontend, before publishing a page. Exactly with this inspection, he raises the first page load and all imagevariations get created. And as an author or admin user, he has / should take this time to wait. After that, the first regular visitor gets presented the cached variations. Hhm, or not?

It output is 6. Thats fine and indicates that the image should be rotated by 270 degrees. The imagesizer does handle this correct. WHat needs to be inspected is the way along through config settings merging by other modules / classes / instances that read / modify and merge those settings before they finally reach the imagesizer. (e.g. Pageimage, ProcessPageEditImageSelect, etc) Thanks, i will look into this, maybe tomorrow, and report back.

Thanks for testing Spica! All what you have found out belongs to sharpening, and indeed there are two instances where currently no memory-check is invoked. Those both (not several!) instances belong to the unsharpMask function, what is enabled by default to create the best possible result for your images. So, as we have to find a balance between fast image processing and to avoid going out of memory, every user should take a bit care with his sites. If you are low on memory, you definetly should limit the max dimensions for upload images. Besides that, you uncovered that we need to describe more features we have under the hood with image processing. With a slide change in the $config settings we can disable the usage of the UnsharpMask for sharpening and switch to another algorithm, what needs much much lesser memory. This unsharp algorythm was there earlier, but we decided to switch to the visually better working USM by default in times where 256M seems to be a common value for memory usage. Conclusion: we will implement memory checking for those two instances currently not covered too, and we will provide flags within the $config settings that enables users to simply switch between sharpening algorythms in systems with low memory. Once again, thanks for your tests and reports, it is much appreciated.