Kiwi Chris

From recent discussions around the future of Processwire, I suspect my use case might be a bit different to others, as rather than using it in the website 'builder' type scenario, I'm using it more as a database management system, where structured data is critical, and quite often 'pages' will never be visualised via a template, and if they are, the template file is effectively a 'report'. In my case I'm tending to build sites where the admin IS the site, and there's often little or no front-end. I've wondered whether I'd be better to learn some framework like Yii or ASP.Net that have CRUD code generators that can generate data entry forms for SQL tables, but I like Processwire's admin, and the permissions system is something I rely on a lot. I've started working on a membership app, that will have quite a few users, and I've immediately run into a problem. I want admin users to be able to store first name, last name, address etc for members, but it's quite conceivable with a large enough list that it's possible to have more than one say 'John Smith', living at different addresses. I know how to write a hook to put an arbitrary auto-incrementing number in the title field (which will also populate the name field), but this seems to me to break the DRY principle. There's already a page ID, and if I'm effectively populating two other indexed database fields with an arbitrary number so that I can uniquely identify records and keep Processwire happy, it doesn't feel quite right. In straight SQL I can use an auto-increment numeric surrogate primary key and create a constraint on multiple columns so I could have as many 'John Smiths' as I like as long as they reside at different addresses, but I'm not sure how I'd go about this in Processwire, as each field resides in its own table. Perhaps the new Combo Profield @ryan recently released might be have potential for this, as all the subfields reside in a common database table, and there's already some capability to edit the database schema, although I'm not sure if this extends to creating custom constraints? That would be a really great feature, as it would allow defining unique records based on a combination of sub-fields. The Combo fieldtype is still a fieldtype though which needs to be added to a template with mandatory fields, so I still potentially have page id, name, and title all essentially duplicating each other's functionality as arbitrary numeric fields. I understand why page name is meant to be mandatory, as a page that has a template file needs to be accessible via a URL, and the page name is part of the routing, however I'm not sure whether it's practical for pages that don't have a template file to simply work off the page ID? What might be useful is a template property setting that can indicate that the id should be used as the name as well, so there's still a 'name' but it's populated at runtime as a reference to the page id. I'm guessing it might be possible to do something like this via a hook?

Nice clean site. Although I'm on the other side of the world, I remember seeing Shetland calendars and other things most years, because in the pre-internet days Mum made a pen-friend connection with a Shetland woman with the same first name as her. I believe the two were put in contact by a publication run by Enid Blyton - yes The Enid Blyton of Famous Five fame. Mum's never been to Shetland nor has her pen friend ever been to New Zealand, but in our family we've always had this kind of long distance connection. I have an idea that my brother's mother-in-law actually has relatives from Shetland, so the connection there is even stronger.

Most likely I wouldn't be able to make any more commits to the core if I got run over by a train. I've always thought this was one of the reasons to use open source, ensuring that when someone smacks the tracks, the code doesn't. But it's not just about being open source, the code also has to be clean and well documented so that others can easily take it on board. That's one reason why I put so much effort into code quality and code documentation. My intention is that the code is always ready for others to understand and work with. As for commercial services/modules, it's the same risk inherent with any product or service you pay for. While not open source, the Pro modules do have the same code quality and documentation as the core, and are not obfuscated or encrypted. If it sets your mind at ease, I'm 46 years old (not 76), and am very healthy. I run and lift every day, eat lots of salads, don't eat red meat, and don't participate in any dangerous activities. Most likely I'll be here for at least another 46 years. But if I'm ever derailed then I know the project would still be in good hands. Never say never. When I was 25, I was fit an healthy, and then got meningitis which nearly killed me. When I was 42, I was also apparently fit and healthy, and then ended up with a twisted bowel which also nearly killed me. I've recently been asked a similar question by some of my clients, "What happens if something happens to you?" I think perhaps a peace of mind option with the pro modules might be to include some sort of clause that allows rights to modify them to provide bug fixes etc, and also explicitly have some sort of clause around what happens if the developer (you) ceases to support them. I've often thought about software licensing and what should happen if the original author is no longer able to support their code, but there's still a need to use it. I know that people need to earn a living, and relying on donations usually won't be sufficient, so licensing software is a fair way to get paid, however it would be useful to have some sort of commercial license that both respects a copyright holders right to earn an income, but also provides peace of mind to licensees in the event of the software being abandoned by the original creator. I understand in normal circumstances, copyright extends past an author's death, but I know that my wife or daughter would not be able to support any code I might write, although they might need revenues generated by it if it's still useful without modification, so the licensing thing is complex.

Thanks. I worked out the first approach which works as a quick fix, but it's good to know the second one will work as well, as I think from an OO programming perspective, it's clearer to use the second option as I am effectively setting a page property, just I'm doing it at runtime, and as a non-persisted property.

Every week or so, I'm finding large amounts of space taken up with orphaned image files on some pages that have an image field but no images, with the image file names starting with a base number then numbered sequentially. I the worst cases this can amount to several hundred images. I run code similar to what @ryan posted in this thread: however after a few days the large number of files are all back again taking up considerable amounts of space. These are pages that no one other than myself has editing access for. The one thing the pages in question have in common is that they do have child pages that have images, and the orphaned files that are duplicated many times are an image that belongs to the first of the sub-pages of the page where the large number of copies are created. I have this function: /* * If a page doesn't have a header image explicitly set, search for the first child page that has a header image, and use that instead. */ function getHeaderImage($page) { if ($page->headerImage) { $headerImage = $page->headerImage; } else { foreach ($page->children as $item) { if ($item->headerImage) { $headerImage = $item->headerImage; break; } } } return $headerImage; } And in the templates affected this: $page->headerImage = getHeaderImage($page); It appears that every time I assign an image this way, a new copy is getting made into the page's /site/assets/files/ folder even though I never save $page. Normally with other page properties, if they're set, but never saved, then they don't persist, but I know images may be different as they correspond to actual files on disk. Is this expected behaviour?

I agree. The way I see it, if there's a low level SQL type that supports given functionality, there should be a fieldtype that corresponds to it, however the ProcessWire specific fieldtypes are there for when additional functionality is required that can't be supported directly through SQL. Speaking of that, it may be a bit of a limited use case, but I've wondered a few times about adding geometry fieldtypes and adding geometry selectors. There's already the FieldtypeMapMarker, but this is using just standard float fields in mySQL, while mySQL and MariaDB have support for geometry datatypes and spacial indexes, and there are times it would be nice to be able to do a query and return all the places within a given distance. I already hacked the FieldtypeMapMarker a bit to allow adding KML overlays exported from Google Earth, or mobile tracker apps, but there's certainly a lot more that could be done to give geospatial data first class treatment if enough people use it.

Tabulator looks interesting, although sometimes I want forms, not tables for data input. The trouble with Profields Table is it seems to be geared around having multiple records in a row format per page. If I understand it correctly, the new combo fieldtype stores things in sql fields in a table in the database, but gives more flexibility around layout, and is a single record per page, so this is much closer to what I want, and may be what tips the balance in terms of me buying the ProFields package. A question though is whether this is actually creating separate fields in the database table, or doing what custom image fields does, and just storing a JSON object in a single field. From an SQL perspective, having separate fields and being able to do sorting and grouping at the SQL level would be more convenient than doing it at the PHP level. I would like the option that I raised elsewhere around ordinary field types, of being able to choose whether to index subfields or not, as some will only ever be retrieved based on selecting on other fields. I guess I should bite the bullet and try building my own custom fieldtypes. I'm at the point I'm comfortable building simple Process admin modules thanks to @bernhard and his excellent tutorial on building custom admin modules, however I think a good tutorial on building custom data driven fieldtypes or custom admin data access modules might be helpful, and it might actually be as useful as adding more features to ProcessWire itself, as with all the various inputfields and Process modules, possibly everything I need is already there for custom SQL data, just I need to join the dots.

I'd say this is a biggie. During lockdown here in NZ (thankfully long over), I had the need to move various client databases to the cloud, and started looking at a headless CMS, Directus, because ProcessWire doesn't let you easily hook up an SQL database, leaving the SQL structures alone and just give you an admin UI and API, and yet looking at what Directus does and what ProcessWire does, I can't see why ProcessWire couldn't be made simple to connect to an existing SQL backend. Input Fields work with a wide range of data, and there are already ProFields like Table and Combo that connect to an SQL table behind the scenes, so maybe having a fieldtype (maybe a pro field?) that can connect to any SQL table would be quite handy. That would make connecting to existing apps that have been built outside of ProcessWire straightforward, and also make migrating data simple if it does make sense to pull it into the ProcessWire way of doing fields, as the external data can be accessed via the API. ProcessWire already has a fields table that contains the definitions for how each field should be managed in the backend, and I guess it should be possible to do something similar linking an existing SQL table, where it is possible to specify any inputfield parameters for each table field, while leaving the actual SQL table alone. I can see a down side, in that some people who are used to SQL tables, and don't quite understand the ProcessWire way might be inclined to use SQL tables for everything when the ProcessWire way of one table per field actually has advantages depending on the data, but if it makes migration easier, and sticks with ProcessWire's philosophy of trying to keep out of your way and not dictate how you work with your data, then perhaps it could be useful?

For years, as a personal project, I've been documenting my little corner of the globe via my personal website www.marlboroughonline.co.nz, which also gets used as my testbed for new ideas with ProcessWire, SEO and other stuff, as I'm not going to have some client on the phone if I break something, but if I do it well, I can earn a few dollars from Google adsense. I've always had a bit of a passion for natural history, being introduced to David Attenborough documentaries when I was a teenager, and actually studying life sciences at university, so it's only natural if you'll excuse the pun, that a part of my site would be dedicated to documenting local plant and animal species in my part of the world. For many years, one of go to reference websites for information about native plants has been New Zealand Plant Conservation Network, and I often include a link from my site to details on a species on the site as an authoritative reference, and I use the site to help identify plants I've photographed. It was only recently that for some reason I decided to have a look at the source code of the site, and it looked suspiciously like it was made with ProcessWire. After a bit more detective work, I found out that it was made by @Robin S and is indeed built with ProcessWire. To get away from the computer, I also teach kids edible gardens at my daughter's school a couple of hours a week. It's part of a wider Enviroschools programme here in NZ, and among other things it covers is teaching kids to recognise noxious weeds. There's a really useful website Weedbusters that's supported by councils and used as a reference for many invasive plant species, and it turns out it's more of Robin's handiwork. I think it's a bit funny that even when I'm supposedly away from ProcessWire engaging in other activities, it seems to follow me around. ?

There's also the wishlist and roadmap on the forums here. https://processwire.com/talk/forum/5-wishlist-roadmap/ @ryan from your perspective do you have a preference where people post feature requests, and is there a reason to use the forum vs github and vice versa? Interesting feature request. I could have used that recently, although in most cases the images I wanted to use were the only ones on the page they belonged to, so I just used a normal page reference and used the image from the page. For pages with more than one image though, it wouldn't work if I wanted to pick a single image. Now that images support additional fields via a template, then I assume behind the scenes there is a page holding this data, so it should be possible use selectors like with regular pages on these fields to find images that match regardless of what page they belong to? That could apply not only for an images reference field, but also for the RTE when inserting images, to provide an image search functionality, like currently is possible when inserting links.

Since others are getting into localsied greetings, Meri Kirihimete! (Although we're actually already a day ahead and Christmas was yesterday in my part of the world.)

Currently, FieldtypeTextarea and FieldtypeText automatically have a fulltext index created on the field content, however there may be scenarios where this has an undesirable impact on performance and storage if the content of the field will never be searched, ie a field will only ever be accessed as a result of selectors using other fields on templates where it occurs. An example where I experienced an issue with the default behaviour was a module that stored a log of data import to a FieldtypeTextarea field, which grew quite long when importing thousands of records. The fulltext index of course indexed the content of this field, although it was unnecessary, resulting in a very large index file. What would be useful, would be an option (maybe a checkbox) when creating a field with one of these fieldtypes to disable indexing. This would mean the default behaviour would remain with the field indexed as usual, however if the field is never going to be used in selectors, checking the box would drop any fulltext or ordinary index, providing performance benefits. Obviously there needs to be a warning associated with checking this option that the field will no longer be able to be found using selectors, and unchecking the option subsequently should create the index(es) if required.

I recently did an import of about 15K records with these modules, and they worked well, however there was one undesirable side effect of storing the log to a standard Processwire text_area field, log_messages. Processwire creates a full text index on text_area and text field types. I had a few issues with field mappings and defining unique identifiers the first couple of times I ran the import, so just used Tracy Debugger to delete the pages, fix the import definition, and run the import again. Unfortunately, over my efforts, the full text index on the field log_messages grew to over 2GB in size, even though the imported CSV file itself was only about 1MB. It took me a bit to figure out why the database had suddenly grown from a few MB to several GB, and once I located the problem, I ran Optimise Table in mySQLAdmin, which cleaned things up and got things back down to a normal size. I suspect in this case, the log_messages field probably doesn't need to have a full text index on it, although I guess if someone has a large number of import pages, maybe there is a case for them to be searchable? If there isn't a need for the log_messages field to be searchable, then perhaps a custom text fieldtype without a full text index might make both the import run more quickly if the field is repeatedly updated during import, and also avoid the issue of creating an enormous index file. I've made a post to the Wishlist and roadmap thread, suggesting an option for textarea fields to have indexing disabled if not required, as this would resolve the issue.

I have a site with about 16K pages, with most of these (15K + ) being templates with just two text fields (including page title), along with several page reference fields, and a couple of numeric fields and all the tables take up just over 20MB of storage, however there is a huge full text index file of over 2.1GB that seems to be totally out of proportion to the size of the database. Does anyone know whether this is normal behaviour with Processwire? From the reading I've done, it seems a full text index of similar size to the table it indexes is not unexpected, but 100x the size of all the tables in the database seems excessive. If this isn't normal, what's the best way to fix this? Edit: It turns out the huge index was related to logging of imports to a text field by this module: I'll post in that thread regarding the issue since it's module related.

@netcarver Here's details direct from PHP manual: https://www.php.net/manual/en/intro.pcntl.php It certainly doesn't recommend PCNTL be enabled for a web server environment. In the user contributed notes, there is a comment about undefined functions which matches what I was experiencing.

I'm not sure what's gone wrong, but I had this module working perfectly earlier in the day , and I imported all but one data table I needed from CSV, but now it fails to run and my browser shows a 500 server error when I attempt to run it. although I updated to the latest Processwire Dev version to address something else, and I'm not sure if that has broken it? I tried copying the latest version of this and the tasker module from Github, as the modules directory doesn't seem to be up to date with the latest, and I wondered if there was a compatibility issue with the latest build of Processwire, but that didn't fix it. Chrome debug tools show a resource like so: admin/page/tasks/api//?cmd=run&id=4820 Opening this resource directly in the browser says that the task has already been run and won't run again. Manually updating the task page and setting 'task is running' to zero triggers an error Tracy Debugger shows : Call to undefined function ProcessWire\pcntl_signal() (/site/modules/Tasker/Tasker.module.php:697) According to some places in PHP docs, pcntl functions shouldn't be called by a web server like Apache, so I'm not sure if that is an issue? The key thing is the module was working just fine, then suddenly stopped working, with import configurations that had previously worked. Any ideas? [Edit] - I found out the problem: I'm running on a VPS with Plesk and PHP as an FPM App doesn't include pcntl but PHP as FastCGI does. The site was one that hadn't been updated in a while and was still running on PHP 7.3 so I updated to 7.4 and in the process, without thinking, changed it to FPM, as most of my other sites are, but this disabled pcntl support. Something to be aware of for anyone else running Plesk.

I think you need $this->page->save(); to actually save the settings, otherwise the values you set are not persistent?

E-commerce is probably one of Processwire's weakest points in terms of a quick, ready to go integration, although hopefully that will change when Padloper 2 is released, but I was doing some browsing and came across a PHP e-commerce platform Aimeos https://aimeos.org/docs/latest/developer/architecture/ that grabbed my attention with this quote: Has anyone had any experience with it? I'm trying to work out how it would need to be integrated, (currently it supports Laravel, Symfony, and Typo3 off the shelf), but if there's an off-the-shelf open source e-commerce package that can be integrated relatively easily with Processwire, that would be nice.

Since I first rebuilt this site in Processwire, it's had a major overhaul, with all competition data now handled on the site rather than in a separate desktop database app. Members are able to access the admin, but using AdminRestrictBranch, some hooks in ready.php and some custom process modules, logged in users are taken to a dashboard that varies depending on their role, so members can only see their own competition images (in a list via Lister Pro) and submit more, while the competition administrator has a dashboard that allows them to manage all aspects of competitions. Competitions close off automatically once their closing date has passed. CustomUploadNames module is used to automatically rename image files when they are uploaded so that once competition entries are downloaded as a zip file, they can be identified by filename. On the front-end slideshows are generated automatically once competition results have been entered, which is done via Lister Pro.

It's possible to hide the page tree, and redirect users on login to some other page. Lister Pro allows you to create your own pages that show data in custom table layout that are editable, and it's not too hard to create your own admin dashboard module with links to several listers. Here's a thread with some discussion of both:

ListerPro extends the Lister class, so it's a superset of functionality of Lister, so any bugs in Lister probably also apply in ListerPro. API reference for both is here: https://processwire.com/api/ref/process-page-lister/ I've found the upgrade to ListerPro well worth it, however if you're prepared to write your own modules some of the things it provides like saved lister layouts can easily be achieved with Lister, with minimal coding. I've used ListerPro to export (to CSV), but not to import. I wonder if the issue here is that has_parent is referring to the page id field rather than the title?

An SQL "expert" with records with duplicate keys? That starts alarm bells ringing for me. Generally any SQL table should have a primary key which by definition must be unique for each record. Of course it's possible they have some sort of composite key based on multiple fields that needs combining to correspond to a unique page name in ProcessWire. There are certainly cases where I've build SQL tables with composite keys, with one scenario being many-many relationships, and that's something that Processwire doesn't handle too well, although there is a module that makes many-many type relationships possible, although it doesn't compare to what you can do with pure SQL. Processwire does handle One-Many relationships fine via page reference fields or Pagetable fields. If you really must have a direct relationship between SQL commands and table structure and your CMS, I actually wonder whether ProcessWire is your best option. I've been doing a bit of investigation of Directus which looks promising, although it's headless, so no templates for output like ProcessWire, just a REST API, from what I can see, no full text indexing, and being more of a direct SQL - CMS mapping, it also lacks the hierarchical parent-child structure that ProcessWire handles so well, as it doesn't make any assumptions about what sort of data structures you have, whereas ProcessWire, while generally very un-opinionated does treat everything as a page in a page hierarchy. For websites, that's generally a pretty reasonable assumption, but if you really don't want that, and just want pure SQL tables then there are alternatives. While something like Directus will give you a direct SQL to CMS mapping, it won't fix bad SQL data, so if you've already started down the ProcessWire path, you want to be really sure it's worth the effort to change. I come from a pure SQL background myself, and it only took me about 20 minutes of reading the ProcessWire documentation to understand how it works, so I don't think it should be hard for someone from an SQL background to adapt. Maybe there's room for a blog post or tutorial showing how to "do it this way in SQL" and ProcessWire equivalent along with what's different.

Apart from working with ProcessWire, my other area of work is SQL databases, where I tend to have nearly all application logic contained in the database through views, stored procedures, user defined functions, triggers etc, so I understand how coming to ProcessWire can take some getting used to. If you simply want to import data from an existing system, then there's no problem to mix SQL queries in PHP and calls to the ProcessWire API to read data from an existing database using PHP's PDO or ODBC abstraction layers and then store fields to ProcessWire fields. I've done this myself before importing data from another system, and it's not overly difficult. Here's a bit of code I wrote a few years ago to import some data from another system into ProcessWire. Obviously the SQL will vary depending on what system you're importing from, and how you've named your ProcessWire fields will be different. This is old code, and untested, although I think it worked at the time. Having the Tracy Debugger module installed is a must have, as it provides a PHP console with code completion for the ProcessWire API, so that you can run ad-hoc bits of PHP code to do stuff like importing data without having to write a module or embed it in a template if it's just one off code that will only be run once. $webpressiondb = new PDO('mysql:host=127.0.0.1;dbname=mol_webpression', $user, $pass); $sql = "SELECT {$site}_docs.docid, {$site}_docs.parentid,custhead, subindex, title, keywords, author, {$site}_docs.modified, created, wbdesc, name, content, template, folder FROM {$site}_docs INNER JOIN {$site}_pages ON "; $sql .= " {$site}_docs.docid = {$site}_pages.docid ORDER BY {$site}_docs.subindex"; $wbdocs = $webpressiondb->query($sql)->fetchAll(); foreach ($wbdocs as $record) { //Need to allow for old records with no title. if (preg_match("/[a-zA-Z0-9]/i", $record['name'])) { print '<h1>' . $record['title'] . '</h1>'; $pwpage = new page(); $pwpage->parent = $pages->get($parent); $pwpage->template = $templates->get("rosina"); $pwpage->title = $record['name']; $pwpage->text3 = $record['title']; $pwpage->text1 = $record['wbdesc']; $pwpage->keywords = $record['keywords']; $pwpage->textHtml1 = $record['content']; $pwpage->wid = $record['docid']; $pwpage->wparent = $record['parentid']; $pwpage->worder = $record['subindex']; $pwpage->wcreated = $record['created']; $pwpage->wmodified = $record['modified']; $pwpage->custhead = $record['custhead']; $pwpage->save(); } } If you have cases where you really need traditional SQL tables with multiple fields in ProcessWire, you can create your own custom fieldtypes to do this, but you need to think carefully about whether you need this. The ProcessWire approach can be slow at bulk inserts, as each field results in a database INSERT statement, however retrieving data, especially if you don't need all fields is fast as it doesn't need to load fields in a query if you don't use them. If you're going to be doing a lot of multiple field inserts on an ongoing basis, and you're also going to want to retrieve most or all of the fields when you read the data, creating a custom fieldtype that's a traditional SQL table might give you performance advantage, but to date, I've generally found I've been able to work within ProcessWire's way of doing things.

@adrian Thanks, that's working as expected again now.

I'm having an issue with PW 3.0.163 and this module 1.1.12 where the initial option to send welcome message is not present, however the 'Re-send welcome message' option is present after saving a user. I think I used an earlier version of the module that didn't require the user to have the permission edit-profile, and that worked OK. I think the permission profile-edit is not being set until after the user is already published, so the initial option to send the welcome message is never shown.