ryan

Today a new version of FormBuilder has been released in the FormBuilder support board (our 50th version) and it has a lot of interesting new features, which we’ll take a closer look at in this post— https://processwire.com/blog/posts/formbuilder-v50/

The next dev version of ProcessWire is in progress but I'll wait till likely next week to bump the version. So far there are 6 new pull requests added since 3.0.179 and 2 issue resolutions, plus some other updates, with plenty more on the way. A lot of focus this week has also been on FormBuilder updates which include new spam filtering options, improved save-to-page options, improved Combo field support, new entries actions (and the ability to add more via modules and hooks), framework updates, and various minor bug fixes. This version has a lot of nice improvements and I'm hoping to have it ready for you in the next week or so. More details soon. Have a great weekend!

@Robin S Uikit 3 originally had LESS only, and we were an early adopter. I would have preferred SCSS at the time, as I didn't know much about LESS. But I soon learned that there's not that much difference between the two. If you know the basics of one, you already know the other. Now I regularly use both LESS and SCSS and consider them equals for my use cases at least, one of which is maintaining the Uikit PW admin theme. Given two choices I would almost always choose what's less popular. ? (I don't use WordPress either). It might be worthwhile for a major CSS framework to also add SCSS support like Uikit has done, but I think in our case it wouldn't be a good use of time. @ukyo There is some stuff we can delete, such as the /src/scss files and /src/js files, which we don't need. But prior to Bernhard's work, I was using Uikit with node and that's not something I'd want to go back to, it was really not a good solution for PW and was the main obstacle towards more happening with AdminThemeUikit. I'm sure it works fine for some, but for me and I imagine many PW users, we don't like having dependencies beyond PHP and PW, especially higher maintenance ones like node and the seemingly endless dependencies on top of it that result. Bernhard solved those obstacles so that now customization of Uikit is easily accessible to all PW users, and while it might consume a little more disk space for the .less files, it's well worth it in my mind. If someone wants to define their own uikit source path, that is also fine too, but the goal here has really been to make the whole thing as simple as possible, so I don't want to require people to download their own separate copy of Uikit, but will definitely get rid of some of the unnecessary files we don't need.

ProcessWire 3.0.179 adds great new admin theme customization tools that put you in full control over the admin styles, thanks to PR #189 from @bernhard — https://processwire.com/blog/posts/pw-3.0.179/

The UserActivity Pro module in ProDevTools was upgraded this week so that it now uses the Javascript Beacon API, enabling quick and reliable identification of when a user has finished a particular activity. It also now keeps track of when an activity is visible to the user or the window is hidden (like minimized or in a different tab), and it is able to report how many unsaved changes a user has made. All of this is visible from the Access > Activity menu in the admin. On the core side, there have been minor updates, so no version bump this week. But there are still a couple new PRs and useful things to look at in the commit log. The plan for next week: Bernhard Baumrock has been working on a PR that makes it very simple to modify and recompile the CSS of our AdminThemeUikit module... like as simple as placing an admin.less file in /site/templates/ directory. I'll save the details for next week, but after using it a bit here I have to say it's very cool. It also makes it really simple to upgrade the Uikit version. Previously I'd been following the build process as outlined in the Uikit instructions and often found myself in a state of confusion with npm errors and strange dependencies, needing upgrades for unidentifiable tools and producing incoherent warnings that never went away and left me with little confidence in the process. Perhaps it was because I started all this back when Uikit 3 was still beta, but the result was that I didn't much like upgrading Uikit or recompiling our AdminThemeUikit CSS, and I don't think anyone else did either. It made it a little difficult for our AdminThemeUikit module to achieve its original mission of being a simple module that many would extend and build from. Well, Bernhard has found a way to skip over all that nonsense and made it much simpler for all of us, so that AdminThemeUikit can finally be what it set out to be. I look forward to integrating this PR hopefully this coming week and think it means a lot of good things for our admin. Also, huge thanks to Pete for upgrading our forums this week! It's a great upgrade.

@bernhard There isn't really a standard, just a lot of options to find which suits you best. For most modules, just simply doing $this->pages, etc., is likely the simplest route to take. The core has some cases where fuel is turned off, which is rare with modules. But for the core I use $this->wire()->pages; or $this->wire('pages'); because it's one thing that's going to work consistently in all core classes since there will always be a wire() method on all Wire derived classes. If you are using $this->wire->pages that is fine, but just note that $this->wire hits __get(), then returns ProcessWire instance, then does it again to get to the pages part. So if it also works with your IDE, then just $this->pages would likely be preferable to $this->wire->pages. In the end, it's kind of micro optimization so I would use whatever you prefer. @matjazp I think this is going to be the same as $this->wire()->apivar in that regard. The only reason I don't use the string argument so much anymore is because phpstorm seems to understand exactly what's being accessed with wire()->apivar but not wire('apivar'). Plus, it'll know if you've mistyped an API var name so long as it's not in a string.

ProcessWire 3.0.178 focuses largely in adding pull requests (PRs), which are code contributions by ProcessWire users. We had quite a few great pull requests pending, and in total we have added 26 of them in 3.0.178— https://processwire.com/blog/posts/pw-3.0.178/

Like what Bernhard mentioned — my IDE knows all about what I'm accessing when I type in $this->wire()->apivar or wire()->apivar, so it can suggest methods, arguments, and tell me when I've typed something wrong. Whereas less of this happens with $this->wire('apiVarName') or wire('apiVarName') — the IDE isn't nearly as helpful. So it's more of catering to the way the IDE works than anything else, and in exchange it makes coding easier, faster and less error prone. It's the same reason you'll sometimes see this in modules: /** @var Pages $pages */ $pages = $this->wire('pages'); That comment in the first line tells the IDE that $pages is referring to the Pages class. But this produces the same result with no /** comment */ necessary: $pages = $this->wire()->pages; In most Wire derived classes, $this->wire->pages (like Bernhard was using above) or $this->pages will also work and the IDE will still know what API var you are accessing. The reason I prefer to call it as a function $this->wire()->pages rather than a property $this->wire->pages (or $this->pages) is because it will work consistently everywhere in PW, even if a class has "fuel" turned off (see useFuel). Having fuel off is necessary in classes that can potentially have properties or field names that could overlap with API var names; an example is the Page class or your own custom Page classes in /site/classes/. The other reason is that $this->wire()->apivar (wire method) is the most efficient access to an API var because it has the fewest hops. Accessing $this->wire->apivar or $this->apivar first goes to $this->__get('apivar') and then $this->wire()->apivar. So $this->wire()->apivar is more direct (one less hop). In reality it probably doesn't matter much though. @Zeka

This week I've been working on a combination of core improvements, optimizations, and fixes, plus a dozen pull requests have been added. Thanks for all of the great pull requests to the core that many of you have submitted. PR authors will appear in our GitHub contributors list once the changes are merged to the master branch (that's apparently how GitHub works). I do think soon we'll focus on getting a new master version out, as 3.0.165 is starting to feel old relative to the current dev branch. You can see all that's been changed and added this week in the dev branch commit log. Next week we'll be doing more of the same, though planning to get into some of the PRs that I didn't this week because they required more review and testing (those that involved more than a few lines of code changes). We're not going to bump the dev branch version till likely next Friday, since this week's work will continue into next week. Thanks for reading and have a great weekend!

This week we focus in on a new and unique Inputfield module added to the core that enables a lot of useful new input capabilities for tags, sortable multiple selection and custom user input. I introduced it in last week's forum post, but it was further improved this week and reached a point where I thought it would need some dedicated documentation, so this post dips into that and goes quite a bit more in-depth than last week’s. I never got around to bumping the version to 3.0.176 because the module wasn't quite done till Thursday (kept updating with improvements), but just in case there's any confusion about versions between last week and this week, I bumped it to 3.0.177 even though it contains what was originally intended for 3.0.176, plus more— https://processwire.com/blog/posts/pw-3.0.177/

Instance refers to an object instance of the "ProcessWire" object, and there can be more than one. When used, it has full access to the instance it connects to, so it's not something that could be done through http. Though if you need to share some data through http then this is very simple to do in PW (especially now with URL/path hooks), but it's something completely different in this case. I'll take your word about the technical semantics. I don't think we intended that broad of a definition for the forum rules, though perhaps they need to be modified to better clarify. I was thinking more of "relating to the government or political party of a country" type political stuff. If someone wants their avatar to communicate they like running, swimming or biking, or that they are against racism, cancer or school shootings, or they want to have a bow to support breast cancer awareness, a puzzle piece to support autism awareness, support diversity, etc., seems fine. Though if they want to open a topic about it, we'd want it in the beer garden or one of the other off-topic boards. I'd also say the difference between positive and negative matters too. Hopefully we veer on the positive side as being a place for inclusion where everyone is welcome and we support and help each other out, at least I think of you all here as my best friends.

@Torsten Baldes I think it works well. We use it for https://processwire.com/modules/, where it pulls (and manipulates) all the data from a separate PW installation using the multi-instance support. I recommend keeping all instances running the exact same PW version. Supporting multi-instance does sometimes require recognition from installed modules, and and I think it's likely some 3rd party modules might not fully support it (though don't know of any specifically), so the fewer 3rd party modules installed I would think the less likely you are to run into issues, but I'm only guessing there. I don't know what's up with the chains and fire in the logo (looks potentially violent), but I just googled it and ended up here. They don't appear to be promoting racism and do not appear to be affiliated with a political or religious group, unless you guys know something I don't? It looks like it's for an exercise/running group and clothing brand with a seemingly positive message "Running is to unite people, not to disconnect or isolate them". Or are the graphic elements the logo inferring something bad or affiliated with a religious/political group?

The latest version on the dev branch will actually get the version bump tomorrow, as I'm currently on a 7-day work schedule with shorter days (factors outside my control). So what I'd usually be doing Friday is happening Saturday or Sunday instead. But the updates in 3.0.176 are well enough defined to write about here, and everything is there already, it's primarily just additional testing that remains before the version bump. Relative to 3.0.175 version 3.0.176 contains about 25 commits or so with most of the focus being resolution of minor issue reports. However, 25 commits is a lot so there are also some new things here too. First off, as requested, PW now supports multiple database read-only configuration settings rather than just one. When more than one is present, it will select one randomly. To use multiple read-only connections, simply specify a regular PHP array of 2 or more associative arrays in your /site/config.php file, like this: $config->dbReader = [ [ 'host' => 'mydb1.domain.com' ], [ 'host' => 'mydb2.domain.com' ], [ 'host' => 'mydb3.domain.com' ], ]; In each array item, you can specify 'host' (as above), along with any other setting (name, user, pass, port, etc.) that differs from the primary DB connection. There's another benefit to having multiple dbReader hosts as well: If the connection for one fails, it'll move on to the next, and keep moving on till it either finds a working connection or finishes the list. Next up, ProcessTemplate (Setup > Templates) gained a "Manage Tags" feature just like the one you have in ProcessField (Setup > Fields). So now it's a lot easier than before to define and manage tags for multiple templates as a group. In addition to this, both ProcessField and ProcessTemplate now have an actual Tags field (like the one available for files/images) rather than just a plain text input. The Template class also gained several new API functions for working with tags on templates. Templates in PW have supported tags for a long time, but now they are backed up by a much better API and admin interface to them. In order to support the new tags inputs in ProcessTemplate and ProcessField, a Tags Inputfield module was developed, named InputfieldTextTags (the name InputfieldTags was already taken). This module is now in the core, and it uses the existing Selectize tags functionality already present in the core, but previously only used by file and image fields. While developing this, I found it was a nice alternative to AsmSelect for sortable multiple selection, so made it available as an additional input option for both Page and Options fields. When used in a multiple-selection context, it ends up being a multiple selection input that takes up no more space than a text input, which can be quite handy for a lot of situations, and more space friendly then AsmSelect or even PageAutocomplete. I've noticed travel websites using a similar solution for multiple selection of destinations or amenities in search forms, and perhaps this one has some front-end potential as well. Since it's just an Inputfield without a dedicated Fieldtype, it is likely to work just as well on front-end forms (like FormBuilder) as it does on admin forms. Longer term I imagine we'll have an optional FieldtypeTextTags module as well, which will be useful for supporting user-entered tags sharable between pages. Currently it does support user entered tags if you choose it as an input option for a regular "text" field (see bottom of Details tab in field editor), and it can optionally be combined with predefined selectable tags as well. While all of these updates are on the dev branch now, look for the version bump this weekend after I've had a little more time testing. Though if you'd like to help test, feel free to grab it now. Thanks for reading and have a great weekend!

@fedeb Looks like I missed a spot on the updates to the Event class, I have fixed that and removed the references to the old fields (location, notes). 1. In ProcessWire a Page can be in a "output formatted" or unformatted state. You can toggle output formatting on with $page->of(true); or toggle it off with $page->of(false); Or you can get the current state with $formatted = $page->of(); When output formatting is on, values returned from the page are intended for output and thus can contain runtime formatting such as entity encoding. This is good for output, but definitely not something you'd want saved the database, so when you are creating pages or saving values on pages you want to have its output formatting disabled, i.e. $page->of(false); This should be done before getting, setting or modifying values that will be saved. 2. The location and notes aren't supposed to be there. I removed them in favor of making the module simpler by just having "title" and "date", and then people can add any additional columns they want. 3. The set() and get() functions are inherited from the WireData class. (Event extends WireData) 4. InputfieldEvents won't be included unless something asks for it (like the page editor in the admin). If you are just using the API side, then the InputfieldEvents module likely will never be loaded or come into play. It's still nice to have in case you ever want to see or edit your data in the admin.

This week I didn't get as much time as I'd hoped to work on the core, but still managed to get a few things done. I did get some small core updates in place, but not enough for a version bump, and got the second covid vaccine, kids back to in-person school (3 hours a day, after a year of being away), and a couple more major module upgrades released. In last week's post, someone asked for an update to the ProcessWireUpgrade module, and I had a look and a realized it was long overdue for one, so I tried to cover a lot of ground there. The new version is now out and improved throughout. Grab this latest version of ProcessWireUpgrade if you are interested. It's always been pretty useful, but it's even more so now. I've also updated the modules directory feed so that it can let you know when Pro module upgrades are available as well (as requested). The ProcessWireUpgrade module reports this Pro module information in a clear manner too. Some improvements have also been made to the core version updating features as well. The other module updated was FieldtypeEvents, a module that demonstrates how to create simple a Fieldtype+Inputfield module pair that has repeatable rows of multiple columns. The last version was made for PW 2.x, so I largely rewrote the module while updating it for PW 3.x. It's been simplified quite a bit so that it can be even more clear how to adapt it to create your own custom module. Being PW 3.x exclusive also opened more doors in keeping it simple but powerful. If you don't mind doing a little customization in adapting the code for your purposes, this module can be every bit as powerful in solving custom needs as Repeaters, but without Repeater overhead. I updated this module because another ProcessWire user was asking about how to store multiple columns of repeatable data for 20-million pages. Repeaters would be too much overhead at that scale, though FieldtypeTable could fit the bill. But at such a large scale, I thought a custom Fieldtype would be even better. So FieldtypeEvents shows you how to do it, and it's [hopefully] simpler than you might think. This has nothing to do with PW, but to follow up from last week, that second dose of the covid vaccine was an interesting experience. I got the shot (jab?) and strangely didn't feel a thing. I walked home and took the band-aid off my arm and noticed there was no sign of anything, no little pin prick or red spot, and a completely clean band-aid. I started to wonder if I'd been fooled or something. If someone sticks a needle in your arm, you should feel it right? And there should be some evidence of it? Well there wasn't (and no I didn't watch the needle, who would do that?). They also told me I'd feel ill for a day or two, and I felt completely fine even the next morning. But then 10 am rolled around and my arm started feeling sore and I was suddenly very tired. So tired that it was laborious to walk, sit at the computer, or do anything other than chill on the couch. That might sound undesirable, but it wasn't uncomfortable, and it was kind of funny being so ridiculously tired, but also really relieved. It meant it was working and my body was getting a download on how to battle covid, and putting all its resources into taking it seriously. It was fascinating, so I just crashed on the couch, relaxed and let it do its thing. By the next day, full energy was back, batteries recharged, upgrades installed. From what I understand, some people have no side effects, but I was one of those that did. Though if anything I enjoyed the different experience for a day, and the comfort of clear evidence it was working.

@Hector Nguyen getcsv() does return an array, but it is an array representing and consuming the memory of just one line from a CSV file (i.e. the columns from 1 row). fgetcsv() is just a layer on top of the fgets() function, which reads one line at a time from a file, which is what makes it memory friendly. On the other hand, PHP functions like file() or file_get_contents() do read the entire file in memory, so they are not memory friendly, even if they are fast. @fedeb I think the best route to take for your groupID+start+end+sequence would be a custom Fieldtype. This would give you all of the benefits of having a repeater, and without any of the overhead. Custom fieldtype may sound complicated, but it's not at all. I've developed a module that can very easily be adapted for this need. See FieldtypeEvents which was created as an example to build exactly this sort of thing from. If you are interested in that route and have any questions, I'm happy to walk you through it.

@fedeb Glad that moving the $parent outside the loop helped there. The reason it helps is because after a $pages->save() is the automatic $pages->uncacheAll(), so the auto-assigned parent from the template is having to be re-loaded on every iteration. By keeping your own copy loaded and assigning it yourself, you are able to avoid that extra overhead in this case. Avoid getting repeaters involved. I wouldn't even experiment with it here. That will at minimum triple the number of pages (assuming every protein page could have a repeater). Repeaters would be just fine if you were working in the thousands-of-pages territory, but in the millions-of-pages territory, it's not going to be worth even attempting. Using a ProFields table field would be the best alternative if you needed it to be queryable data. If you didn't need it to be queryable data (groupID, start, end, sequence), I would leave them as they are, space-separated in a plain textarea field — they can easily be parsed out at runtime so you can access them as as properties of the page. (If that suits your need, let me know and I'll get into how that can be done). When working at large scale, it's also always good to consider custom building a Fieldtype module for the purpose too (that's another topic, but we can get into it too). For your groupID, if the same groupID is referenced by multiple proteins, and there is more information about each "group" (other than just an ID) then I think it would make sense for it to be a Page reference field. What is the max number of groupID+start+end+sequence rows that a protein can have? If there is a natural limit and it's not large, then that would open up some new storage possibilities too. Another optimization you can make in your loop: $page->sort = $i; This prevents it from having to detect and auto-assign a sort value based on the quantity of children the parent page has. For the $page->name, if each page will have a unique "protein-name" then you might also consider using that rather than the ("protein" . $i), as it will be more reflective of the page than a generic index number.

@cb2004 Got it. I'll put out an update to the ProcessWireUpgrade module this week. Support for identifying the latest version of Pro modules is a function of the modules directory rather than the upgrades module. I've been meaning to do this, so thanks for the reminder. I have gone ahead and updated it so that it can now identify the latest versions of all Pro modules. Though I can't add support for download+install upgrades of Pro modules, as they are access controlled so there can't be public download URLs for these. I also think that in general it's always better to install or upgrade modules directly on the file system, as that prevents permissions problems (for when apache is not running as your user account), and makes it easier to troubleshoot and resolve issues when installing or upgrading modules.

@Hector Nguyen This is cool to see generators in action. Though as far as I know, PHP's fgetcsv() never loads the whole file in memory at the same time, regardless of which method is used to call it. I think it just loads one line at a time (?), but this reminds me that an optimization to fgetcsv() is to tell it what the longest possible line might be (as 2nd argument), so that it doesn't have to figure it out. Fedeb's example has 0 as the 2nd argument to fgetcsv(), which means "let PHP figure it out", so some overhead could be reduced here by giving it a number like 1024 or whatever the largest line length (in bytes) might be. There may be other benefits to using generators here though? I haven't experimented with them much yet so am curious.

Unless I'm forgetting something, the $pages->uncache($page); won't help here because $page is a newly created Page that wasn't loaded from the database. So it's not going to be cached either. Uncaching pages is potentially useful when iterating through large groups of existing pages. For instance, if you are rendering or exporting something large from the contents of existing pages, you might like to $pages->uncacheAll() after getting through a thousand of them to clear room for another paginated batch. Though nowadays we have $pages->findMany() and $pages->findRaw(), so there are fewer instances were you would even need to use uncache or uncacheAll, if ever. ProcessWire actually does an uncacheAll() internally after saving a page already. This is necessary because changes to a page or additions/deletions to the page tree may affect other pages, and we don't want any potential for old cached data to appear in future $pages->find() or other operations. Just one example is if we called $parent->children() before a save, and then after the save called it again, we'd want our new page to be in the children rather than having it return the previously cached value. There are a lot of similar cases, so the safest bet is for PW to uncache the results of future page get/find operations after a save as the default behavior. So that's the way it's always done it. As far as I can tell from fedeb's example (and often other with import operations), it may be better to tell PW to skip this "uncacheAll-after-save" behavior. That's because imports often involve Page reference fields, and you don't want PW to have to reload referenced pages after every save. So you could potentially reduce overhead by telling it not to uncache after save, i.e. $pages->save($page, [ 'uncacheAll' => false ]); I'm not sure if fedeb's import involves loading of any other pages, whether for page reference fields, or anything else. So it may not matter one way or the other here, but wanted to mention it just in case. I know about ProcessWire tuning, but not about MySQL server tuning. When dealing with 20 million rows that seems like getting into the territory where optimizations to the DB configuration deserve a lot of focus, so I would bet that BitPoet's suggestions are going to make the most difference.

@fedeb That's the largest quantity of pages I've heard of anyone creating in ProcessWire, by a pretty large margin. So you are in somewhat uncharted territory. But that's really cool you are doing that. I would be curious how different the graph would be if you split it up into batches so that you aren't creating more than a certain quantity per execution/runtime. For instance, maybe you create 10k in one execution and another 10k in the next, etc., or something like that. Would the same slowdown still occur? If so, I would start to think it might be the database index and increased overhead in maintaining that index as the quantity increases. On the flip side, if restarting the process to create each set in batches solves the slowdown, then I would think it might be memory or resource related. A couple things you can do to potentially (?) improve your page creation time: 1. At the top of your code (before the loop) put: $template = $templates->get('protein'); Then within the loop set: $page->template = $template; 2. I don't see a parent page assignment. How are you doing that? Double check that you aren't asking PW to load the parent page every time in the loop and instead handle it like with the template in #1 above. 3. What kind of fields are on your "protein" template? Depending on their type, there may be potential optimizations. Especially if any are Page references. Can you paste in a line or two from the CSV? 4. If you can assign a $page->name = "protein" . $i; rather than having PW auto-generate a name from the title, that will save some resources too.

@cb2004 I'm not sure I understand what you mean by "removing older official repositories". Can you expand on this? Which social embeds? I often use the MarkupSocialShareButtons module for this stuff. But if there are oembed providers for social links then a similar strategy would work. I know that Twitter has an oembed service, and Facebook apparently used to, but then killed it. I was originally thinking about expanding this module (or adding another) to support any oembed provider, except there is a TextformatterOEmbed module that apparently does this so thought I might try that one out first. Within the last few weeks they opened it up to everyone here. Previously it was just people 65+ or with health conditions, etc. I think the vaccinations are going well for the people that want them (which seems to be the majority), but apparently there is still a portion of the population that doesn't want to get the vaccine, so that could keep the virus spreading and mutating as long as that remains the case.

@markus_blue_tomato Great, glad to hear it's working well! @StanLindsey This would be very simple to add, I'll plan to add it this week. Question: would just an array of DB hosts be adequate, or would it need separate configuration (host plus db name, user, pass, port, etc.) for each of the readonly db hosts?

@markus_blue_tomato Sorry to hear that, I hope they have it available soon. I figured we'd be the last to get it here because the for-profit healthcare system in the US doesn't often lend itself well to public health, unless you are wealthy. (Just getting a covid test was $400). Luckily it seems the vaccine isn't being handled by the healthcare companies, and it's free. My parents got their vaccine at an appliance store drive-through, my wife got hers at the grocery store, and I got mine at the office of some technology company in our town square. That might sound sketchy but they are all legitimate and it seems to be working well for once.

It's spring break here and my kids are going back to school next week after being out for more than a year. Since it's a break week, the weather is great, and it's also the last week of the year-long covid break from school, I've spent a little less time at the computer this week. I've focused on some smaller module projects rather than the core. More specifically: posted a major update and refactor of the TextformatterHannaCode module, and a completely rewritten TextformatterVideoEmbed module. While making these updates, I've also made note of and attempted to resolve any reported issues in the GitHub repositories. Next week, it's back to the core, with both issue resolutions and pull requests scheduled for upcoming versions. Next week I also get my 2nd shot of covid vaccine, and I'm told it may slow me down a bit for a day, but will be well worth it. I had a day of tiredness from the 1st shot, but it was greatly outweighed by feelings of gratitude and reduction of worry. I highly recommend it as soon as you can get it, if you haven't already.