Leaderboard

Well, yes and no. Two migration modules already exist in ProcessWire, but neither suited my needs: “Migrations” by @LostKobrakai seems effective but quite onerous to use and has been deprecated in favour of “RockMigrations” RockMigrations by @bernhard is simpler and has a nice declarative method: migrate(). However, it is ideally suited to “headless” development, where the API is used in preference to the Admin UI. This is great for professional PW developers, but for occasional developers like me, it is much easier to use the UI rather than just the API. In addition there @adrian's ProcessMigrator which is designed for migrating page trees. Concept I wanted something to achieve the following: To allow development to take place (in a separate environment on a copy of the live database, or on a test database with the same structure) using the Admin UI. When finished, to permit a declarative approach to defining the migration and implementing it (again in the UI). To allow testing of the migration in a test environment on a copy of the live database. To allow roll-back of a migration if installation causes problems (ideally while testing rather than after implementation!). To provide a record of changes applied. Although not originally intended, the module I developed also allows the selective reversion of parts of the database by exporting migration data from a backup copy. Also, if changes are made directly on the live system (presumably simple, low-risk mods – although not best practice), it allows reverse migration to the development system in a similar fashion. I should emphasise that what I have built is more of a 'proof of concept' than a fully-fledged module. The code is pretty hacky and uses some stuff outside of the module itself. Lots of validation is missing. However, I have used it successfully in a number of small tests and a medium-sized live migration. If there is sufficient interest, I will tidy the code and make it available, but it would still need input from better coders and PW-savants than me to make it into something more widely usable. EDIT: Please note that the module has moved on a bit from this original post - the design has changed somewhat to make it more robust and flexible and additional features have been added. Please see the help file for full details. I still consider it to be at alpha stage, however, so use with care - test before making migrations and always take backups first. Design The module has the following principal components: A PW module “ProcessMigrateData”, bundled with a bootstrap migration in the same ProcessMigrateData folder, to be placed in the site/modules folder; A Page Class “MigrationPage” to be placed in the site/classes folder; Php files migrationActions.php and migrationControl.php to be placed in the site/templates/RuntimeMarkup folder (and migrationActions.js to be placed in site/templates/RuntimeMarkup/scripts). There are also a methods which need to be put in class DefaultPage and a functions in the init.php file. The module requires the FieldtypeRuntimeMarkup module. Migration definitions are held in .json files in the ProcessMigrateData/migrations/{migration name} folder (I might move this). This folder contains up to 2 sub-folders - “new” and “old” which each contain a file called a migration.json file, which defines the scope of the migration in terms of fields, templates and pages, and also one or more of fields.json, templates.json, pages.json and remove.json. The first 3 of these files contain the field, template and file definitions within the migration scope and the remove.json file simply lists fields, templates and pages to be removed. These migration files are mirrored by pages of template “Migration” under a parent /migrations/ of template “Migrations”. The mirroring happens in two ways: If a new migration is created in the module (from the Setup -> DB Migration menu), then initially no json files exist. The json files are created, after the scope of the migration is defined on the page, by running “Export Data” from the eponymous button. If json files exist, but there is no matching migration page, then the latter is created by the module on accessing the DB Migration admin page. In this case, we are in the “target” database so there is no “Export Data” button, but instead “Install” and/or “Uninstall” buttons. Migrations therefore either view the current environment as a “source” (type 1) or a “target” (type 2). Installation The module creates templates called Migration and Migrations and a page below the root named ‘migrations’. Open the admin page “Setup -> DB Migration” to create a migration. One (“bootstrap” is already installed) and cannot be modified. The pic below illustrates the DB Migrations page in the source environment. The status of a migration (as a source page) can be ‘pending’ or ‘exported’. ‘Pending’ means either that the migration data have not yet been exported or that the current export files differ from the source database. On opening this page in the target environment, the individual Migration pages (type 2) are created from the definitions in their respective /new/migration.json file. The pic below illustrates the DB Migrations page in the target environment. In a target environment, a migration status can usually be ‘indeterminate’, ‘installed’ or ‘uninstalled’. ‘Indeterminate’ means either that the migration has not yet been installed (so no ‘old’ files containing the uninstall definition exist yet) or that the current state matches neither the ‘new’ or the ‘old’ state. ‘Installed’ means that the current state matches the ‘new’ definition and ‘uninstalled’ means that it matches the ‘old’ definition (i.e. it has been actively uninstalled rather than not yet installed). When carrying out development work, you keep a note of what fields, templates and pages you have added, changed or removed. The module does not track this – it is a declarative approach, not a macro recorder. Also, it does not handle other components such as Hanna codes and Formbuilder forms. These come equipped with their own export/import functions. You can update a migration page as you go along, rather than keep a separate note of changed components. The migration page also allows you to document the migration and add any number of “snippets”. These snippets do not do anything, but can be a convenient place to store (for example) Hanna code exports for pasting into the target environment and help to make the page a comprehensive record of the migration. See example below: Note that migration pages just define the scope of the migration. It is entirely feasible for other parts of the dev database to be changed which are outside this scope and which will therefore not be migrated. After sync'ing code files to the target environment, the new migration will be listed on the setup page. On the migration page, in the target environment, there are “preview” buttons to see what changes will be implemented. The migration can then be 'installed'. See example of the migration page in ‘installation’ mode below: That's the gist of it, but inevitably there are complications. Happy to discuss and share further if there is interest in this.

Hi guys, that was a few days of fun for me. A good learning experience. I have done some experimental things (for me "experimental) ? The result: https://github.com/eversthomas/pw-maturana I have sass a small blog metainformation in header a responsive menu with hamburger my cookie consent a card component (repeater field) My experimental website: https://pwblog.end-linkage.de Greetz from germany Tom. PS: Everything is done with direct output strategy with includes. Actually i have to learn the delayed output strategy.

The module was never published in the module directory :-)

Where this may not work is where you have a new page reference field that needs to access pages with a given template when that template does not yet exist, or a given parent page when that parent page does not yet exist. In that case you're going to need to add a page or template (or both) before you add the field. @adrian mentioned having to loop through several times to ensure all dependencies are met, and I don't think it's possible to avoid this. Some field types don't have dependencies, so it makes sense to process them first, but page references will have dependencies that may or may not have been met, and if not, you'll need to install any templates and pages then loop back and check whether the page reference fields have their dependencies met. Where it gets really messy is if you have a template that depends on one page reference field, which happens to be the template used by another page reference field. eg: customer->billing-contact (page reference to contact template) , invoice->customer (page reference to customer template), so invoice template can't be made till customer page field exists, which in turn depends on template that has a page field that references contact template. In this case adding fields > templates > pages in that order won't work. FWIW, it's hard to make dependency tracking work, and even the big guys have issues. I recently had a .Net Xamarin Forms app project using Microsoft Visual Studio, and circular references between dependencies between third party libraries was blocking me from updating what I needed to get it working. In the end I had to delete all the dependencies and then add them back in with the updated versions to get it to work! I think what I'd be happy to settle for as a developer with ProcessWire, would be to be able to set up a single list of objects in order, specifying whether they're a page, field, template, or module, etc with the ability to reorder them if necessary, similar to in the template editor you can re-order fields. Although this means I'd need to manually figure out what order things should be in, if I'm creating the code, even if it's via ProcessWire admin rather than the API, I should know what depends on what. It might simplify the layout of your UI a bit, as you'd simply have object type, object name, whether to add/update or remove it, although it would require some ajax callback if you want a lookup on the list of objects to make sure the object with the name you've specified actually exists as the type you've specified, although maybe not absolutely essential as currently you've just got a text field to add your object names.

You may have a specific reason for going this route, but first things first: Instagram normally prohibits this sort of use. They used to name "scraping" and "caching" specifically, but current version of their ToS just disallows "collecting information in an automated way without our express permission". So I'd say that while you may be able to do this technically, you probably shouldn't, especially if any of the data pulled from IG is going to be displayed publicly. You can of course request permission for this, but I'd wager that there's a very low chance of that working out ?‍♂️ Legal aspect aside, whether PW is a sensible platform for something like this would depend a lot on the specific case. For storing, displaying and searching through structured data? Sure, why not. Though in such a case it might make more sense to write the scraper in PHP as well, so you could just bootstrap PW. Alternatively the scraper could store data locally or in a DB, and PW would then read it from there... or you could provide an API endpoint at PW side ?

Admin Restrict Branch Select is an add-on / companion module for Admin Restrict Branch. With this module enabled, you can manually select more than one branch parent per user via the branch_parent field, and users with more than one option selected will be able to switch between those while editing site content. Switching between branches is done via a select field injected at the top of the page tree. Note that users are still limited to one branch at a time: this module will not make it possible to view multiple branches at the same time. When the module is installed, it will automatically update the branch_parent field if deemed necessary, i.e. if it isn't yet configured to allow selecting multiple pages. You can make changes to the field later, if you want to restrict selectable options by template, use asmSelect instead of PageListSelectMultiple, etc. This is an early beta release, so be sure to test carefully before enabling this module in production! https://processwire.com/modules/admin-restrict-branch-select/ https://github.com/teppokoivula/AdminRestrictBranchSelect

Thanks so much! I never found something useful like that. I guess I was using the wrong keywords

Good question. I doubt that MarkupSimpleNavigation does this, my guess goes more towards the template code it is used in. The "1" likes to appear when you accidentally output the return value of include(), most common when mixing up <?= and <?php (but "echo include('something.php')" has been encountered in the wild too). The "Array" points towards an echo $something where $something is an array. It might even be another <?= vs. <?php issue like this one: <?= $myvar = array(1,2,3); ?> So, to get to the culprit, check your PHP open tags first, then look for superfluous "echo" calls.

Thanks — looks like the modules directory defaults to master branch, while GitHub defaults to main instead. Fixed now.

Hi @bernhard - have you tried the "Clear Session & Cookies" option - I think that does what you are looking for.

The sequence is: remove (pages > templates > fields); add/change (fields > templates > pages). Within each type (e.g. pages), the sequence is in the order listed in the entry. I think this caters for most situations - e.g. a new page select field followed by a changed template, followed by pages in the right dependency order - but it may not cover everything (particularly making the uninstall happen correctly too). In mitigation, unsuccessful migrations can be reviewed via the previews (see example below) and more complex migrations can be broken into smaller migrations to better handle dependencies (I had thought of providing dependency linkage between migrations, but at the moment I think that adds too much complexity). Pic below shows an example preview with one 're-purposed' field and one new one - this can be reviewed before implementation and also afterwards if the install was not complete. I will take a closer look at that to see if it helps any. Quite! IMHO it is the main area where PW is lacking, particularly given that it is such a good tool for heavy-duty apps as well as just websites - a point that has been well made by @Kiwi Chris. I was surprised and a bit disappointed by how little attention this has had in terms of core modules. Not sure if I need to loop round like this (and also as @adrian indicates). Further testing may indicate that it is necessary. Well, they could but the tools are not complete (e.g. fields does not handle select options and pages does not exist except 'in development'); for anything but the simplest migration, you have to do several manual steps in the right order (coupled with working out what didn't implement correctly) - this can be a real pain if you are testing several times then needing to implement on the live version and can run a real risk of making mistakes when doing it for the n'th time; there is no documentation of what you have done; there is no ability to uninstall other than to repeat the manual steps correctly in reverse order. For very simple migrations (e.g. one changed template), then it is a feasible approach. I wanted something that installs fast and consistently every time - particularly when it is interdependent with changed code and I want to minimise downtime. Also, it is not just a case of 'not wanting to learn'. Code-based migrations need familiarity through frequent use and work best in the context of code-based app development where the same comment applies. I'll do a bit more testing and try and cover some of the points made above, then do this. Meanwhile, the main task is to refactor and restructure the code which is a truly horrible lot of spaghetti at the moment. If this is to go any further, I will then need some help, particularly with turning it into a more 'professional' module, as my PHP (particularly OOP) skills are limited and I have never previously done anything beyond simple Process modules .

Wow, looks like you have put a lot of work into that ? I think every step towards better PW migrations is important and very welcome ? https://github.com/BernhardBaumrock/RockMigrations/blob/bb43552f55ef7ff57533083f4d886c3aa00a8e41/RockMigrations.module.php#L1956-L2000 $rm->migrate([ 'fields' => [...], 'templates' => [...], 'pages' => [...], ]); Maybe I'm missing something, but I thought I had that problem too when developing the migrate() method and it turned out to be quite easy: I create fields, then templates, then I setup the fields and templates again (now that fields and templates exist in the system the references can properly be set) and finally I create all pages. In my scenarios this has worked perfectly for several months (years?) now ? I have to think about that sentence ? Maybe you could elaborate a little more on that? I try to understand your workflows better. I always thought that if somebody does not want to learn how to use code-based migrations he/she could simply use PW's import/export tools?! Do you think you could create a quickstart-screencast using some free tool like https://screencast-o-matic.com/home to show the workflow when using your migrations?

A question: Do new and changed fields and/or pages get processed in the order they're specified? I'm thinking of the scenario where a new page field is added that depends on a template and page that may also need to be added. That's one of the scenarios where using the built in field inport/export doesn't work too well if there are dependencies. It will advise you of them, but then you have to go back and import the dependencies then re-do the import, whereas if it's possible to specify order so that dependencies are met before an object is processed, it would make migrations more robust. This isn't an issue for declarative development via the API rather than the admin UI, as the developer can order their code so that dependencies are processed in order, but if changes are made via the admin UI this is a potential problem.

contributions.template.name=mytemplate Maybe add the name subfield? Wild guesss only!

I hope everyone is having a good week! Commits to the core this week include 10 issue report resolutions (so far). This will likely continue next week as well, and then I'll bump the version up then. Also included this week is an CKEditor upgrade from 4.14.0 to version 4.16.0. While that may sound like a minor CKEditor version bump, there's actually quite a list of updates in the CKEditor 4.x changelog, including a few security-related fixes, though none that I think are likely to affect PW users. I do still have a couple of core feature requests in progress as well, but there's more work still to do on those. Nothing too exciting this week, but I like to check in and say hello either way. I hope you all have a great weekend!

If you don't like to use a hook for this, there is a module: https://processwire.com/modules/process-page-list-multiple-sorting/

Agreed. Reading the last page of this thread I was surprised this wasn’t already the case. Could be as simple as just taking a custom selector string instead of the current current single page dropdown. I suppose the idea was to have something very robust for users, since it can show up in individual page settings.

Eine super Seite - nicht nur das Design, auch der Content // Awesome Site Marc!

// Should be automatically sorted like the page-table field. $page->pt_contents->get("template=part_text");

Doing a refresh in the modules section should be enough to get permissions working.