MarkE

Horses for courses, I think @opalepatrick. PW is undoubtedly an excellent tool for admin-type applications. I have done a couple (a membership system with an integrated public website and a holiday cottage booking system which is loosely-coupled to a public website) and have found that it works well. However, you do need to think about the design quite carefully if you want it to perform elegantly and be maintainable. So far I haven't found any really good tutorials on this topic. The systems I have built tend to have the following characteristics: Use PW's page-based structure as far as possible for the database and straightforward edits; Use existing modules that suit the purpose to avoid re-inventing the wheel - I find ConnectPageFields and RuntimeMarkup/RuntimeOnly particularly useful; Use process modules for custom user interfaces - typically using MarkupAdminDataTable; Use the access control tools as required - I also use the RestrictTabView module; Use Listerpro for ad-hoc reporting; Tweak the interface as necessary with js. Inevitably you will take some wrong turnings, but PW is a great prototyping tool and genrally not too much time is wasted (certainly not compared with doing the design wrong in pure php). And the forum is great too, if your brain starts to hurt, as mine seems to frequently ?

Just a thought- have you tried setEncodeEntities(false) before writing the rows?

Version is now 0.0.13 - bug fixes and enhanced handling of image/file links in RTE fields. BTW, @teppo, in PW weekly, characterises this module thus: "DbMigrate focuses more on migrating page content, while RockMigrations is more about the structure of the site". I'm not sure this is really correct: while I have made an effort to deal with page migrations, the module was not designed to be used for mass page migrations. The driver behind including page as well as template and field migrations was to facilitate migrations that cut across code, database structure and pages - typically involving settings-type pages which are maintained by the developer, not general users. If it assists in other page migrations as well then that is a beneficial side-effect, but be aware that I have only tested the page migrations with a limited set of field types (and no pro fields).

Latest version on github is v0.0.12. This fixes a number of bugs and also adds 2 new features: When a migration has been 'locked', so that it can no longer be changed/uninstalled etc., the 'preview' button is replaced with 'review'. The review shows all the changes made by the migration (on the assumption it was properly installed before locking); it is completely derived from the json files (which should be kept unchanged on the system) and therefore is completely independent of the current database state. BTW, the purpose of locking is to prevent inadvertant overwriting of other migrations if there is an overlapping scope (although warning messages are issued if this is a risk). Hopefully it is now (mostly) multi-language enabled. If anyone needs this and finds something is missing, let me know. Obviously you will need to add your own translations ? I've been using the module a fair bit and am pretty happy with it so far. It has been very useful when updating help pages (using @Macrura's excellent AdminHelp module) on the dev system - I can then release them as a consistent batch to the live system. I have also found that it is a good way of migrating language pages as it will migrate the language .json files automatically with the pages - all that's needed is a migration item with selector "template=language".

v0.0.6 allows page selectors that result in multiple levels. The use of “sort=path” is permitted in selectors, even though this normally has no effect. If it is used, the pages will be sorted in parent-child order, using the ‘natural’ sort order (e.g. XYZ100 is greater than XYZ20). This means that parents should be installed before children. For ‘removed’ pages, the order is reversed so that children are deleted before parents.

Quite easy, actually - just replace the return statement in the snippet with return strnatcmp($a->path, $b->path);

Thanks @Robin S. I installed PagePaths, but sort=path still didn't seem to work. By way of context, I was testing out my new DbMigrate module. I realised that if a page selector was used which resulted in multiple levels then the pages would need to be sorted in parent-child order before exporting to the json file, so that the subsequent import didn't try and load parents before their children. While my test may only have had 2 levels, in theory there could be any number of levels. "sort=parent.sort, sort=sort" didn't seem to do the business either. I notice that in @adrian's ProcessMigrator module, he just counts the number of path segments and then sorts by that, so that all parents are loaded first and then all children, then all grandchildren etc. I wanted the children to follow on immediately from their parents in the json which makes for a more human-readable output. (Why bother? My plan was to use the DbMigrate module as a way of exporting pages such that they could in theory be loaded in a different environment, possibly not even PW ?. In doing this, I find being able to visually review the data is helpful. Also, it seems more logical and more likely to fit a subsequent import process). I'm puzzled as to why PagePaths doesn't work, but my snippet seems to do the business - although it is a bit of a hack, it may be the best way to go as I would ideally like to refine the sorting so that numeric elements of strings are fully respected - eg. /bkg20/ should be before /bkg101/.

FWIW, I came across a similar problem with my DbMigrate module. In my case, it happened if the admin root was not just /processwire/. My solution was to use $pages->get(2) rather than $urls->admin for the admin root. This may be completely irrelevant in your case, of course ?

This seems to work (N.B. this is with a php array not a PageArray) $pages = $this->wire()->pages->find($selector); $pages = $pages->getArray(); // want them as a php array not an object if (strpos($selector, 'sort=path')) { usort($pages,function ($a, $b) { return ($a->path >= $b->path) ? 1 : -1; }); }

I mean path as in url (which doesn't work either). path is a property of page so, per the docs, should be available in sort=, but doesn't seem to be. Looks like I'll need to do the sorting after the selection ? Thanks for the comments.

I'm trying to get them in parent-child order. There are two levels below the 'has_parent' and I wanted to get each parent followed by its children. I thought that sorting by path would achieve that but, as you say, maybe its not available in a selector. Strange, because the docs say "This behavior can be overridden by specifying your own sort=property (where property is any property or field name)" (my emphasis). sort=sort doesn't work for 2 levels like this.

I have a selector, e.g. "has_parent=/property/bawd-hall/bookings/, sort=path" The results (in this case several hundred) are not sorted by path, they are sorted by id. Changing 'path' to 'name' or any other property/field works fine. It makes no difference what the first part of the selector is (I've tried several) - the sort=path always returns a sort order of id. No doubt I am being dim, but any ideas?

Hopefully the bugs in 0.0.4 have been fixed in v0.0.5 FWIW, the issue was to do with the operation of hooks when adding pages via a migration. The module was designed for migrating developments, not for mass updating of user pages. It was therefore assumed that any pages being migrated would be of a ‘site-settings’ nature, not user pages. However, the module allows the migration of any pages and the host PW application may make use of page hooks. All page actions in the module allow hooks to run. To enable users to modify this behaviour, session variables are set for the duration of the following methods: • installPages() – new/changed pages – ‘dbMigrate_installPages’ is set to true • removeItems() – all item removals – ‘dbMigrate_removeItems’ is set to true These can then be referenced in the application code as required. Note that in removals, all pages are trashed (so that hooks can operate) then deleted (so they are no longer in the trash). So it is possible to use the module more generally (e.g. in ‘rescue’ mode) but test carefully first! Please report any bugs (or indeed successful use). I'd also be grateful for feedback (particularly from module developers @adrian and @bernhard, who have previously commented, but also from any other interested persons) on whether this should be added to the modules library and, if so, whether anything needs to be done to it first.

EDIT: Don't try this just yet, there seems to be a slight bug! Version 0.0.4 on https://github.com/MetaTunes/ProcessDbMigrate To upgrade, place the files in the module folder and refresh modules. This version is partly a code tidy-up, but also adds 2 useful things to the module settings: A (collapsed) help field which contains the help.md text A feature which enables the current database to be named (e.g. Development, Test, Production, Client_1 or whatever). Migrations sourced from a named database will be treated as 'installable' in any database of a different name (or unnamed), but as 'exportable' in a database with the same name. This means, for example, that you can copy a production database as a new development database and rename it to be the same as your original development database, so that any migrations sourced from that development database will be shown as exportable, not installable, in the new database. You can also request that the current database name is notified in every admin page (in case you forget which environment you are in!). The feature is optional - if not used, any migrations will be treated as installable in every other database. The module now does pretty much everything I originally wanted - it just needs a bit more use to flush out remaining bugs. It is also possible that further field types may be needed - there are some slight imperfections with pagetables and I do not have pro fields, so can't test those. And I am sure the code can be improved ?

Version 0.0.2 now on GitHub https://github.com/MetaTunes/ProcessDbMigrate This version more fully allows for different page ids in source and target systems. A meta value (idMap) maintains the mapping. This allows the replacement of links in RTE fields provided the relevant pages are all in the migration. Also, all existing image variants are migrated. EDIT: Now 0.0.3 fixes install problem and adds upgrade via modules -> refresh.

Yeah, I did that and then came across the /> issue. Trouble is my diff method will report any diffs in the source unless I exempt them. I don't like it if you can't predict what will be returned. I think I'll stick with preg_replace for now since the parsing is very limited and see if it works out OK.

There were other issues too, like '/>' vs '>' as the img tag end. Eventually I decided to ditch the DOMDocument and just use a simple preg_replace: protected function replaceImgSrc($html, $idMapArray) { if (strpos($html,'<img') === false) return $html; //return early if no images are embedded in html foreach ($idMapArray as $origId => $destId) { bd([$origId, $destId], 'Id pair'); $re = '/(<img.*\/files\/)' . $origId . '(\/.*>)/m'; $html = preg_replace($re, '${1}' . $destId . '$2', $html); } return $html; } Any reason @adrian why you went the DOMDocument route? I'll post an updated script to GitHub shortly, then maybe someone will find some holes in it!

TBH, neither have I*. However, I have had to migrate pages, sometimes with images. For example, pages that are used to hold site settings. Also, I intended that the module might be used in 'rescue' mode as explained in the original post, which might involve migrating 'content'. Since the module does allow migrating pages, prompted by @adrian, I thought I would try and include RTE fields if I could. *Correction - I meant RTE fields with images. Even the migration pages themselves have an RTE field, but I hadn't expected to put images in it, although that is possible.

@adrian, your suggestions have been invaluable! I think I have it working OK using ids - basically the 'new' pages all store a meta value for the related old page id so that mapping is possible (of course all pages with the source images must be included in the migration). That means that I only have to do one 'translation' - in the target system, replacing the old id's with the new ones. I used the code in your nameImagePathId() method for this - amended as required: protected function replaceImgSrc($page, $field, $idMapArray) { $files = $this->wire()->config->urls->files; $html = $page->$field; if (strpos($html,'<img') === false) return $html; //return early if no images are embedded in html $dom = new DOMDocument(); @$dom->loadHTML(mb_convert_encoding($html, 'HTML-ENTITIES', 'UTF-8')); foreach ($dom->getElementsByTagName('img') as $img) { $src = $img->getAttribute('src'); bd($src, 'Image src for ' . $page); $origId = basename(dirname($src)); $destId = (isset($idMapArray[$origId])) ? $idMapArray[$origId] : $origId; $img->setAttribute( 'src', $files . $destId . '/' . basename($img->getAttribute('src'))); bd($img->getAttribute('src'), 'reset img src'); } return preg_replace('/^<!DOCTYPE.+?>/', '', str_replace( array('<html>', '</html>', '<body>', '</body>'), array('', '', '', ''), $dom->saveHTML())); } $idMapArray is just an array of oldId => newId pairs. The only slight problem is that this introduces line breaks ( \n ) at the start and end of the html and I can't see why.

Thanks @adrian. That's roughly what I was thinking of - except that in my case there is no zip. However, I was thinking it might be simpler to use the original page id, not the path. I'll work through your code and see how well it fits. Hope it's OK to use it (with credits!) if it works in my situation.

Thanks for the tip, @adrian. In my tests to date, this hadn't proved to be an issue, because I was leaving the uploaded image in place. In other words, the image field on the page pointed to the folder for that (target) page id, but the RTE field pointed to the original (source) folder which I had left in place, so I didn't spot the problem. However as I said, I realise that there is a potential issue with leaving the original image in its folder. I was going to just delete it after the migration installation, but this yields several problems: RTE images, as you point out Uninstalling and re-installing won't work unless you reinstate the original If you use the development environment for testing by restoring different database versions then the original file is lost. So now I am trying to find a way of handling all these. BTW, I realise that the last one is not good practice as /files/id/ conflicts could arise anyway - but I have a personal problem in that, having moved to PHP7.0, my test environment needs an upgrade before I can use it again, so I was making do with using the dev environment as a test environment too. Currently my thinking is to use a special-purpose files directory within the migration package for the images to be uploaded, but I've yet to work that through.

I have made a few minor amendments to the code at https://github.com/MetaTunes/ProcessDbMigrate, so anyone who has downloaded an earlier version might wish to update their copy. I found a few bugs with the image files which I've hopefully fixed - but there is a residual issue: Because the target database might have different page ids from the source database, the module uses page paths not ids for referencing. However images and files are stored in folders using ids. In order to migrate a page with files/images, it is necessary to upload the files in their related folders. The module will then put them in the right folder for the target system, but problems could arise if there is a page in the target system with images/files and its id is the same as the source page id. I'll scratch my head a bit over that one! I just used the module to migrate a site from my first prototype (see the OP) to this version and it worked fine.

You can create/modify the fields/templates/pages however you like in the development environment. Then create the migration page by just defining what has been added/changed/deleted. Sure, but why bother (see the above)? Minor tweaks by hand are OK if the json file isn't quite what you want, but better to let the code generate it. My module does not track changes - that can get very messy. You just define the scope of changes (in the right dependency order) and it picks up the current state - not how it got there. Perhaps this might work better than the real-time hooking. A suitable compromise that might be workable is a separate component that just logs what has changed since last time (without knowing how). That could then build a draft migration page, but the sequence may need to be hand-sorted as getting the system to work out the dependencies could be tricky. In theory, you could use json files to snapshot the whole database and then take diffs from that to create the migration page, but that could be pretty resource-intensive - at the very least you would want to restrict the page tree to exclude user pages which are not maintained in the dev. It definitely has that advantage, provided you took a snapshot before you started work on the changes. On the other hand (a) it is a good idea to document what you are doing ? and (b) if you are working on 2 or more sets of (disjoint) changes, your approach would bundle them as one. So while it may be a good idea (and maybe achievable as per the above comments), you would definitely want it to be optional - e.g. have a button "Create migration page from snapshot". I couldn't agree more, and would appreciate @ryan's take on this. It is the only thing about PW that irritates me. I'm not sure he would agree - the whole import/export stuff seemed to have been left unfinished years ago - for example this. And by all means look at my code, but you might wish to wear gloves ?

For those that like a screencast, I hope this helps (I've broken it down into logical steps): Install the module in your development environment (making sure you have FieldtypeRuntimeOnly installed first). Then open the "Database Migrations" setup page and refresh it. You will see that it has automatically installed a 'bootstrap' migration. Install.mp4 Create a new migration page to hold the scope definition of your new migration - just enter the basic details and save it at this stage. New_migration.mp4 Make the changes you want in the development environment (of course, you may have already done this ? ). We will add a new page and a couple of children. New_pages.mp4 Then a couple of fields (one a page ref with the new page as parent) and a template. Fields_template.mp4 Add a page using the new template. Snafu.mp4 Now go back to the migration page and define the affected elements. Use the preview to see the effect, then "export" the migration if you are happy. Export.mp4 The next step is to install the new migration in the target environment. Sync the code files (including the .json files created by the migration and any new images/files in assets/), install ProcessDbMigrate in the target if necessary and go to the setup page. You can preview the migration before installing it. Install_migration.mp4 If necessary, you can uninstall the migration (and the module), but the code files will remain. Uninstall.mp4 End of show!

Code is now posted at https://github.com/MetaTunes/ProcessDbMigrate Many thanks once more to those who gave me ideas and help in doing my first 'proper' module - in particular @Kiwi Chris, @bernhard and @adrian. Please test and feedback. But be gentle in your criticism of the code ?. Most importantly - this is a proof of concept only at this stage and it makes changes to your files and databases so please do not use on production sites and do back up everything beforehand - use at your own risk! To install: Place the ProcessDbMigrate folder in your site/modules directory. Make sure your environment meets the requirements - you need @Robin S's FieldtypeRuntimeOnly to be installed first. The earliest PW version I have tested it with is 3.0.148, but it might work on earlier 3.0.xxx versions. Please let me know if it works with earlier versions. Having satisfied the dependencies, install the module. You will see that there is an extensive help.md file - please read it, particularly if you get stuck. @bernhard asked for a screencast - I will do that next - hopefully it will make things clearer.