Jump to content


  • Content Count

  • Joined

  • Last visited

  • Days Won


MarkE last won the day on April 8

MarkE had the most liked content!

Community Reputation

128 Excellent

About MarkE

  • Rank
    Sr. Member

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

  1. EDIT: Don't try this just yet, there seems to be a slight bug! Version 0.0.4 on https://github.com/MetaTunes/ProcessDbMigrate To upgrade, place the files in the module folder and refresh modules. This version is partly a code tidy-up, but also adds 2 useful things to the module settings: A (collapsed) help field which contains the help.md text A feature which enables the current database to be named (e.g. Development, Test, Production, Client_1 or whatever). Migrations sourced from a named database will be treated as 'installable' in any database of a different name (or unnamed), but as 'exportable' in a database with the same name. This means, for example, that you can copy a production database as a new development database and rename it to be the same as your original development database, so that any migrations sourced from that development database will be shown as exportable, not installable, in the new database. You can also request that the current database name is notified in every admin page (in case you forget which environment you are in!). The feature is optional - if not used, any migrations will be treated as installable in every other database. The module now does pretty much everything I originally wanted - it just needs a bit more use to flush out remaining bugs. It is also possible that further field types may be needed - there are some slight imperfections with pagetables and I do not have pro fields, so can't test those. And I am sure the code can be improved πŸ˜‰
  2. Version 0.0.2 now on GitHub https://github.com/MetaTunes/ProcessDbMigrate This version more fully allows for different page ids in source and target systems. A meta value (idMap) maintains the mapping. This allows the replacement of links in RTE fields provided the relevant pages are all in the migration. Also, all existing image variants are migrated. EDIT: Now 0.0.3 fixes install problem and adds upgrade via modules -> refresh.
  3. Yeah, I did that and then came across the /> issue. Trouble is my diff method will report any diffs in the source unless I exempt them. I don't like it if you can't predict what will be returned. I think I'll stick with preg_replace for now since the parsing is very limited and see if it works out OK.
  4. There were other issues too, like '/>' vs '>' as the img tag end. Eventually I decided to ditch the DOMDocument and just use a simple preg_replace: protected function replaceImgSrc($html, $idMapArray) { if (strpos($html,'<img') === false) return $html; //return early if no images are embedded in html foreach ($idMapArray as $origId => $destId) { bd([$origId, $destId], 'Id pair'); $re = '/(<img.*\/files\/)' . $origId . '(\/.*>)/m'; $html = preg_replace($re, '${1}' . $destId . '$2', $html); } return $html; } Any reason @adrian why you went the DOMDocument route? I'll post an updated script to GitHub shortly, then maybe someone will find some holes in it!
  5. TBH, neither have I*. However, I have had to migrate pages, sometimes with images. For example, pages that are used to hold site settings. Also, I intended that the module might be used in 'rescue' mode as explained in the original post, which might involve migrating 'content'. Since the module does allow migrating pages, prompted by @adrian, I thought I would try and include RTE fields if I could. *Correction - I meant RTE fields with images. Even the migration pages themselves have an RTE field, but I hadn't expected to put images in it, although that is possible.
  6. @adrian, your suggestions have been invaluable! I think I have it working OK using ids - basically the 'new' pages all store a meta value for the related old page id so that mapping is possible (of course all pages with the source images must be included in the migration). That means that I only have to do one 'translation' - in the target system, replacing the old id's with the new ones. I used the code in your nameImagePathId() method for this - amended as required: protected function replaceImgSrc($page, $field, $idMapArray) { $files = $this->wire()->config->urls->files; $html = $page->$field; if (strpos($html,'<img') === false) return $html; //return early if no images are embedded in html $dom = new DOMDocument(); @$dom->loadHTML(mb_convert_encoding($html, 'HTML-ENTITIES', 'UTF-8')); foreach ($dom->getElementsByTagName('img') as $img) { $src = $img->getAttribute('src'); bd($src, 'Image src for ' . $page); $origId = basename(dirname($src)); $destId = (isset($idMapArray[$origId])) ? $idMapArray[$origId] : $origId; $img->setAttribute( 'src', $files . $destId . '/' . basename($img->getAttribute('src'))); bd($img->getAttribute('src'), 'reset img src'); } return preg_replace('/^<!DOCTYPE.+?>/', '', str_replace( array('<html>', '</html>', '<body>', '</body>'), array('', '', '', ''), $dom->saveHTML())); } $idMapArray is just an array of oldId => newId pairs. The only slight problem is that this introduces line breaks ( \n ) at the start and end of the html and I can't see why.
  7. Thanks @adrian. That's roughly what I was thinking of - except that in my case there is no zip. However, I was thinking it might be simpler to use the original page id, not the path. I'll work through your code and see how well it fits. Hope it's OK to use it (with credits!) if it works in my situation.
  8. Thanks for the tip, @adrian. In my tests to date, this hadn't proved to be an issue, because I was leaving the uploaded image in place. In other words, the image field on the page pointed to the folder for that (target) page id, but the RTE field pointed to the original (source) folder which I had left in place, so I didn't spot the problem. However as I said, I realise that there is a potential issue with leaving the original image in its folder. I was going to just delete it after the migration installation, but this yields several problems: RTE images, as you point out Uninstalling and re-installing won't work unless you reinstate the original If you use the development environment for testing by restoring different database versions then the original file is lost. So now I am trying to find a way of handling all these. BTW, I realise that the last one is not good practice as /files/id/ conflicts could arise anyway - but I have a personal problem in that, having moved to PHP7.0, my test environment needs an upgrade before I can use it again, so I was making do with using the dev environment as a test environment too. Currently my thinking is to use a special-purpose files directory within the migration package for the images to be uploaded, but I've yet to work that through.
  9. I have made a few minor amendments to the code at https://github.com/MetaTunes/ProcessDbMigrate, so anyone who has downloaded an earlier version might wish to update their copy. I found a few bugs with the image files which I've hopefully fixed - but there is a residual issue: Because the target database might have different page ids from the source database, the module uses page paths not ids for referencing. However images and files are stored in folders using ids. In order to migrate a page with files/images, it is necessary to upload the files in their related folders. The module will then put them in the right folder for the target system, but problems could arise if there is a page in the target system with images/files and its id is the same as the source page id. I'll scratch my head a bit over that one! I just used the module to migrate a site from my first prototype (see the OP) to this version and it worked fine.
  10. You can create/modify the fields/templates/pages however you like in the development environment. Then create the migration page by just defining what has been added/changed/deleted. Sure, but why bother (see the above)? Minor tweaks by hand are OK if the json file isn't quite what you want, but better to let the code generate it. My module does not track changes - that can get very messy. You just define the scope of changes (in the right dependency order) and it picks up the current state - not how it got there. Perhaps this might work better than the real-time hooking. A suitable compromise that might be workable is a separate component that just logs what has changed since last time (without knowing how). That could then build a draft migration page, but the sequence may need to be hand-sorted as getting the system to work out the dependencies could be tricky. In theory, you could use json files to snapshot the whole database and then take diffs from that to create the migration page, but that could be pretty resource-intensive - at the very least you would want to restrict the page tree to exclude user pages which are not maintained in the dev. It definitely has that advantage, provided you took a snapshot before you started work on the changes. On the other hand (a) it is a good idea to document what you are doing πŸ˜‰ and (b) if you are working on 2 or more sets of (disjoint) changes, your approach would bundle them as one. So while it may be a good idea (and maybe achievable as per the above comments), you would definitely want it to be optional - e.g. have a button "Create migration page from snapshot". I couldn't agree more, and would appreciate @ryan's take on this. It is the only thing about PW that irritates me. I'm not sure he would agree - the whole import/export stuff seemed to have been left unfinished years ago - for example this. And by all means look at my code, but you might wish to wear gloves πŸ˜…
  11. For those that like a screencast, I hope this helps (I've broken it down into logical steps): Install the module in your development environment (making sure you have FieldtypeRuntimeOnly installed first). Then open the "Database Migrations" setup page and refresh it. You will see that it has automatically installed a 'bootstrap' migration. Install.mp4 Create a new migration page to hold the scope definition of your new migration - just enter the basic details and save it at this stage. New_migration.mp4 Make the changes you want in the development environment (of course, you may have already done this πŸ™‚ ). We will add a new page and a couple of children. New_pages.mp4 Then a couple of fields (one a page ref with the new page as parent) and a template. Fields_template.mp4 Add a page using the new template. Snafu.mp4 Now go back to the migration page and define the affected elements. Use the preview to see the effect, then "export" the migration if you are happy. Export.mp4 The next step is to install the new migration in the target environment. Sync the code files (including the .json files created by the migration and any new images/files in assets/), install ProcessDbMigrate in the target if necessary and go to the setup page. You can preview the migration before installing it. Install_migration.mp4 If necessary, you can uninstall the migration (and the module), but the code files will remain. Uninstall.mp4 End of show!
  12. Code is now posted at https://github.com/MetaTunes/ProcessDbMigrate Many thanks once more to those who gave me ideas and help in doing my first 'proper' module - in particular @Kiwi Chris, @bernhard and @adrian. Please test and feedback. But be gentle in your criticism of the code 😜. Most importantly - this is a proof of concept only at this stage and it makes changes to your files and databases so please do not use on production sites and do back up everything beforehand - use at your own risk! To install: Place the ProcessDbMigrate folder in your site/modules directory. Make sure your environment meets the requirements - you need @Robin S's FieldtypeRuntimeOnly to be installed first. The earliest PW version I have tested it with is 3.0.148, but it might work on earlier 3.0.xxx versions. Please let me know if it works with earlier versions. Having satisfied the dependencies, install the module. You will see that there is an extensive help.md file - please read it, particularly if you get stuck. @bernhard asked for a screencast - I will do that next - hopefully it will make things clearer.
  13. Snap! The website is here but that is like the tip of the iceberg. Behind the scenes (i.e. in the admin) it does membership records, mailings, events management, news reporting, subscriptions etc. No credit card facility - we use direct debits courtesy of GoCardless instead - cheaper and more secure. Members access their details, book events etc. via the My NCOG page (no passwords - we use a one-time email token instead as we have a slightly senior group who are not reliable password users πŸ˜…). There are still quite a lot of rough edges, but it has been running for a couple of years now and does the business.
  14. I think this is an important topic and @pideluxe makes some useful suggestions. The subsequent discussion provides insights into what is attractive about PW and what people think is missing. I am a relative newcomer to PW and am only an amateur - in two senses: (a) as a retiree, I can do stuff for for friends and family without commercial pressures and (b) I have never had any training (and it shows πŸ˜‰ ). What attracted me to PW was the ability to build custom apps integrated into a website and give others the ability to manage the content. Having used WP (aargh - customisation is a nightmare and the admin is really clunky), CodeIgniter (well-designed, but hard work!) and some simple CMS solutions, I wanted something that was easy to use and took out a lot of the hard coding work, but was flexible and capable. PW ticked all the boxes. As well as straight CMS, I have now built two quite complex apps - a club management system and a self-catering cottage management system. PW has been a great tool for the job. So the question is, why is it not more widely known and used? I think there are two issues, both of which have been mentioned by others: The project does appear to be very reliant on Ryan (albeit maybe less than previously owing to some of the excellent contributors to the ecosystem). This was not a problem for me (in fact a positive, because it has meant a clear vsion for the project), but is a negative point for some. My daughter, who is tech director of an e-marketing company, prefers WP even though it is technically inferior because of the greater assurance of continuity and the larger pool of people with relevant skills. The OP suggests some ways in which this perception could be changed, but I suspect that may not be possible without @ryan's active involvement. Over time, the user group has become more technical so that the appeal is narrowing. In some ways, this is a consequence of PW's success in enabling some pretty sophisticated sites/apps - this has attracted technophiles whose needs have further driven it in this direction. This has happened without any active decision on anyone's part. Some may feel very comfortable with this and think that it is a perfectly viable niche. I am not so sure - the risk is that the ease-of-use aspects are increasingly downplayed so that ultimately the comparison is with, say, Laravel, rather than WP. The bottom line is that PW's initial attraction of being both a 'simple' CMS and a sophisticated app-building tool risk it ending up being between a rock and a hard place. That would be a great shame because IMHO it is better, technically, than the competition. You may infer from my comments above that I think that part of the 'problem' is that the project is largely technically-driven - inevitable given the nature of those involved. This is not a critcism, just a fact of life. But perhaps the community needs to get more input from outside and think about PW's marketplace more broadly. FWIW, I agree with many of the suggestions above, including Pretty much all of @OllieMackJames's initial post. BTW, this one - is one of the few things that actually irritates me about PW, which is why I am trying to do something about it (see https://processwire.com/talk/topic/25307-oh-no-not-another-migration-module/, but I really think something like this should be in the core).
  • Create New...