adrian

Migrator

Recommended Posts

What do you see in your 1022 folder if you move the exit down to after line 317 and then again after 318?

I am assuming there is something in the create_zip function that isn't working on your local install. Maybe it's the ZIPARCHIVE::OVERWRITE?

exit after line 316  /1203/data.json & files.zip

exit after line 317 /1203/data.json & files.zip plus this message: Warning: Illegal string offset 'currentpath' in  :\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2100

exit after line 318 /1203/files.zip only no json file plus the same warning mssg as above.

Each time I deleted the files in /1203/ before running the next test.

Share this post


Link to post
Share on other sites

@tobaco - thanks for the feedback - obviously I need to look further into users/roles migrations. Your exported json is showing that the home page is created by a user named "cms" with no role. Looking at the created_users_id for the home page, this makes sense. I'll try to work on this shortly, along with the rest of the multi-language issues.

@NooseLadder - thanks for that - I have pushed an update which I hope will fix things for you - can you please let me know how it goes.

Share this post


Link to post
Share on other sites

Hi Adrian,

Thanks for the update but this still does not work. Installed version 0.1.2.

It seems that the json and zip files are being created in the folder but the json is not being added to the zip prior to download.

I commented out line 320

//unlink($this->session->jsonFilename);

The zip file is downloaded without the json file included and the json file remains in the files/folder.

If you want me run any more tests please let me know.

Share this post


Link to post
Share on other sites

Hi This is the result:

Array ( [currentpath] => C:/xampp/htdocs/mysite/site/assets/files/1062/data.json [newpath] => C:/xampp/htdocs/mysite/site/assets/files/1062/data.json )

Share this post


Link to post
Share on other sites

I am thinking this might be a Windows vs Unix slash/path issue, but I am not exactly sure where. Does xammp translate forward slashes to back slashes for disk paths - is that why the results you just showed have forward slashes? But then if that was the problem then you'd think it would also affect the template files and images in the zip as well. Maybe it's time I set up an xampp installation on my Windows VM.

Please grab the latest version from Github and try adding the following after line 2120:

error_log('current:'.$file.'|new:'.pathinfo($file, PATHINFO_BASENAME));

and let me know what ends up in your log.

In other news, the latest version should now solve the role issue that @tobaco was having with system created pages and also it should now work properly when attempting to migrate a site with the Home page set as the parent. You can now import Home into Home and it should work properly, allowing full migration of an entire site's page tree - at least something productive came of my efforts today :)

Share this post


Link to post
Share on other sites

Thanks again. If nothing was logged that suggests that $valid_files is empty, or that create_zip never gets called for the json file, but I can't see how that would be possible.

Would you mind playing around a little more inside that create_zip function and try to figure out the content of $valid_files when trying to add the json file to the zip. Try this after 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));

Also, I see that you edited your post since initially posting. Those errors you did get are strange - seems like you might be trying to export a page with no template. Do you have any ideas on those errors?

Thanks again for your help with this. I hope it doesn't turn out to be some version specific bug in xammp :)

  • Like 1

Share this post


Link to post
Share on other sites

Another significant update this morning which corrects a major omission in critical functionality. Migrator now handles migration of template context field settings.

Now to tackle these new multi-language issues :)

  • Like 1

Share this post


Link to post
Share on other sites

Thanks again. If nothing was logged that suggests that $valid_files is empty, or that create_zip never gets called for the json file, but I can't see how that would be possible.

Would you mind playing around a little more inside that create_zip function and try to figure out the content of $valid_files when trying to add the json file to the zip. Try this after 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));

Also, I see that you edited your post since initially posting. Those errors you did get are strange - seems like you might be trying to export a page with no template. Do you have any ideas on those errors?

Thanks again for your help with this. I hope it doesn't turn out to be some version specific bug in xammp :)

Hi Adrian,

I will play around and report back. The error I deleted I think were due to me not selecting a parent. I'll test again and let you know. Thanks.

BTW, What do you use as a localhost server?

You have to forgive my ignorance here: All this time I have been looking in the wrong error_log file. I have now found the correct one and will go through the lines.

Edited by NooseLadder

Share this post


Link to post
Share on other sites

I am thinking this might be a Windows vs Unix slash/path issue, but I am not exactly sure where. Does xammp translate forward slashes to back slashes for disk paths - is that why the results you just showed have forward slashes? But then if that was the problem then you'd think it would also affect the template files and images in the zip as well. Maybe it's time I set up an xampp installation on my Windows VM.

Please grab the latest version from Github and try adding the following after line 2120:

error_log('current:'.$file.'|new:'.pathinfo($file, PATHINFO_BASENAME));

and let me know what ends up in your log.

In other news, the latest version should now solve the role issue that @tobaco was having with system created pages and also it should now work properly when attempting to migrate a site with the Home page set as the parent. You can now import Home into Home and it should work properly, allowing full migration of an entire site's page tree - at least something productive came of my efforts today :)

After finding the correct error_log file I reran this line and got the following:

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Warning:  Illegal string offset 'currentpath' in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

And the same set of errors after running this after line 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));
Edited by NooseLadder
  • Like 1

Share this post


Link to post
Share on other sites

Hi Adrian,

I managed to get this to work by comment out the following on line 2106:

$valid_files = array();
        if(is_array($files)) {
            foreach($files as $file) {
                //if(file_exists($file['currentpath']) || file_exists($file[0]))  /* <-- This line */
$valid_files[] = $file;
            }
        }

I've tried an import into another site and it works  :-)

There may be implications I'm not aware of by doing this?

Edit: Just tried with latest version 0.1.9 and that works too. Line 2106 in version 0.1.6 is now line 2144 in version 0.1.9

I did get this warning after the import to the new site both in versions 0.1.6 and 0.1.9:

Warning: rmdir(C:/xampp/htdocs/mysite/site/assets/files/1022/migratorfiles): Directory not empty in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2182
Edited by NooseLadder
  • Like 2

Share this post


Link to post
Share on other sites

@NooseLadder - thanks so much for your help with this. So if things are working with that file_exists check removed then I think we can assume it's a windows path issue. Weird that the addfile to the zip works, but file_exists fails. I am curious about the Directory not empty warning. Does the migratorfiles directory actually get removed? Again, this seems like a windows path issue because I am using a recursive directory delete function and it works here. I really might need that windows xammp setup to get all these sorted out. But for now, I have removed the file_exists check from the latest committed version as I don't really think it should be necessary as the $files array shouldn't contain any files that don't exist anyway.


Another big enhancement this morning - I think all multi-language features should now be working!

@tobaco - would you please check to see if the latest version fixes all the issues you were having with multi-language page names etc?

  • Like 3

Share this post


Link to post
Share on other sites

Hi Adrian,

I'm happy to help, thanks.

The folder C:/xampp/htdocs/mysite/site/assets/files/1022/migratorfiles is not there. The folder C:/xampp/htdocs/mysite/site/assets/files/1022/ contains some images that I uploaded for a specific page, a thumbnail and the main image. I don't understand why that folder was used to append the migratorfiles folder, maybe because it is the last folder listed in the in the $files array?

  • Like 1

Share this post


Link to post
Share on other sites

Glad to hear the migratorfiles temp folder is being deleted as expected. I am also not sure why 1022 contains some images. The numbered folder that migratorfiles should be saved to is the one associated with the Migrator module page. If you browse your page tree: Admin > Setup > Migrator > edit that should show ?id=xxxx

The id should be the folder where migratorfiles is being created. Do these match, or does 1022 really belong to another page in your site's tree?

Share this post


Link to post
Share on other sites

I have just finalized support for all Profields field types: Table, PageTable, Multiplier, and Texareas

Please check out the latest version and test away.

I think this completes enough functionality now that I am ready to release this to the modules directory once I get some feedback to verify that multi-language functionality and the Profields additions are working well. 

The official release will probably still be "alpha" for a little while, but hopefully getting it in the modules directory will get some more people testing!

  • Like 1

Share this post


Link to post
Share on other sites

@adrian

sorry for the delay, i just saw your post.

i tried to import the the data, but got the following error right after submitting the form:

Recoverable Fatal Error: Object of class stdClass could not be converted to string (line 38 of /cms/wire/modules/LanguageSupport/FieldtypeTextLanguage.module) 

another thing:

is it possible to point the importer to an already uploaded (e.g. via ftp) file instead of uploading the file via the form.

i think most hosting environments will struggle with uploading large files.

THANKS!

  • Like 1

Share this post


Link to post
Share on other sites

Glad to hear the migratorfiles temp folder is being deleted as expected. I am also not sure why 1022 contains some images. The numbered folder that migratorfiles should be saved to is the one associated with the Migrator module page. If you browse your page tree: Admin > Setup > Migrator > edit that should show ?id=xxxx

The id should be the folder where migratorfiles is being created. Do these match, or does 1022 really belong to another page in your site's tree?

Hi Adrian,

Admin>Setup>Migrator>edit ?id=1062

1022 belongs to another page in the tree and contains the images for that page.

Share this post


Link to post
Share on other sites

@tobaca,

I just tested the json file you sent me last week and got the same error. If that is the file you are still using, then that is the reason. You need to export again with the new version of Migrator so that the json includes the new multi-language information. Could you please try that and let me know how it goes. If it still doesn't work, please send me your newly exported json file. Also, I have a think about pointing to an already uploaded file.

@NooseLadder - that is definitely strange - it is making me think that: $this->page->filesManager()->path() is somehow not working for you because it doesn't make any sense that this is not pointing to Migrator's admin page id. Anyone else out there running xammp on Windows who feels like testing this ?

Share this post


Link to post
Share on other sites

......that is definitely strange - it is making me think that: $this->page->filesManager()->path() is somehow not working for you because it doesn't make any sense that this is not pointing to Migrator's admin page id. Anyone else out there running xammp on Windows who feels like testing this ?

Tell me what to do and I'll test. There's 9 pages in this thread and I don't know where to start  :)

Share this post


Link to post
Share on other sites

@tobaco - that is one serious test case of an import file :)

It's been a mission, but I have narrowed down the problem to the "content_variants" repeater field. If you use the Edit Imported Content option and remove that field from the list, it imports everything else just fine. See image of page tree :)

post-985-0-09070500-1404275990_thumb.png

Migrator normally works just fine with repeater fields, but I see that you have a page field in the repeater field's template, which is something I haven't tested before, especially together with the muit-language element. Not sure exactly what the issue is yet, so I need to investigate more.

For now, I'd appreciate it if you could confirm on your end that the import works if you omit that repeater field during the import, just so I know we are experiencing exactly the same results.

You will need the latest version of Migrator from github as I had to set PHP's max_execution_time because your import was so complex.

Thanks for continuing to help out with this - getting real life tests like this is very valuable.

@kongondo - thanks for offering. You are right, this thread is a behemoth - I will start a new one as soon as I am ready to release, but for now to test:

  1. Grab the module from: https://github.com/adrianbj/ProcessMigrator and install
  2. Make sure debug mode is on
  3. Run the module (Setup > Migrator)
  4. Choose Export
  5. Choose a Parent Page - to keep things simple, maybe something like the default PW's About page
  6. Export
  7. Check that the downloaded zip file contains a data.json file
  8. Let me know if you get any path related errors like @NooseLadder
  9. If you're feeling energetic, comment out line 335: //unlink($zipFilename);
  10. Run again and find out the id of the assets/files folder where the zip is stored. It should match the id of the Migrator Admin > Setup > Migrator page id, but some reason @NooseLadder is seeing it in one of his normal page id folders. Seems like it's a windows/xampp issue.

Thanks so much!

Share this post


Link to post
Share on other sites

@adrian

thanks for your hard work!

I tried the import omitting the repeater field an got the following response:

Bildschirmfoto%202014-07-02%20um%2009.02

That happened, when i tried to replace the content.

After that I deleted alle the pages I already had in the import system and imported again. I got a similar message and no pages were imported.

After a second try, some pages but not all got imported. It always displayed the »This page may not be deleted« message.

When i tried »Append« instead of »Replace« the import seems to work, I got this screen:

Bildschirmfoto%202014-07-02%20um%2009.22

I hope this helps to narrow down the issue.

Thanks again!

  • Like 1

Share this post


Link to post
Share on other sites

Thanks again for testing. I should have mentioned to try on a clean install. I believe the "This page may not be deleted" message is referring to various system pages. The replace method deletes all children of the Import Parent Page before creating them again. Append or Overwrite doesn't do this. This is not an issue when migrating page trees below home, but if you choose home, then http404 and all the pages under Admin are selected, so obviously I need a check to deal with system pages that are not deleteable to prevent the error.

I'll also look further into the icon_select error - I saw this a few times too - for some reason that page field is not being fully configured during the import, even though the other page fields are importing perfectly.

I am glad to see the 196 pages imported though - that looks like the full set :)

I don't really have any time for this for the next few days, but hopefully I can get these things sorted out after that.

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By netcarver
      Part 1 of a 2 part Module & Service Reveal.
      I'm currently working on a new module: ModuleReleaseNotes that was inspired by the work I originally did on making Ryan's ProcessWireUpgrades module "release" aware. In the end, I decided to ditch the approach I was originally taking and instead work on a module that hooked in to the UpgradeConfirmation dialog and the module edit page.
      Aims
      My aims for this module are as follows...
      Make discovery of a module's changes prior to an upgrade a trivial task. Make breaking changes very obvious. Make reading of a module's support documentation post-install a trivial task. Make module authors start to think about how they can improve the change discovery process for their modules. Make sure the display of information from the module support files/commit messages doesn't introduce a vulnerability. Looking at these in turn...
      Making discovery of a module's changes prior to upgrade a trivial task.
      This is done by adding a "What's changed section" to the upgrade confirmation dialog.  This section takes a best-effort approach to showing what's changed between the installed version and the updated version that's available via the module repository.
      At present, it is only able to talk to github-hosted repositories in order to ask them for the release notes, the changelog file (if present) and a list of commits between the git tag that matches the installed version and the tag matching the latest version.
      It will display the Release Notes (if the author is using the feature), else it will display the commits between the tags (if tagging is used by the module author) else it will show the changelog file (if present) else it will show the latest N commits on the master branch (N, of course, being configurable to your liking.)
      An example of the Github Release Notes pulled in for you, taken from Mike Rockett's TextformatterTypographer Module...

      An example of a tag-to-tag commit list from the same module...

      An example of a changelog - formatted to show just the changes (formatting styles will change)...

      Finally, an example of a fallback list of commits - sorry Adrian ...

       
      Making breaking changes obvious.
      This is currently done by searching for a set of configurable search strings. Later versions may be able to support breaking change detection via use of Semantic Versioning - but this may require some way of signalling the use of this versioning standard on a module-by-module basis.
      For now, then, you can customise the default set of change markers. Here I have added my own alias to the list of breaking change markers and the changes section of the changelog is styled accordingly (these will be improved)...

       
      Make reading of a module's support documentation, post-install, a trivial task.
      This is done by making some of the support files (like the README, CHANGELOG and LICENSE files) readable from the module's information/settings screen. There is an option to control the initial open/closed state of this section...

      Here is Tracy's README file from within the module settings page... 

       
      Make module authors start to think about how they can improve the change discovery process for their modules.
      There are notes in each of the sections displayed on the upgrade confirmation page that help authors use each of the features...

       
      Make sure display of external information doesn't introduce a vulnerability.
      This is an ongoing concern, and is the thing that is most likely to delay or prevent this module's release lead to this module's withdrawl should a vulnerability be found. Currently, output is formatted either via Markdown + HTML Purifier (if it was originally a Markdown file) or via htmlspecialchars() if it has come from a plaintext file.
      If you discover a vulnerability, please get in contact with me via the forum PM system.
       
      Ongoing...
      For now, I've concentrated on integration with GitHub, as most people use that platform to host their code. I know a few people are hosting their repositories with BitBucket (PWFoo comes to mind) and some with GitLab (Mike Rockett?) and I would eventually like to have adaptor implementations for these providers (and perhaps GitKraken) - but for now, GitHub rules and the other hosts are unsupported.
       
      Links
      Github: ModuleReleaseNotes
      PW Module Repository: Here
    • By FrancisChung
      Hi,
      I was wondering if anyone knew what this did? 

      If so, do you know what modules it actually impacts? Is there another log that lists the action it took?

      I'm having some weird problems with PHP duplicate declaration per link below, since this module was run.
       
       
    • By tpr
      AdminOnSteroids
      Various admin tweaks to enhance ProcessWire admin.
      http://modules.processwire.com/modules/admin-on-steroids/
      https://github.com/rolandtoth/AdminOnSteroids

    • By Macrura
      Field Descriptions Extended
      This module enables you to extend field descriptions by dividing short descriptions with a longer text that is revealed in a toggle.
      Github: https://github.com/outflux3/FieldDescriptionsExtended
      Extending your field descriptions using the standard field's description field.
      Once this module is installed, it will automatically search your description field for the presence of 5 dashes (-----).
      Any content above the 5 dashes will be visible and the content below the dashes will be hidden. A 'More...' link will appear at the end of the short description which when clicked will reveal the rest of the description.
      Using Simple Markdown Editor with the description field
      If you have Simple Markdown Editor (InputfieldSimpleMDE) installed, you can enable the field description to have that editor.
      *When using Simple MDE, you can use the button (Insert Horizontal Line) instead of typing 5 dashes. More about SimpleMDE.
      Extending your field descriptions using content from a ProcessWire Page for the field description.
      You may use the content from a ProcessWire page as a field description. This would allow you to easily insert images, links, and use hanna codes.
      To use page content for your field descriptions, please follow these instructions:
      Install Select Fields module (FieldtypeFields) http://modules.processwire.com/modules/fieldtype-fields/ Create a new field using this field type, e.g. field_select. Add the field to any template you will be using for your field descriptions. Setup your help pages (for example under a settings branch) where you will store the field description content,using the template containing the Field Select. Add content to a page and select the field where that content should show. To show a short text before the link to the longer content, separate them with 5 dashes Be sure to update your settings on this page, first enable page content descriptions,then specify the name of the Select Fields field, template to search, and content field. If you create a field description using this method, please note that the description field must be blank for contexts where you want the page content to appear.
      You can freely use template context for field descriptions, but the Page Content method is not context sensitive and will display under all contexts where the description is blank.
      ----
      original post:
      This is a new module, hope to release soon, which allows extended field descriptions, in currently 2 ways.
      The main feature of the module is that you can have a short description and then a 'more...' link which drops down a longer block of text.
      This is achieved by separating the intro/visible text and the rest with 5 dashes.

      Example setup:

      the 2nd way is if you are using AdminThemeUiKit, you can show extended field instructions in a panel. The content of the panel is edited on a regular PW page. This use case would probably not be that common, but if you had a field that required some extended instructions for how to use, this could be useful; Also, since this allows you to target information and instructions down at the field level, it could reduce the amount of documentation needed on a global level, since it is a lot more context targeted.