Jump to content

Migrator


adrian

Recommended Posts

What do you see in your 1022 folder if you move the exit down to after line 317 and then again after 318?

I am assuming there is something in the create_zip function that isn't working on your local install. Maybe it's the ZIPARCHIVE::OVERWRITE?

exit after line 316  /1203/data.json & files.zip

exit after line 317 /1203/data.json & files.zip plus this message: Warning: Illegal string offset 'currentpath' in  :\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2100

exit after line 318 /1203/files.zip only no json file plus the same warning mssg as above.

Each time I deleted the files in /1203/ before running the next test.

Link to comment
Share on other sites

@tobaco - thanks for the feedback - obviously I need to look further into users/roles migrations. Your exported json is showing that the home page is created by a user named "cms" with no role. Looking at the created_users_id for the home page, this makes sense. I'll try to work on this shortly, along with the rest of the multi-language issues.

@NooseLadder - thanks for that - I have pushed an update which I hope will fix things for you - can you please let me know how it goes.

Link to comment
Share on other sites

Hi Adrian,

Thanks for the update but this still does not work. Installed version 0.1.2.

It seems that the json and zip files are being created in the folder but the json is not being added to the zip prior to download.

I commented out line 320

//unlink($this->session->jsonFilename);

The zip file is downloaded without the json file included and the json file remains in the files/folder.

If you want me run any more tests please let me know.

Link to comment
Share on other sites

I am thinking this might be a Windows vs Unix slash/path issue, but I am not exactly sure where. Does xammp translate forward slashes to back slashes for disk paths - is that why the results you just showed have forward slashes? But then if that was the problem then you'd think it would also affect the template files and images in the zip as well. Maybe it's time I set up an xampp installation on my Windows VM.

Please grab the latest version from Github and try adding the following after line 2120:

error_log('current:'.$file.'|new:'.pathinfo($file, PATHINFO_BASENAME));

and let me know what ends up in your log.

In other news, the latest version should now solve the role issue that @tobaco was having with system created pages and also it should now work properly when attempting to migrate a site with the Home page set as the parent. You can now import Home into Home and it should work properly, allowing full migration of an entire site's page tree - at least something productive came of my efforts today :)

Link to comment
Share on other sites

Thanks again. If nothing was logged that suggests that $valid_files is empty, or that create_zip never gets called for the json file, but I can't see how that would be possible.

Would you mind playing around a little more inside that create_zip function and try to figure out the content of $valid_files when trying to add the json file to the zip. Try this after 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));

Also, I see that you edited your post since initially posting. Those errors you did get are strange - seems like you might be trying to export a page with no template. Do you have any ideas on those errors?

Thanks again for your help with this. I hope it doesn't turn out to be some version specific bug in xammp :)

  • Like 1
Link to comment
Share on other sites

Another significant update this morning which corrects a major omission in critical functionality. Migrator now handles migration of template context field settings.

Now to tackle these new multi-language issues :)

  • Like 1
Link to comment
Share on other sites

Thanks again. If nothing was logged that suggests that $valid_files is empty, or that create_zip never gets called for the json file, but I can't see how that would be possible.

Would you mind playing around a little more inside that create_zip function and try to figure out the content of $valid_files when trying to add the json file to the zip. Try this after 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));

Also, I see that you edited your post since initially posting. Those errors you did get are strange - seems like you might be trying to export a page with no template. Do you have any ideas on those errors?

Thanks again for your help with this. I hope it doesn't turn out to be some version specific bug in xammp :)

Hi Adrian,

I will play around and report back. The error I deleted I think were due to me not selecting a parent. I'll test again and let you know. Thanks.

BTW, What do you use as a localhost server?

You have to forgive my ignorance here: All this time I have been looking in the wrong error_log file. I have now found the correct one and will go through the lines.

Edited by NooseLadder
Link to comment
Share on other sites

I am thinking this might be a Windows vs Unix slash/path issue, but I am not exactly sure where. Does xammp translate forward slashes to back slashes for disk paths - is that why the results you just showed have forward slashes? But then if that was the problem then you'd think it would also affect the template files and images in the zip as well. Maybe it's time I set up an xampp installation on my Windows VM.

Please grab the latest version from Github and try adding the following after line 2120:

error_log('current:'.$file.'|new:'.pathinfo($file, PATHINFO_BASENAME));

and let me know what ends up in your log.

In other news, the latest version should now solve the role issue that @tobaco was having with system created pages and also it should now work properly when attempting to migrate a site with the Home page set as the parent. You can now import Home into Home and it should work properly, allowing full migration of an entire site's page tree - at least something productive came of my efforts today :)

After finding the correct error_log file I reran this line and got the following:

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Warning:  Illegal string offset 'currentpath' in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

And the same set of errors after running this after line 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));
Edited by NooseLadder
  • Like 1
Link to comment
Share on other sites

Hi Adrian,

I managed to get this to work by comment out the following on line 2106:

$valid_files = array();
        if(is_array($files)) {
            foreach($files as $file) {
                //if(file_exists($file['currentpath']) || file_exists($file[0]))  /* <-- This line */
$valid_files[] = $file;
            }
        }

I've tried an import into another site and it works  :-)

There may be implications I'm not aware of by doing this?

Edit: Just tried with latest version 0.1.9 and that works too. Line 2106 in version 0.1.6 is now line 2144 in version 0.1.9

I did get this warning after the import to the new site both in versions 0.1.6 and 0.1.9:

Warning: rmdir(C:/xampp/htdocs/mysite/site/assets/files/1022/migratorfiles): Directory not empty in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2182
Edited by NooseLadder
  • Like 2
Link to comment
Share on other sites

@NooseLadder - thanks so much for your help with this. So if things are working with that file_exists check removed then I think we can assume it's a windows path issue. Weird that the addfile to the zip works, but file_exists fails. I am curious about the Directory not empty warning. Does the migratorfiles directory actually get removed? Again, this seems like a windows path issue because I am using a recursive directory delete function and it works here. I really might need that windows xammp setup to get all these sorted out. But for now, I have removed the file_exists check from the latest committed version as I don't really think it should be necessary as the $files array shouldn't contain any files that don't exist anyway.


Another big enhancement this morning - I think all multi-language features should now be working!

@tobaco - would you please check to see if the latest version fixes all the issues you were having with multi-language page names etc?

  • Like 3
Link to comment
Share on other sites

Hi Adrian,

I'm happy to help, thanks.

The folder C:/xampp/htdocs/mysite/site/assets/files/1022/migratorfiles is not there. The folder C:/xampp/htdocs/mysite/site/assets/files/1022/ contains some images that I uploaded for a specific page, a thumbnail and the main image. I don't understand why that folder was used to append the migratorfiles folder, maybe because it is the last folder listed in the in the $files array?

  • Like 1
Link to comment
Share on other sites

Glad to hear the migratorfiles temp folder is being deleted as expected. I am also not sure why 1022 contains some images. The numbered folder that migratorfiles should be saved to is the one associated with the Migrator module page. If you browse your page tree: Admin > Setup > Migrator > edit that should show ?id=xxxx

The id should be the folder where migratorfiles is being created. Do these match, or does 1022 really belong to another page in your site's tree?

Link to comment
Share on other sites

I have just finalized support for all Profields field types: Table, PageTable, Multiplier, and Texareas

Please check out the latest version and test away.

I think this completes enough functionality now that I am ready to release this to the modules directory once I get some feedback to verify that multi-language functionality and the Profields additions are working well. 

The official release will probably still be "alpha" for a little while, but hopefully getting it in the modules directory will get some more people testing!

  • Like 1
Link to comment
Share on other sites

@adrian

sorry for the delay, i just saw your post.

i tried to import the the data, but got the following error right after submitting the form:

Recoverable Fatal Error: Object of class stdClass could not be converted to string (line 38 of /cms/wire/modules/LanguageSupport/FieldtypeTextLanguage.module) 

another thing:

is it possible to point the importer to an already uploaded (e.g. via ftp) file instead of uploading the file via the form.

i think most hosting environments will struggle with uploading large files.

THANKS!

  • Like 1
Link to comment
Share on other sites

Glad to hear the migratorfiles temp folder is being deleted as expected. I am also not sure why 1022 contains some images. The numbered folder that migratorfiles should be saved to is the one associated with the Migrator module page. If you browse your page tree: Admin > Setup > Migrator > edit that should show ?id=xxxx

The id should be the folder where migratorfiles is being created. Do these match, or does 1022 really belong to another page in your site's tree?

Hi Adrian,

Admin>Setup>Migrator>edit ?id=1062

1022 belongs to another page in the tree and contains the images for that page.

Link to comment
Share on other sites

@tobaca,

I just tested the json file you sent me last week and got the same error. If that is the file you are still using, then that is the reason. You need to export again with the new version of Migrator so that the json includes the new multi-language information. Could you please try that and let me know how it goes. If it still doesn't work, please send me your newly exported json file. Also, I have a think about pointing to an already uploaded file.

@NooseLadder - that is definitely strange - it is making me think that: $this->page->filesManager()->path() is somehow not working for you because it doesn't make any sense that this is not pointing to Migrator's admin page id. Anyone else out there running xammp on Windows who feels like testing this ?

Link to comment
Share on other sites

......that is definitely strange - it is making me think that: $this->page->filesManager()->path() is somehow not working for you because it doesn't make any sense that this is not pointing to Migrator's admin page id. Anyone else out there running xammp on Windows who feels like testing this ?

Tell me what to do and I'll test. There's 9 pages in this thread and I don't know where to start  :)

Link to comment
Share on other sites

@tobaco - that is one serious test case of an import file :)

It's been a mission, but I have narrowed down the problem to the "content_variants" repeater field. If you use the Edit Imported Content option and remove that field from the list, it imports everything else just fine. See image of page tree :)

post-985-0-09070500-1404275990_thumb.png

Migrator normally works just fine with repeater fields, but I see that you have a page field in the repeater field's template, which is something I haven't tested before, especially together with the muit-language element. Not sure exactly what the issue is yet, so I need to investigate more.

For now, I'd appreciate it if you could confirm on your end that the import works if you omit that repeater field during the import, just so I know we are experiencing exactly the same results.

You will need the latest version of Migrator from github as I had to set PHP's max_execution_time because your import was so complex.

Thanks for continuing to help out with this - getting real life tests like this is very valuable.

@kongondo - thanks for offering. You are right, this thread is a behemoth - I will start a new one as soon as I am ready to release, but for now to test:

  1. Grab the module from: https://github.com/adrianbj/ProcessMigrator and install
  2. Make sure debug mode is on
  3. Run the module (Setup > Migrator)
  4. Choose Export
  5. Choose a Parent Page - to keep things simple, maybe something like the default PW's About page
  6. Export
  7. Check that the downloaded zip file contains a data.json file
  8. Let me know if you get any path related errors like @NooseLadder
  9. If you're feeling energetic, comment out line 335: //unlink($zipFilename);
  10. Run again and find out the id of the assets/files folder where the zip is stored. It should match the id of the Migrator Admin > Setup > Migrator page id, but some reason @NooseLadder is seeing it in one of his normal page id folders. Seems like it's a windows/xampp issue.

Thanks so much!

Link to comment
Share on other sites

@adrian

thanks for your hard work!

I tried the import omitting the repeater field an got the following response:

Bildschirmfoto%202014-07-02%20um%2009.02

That happened, when i tried to replace the content.

After that I deleted alle the pages I already had in the import system and imported again. I got a similar message and no pages were imported.

After a second try, some pages but not all got imported. It always displayed the »This page may not be deleted« message.

When i tried »Append« instead of »Replace« the import seems to work, I got this screen:

Bildschirmfoto%202014-07-02%20um%2009.22

I hope this helps to narrow down the issue.

Thanks again!

  • Like 1
Link to comment
Share on other sites

Thanks again for testing. I should have mentioned to try on a clean install. I believe the "This page may not be deleted" message is referring to various system pages. The replace method deletes all children of the Import Parent Page before creating them again. Append or Overwrite doesn't do this. This is not an issue when migrating page trees below home, but if you choose home, then http404 and all the pages under Admin are selected, so obviously I need a check to deal with system pages that are not deleteable to prevent the error.

I'll also look further into the icon_select error - I saw this a few times too - for some reason that page field is not being fully configured during the import, even though the other page fields are importing perfectly.

I am glad to see the 196 pages imported though - that looks like the full set :)

I don't really have any time for this for the next few days, but hopefully I can get these things sorted out after that.

  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Similar Content

    • By MarkE
      This fieldtype and inputfield bundle was built for storing measurement values within a field, rendering them in a variety of formats and converting them to other units or otherwise modifying them via the API.
      The API consists of a number of predefined functions, some of which include...
      render() for rendering the measurement object, valueAs() for converting the value to another unit value, convertTo() for converting the whole measurement object to different units, and add() and subtract() for for modifying the stored value by the value (converted as required) in another measurement. In the admin the inputfield includes a checkbox (which can be optionally disabled) for converting values on page save. For an example if a value was typed in as centimeters, the unit was changed to metres, and the page saved with this checkbox selected, said value would be automatically converted so that e.g. 170 cm becomes 1.7 m.

      A simple length field using Fieldtype Measurement and Inputfield Measurement.
      Combination units (e.g. feet and inches) are also supported.
      Please note that this module is 'proof of concept' at the moment - there are limited units available and quite a lot of code tidying to do. More units will be added shortly.
      See the GitHub at https://github.com/MetaTunes/FieldtypeMeasurement for full details and updates.
    • By tcnet
      File Manager for ProcessWire is a module to manager files and folders from the CMS backend. It supports creating, deleting, renaming, packing, unpacking, uploading, downloading and editing of files and folders. The integrated code editor ACE supports highlighting of all common programming languages.
      https://github.com/techcnet/ProcessFileManager

      Warning
      This module is probably the most powerful module. You might destroy your processwire installation if you don't exactly know what you doing. Be careful and use it at your own risk!
      ACE code editor
      This module uses ACE code editor available from: https://github.com/ajaxorg/ace

      Dragscroll
      This module uses the JavaScript dragscroll available from: http://github.com/asvd/dragscroll. Dragscroll adds the ability to drag the table horizontally with the mouse pointer.
      PHP File Manager
      This module uses a modified version of PHP File Manager available from: https://github.com/alexantr/filemanager
       
    • By tcnet
      This module implements the website live chat service from tawk.to. Actually the module doesn't have to do much. It just need to inserted a few lines of JavaScript just before the closing body tag </body> on each side. However, the module offers additional options to display the widget only on certain pages.
      Create an account
      Visit https://www.tawk.to and create an account. It's free! At some point you will reach a page where you can copy the required JavaScript-code.

      Open the module settings and paste the JavaScript-code into the field as shown below. Click "Submit" and that's all.

      Open the module settings
      The settings for this module are located int the menu Modules=>Configure=>LiveChatTawkTo.

       
    • By tcnet
      Session Viewer is a module for ProcessWire to list session files and display session data. This module is helpful to display the session data of a specific session or to kick out a logged in user by simply delete his session file. After installation the module is available in the Setup menu.

      The following conditions must be met for the module to work properly:
      Session files
      Session data must be stored in session files, which is the default way in ProcessWire. Sessions stored in the database are not supported by this module. The path to the directory where the session files are stored must be declared in the ProcessWire configuration which is by default: site/assets/sessions.
      Serialize handler
      In order to transform session data easier back to a PHP array, the session data is stored serialized. PHP offers a way to declare a custom serialize handler. This module supports only the default serialize handlers: php, php_binary and php_serialize. WDDX was dropped in PHP 7.4.0 and is therefore not supported by this module as well as any other custom serialize handler. Which serialize handler is actually used you can find out in the module configuration which is available under Modules=>Configure=>SessionViewer.

      Session data
      The session data can be displayed in two different ways. PHP's default output for arrays print_r() or by default for this module nice_r() offered on github: https://github.com/uuf6429/nice_r. There is a setting in the module configuration if someone prefers print_r(). Apart from the better handling and overview of the folded session data the output of nice_r() looks indeed nicer.

      Links
      ProcessWire module directory
      github.com
    • By Robin S
      Repeater Easy Sort
      Adds a compact "easy-sort" mode to Repeater and Repeater Matrix, making those fields easier to sort when there are a large number of items.
      The module also enhances Repeater Matrix by allowing a colour to be set for each matrix type. This colour is used in the item headers and in the "add new" links, to help visually distinguish different matrix types in the inputfield.
      Screencasts
      A Repeater field

      A Repeater Matrix field with custom header colours

      Easy-sort mode
      Each Repeater/Matrix item gets an double-arrow icon in the item header. Click this icon to enter easy-sort mode.
      While in easy-sort mode:
      The items will reduce in width so that more items can be shown on the screen at once. The minimum width is configurable in the field settings. Any items that were in an open state are collapsed, but when you exit easy-sort mode the previously open items will be reopened. You can drag an item left/right/up/down to sort it within the items. The item that you clicked the icon for is shown with a black background. This makes it easier to find the item you want to move in easy-sort mode. You can click an item header to open the item. An "Exit easy-sort mode" button appears at the bottom of the inputfield. Configuration
      In the field settings for Repeater and Repeater Matrix fields you can define a minimum width in pixels for items in easy-sort mode. While in easy-sort mode the items will be sized to neatly fill the available width on any screen size but will never be narrower than the width you set here.
      In the field settings for Repeater Matrix you can define a custom header colour for each matrix type using an HTML "color" type input. The default colour for this type of input is black, so when black is selected in the input it means that no custom colour will be applied to the header.
      Exclusions
      The easy-sort mode is only possible on Repeater/Matrix fields that do not use the "item depth" option.
       
      https://github.com/Toutouwai/RepeaterEasySort
      https://processwire.com/modules/repeater-easy-sort/
×
×
  • Create New...