Module Migrator

[NooseLadder]

What do you see in your 1022 folder if you move the exit down to after line 317 and then again after 318?

I am assuming there is something in the create_zip function that isn't working on your local install. Maybe it's the ZIPARCHIVE::OVERWRITE?

exit after line 316  /1203/data.json & files.zip

exit after line 317 /1203/data.json & files.zip plus this message: Warning: Illegal string offset 'currentpath' in  :\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2100

exit after line 318 /1203/files.zip only no json file plus the same warning mssg as above.

Each time I deleted the files in /1203/ before running the next test.

[adrian]

@tobaco - thanks for the feedback - obviously I need to look further into users/roles migrations. Your exported json is showing that the home page is created by a user named "cms" with no role. Looking at the created_users_id for the home page, this makes sense. I'll try to work on this shortly, along with the rest of the multi-language issues.

@NooseLadder - thanks for that - I have pushed an update which I hope will fix things for you - can you please let me know how it goes.

[NooseLadder]

Hi Adrian,

Thanks for the update but this still does not work. Installed version 0.1.2.

It seems that the json and zip files are being created in the folder but the json is not being added to the zip prior to download.

I commented out line 320

//unlink($this->session->jsonFilename);

The zip file is downloaded without the json file included and the json file remains in the files/folder.

If you want me run any more tests please let me know.

[adrian]

Sorry to hear that - can you please try this after line 318:

print_r($allfiles);exit;

[NooseLadder]

Hi This is the result:

Array ( [currentpath] => C:/xampp/htdocs/mysite/site/assets/files/1062/data.json [newpath] => C:/xampp/htdocs/mysite/site/assets/files/1062/data.json )

[adrian]

I am thinking this might be a Windows vs Unix slash/path issue, but I am not exactly sure where. Does xammp translate forward slashes to back slashes for disk paths - is that why the results you just showed have forward slashes? But then if that was the problem then you'd think it would also affect the template files and images in the zip as well. Maybe it's time I set up an xampp installation on my Windows VM.

Please grab the latest version from Github and try adding the following after line 2120:

error_log('current:'.$file.'|new:'.pathinfo($file, PATHINFO_BASENAME));

and let me know what ends up in your log.

In other news, the latest version should now solve the role issue that @tobaco was having with system created pages and also it should now work properly when attempting to migrate a site with the Home page set as the parent. You can now import Home into Home and it should work properly, allowing full migration of an entire site's page tree - at least something productive came of my efforts today

[NooseLadder]

Hi Adrian,

I did not get any error logged.

Regarding the forward slash/backslash question please see the answer from Stackoverflow. I do not know if this helps or not?

Rgds,

Keith

[adrian]

Thanks again. If nothing was logged that suggests that $valid_files is empty, or that create_zip never gets called for the json file, but I can't see how that would be possible.

Would you mind playing around a little more inside that create_zip function and try to figure out the content of $valid_files when trying to add the json file to the zip. Try this after 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));

Also, I see that you edited your post since initially posting. Those errors you did get are strange - seems like you might be trying to export a page with no template. Do you have any ideas on those errors?

Thanks again for your help with this. I hope it doesn't turn out to be some version specific bug in xammp

[adrian]

Another significant update this morning which corrects a major omission in critical functionality. Migrator now handles migration of template context field settings.

Now to tackle these new multi-language issues

[NooseLadder]

Thanks again. If nothing was logged that suggests that $valid_files is empty, or that create_zip never gets called for the json file, but I can't see how that would be possible.

Would you mind playing around a little more inside that create_zip function and try to figure out the content of $valid_files when trying to add the json file to the zip. Try this after 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));

Also, I see that you edited your post since initially posting. Those errors you did get are strange - seems like you might be trying to export a page with no template. Do you have any ideas on those errors?

Thanks again for your help with this. I hope it doesn't turn out to be some version specific bug in xammp

Hi Adrian,

I will play around and report back. The error I deleted I think were due to me not selecting a parent. I'll test again and let you know. Thanks.

BTW, What do you use as a localhost server?

You have to forgive my ignorance here: All this time I have been looking in the wrong error_log file. I have now found the correct one and will go through the lines.

Edited June 29, 2014 by NooseLadder

[NooseLadder]

I am thinking this might be a Windows vs Unix slash/path issue, but I am not exactly sure where. Does xammp translate forward slashes to back slashes for disk paths - is that why the results you just showed have forward slashes? But then if that was the problem then you'd think it would also affect the template files and images in the zip as well. Maybe it's time I set up an xampp installation on my Windows VM.

Please grab the latest version from Github and try adding the following after line 2120:

error_log('current:'.$file.'|new:'.pathinfo($file, PATHINFO_BASENAME));

and let me know what ends up in your log.

In other news, the latest version should now solve the role issue that @tobaco was having with system created pages and also it should now work properly when attempting to migrate a site with the Home page set as the parent. You can now import Home into Home and it should work properly, allowing full migration of an entire site's page tree - at least something productive came of my efforts today

After finding the correct error_log file I reran this line and got the following:

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Warning:  Illegal string offset 'currentpath' in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

And the same set of errors after running this after line 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));

Edited June 29, 2014 by NooseLadder

[NooseLadder]

Hi Adrian,

I managed to get this to work by comment out the following on line 2106:

$valid_files = array();
        if(is_array($files)) {
            foreach($files as $file) {
                //if(file_exists($file['currentpath']) || file_exists($file[0]))  /* <-- This line */
$valid_files[] = $file;
            }
        }

I've tried an import into another site and it works 

There may be implications I'm not aware of by doing this?

Edit: Just tried with latest version 0.1.9 and that works too. Line 2106 in version 0.1.6 is now line 2144 in version 0.1.9

I did get this warning after the import to the new site both in versions 0.1.6 and 0.1.9:

Warning: rmdir(C:/xampp/htdocs/mysite/site/assets/files/1022/migratorfiles): Directory not empty in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2182

Edited June 29, 2014 by NooseLadder

[adrian]

@NooseLadder - thanks so much for your help with this. So if things are working with that file_exists check removed then I think we can assume it's a windows path issue. Weird that the addfile to the zip works, but file_exists fails. I am curious about the Directory not empty warning. Does the migratorfiles directory actually get removed? Again, this seems like a windows path issue because I am using a recursive directory delete function and it works here. I really might need that windows xammp setup to get all these sorted out. But for now, I have removed the file_exists check from the latest committed version as I don't really think it should be necessary as the $files array shouldn't contain any files that don't exist anyway.

Another big enhancement this morning - I think all multi-language features should now be working!

@tobaco - would you please check to see if the latest version fixes all the issues you were having with multi-language page names etc?

[NooseLadder]

Hi Adrian,

I'm happy to help, thanks.

The folder C:/xampp/htdocs/mysite/site/assets/files/1022/migratorfiles is not there. The folder C:/xampp/htdocs/mysite/site/assets/files/1022/ contains some images that I uploaded for a specific page, a thumbnail and the main image. I don't understand why that folder was used to append the migratorfiles folder, maybe because it is the last folder listed in the in the $files array?

[adrian]

Glad to hear the migratorfiles temp folder is being deleted as expected. I am also not sure why 1022 contains some images. The numbered folder that migratorfiles should be saved to is the one associated with the Migrator module page. If you browse your page tree: Admin > Setup > Migrator > edit that should show ?id=xxxx

The id should be the folder where migratorfiles is being created. Do these match, or does 1022 really belong to another page in your site's tree?

[adrian]

I have just finalized support for all Profields field types: Table, PageTable, Multiplier, and Texareas

Please check out the latest version and test away.

I think this completes enough functionality now that I am ready to release this to the modules directory once I get some feedback to verify that multi-language functionality and the Profields additions are working well. 

The official release will probably still be "alpha" for a little while, but hopefully getting it in the modules directory will get some more people testing!

[Torsten Baldes]

@adrian

sorry for the delay, i just saw your post.

i tried to import the the data, but got the following error right after submitting the form:

Recoverable Fatal Error: Object of class stdClass could not be converted to string (line 38 of /cms/wire/modules/LanguageSupport/FieldtypeTextLanguage.module) 

another thing:

is it possible to point the importer to an already uploaded (e.g. via ftp) file instead of uploading the file via the form.

i think most hosting environments will struggle with uploading large files.

THANKS!

[NooseLadder]

Glad to hear the migratorfiles temp folder is being deleted as expected. I am also not sure why 1022 contains some images. The numbered folder that migratorfiles should be saved to is the one associated with the Migrator module page. If you browse your page tree: Admin > Setup > Migrator > edit that should show ?id=xxxx

The id should be the folder where migratorfiles is being created. Do these match, or does 1022 really belong to another page in your site's tree?

Hi Adrian,

Admin>Setup>Migrator>edit ?id=1062

1022 belongs to another page in the tree and contains the images for that page.

[Nico Knoll]

Well I think it's alright then. If 1022 contains the images for 1022 it was a successful import.

[adrian]

@tobaca,

I just tested the json file you sent me last week and got the same error. If that is the file you are still using, then that is the reason. You need to export again with the new version of Migrator so that the json includes the new multi-language information. Could you please try that and let me know how it goes. If it still doesn't work, please send me your newly exported json file. Also, I have a think about pointing to an already uploaded file.

@NooseLadder - that is definitely strange - it is making me think that: $this->page->filesManager()->path() is somehow not working for you because it doesn't make any sense that this is not pointing to Migrator's admin page id. Anyone else out there running xammp on Windows who feels like testing this ?

[Torsten Baldes]

@adrian

actually, i did export again with the new version. do i have to do something than replacing the module to update?

here's the json: http://cl.ly/3V0a2a2P2203 

thanks!

[kongondo]

......that is definitely strange - it is making me think that: $this->page->filesManager()->path() is somehow not working for you because it doesn't make any sense that this is not pointing to Migrator's admin page id. Anyone else out there running xammp on Windows who feels like testing this ?

Tell me what to do and I'll test. There's 9 pages in this thread and I don't know where to start 

[adrian]

@tobaco - that is one serious test case of an import file

It's been a mission, but I have narrowed down the problem to the "content_variants" repeater field. If you use the Edit Imported Content option and remove that field from the list, it imports everything else just fine. See image of page tree

Migrator normally works just fine with repeater fields, but I see that you have a page field in the repeater field's template, which is something I haven't tested before, especially together with the muit-language element. Not sure exactly what the issue is yet, so I need to investigate more.

For now, I'd appreciate it if you could confirm on your end that the import works if you omit that repeater field during the import, just so I know we are experiencing exactly the same results.

You will need the latest version of Migrator from github as I had to set PHP's max_execution_time because your import was so complex.

Thanks for continuing to help out with this - getting real life tests like this is very valuable.

@kongondo - thanks for offering. You are right, this thread is a behemoth - I will start a new one as soon as I am ready to release, but for now to test:

1. Grab the module from: https://github.com/adrianbj/ProcessMigrator and install
2. Make sure debug mode is on
3. Run the module (Setup > Migrator)
4. Choose Export
5. Choose a Parent Page - to keep things simple, maybe something like the default PW's About page
6. Export
7. Check that the downloaded zip file contains a data.json file
8. Let me know if you get any path related errors like @NooseLadder
9. If you're feeling energetic, comment out line 335: //unlink($zipFilename);
10. Run again and find out the id of the assets/files folder where the zip is stored. It should match the id of the Migrator Admin > Setup > Migrator page id, but some reason @NooseLadder is seeing it in one of his normal page id folders. Seems like it's a windows/xampp issue.

Thanks so much!

[Torsten Baldes]

@adrian

thanks for your hard work!

I tried the import omitting the repeater field an got the following response:

That happened, when i tried to replace the content.

After that I deleted alle the pages I already had in the import system and imported again. I got a similar message and no pages were imported.

After a second try, some pages but not all got imported. It always displayed the »This page may not be deleted« message.

When i tried »Append« instead of »Replace« the import seems to work, I got this screen:

I hope this helps to narrow down the issue.

Thanks again!

[adrian]

Thanks again for testing. I should have mentioned to try on a clean install. I believe the "This page may not be deleted" message is referring to various system pages. The replace method deletes all children of the Import Parent Page before creating them again. Append or Overwrite doesn't do this. This is not an issue when migrating page trees below home, but if you choose home, then http404 and all the pages under Admin are selected, so obviously I need a check to deal with system pages that are not deleteable to prevent the error.

I'll also look further into the icon_select error - I saw this a few times too - for some reason that page field is not being fully configured during the import, even though the other page fields are importing perfectly.

I am glad to see the 196 pages imported though - that looks like the full set

I don't really have any time for this for the next few days, but hopefully I can get these things sorted out after that.