adrian

Migrator

Recommended Posts

What do you see in your 1022 folder if you move the exit down to after line 317 and then again after 318?

I am assuming there is something in the create_zip function that isn't working on your local install. Maybe it's the ZIPARCHIVE::OVERWRITE?

exit after line 316  /1203/data.json & files.zip

exit after line 317 /1203/data.json & files.zip plus this message: Warning: Illegal string offset 'currentpath' in  :\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2100

exit after line 318 /1203/files.zip only no json file plus the same warning mssg as above.

Each time I deleted the files in /1203/ before running the next test.

Share this post


Link to post
Share on other sites

@tobaco - thanks for the feedback - obviously I need to look further into users/roles migrations. Your exported json is showing that the home page is created by a user named "cms" with no role. Looking at the created_users_id for the home page, this makes sense. I'll try to work on this shortly, along with the rest of the multi-language issues.

@NooseLadder - thanks for that - I have pushed an update which I hope will fix things for you - can you please let me know how it goes.

Share this post


Link to post
Share on other sites

Hi Adrian,

Thanks for the update but this still does not work. Installed version 0.1.2.

It seems that the json and zip files are being created in the folder but the json is not being added to the zip prior to download.

I commented out line 320

//unlink($this->session->jsonFilename);

The zip file is downloaded without the json file included and the json file remains in the files/folder.

If you want me run any more tests please let me know.

Share this post


Link to post
Share on other sites

Sorry to hear that - can you please try this after line 318:

print_r($allfiles);exit;

Share this post


Link to post
Share on other sites

Hi This is the result:

Array ( [currentpath] => C:/xampp/htdocs/mysite/site/assets/files/1062/data.json [newpath] => C:/xampp/htdocs/mysite/site/assets/files/1062/data.json )

Share this post


Link to post
Share on other sites

I am thinking this might be a Windows vs Unix slash/path issue, but I am not exactly sure where. Does xammp translate forward slashes to back slashes for disk paths - is that why the results you just showed have forward slashes? But then if that was the problem then you'd think it would also affect the template files and images in the zip as well. Maybe it's time I set up an xampp installation on my Windows VM.

Please grab the latest version from Github and try adding the following after line 2120:

error_log('current:'.$file.'|new:'.pathinfo($file, PATHINFO_BASENAME));

and let me know what ends up in your log.

In other news, the latest version should now solve the role issue that @tobaco was having with system created pages and also it should now work properly when attempting to migrate a site with the Home page set as the parent. You can now import Home into Home and it should work properly, allowing full migration of an entire site's page tree - at least something productive came of my efforts today :)

Share this post


Link to post
Share on other sites

Hi Adrian,

I did not get any error logged.

Regarding the forward slash/backslash question please see the answer from Stackoverflow. I do not know if this helps or not?

Rgds,

Keith

Share this post


Link to post
Share on other sites

Thanks again. If nothing was logged that suggests that $valid_files is empty, or that create_zip never gets called for the json file, but I can't see how that would be possible.

Would you mind playing around a little more inside that create_zip function and try to figure out the content of $valid_files when trying to add the json file to the zip. Try this after 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));

Also, I see that you edited your post since initially posting. Those errors you did get are strange - seems like you might be trying to export a page with no template. Do you have any ideas on those errors?

Thanks again for your help with this. I hope it doesn't turn out to be some version specific bug in xammp :)

  • Like 1

Share this post


Link to post
Share on other sites

Another significant update this morning which corrects a major omission in critical functionality. Migrator now handles migration of template context field settings.

Now to tackle these new multi-language issues :)

  • Like 1

Share this post


Link to post
Share on other sites

Thanks again. If nothing was logged that suggests that $valid_files is empty, or that create_zip never gets called for the json file, but I can't see how that would be possible.

Would you mind playing around a little more inside that create_zip function and try to figure out the content of $valid_files when trying to add the json file to the zip. Try this after 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));

Also, I see that you edited your post since initially posting. Those errors you did get are strange - seems like you might be trying to export a page with no template. Do you have any ideas on those errors?

Thanks again for your help with this. I hope it doesn't turn out to be some version specific bug in xammp :)

Hi Adrian,

I will play around and report back. The error I deleted I think were due to me not selecting a parent. I'll test again and let you know. Thanks.

BTW, What do you use as a localhost server?

You have to forgive my ignorance here: All this time I have been looking in the wrong error_log file. I have now found the correct one and will go through the lines.

Edited by NooseLadder

Share this post


Link to post
Share on other sites

I am thinking this might be a Windows vs Unix slash/path issue, but I am not exactly sure where. Does xammp translate forward slashes to back slashes for disk paths - is that why the results you just showed have forward slashes? But then if that was the problem then you'd think it would also affect the template files and images in the zip as well. Maybe it's time I set up an xampp installation on my Windows VM.

Please grab the latest version from Github and try adding the following after line 2120:

error_log('current:'.$file.'|new:'.pathinfo($file, PATHINFO_BASENAME));

and let me know what ends up in your log.

In other news, the latest version should now solve the role issue that @tobaco was having with system created pages and also it should now work properly when attempting to migrate a site with the Home page set as the parent. You can now import Home into Home and it should work properly, allowing full migration of an entire site's page tree - at least something productive came of my efforts today :)

After finding the correct error_log file I reran this line and got the following:

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Warning:  Illegal string offset 'currentpath' in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

And the same set of errors after running this after line 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));
Edited by NooseLadder
  • Like 1

Share this post


Link to post
Share on other sites

Hi Adrian,

I managed to get this to work by comment out the following on line 2106:

$valid_files = array();
        if(is_array($files)) {
            foreach($files as $file) {
                //if(file_exists($file['currentpath']) || file_exists($file[0]))  /* <-- This line */
$valid_files[] = $file;
            }
        }

I've tried an import into another site and it works  :-)

There may be implications I'm not aware of by doing this?

Edit: Just tried with latest version 0.1.9 and that works too. Line 2106 in version 0.1.6 is now line 2144 in version 0.1.9

I did get this warning after the import to the new site both in versions 0.1.6 and 0.1.9:

Warning: rmdir(C:/xampp/htdocs/mysite/site/assets/files/1022/migratorfiles): Directory not empty in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2182
Edited by NooseLadder
  • Like 2

Share this post


Link to post
Share on other sites

@NooseLadder - thanks so much for your help with this. So if things are working with that file_exists check removed then I think we can assume it's a windows path issue. Weird that the addfile to the zip works, but file_exists fails. I am curious about the Directory not empty warning. Does the migratorfiles directory actually get removed? Again, this seems like a windows path issue because I am using a recursive directory delete function and it works here. I really might need that windows xammp setup to get all these sorted out. But for now, I have removed the file_exists check from the latest committed version as I don't really think it should be necessary as the $files array shouldn't contain any files that don't exist anyway.


Another big enhancement this morning - I think all multi-language features should now be working!

@tobaco - would you please check to see if the latest version fixes all the issues you were having with multi-language page names etc?

  • Like 3

Share this post


Link to post
Share on other sites

Hi Adrian,

I'm happy to help, thanks.

The folder C:/xampp/htdocs/mysite/site/assets/files/1022/migratorfiles is not there. The folder C:/xampp/htdocs/mysite/site/assets/files/1022/ contains some images that I uploaded for a specific page, a thumbnail and the main image. I don't understand why that folder was used to append the migratorfiles folder, maybe because it is the last folder listed in the in the $files array?

  • Like 1

Share this post


Link to post
Share on other sites

Glad to hear the migratorfiles temp folder is being deleted as expected. I am also not sure why 1022 contains some images. The numbered folder that migratorfiles should be saved to is the one associated with the Migrator module page. If you browse your page tree: Admin > Setup > Migrator > edit that should show ?id=xxxx

The id should be the folder where migratorfiles is being created. Do these match, or does 1022 really belong to another page in your site's tree?

Share this post


Link to post
Share on other sites

I have just finalized support for all Profields field types: Table, PageTable, Multiplier, and Texareas

Please check out the latest version and test away.

I think this completes enough functionality now that I am ready to release this to the modules directory once I get some feedback to verify that multi-language functionality and the Profields additions are working well. 

The official release will probably still be "alpha" for a little while, but hopefully getting it in the modules directory will get some more people testing!

  • Like 1

Share this post


Link to post
Share on other sites

@adrian

sorry for the delay, i just saw your post.

i tried to import the the data, but got the following error right after submitting the form:

Recoverable Fatal Error: Object of class stdClass could not be converted to string (line 38 of /cms/wire/modules/LanguageSupport/FieldtypeTextLanguage.module) 

another thing:

is it possible to point the importer to an already uploaded (e.g. via ftp) file instead of uploading the file via the form.

i think most hosting environments will struggle with uploading large files.

THANKS!

  • Like 1

Share this post


Link to post
Share on other sites

Glad to hear the migratorfiles temp folder is being deleted as expected. I am also not sure why 1022 contains some images. The numbered folder that migratorfiles should be saved to is the one associated with the Migrator module page. If you browse your page tree: Admin > Setup > Migrator > edit that should show ?id=xxxx

The id should be the folder where migratorfiles is being created. Do these match, or does 1022 really belong to another page in your site's tree?

Hi Adrian,

Admin>Setup>Migrator>edit ?id=1062

1022 belongs to another page in the tree and contains the images for that page.

Share this post


Link to post
Share on other sites

Well I think it's alright then. If 1022 contains the images for 1022 it was a successful import.

Share this post


Link to post
Share on other sites

@tobaca,

I just tested the json file you sent me last week and got the same error. If that is the file you are still using, then that is the reason. You need to export again with the new version of Migrator so that the json includes the new multi-language information. Could you please try that and let me know how it goes. If it still doesn't work, please send me your newly exported json file. Also, I have a think about pointing to an already uploaded file.

@NooseLadder - that is definitely strange - it is making me think that: $this->page->filesManager()->path() is somehow not working for you because it doesn't make any sense that this is not pointing to Migrator's admin page id. Anyone else out there running xammp on Windows who feels like testing this ?

Share this post


Link to post
Share on other sites

......that is definitely strange - it is making me think that: $this->page->filesManager()->path() is somehow not working for you because it doesn't make any sense that this is not pointing to Migrator's admin page id. Anyone else out there running xammp on Windows who feels like testing this ?

Tell me what to do and I'll test. There's 9 pages in this thread and I don't know where to start  :)

Share this post


Link to post
Share on other sites

@tobaco - that is one serious test case of an import file :)

It's been a mission, but I have narrowed down the problem to the "content_variants" repeater field. If you use the Edit Imported Content option and remove that field from the list, it imports everything else just fine. See image of page tree :)

post-985-0-09070500-1404275990_thumb.png

Migrator normally works just fine with repeater fields, but I see that you have a page field in the repeater field's template, which is something I haven't tested before, especially together with the muit-language element. Not sure exactly what the issue is yet, so I need to investigate more.

For now, I'd appreciate it if you could confirm on your end that the import works if you omit that repeater field during the import, just so I know we are experiencing exactly the same results.

You will need the latest version of Migrator from github as I had to set PHP's max_execution_time because your import was so complex.

Thanks for continuing to help out with this - getting real life tests like this is very valuable.

@kongondo - thanks for offering. You are right, this thread is a behemoth - I will start a new one as soon as I am ready to release, but for now to test:

  1. Grab the module from: https://github.com/adrianbj/ProcessMigrator and install
  2. Make sure debug mode is on
  3. Run the module (Setup > Migrator)
  4. Choose Export
  5. Choose a Parent Page - to keep things simple, maybe something like the default PW's About page
  6. Export
  7. Check that the downloaded zip file contains a data.json file
  8. Let me know if you get any path related errors like @NooseLadder
  9. If you're feeling energetic, comment out line 335: //unlink($zipFilename);
  10. Run again and find out the id of the assets/files folder where the zip is stored. It should match the id of the Migrator Admin > Setup > Migrator page id, but some reason @NooseLadder is seeing it in one of his normal page id folders. Seems like it's a windows/xampp issue.

Thanks so much!

Share this post


Link to post
Share on other sites

@adrian

thanks for your hard work!

I tried the import omitting the repeater field an got the following response:

Bildschirmfoto%202014-07-02%20um%2009.02

That happened, when i tried to replace the content.

After that I deleted alle the pages I already had in the import system and imported again. I got a similar message and no pages were imported.

After a second try, some pages but not all got imported. It always displayed the »This page may not be deleted« message.

When i tried »Append« instead of »Replace« the import seems to work, I got this screen:

Bildschirmfoto%202014-07-02%20um%2009.22

I hope this helps to narrow down the issue.

Thanks again!

  • Like 1

Share this post


Link to post
Share on other sites

Thanks again for testing. I should have mentioned to try on a clean install. I believe the "This page may not be deleted" message is referring to various system pages. The replace method deletes all children of the Import Parent Page before creating them again. Append or Overwrite doesn't do this. This is not an issue when migrating page trees below home, but if you choose home, then http404 and all the pages under Admin are selected, so obviously I need a check to deal with system pages that are not deleteable to prevent the error.

I'll also look further into the icon_select error - I saw this a few times too - for some reason that page field is not being fully configured during the import, even though the other page fields are importing perfectly.

I am glad to see the 196 pages imported though - that looks like the full set :)

I don't really have any time for this for the next few days, but hopefully I can get these things sorted out after that.

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By BitPoet
      As threatened in the Pub sub forum in the "What are you currently building?" thread, I've toyed around with Collabora CODE and built file editing capabilities for office documents (Libre-/OpenOffice formats and MS Office as well as a few really old file types) into a PW module.
      If you are running OwnCloud or NextCloud, you'll perhaps be familiar with the Collabora app for this purpose.
      LoolEditor
      Edit office files directly in ProcessWire
      Edit your docx, odt, pptx, xlsx or whatever office files you have stored in your file fields directly from ProcessWire's page editor. Upload, click the edit icon, make your changes and save. Can be enabled per field, even in template context.
      Currently supports opening and saving of office documents. Locking functionality is in development.
      See the README on GitHub for installation instructions. You should be reasonably experienced with configuring HTTPS and running docker images to get things set up quickly.
      Pull requests are welcome!
      Here is a short demonstration:

    • By Robin S
      An Images field allows you to:
      Rename images by clicking the filename in the edit panel or in list view. Replace images, keeping metadata and filename (when possible) by dropping a new image on the thumbnail in the edit panel. Introduced here. But neither of these things is possible in File fields, which prompted this module. The way that files are renamed or replaced in this module is not as slick as in the Images field but it gets the job done. The most time-consuming part was dealing with the UI differences of the core admin themes. @tpr, gives me even more respect for the work that must go into AdminOnSteroids.
      Most of the code to support the rename/replace features is already present in InputfieldFile - there is just no UI for it currently. So hopefully that means these features will be offered in the core soon and this module can become obsolete.
       
      Files Rename Replace
      Allows files to be renamed or replaced in Page Edit.

      Usage
      Install the Files Rename Replace module.
      If you want to limit the module to certain roles only, select the roles in the module config. If no roles are selected then any role may rename/replace files.
      In Page Edit, click "Rename/Replace" for a file...
      Rename
      Use the text input to edit the existing name (excluding file extension).
      Replace
      Use the "Replace with" select to choose a replacement file from the same field. On page save the file will be replaced with the file you selected. Metadata (description, tags) will be retained, and the filename also if the file extensions are the same.
      Tip: newly uploaded files will appear in the "Replace with" select after the page has been saved.
       
      https://github.com/Toutouwai/FilesRenameReplace
      http://modules.processwire.com/modules/files-rename-replace/
    • By d'Hinnisdaël
      So I decided to wade into module development and created a wrapper module around ImageOptim, a service that compresses and optimizes images in the cloud. ImageOptim currently handles JPG, PNG and GIF files and, depending on the settings you use, shaves off between 15% and 60% in filesize. Great for bandwidth and great for users, especially on mobile.
      This module handles the part of uploading images to ImageOptim via their official API, downloading the optimized version and storing it alongside the original image.
       

       
      Download & Info
      GitHub / Module directory / Readme / Usage
       
      Why ImageOptim?
      There are other image optimization services out there, some of them free, that have outstanding ProcessWire modules. A few things make ImageOptim the best tool for most of my customers: It's not free, i.e. it will probably be around for a while and offers support. However, it's cheaper than some of the bigger competitors like Cloudinary. And it does PNG compression better than any of the free services out there, especially those with alpha channels.
       
      Installation
      Install the module like any other ProcessWire module, by either copying the folder into your modules folder or installing it via the admin. See above for downloads links on GitHub and in the module directory.
       
      Requirements
      To be able to upload images to the service, allow_url_fopen must be set on the server. The module will abort installation if that's not the case.
      I have only tested the module on ProcessWire 3.x installations. I don't see why it shouldn't work in 2.x, if anyone wants to try it out and report back.
       
      ImageOptim account
      To compress images, you first need to sign up for an ImageOptim account. They offer free trials to try the service.
       
      Usage (manual optimization)
      Images can be optimized by calling the optimize() method on any image. You can pass an options array to set ImageOptim API parameters.
      $image->size(800,600)->optimize()->url $image->optimize(['quality' => 'low', 'dpr' => 2]) // Set quality to low and enable hi-dpi mode
      Automatic optimization
      The module also has an automatic mode that optimizes all image variations after resizing. This is the recommended way to use this module since it leaves the original image uncompressed, but optimizes all derivative images.
      $image->size(800,600)->url // nothing to do here; image is optimized automatically
      To change compression setting for single images, you can pass an options array along with the standard ImageResizer options. Passing false disables optimization.
      $image->size(800, 600, ['optimize' => 'medium']) $image->size(800, 600, ['optimize' => ['quality' => 'low', 'dpr' => 2]]) $image->size(800, 600, ['optimize' => false])
      For detailed usage instructions and all API parameters, see the usage instructions on GitHub.
       
      Filenames
      Optimized images will be suffixed, e.g. image.jpg becomes image.optim.jpg. You can configure the suffix in the module settings.
       
      Roadmap
      Asynchronous processing. Not really high on the list. Image variations need to be created anyway, so waiting a few seconds longer on first load is preferable to adding complexity to achieve async optimization. Optimize image variations created by other modules. CroppableImage comes to mind. I don't use any of these, so if somebody wants to help out and submit a pull request — all for it! Add a dedicated page in the setup menu with a dashboard and detailed statistics. ImageOptim's API is very barebones for now, so not sure if that's feasible or even necessary.
      Stability
      I've been using this module on production sites for some time now, without hiccups. If you do notice oddities, feel free to comment here or investigate and submit PRs.
    • By daniels
      General
      This is a lightweight alternative to other newsletter & newsletter-subscription modules.
      It can update / create a subscriber in a list in Mailchimp with MailChimp API 3.0. It does not provide any forms or validation, so you can feel free to use your own. To protect your users, it does not save any user data in logs or sends them to an admin.
      This module fits your needs if you...
      ...use Mailchimp as your newsletter / email-automation tool ...want to let users subscribe to your newsletter on your website ...want to use your own form, validation and messages (with or without the wire forms) ...don't want any personal user data saved in any way in your ProcessWire environment (cf. EU data regulation terms) ...like to subscribe users to different lists ...like the Mailchimp UI for creating / sending / reviewing email campaigns You can find it here: https://github.com/danielstieber/SubscribeToMailchimp
      Let me know what you think and if I should add it to the Modules Directory.
      Setup
      Log into your Mailchimp account and go to  Profile > Extras > API Keys. If you don't have an API Key, create a new one. Copy your API Key and paste it in the module settings (Processwire > Modules > Site > SubscribeToMailchimp). Back in Mailchimp, go to the list, where you want your new subscribers. Go to Settings > List name and defaults. Copy the List ID an paste it in to the module settings.
      Usage
      To use the module, you need to load it into your template:
      $mc = $modules->get("SubscribeToMailchimp"); Now you can pass an email address to the module and it will try to edit (if the user exists) or create a new subscriber in your list.
      $mc->subscribe('john.doe@example.com'); You can also pass a data array, to add additional info.
      $mc->subscribe('john.doe@example.com', ['FNAME' => 'John', 'LNAME' => 'Doe']); You can even choose an alternative list, if you don't want this subscriber in your default list.
      $mc->subscribe('john.doe@example.com', ['FNAME' => 'John', 'LNAME' => 'Doe'], 'abcdef1356'); // Subscribe to List ID abcdef1356  
      Important Notes
      This module does not do any data validation. Use a sever-sided validation like Valitron Make sure that you have set up your fields in your Mailchimp list. You can do it at Settings > List fields and *|MERGE|* tags Example
      Example usage after a form is submitted on your page:
      // ... validation of form data $mc = $modules->get("SubscribeToMailchimp"); $email = $input->post->email; $subscriber = [ 'FNAME' => $input->post->firstname, 'LNAME' => $input->post->lastname, ]; $mc->subscribe($email, $subscriber);  
      Troubleshooting
      In case of trouble check your ProcessWire warning logs.
      I can't see the subscriber in the list
      If you have enabled double opt-in (it is enabled by default) you will not see the subscriber, until he confirmed the subscription in the email sent by Mailchimp
      I get an error in my ProccessWire warning logs
      Check if you have the right List ID and API Key. Check if you pass fields, that exist in your list. Check if you pass a valid email address. Go to Mailchimps Error Glossary for more Information
      How To Install
      Download the zip file at Github or clone directly the repo into your site/modules If you downloaded the zip file, extract it in your sites/modules directory. You might have to change the folders name to 'SubscribeToMailchimp'. Goto the modules admin page, click on refresh and install it  
    • By PWaddict
      Module for hiding pages for non-superusers.
      Download
      HidePages
      Requirements
      ProcessWire 3.x Changelog
      1.0.1 (31 March 2018)
      Module renamed to prevent confusion 1.0.0 (30 March 2018)
      Initial release