adrian

Module
Migrator

294 posts in this topic

What do you see in your 1022 folder if you move the exit down to after line 317 and then again after 318?

I am assuming there is something in the create_zip function that isn't working on your local install. Maybe it's the ZIPARCHIVE::OVERWRITE?

exit after line 316  /1203/data.json & files.zip

exit after line 317 /1203/data.json & files.zip plus this message: Warning: Illegal string offset 'currentpath' in  :\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2100

exit after line 318 /1203/files.zip only no json file plus the same warning mssg as above.

Each time I deleted the files in /1203/ before running the next test.

Share this post


Link to post
Share on other sites

@tobaco - thanks for the feedback - obviously I need to look further into users/roles migrations. Your exported json is showing that the home page is created by a user named "cms" with no role. Looking at the created_users_id for the home page, this makes sense. I'll try to work on this shortly, along with the rest of the multi-language issues.

@NooseLadder - thanks for that - I have pushed an update which I hope will fix things for you - can you please let me know how it goes.

Share this post


Link to post
Share on other sites

Hi Adrian,

Thanks for the update but this still does not work. Installed version 0.1.2.

It seems that the json and zip files are being created in the folder but the json is not being added to the zip prior to download.

I commented out line 320

//unlink($this->session->jsonFilename);

The zip file is downloaded without the json file included and the json file remains in the files/folder.

If you want me run any more tests please let me know.

Share this post


Link to post
Share on other sites

Sorry to hear that - can you please try this after line 318:

print_r($allfiles);exit;

Share this post


Link to post
Share on other sites

Hi This is the result:

Array ( [currentpath] => C:/xampp/htdocs/mysite/site/assets/files/1062/data.json [newpath] => C:/xampp/htdocs/mysite/site/assets/files/1062/data.json )

Share this post


Link to post
Share on other sites

I am thinking this might be a Windows vs Unix slash/path issue, but I am not exactly sure where. Does xammp translate forward slashes to back slashes for disk paths - is that why the results you just showed have forward slashes? But then if that was the problem then you'd think it would also affect the template files and images in the zip as well. Maybe it's time I set up an xampp installation on my Windows VM.

Please grab the latest version from Github and try adding the following after line 2120:

error_log('current:'.$file.'|new:'.pathinfo($file, PATHINFO_BASENAME));

and let me know what ends up in your log.

In other news, the latest version should now solve the role issue that @tobaco was having with system created pages and also it should now work properly when attempting to migrate a site with the Home page set as the parent. You can now import Home into Home and it should work properly, allowing full migration of an entire site's page tree - at least something productive came of my efforts today :)

Share this post


Link to post
Share on other sites

Hi Adrian,

I did not get any error logged.

Regarding the forward slash/backslash question please see the answer from Stackoverflow. I do not know if this helps or not?

Rgds,

Keith

Share this post


Link to post
Share on other sites

Thanks again. If nothing was logged that suggests that $valid_files is empty, or that create_zip never gets called for the json file, but I can't see how that would be possible.

Would you mind playing around a little more inside that create_zip function and try to figure out the content of $valid_files when trying to add the json file to the zip. Try this after 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));

Also, I see that you edited your post since initially posting. Those errors you did get are strange - seems like you might be trying to export a page with no template. Do you have any ideas on those errors?

Thanks again for your help with this. I hope it doesn't turn out to be some version specific bug in xammp :)

1 person likes this

Share this post


Link to post
Share on other sites

Another significant update this morning which corrects a major omission in critical functionality. Migrator now handles migration of template context field settings.

Now to tackle these new multi-language issues :)

1 person likes this

Share this post


Link to post
Share on other sites

Thanks again. If nothing was logged that suggests that $valid_files is empty, or that create_zip never gets called for the json file, but I can't see how that would be possible.

Would you mind playing around a little more inside that create_zip function and try to figure out the content of $valid_files when trying to add the json file to the zip. Try this after 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));

Also, I see that you edited your post since initially posting. Those errors you did get are strange - seems like you might be trying to export a page with no template. Do you have any ideas on those errors?

Thanks again for your help with this. I hope it doesn't turn out to be some version specific bug in xammp :)

Hi Adrian,

I will play around and report back. The error I deleted I think were due to me not selecting a parent. I'll test again and let you know. Thanks.

BTW, What do you use as a localhost server?

You have to forgive my ignorance here: All this time I have been looking in the wrong error_log file. I have now found the correct one and will go through the lines.

Edited by NooseLadder

Share this post


Link to post
Share on other sites

I am thinking this might be a Windows vs Unix slash/path issue, but I am not exactly sure where. Does xammp translate forward slashes to back slashes for disk paths - is that why the results you just showed have forward slashes? But then if that was the problem then you'd think it would also affect the template files and images in the zip as well. Maybe it's time I set up an xampp installation on my Windows VM.

Please grab the latest version from Github and try adding the following after line 2120:

error_log('current:'.$file.'|new:'.pathinfo($file, PATHINFO_BASENAME));

and let me know what ends up in your log.

In other news, the latest version should now solve the role issue that @tobaco was having with system created pages and also it should now work properly when attempting to migrate a site with the Home page set as the parent. You can now import Home into Home and it should work properly, allowing full migration of an entire site's page tree - at least something productive came of my efforts today :)

After finding the correct error_log file I reran this line and got the following:

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Notice:  Undefined offset: 0 in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

[29-Jun-2014 11:47:45 Europe/London] PHP Warning:  Illegal string offset 'currentpath' in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2106

And the same set of errors after running this after line 2108:

if($filetype == 'json') error_log(print_r($valid_files, true));
Edited by NooseLadder
1 person likes this

Share this post


Link to post
Share on other sites

Hi Adrian,

I managed to get this to work by comment out the following on line 2106:

$valid_files = array();
        if(is_array($files)) {
            foreach($files as $file) {
                //if(file_exists($file['currentpath']) || file_exists($file[0]))  /* <-- This line */
$valid_files[] = $file;
            }
        }

I've tried an import into another site and it works  :-)

There may be implications I'm not aware of by doing this?

Edit: Just tried with latest version 0.1.9 and that works too. Line 2106 in version 0.1.6 is now line 2144 in version 0.1.9

I did get this warning after the import to the new site both in versions 0.1.6 and 0.1.9:

Warning: rmdir(C:/xampp/htdocs/mysite/site/assets/files/1022/migratorfiles): Directory not empty in C:\xampp\htdocs\mysite\site\modules\ProcessPageTreeMigrator\ProcessMigrator.module on line 2182
Edited by NooseLadder
2 people like this

Share this post


Link to post
Share on other sites

@NooseLadder - thanks so much for your help with this. So if things are working with that file_exists check removed then I think we can assume it's a windows path issue. Weird that the addfile to the zip works, but file_exists fails. I am curious about the Directory not empty warning. Does the migratorfiles directory actually get removed? Again, this seems like a windows path issue because I am using a recursive directory delete function and it works here. I really might need that windows xammp setup to get all these sorted out. But for now, I have removed the file_exists check from the latest committed version as I don't really think it should be necessary as the $files array shouldn't contain any files that don't exist anyway.


Another big enhancement this morning - I think all multi-language features should now be working!

@tobaco - would you please check to see if the latest version fixes all the issues you were having with multi-language page names etc?

3 people like this

Share this post


Link to post
Share on other sites

Hi Adrian,

I'm happy to help, thanks.

The folder C:/xampp/htdocs/mysite/site/assets/files/1022/migratorfiles is not there. The folder C:/xampp/htdocs/mysite/site/assets/files/1022/ contains some images that I uploaded for a specific page, a thumbnail and the main image. I don't understand why that folder was used to append the migratorfiles folder, maybe because it is the last folder listed in the in the $files array?

1 person likes this

Share this post


Link to post
Share on other sites

Glad to hear the migratorfiles temp folder is being deleted as expected. I am also not sure why 1022 contains some images. The numbered folder that migratorfiles should be saved to is the one associated with the Migrator module page. If you browse your page tree: Admin > Setup > Migrator > edit that should show ?id=xxxx

The id should be the folder where migratorfiles is being created. Do these match, or does 1022 really belong to another page in your site's tree?

Share this post


Link to post
Share on other sites

I have just finalized support for all Profields field types: Table, PageTable, Multiplier, and Texareas

Please check out the latest version and test away.

I think this completes enough functionality now that I am ready to release this to the modules directory once I get some feedback to verify that multi-language functionality and the Profields additions are working well. 

The official release will probably still be "alpha" for a little while, but hopefully getting it in the modules directory will get some more people testing!

1 person likes this

Share this post


Link to post
Share on other sites

@adrian

sorry for the delay, i just saw your post.

i tried to import the the data, but got the following error right after submitting the form:

Recoverable Fatal Error: Object of class stdClass could not be converted to string (line 38 of /cms/wire/modules/LanguageSupport/FieldtypeTextLanguage.module) 

another thing:

is it possible to point the importer to an already uploaded (e.g. via ftp) file instead of uploading the file via the form.

i think most hosting environments will struggle with uploading large files.

THANKS!

1 person likes this

Share this post


Link to post
Share on other sites

Glad to hear the migratorfiles temp folder is being deleted as expected. I am also not sure why 1022 contains some images. The numbered folder that migratorfiles should be saved to is the one associated with the Migrator module page. If you browse your page tree: Admin > Setup > Migrator > edit that should show ?id=xxxx

The id should be the folder where migratorfiles is being created. Do these match, or does 1022 really belong to another page in your site's tree?

Hi Adrian,

Admin>Setup>Migrator>edit ?id=1062

1022 belongs to another page in the tree and contains the images for that page.

Share this post


Link to post
Share on other sites

Well I think it's alright then. If 1022 contains the images for 1022 it was a successful import.

Share this post


Link to post
Share on other sites

@tobaca,

I just tested the json file you sent me last week and got the same error. If that is the file you are still using, then that is the reason. You need to export again with the new version of Migrator so that the json includes the new multi-language information. Could you please try that and let me know how it goes. If it still doesn't work, please send me your newly exported json file. Also, I have a think about pointing to an already uploaded file.

@NooseLadder - that is definitely strange - it is making me think that: $this->page->filesManager()->path() is somehow not working for you because it doesn't make any sense that this is not pointing to Migrator's admin page id. Anyone else out there running xammp on Windows who feels like testing this ?

Share this post


Link to post
Share on other sites

......that is definitely strange - it is making me think that: $this->page->filesManager()->path() is somehow not working for you because it doesn't make any sense that this is not pointing to Migrator's admin page id. Anyone else out there running xammp on Windows who feels like testing this ?

Tell me what to do and I'll test. There's 9 pages in this thread and I don't know where to start  :)

Share this post


Link to post
Share on other sites

@tobaco - that is one serious test case of an import file :)

It's been a mission, but I have narrowed down the problem to the "content_variants" repeater field. If you use the Edit Imported Content option and remove that field from the list, it imports everything else just fine. See image of page tree :)

post-985-0-09070500-1404275990_thumb.png

Migrator normally works just fine with repeater fields, but I see that you have a page field in the repeater field's template, which is something I haven't tested before, especially together with the muit-language element. Not sure exactly what the issue is yet, so I need to investigate more.

For now, I'd appreciate it if you could confirm on your end that the import works if you omit that repeater field during the import, just so I know we are experiencing exactly the same results.

You will need the latest version of Migrator from github as I had to set PHP's max_execution_time because your import was so complex.

Thanks for continuing to help out with this - getting real life tests like this is very valuable.

@kongondo - thanks for offering. You are right, this thread is a behemoth - I will start a new one as soon as I am ready to release, but for now to test:

  1. Grab the module from: https://github.com/adrianbj/ProcessMigrator and install
  2. Make sure debug mode is on
  3. Run the module (Setup > Migrator)
  4. Choose Export
  5. Choose a Parent Page - to keep things simple, maybe something like the default PW's About page
  6. Export
  7. Check that the downloaded zip file contains a data.json file
  8. Let me know if you get any path related errors like @NooseLadder
  9. If you're feeling energetic, comment out line 335: //unlink($zipFilename);
  10. Run again and find out the id of the assets/files folder where the zip is stored. It should match the id of the Migrator Admin > Setup > Migrator page id, but some reason @NooseLadder is seeing it in one of his normal page id folders. Seems like it's a windows/xampp issue.

Thanks so much!

Share this post


Link to post
Share on other sites

@adrian

thanks for your hard work!

I tried the import omitting the repeater field an got the following response:

Bildschirmfoto%202014-07-02%20um%2009.02

That happened, when i tried to replace the content.

After that I deleted alle the pages I already had in the import system and imported again. I got a similar message and no pages were imported.

After a second try, some pages but not all got imported. It always displayed the »This page may not be deleted« message.

When i tried »Append« instead of »Replace« the import seems to work, I got this screen:

Bildschirmfoto%202014-07-02%20um%2009.22

I hope this helps to narrow down the issue.

Thanks again!

1 person likes this

Share this post


Link to post
Share on other sites

Thanks again for testing. I should have mentioned to try on a clean install. I believe the "This page may not be deleted" message is referring to various system pages. The replace method deletes all children of the Import Parent Page before creating them again. Append or Overwrite doesn't do this. This is not an issue when migrating page trees below home, but if you choose home, then http404 and all the pages under Admin are selected, so obviously I need a check to deal with system pages that are not deleteable to prevent the error.

I'll also look further into the icon_select error - I saw this a few times too - for some reason that page field is not being fully configured during the import, even though the other page fields are importing perfectly.

I am glad to see the 196 pages imported though - that looks like the full set :)

I don't really have any time for this for the next few days, but hopefully I can get these things sorted out after that.

1 person likes this

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By psy
      ProcessWire DropboxAPI on GitHub: https://github.com/clipmagic/DropboxAPI
       
      This module is a wrapper for Kunal Varma's Dropbox PHP SDK https://github.com/kunalvarma05/dropbox-php-sdk
      It was created to perform a specific function for a client, ie upload PDF files on a PW website to a specific Dropbox account folder.
      The PDF files, created using @Wanze's excellent PagesToPdf module using the WirePDF functions, are generated from Formbuilder forms completed by front-end site visitors.  Works a treat!
      There's more that could be done to allow ProcessWire to take advantage of all the features of the SDK, eg downloads, multiple Dropbox accounts, etc. You are welcome to request changes and/or fork the GitHub project to extend the feature set.
      Enjoy!
      System requirements
      PHP 5.6.4 or greater Composer The PHP mbstring extension General information
      This module enables you to access a single Dropbox (www.dropbox.com) account to upload files from a ProcessWire website.
      All kudos to (https://github.com/kunalvarma05/dropbox-php-sdk) for the PHP API.
      First steps
      Visit (https://www.dropbox.com/developers) and read the documentation. Log into Dropbox and create a new application. It's recommended to limit ProcessWire App access to a specified folder Make a note of the App key, the App secret and the name of the Dropbox folder Installation
      Download the zip file into your site/modules folder then expand the zip file. Next, login to ProcessWire > go to Modules > click "Refresh". You should see a note that a new module was found. Install the DropboxAPI module. Configure the module with your App key, App secret and your Call Back URL You need to generate a Dropbox access token to enable your site to communicate with the nominated Dropbox account. Dropbox will generate a token for you or you can create a page for the front end of your ProcessWire site with a template to submit the token request to Dropbox, eg:
      <?php namespace ProcessWire; $drop = $modules->get('DropboxAPI'); if ($input->get->code && $input->get->state) { $code = $sanitizer->text($input->get->code); $state = $sanitizer->text($input->get->state); //Fetch the AccessToken $accessToken = $drop->getAccessToken($code, $state); echo "Copy/paste this code into the module configuration: " . $accessToken; } else { echo "<p><a href='" . $drop->getAuthURL() . "'>Log in with Dropbox</a></p>"; } ?> Once you have entered the token in the module configuration, you can unpublish this page.
      Usage
      Read the dropbox-php-sdk documentation!
      An example template for sending a file to a Dropbox App folder from ProcessWire:
      <?php namespace ProcessWire; use Kunnu\Dropbox\Dropbox; use Kunnu\Dropbox\DropboxApp; use Kunnu\Dropbox\DropboxFile; // send pdf to Dropbox $drop = $modules->get('DropboxAPI'); $app = new DropboxApp($drop->app_key, $drop->app_secret, $drop->authorization_code); if ($app) { //Configure Dropbox service $dropbox = new Dropbox($app); $dropboxFile = new DropboxFile('/path/to/myfilename.pdf'); $file = $dropbox->upload($dropboxFile, "/myfilename.pdf", ['autorename' => true]); //Uploaded File meta data if ($file) { $success = $file->getName() . " uploaded to Dropbox"; $drop->log($success); } }  
       
    • By sirhc
      Hi all,
      Im trying to install this core module and got the error of not having the ImageMagick library installed.
      But if im checking for the version of my ImageMagick installed on my server via SSH im getting this response;
      identify -version
      Version: ImageMagick 6.9.4-9 Q16 i386 2016-06-21 http://www.imagemagick.org
      Copyright: Copyright (C) 1999-2016 ImageMagick Studio LLC
      License: http://www.imagemagick.org/script/license.php
      Features: Cipher DPC 
      Delegates (built-in): bzlib freetype gslib jng jpeg png ps tiff xml zlib
      Does this even mean the library is installed? And if yes, why cant i install my module?
      Pleaassee help me
      Greetz.
    • By kixe
      FieldtypeColor is on github

      Fieldtype stores a 32bit integer value reflecting a RGBA value.
      Input
      5 types of Inputfields provided
      Html5 Inputfield of type='color' (if supported by browser)    Inputfield type='text' expecting a 24bit hexcode string (RGB). Input format: '#4496dd'.
      The background color of the input field shows selected color Inputfield of type='text' expecting 32bit hexcode strings (RGB + alpha channel) Input format: '#fa4496dd' Inputfield with Spectrum Color Picker
      (Options modifiable) Inputfield type='text' with custom JavaScript and/or CSS (since version 1.0.3) Output
      Define output format under 'Details' tab in field settings. Select from the following 8 options
      string 6-digit hex color. Example: '#4496dd' string 8-digit hex color (limited browser support).  Example: '#fa4496dd' string CSS color value RGB. Example: 'rgb(68, 100, 221)' string CSS color value RGB. Example: 'rgba(68, 100, 221, 0.98)' string CSS color value RGB. Example: 'hsl(227, 69.2%, 56.7%)' string CSS color value RGB. Example: 'hsla(227, 69.2%, 56.7%, 0.98)' string 32bit raw hex value. Example: 'fa4496dd' int 32bit. Example: '4198799069' (unformatted storage value) The Fieldtype includes
      Spectrum Color Picker by Brian Grinstead

      SCREENSHOTS
      Input type=text with changing background and font color (for better contrast)

      Input type=color (in Firefox)


      Javascript based input (Spectrum Color Picker)


      Settings Output

       
      Settings Input

    • By zlitrox
      Hi!
      I'm relatively new to the world of Processwire, but so far I'm really impressed by this CMS and its ease, power and speed.
      I've been looking at the different field types and also existing CKEditor modules (pwimage and pwlink).
      Here's what i want to achieve:
      I would like to use or create a own module which allows my admin users to upload pdf files inline in a CKEditor field (page content) the same way images are handled. It should be possible to upload a pdf file and specify its name and from the file uploaded and name i want to show an icon or image inline in editor and the output for that upload should be customized based on the module. Lets make it simple and say that i upload datasheet-1.pdf and want it to be named "My product datasheet", i want the output to be:
      <a href="{ link to uploaded pdf file }">My product datasheet</a>  
      Also the uploaded pdf files should be related to the page the user is editing.
       
      Is this doable? 
       
      Any guidance would be greatly appreciated! Thanks in advance.
      PS. I'm a PHP programmer so i know it would require some custom code, but the real question is where do i start, what should it take and is it even possible to do?
    • By flydev
      Auth2Login for ProcessWire
      A Module which give you ability to login an existing user using your favorite thrid-party OAuth2 provider (i.e. Facebook, GitHub, Google, LinkedIn, etc.)..
      You can login from the backend to the backend directly or render a form on the frontend and redirect the user to a choosen page.
      Built on top of ThePhpLeague OAuth2-Client lib.
      Registration is not handled by this module but planned.
       
      Howto Install
      Install the module following this procedure:
       - http://modules.processwire.com/modules/oauth2-login/
       - https://github.com/flydev-fr/OAuth2Login
      Next step, in order to use a provider, you need to use Composer to install each provider
      ie: to install Google, open a terminal, go to your root directory of pw and type the following command-line: composer require league/oauth2-google
      Tested providers/packages :
          Google :  league/oauth2-google     Facebook: league/oauth2-facebook     Github: league/oauth2-github     LinkedIn: league/oauth2-linkedin
      More third-party providers are available there. You should be able to add a provider by simply adding it to the JSON config file.

      Howto Use It
      First (and for testing purpose), you should create a new user in ProcessWire that reflect your real OAuth2 account information. The important informations are, Last Name, First Name and Email. The module will compare existing users by firstname, lastname and email; If the user match the informations, then he is logged in.
      ie, if my Google fullname is John Wick, then in ProcessWire, I create a new user  Wick-John  with email  johnwick@mydomain.com
      Next step, go to your favorite provider and create an app in order to get the ClientId and ClientSecret keys. Ask on the forum if you have difficulties getting there.
      Once you got the keys for a provider, just paste it into the module settings and save it. One or more button should appear bellow the standard login form.
      The final step is to make your JSON configuration file.
      In this sample, the JSON config include all tested providers, you can of course edit it to suit your needs :
      { "providers": { "google": { "className": "Google", "packageName": "league/oauth2-google", "helpUrl": "https://console.developers.google.com/apis/credentials" }, "facebook": { "className": "Facebook", "packageName": "league/oauth2-facebook", "helpUrl": "https://developers.facebook.com/apps/", "options": { "graphApiVersion": "v2.10", "scope": "email" } }, "github": { "className": "Github", "packageName": "league/oauth2-github", "helpUrl": "https://github.com/settings/developers", "options": { "scope": "user:email" } }, "linkedin": { "className": "LinkedIn", "packageName": "league/oauth2-linkedin", "helpUrl": "https://www.linkedin.com/secure/developer" } } }  
      Backend Usage
      In ready.php, call the module :
      if($page->template == 'admin') { $oauth2mod = $modules->get('Oauth2Login'); if($oauth2mod) $oauth2mod->hookBackend(); }  
      Frontend Usage
      Small note: At this moment the render method is pretty simple. It output a InputfieldForm with InputfieldSubmit(s) into wrapped in a ul:li tag. Feedbacks and ideas welcome!
      For the following example, I created a page login and a template login which contain the following code :
      <?php namespace ProcessWire; if(!$user->isLoggedin()) { $options = array( 'buttonClass' => 'my_button_class', 'buttonValue' => 'Login with {provider}', // {{provider}} keyword 'prependMarkup' => '<div class="wrapper">', 'appendMarkup' => '</div>' ); $redirectUri = str_lreplace('//', '/', $config->urls->httpRoot . $page->url); $content = $modules->get('Oauth2Login')->config( array( 'redirect_uri' => $redirectUri, 'success_uri' => $page->url ) )->render($options); }

       
      Screenshot