adrian

Migrator

Recommended Posts

Ok. I installed 2.8.5 with the ProcessMigrator that you suggested + MigratorWordpress. It seemed to surpass the initial error but now after uploading the wordpress XML file is showing me this error:

 Can't save page 1: /: Selected homepage template cannot be used because it does not define access.

After that, going back to the home screen I only see two pages created (Categories & Tags).

I noticed that the error above is showing only when I use replace/overwrite. If I choose Append, it was showing that all 571 posts were processed properly but in the pages I did not see them.

I decided to try and convert the posts to pages (with comments) and this time I was able to see all the pages created. Now the issue is that I do not see the comments so I guess I should redo the import but setup a template with all the fields created in advance and have them added to a template that would be used for the imported pages. On top of that I did not have installed the comments field module so it seems like it might work once I rethink the whole process again.

Do you have any idea how to make replace/overwrite work without the error saving the page?

P.S. It looks like if I use Append, there are no comments attached to the page even though I have the module now installed and created a template with all fields that were initially created during the Append. So I need to figure out how to define the access for the selected template and try again.

Share this post


Link to post
Share on other sites

@MilenKo - I don't think that error has anything to do with Migrator. Please google this for the access issue: 

site:processwire.com "Selected homepage template cannot be used because it does not define access"

Once that is sorted, I think everything else should work fine. I have only ever imported WP posts into a subpage, like "blog", but importing into home should also be fine.

Append vs Replace/Overwrite should both work fine, but if you are importing to a new install, I would go with append.

I would suggest reducing your xml to just a few posts while you try different things - just so each test is much faster.

BTW, I have never needed to change the post type for normal WP blog post type pages - they just worked as is.

Late here, so won't be able to respond again until tomorrow, so good luck!

Share this post


Link to post
Share on other sites

Adrian, thanks for the points. The need to switch the post type is that the themes I used were using custom post types so most of the converters are made to just look into the standard type - posts or else, but not news (for example). Switching the post type is not an issue and is semi automatic (semi, as I had to do around 100 posts at once to avoid the server timeout.)

I will google the error and see where would that lead. Even if the pages are not properly inserted, but the comments are assigned to a title, that would fully satisfy my needs. Again, I have planned the post cleanup no matter of the CMS but PW is now my top prefference.

Share this post


Link to post
Share on other sites

@MilenKo - if you fix that error and posts are still not being imported fully, PM me your XML file (if you can) and I'll take a look - it should work fine.

  • Like 1

Share this post


Link to post
Share on other sites

I will Adrian. The posts are imported correctly but the comments are not there. This is the only issue. I have a feeling that I might have not set up properly the comments field or something else. There are no issues with sending you the XML file  but I would first try to recover the issue myself using your tool and other infos and in case I am failing, will use the Joker ;)

Share this post


Link to post
Share on other sites

@adrian Looks like I had some progress by creating a simple template matching most of the field names etcl with the wordpress one. Installed the FieldTypeComments and imported the XML. For an unknown reason the name/content of my home page changed to one of the pages from the import as well as the comments are imported in repeating one commment several times (ex, Comment1 x10, Comment 2 x10 etc.) I know I need to go over every page and make the fields properly so that won't be an issue even if I have to work over all comments in the page, but in the admin of the comments I just see them with Pending status and I am unable to just delete the repeating ones.

Share this post


Link to post
Share on other sites

@MilenKo - I don't know why this is proving so problematic for you. I haven't ever imported a WP blog into the PW homepage - always into /blog/, but it has always worked perfectly and I have never had to create any fields ahead of time, including the comment fields. The pending status is a known issue. I had been waiting on the response to this: https://github.com/ryancramerdesign/ProcessWire/issues/1034, which ended up here: https://github.com/processwire/processwire-issues/issues/88 - it looks like it can be solved now.

 

  • Like 1

Share this post


Link to post
Share on other sites

@adrian, I did not import the WP blog directly to homepage but created a page called Import and setup the module to point to it. Somehow I ended up with the home page changed to one of the WP Posts, and inside the include folder I got all the pages with the comments. If you feel like having some time, I would send you the XML archive to try and see if it works for you. Thanks in advance ;)

 

Share this post


Link to post
Share on other sites

Btw, thinking of the conversion from WP to PW, it seems like I need to export from the PW profile the pages, images, comments etc. to the theme profile as well so the fun will be on MAX but it worth every effort knowing how many years I have tried to work the things out with WP and how for a really short time I got my first theme (thanks to you and others).

Share this post


Link to post
Share on other sites

Hi @MilenKo - thanks for the XML file - it was very helpful.

I have committed a new version of Migrator to Github that fixes the duplicate comments issue and also now properly sets the approved status of comments.

Note that I didn't test your complete file (because it would take quite a long time I expect). I cut it down to 5 posts, but with those everything is now working as expected. I didn't see the issue with the homepage being renamed, but maybe there is one post in there that is doing something strange. Could you please test with a limited set to make sure everything is working at your end. And then try with all and let me know about the homepage issue. If you could narrow it down to a particular post then I should be able to fix.

It's probably a good idea to start with a completely fresh PW install (after all your previous attempts, something might be messed up by now) - you can even let Migrator install the comments module and set up the field etc. 

Thanks,
Adrian

  • Like 2

Share this post


Link to post
Share on other sites

@adrian Thank you very much for the update and I am glad I helped a bit in the process.

I will try to import a few posts first and see how it goes after which I will try to import the whole database. If the posts are imported properly but only one changes the homepage - I can totally live with this, but I understand your point of having that fixed so that it does not affect somebody else using the module.

To import the WP to PW I am definitely using the latest PW 3.X with a blank profile. After I do that, I will just need to import my existing profile I have almost finished. But let's do this step by step and at the end we will have a nice and smooth approach.

Will let you know as soon as finish both tests.

  • Like 2

Share this post


Link to post
Share on other sites
Just now, MilenKo said:

To import the WP to PW I am definitely using the latest PW 3.X with a blank profile.

Oh yes, I should mention that I did my testing with 3.0.55, but it should still work in 2.x

Thanks for helping to narrow down what might be causing that homepage change. I think it will be a matter of doing chunks of posts (maybe 20 at a time, or something like that) and then narrowing down from there. There must be something weird :)

  • Like 1

Share this post


Link to post
Share on other sites

Oh, I should be the one thanking you.

I was thinking about the post import - would it be possible to do that limitting via the API and some coding versus cutting the posts manually? Saying the import goes in a step of 20, 50 etc. and keeps on doing it until the last one is processed. This way you might import a bigger DB without the need of messing around with the XML.

Share this post


Link to post
Share on other sites
Just now, MilenKo said:

I was thinking about the post import - would it be possible to do that limitting via the API and some coding versus cutting the posts manually? Saying the import goes in a step of 20, 50 etc. and keeps on doing it until the last one is processed. This way you might import a bigger DB without the need of messing around with the XML.

It would be possible (and probably a good idea) to add this functionality to Migrator, but it's beyond what I have time for at the moment I'm afraid.

  • Like 2

Share this post


Link to post
Share on other sites

Huh, it is true - ideas we could have plenty, but there are only 24 hours in 1 day and they are never enough ;)

Thank you again for being so helpful...

  • Like 2

Share this post


Link to post
Share on other sites

Hello @adrian I tested the importer and got an error. Here are the steps I took and the error to just check if I did everything right or missed something:

1. I cut the whole WordPress file (14-15Mb size)  in 6 pieces of 2Mb to make it much smaller and reduce the load etc. as per your instructions. To reduce the size of the file I used a simple app called wxrsplitter that I know worked for WP before.

2. Installed fresh PW 3.0.55 with blank profile and only added ProcessMigrator & MigratorWordpress (I downloaded latest update for ProcessMigrator from the git source but MigratorWordpress was kept the same as I did not se any updated files).

3. Created a page Wordpress and assigned it as parent at the Import page.

4. Selected to import user details.

5. Selected to Import created/modified dates.

6. Choose to automatically download and install missing filetypes.

7. As data source used WordPress Migrator and pointed to the first chunk.

The error I got after the import is:

Error: Call to a member function eq() on a non-object (line 1704 of C:\OpenServer\domains\recipemagician.pw\site\assets\cache\FileCompiler\site\modules\ProcessMigrator\ProcessMigrator.module) 

This error message was shown because: you are logged in as a Superuser. Error has been logged.

If I refresh the page, then it continued and finished with the informing message of how many pages were imported successfully. The same thing is applied for the rest of the files or if I use the file at once (which by obvious reasons takes more time to import but does not fail other than the error).

So I pushed the whole file for the test and was able to import 559 posts as child pages of Wordpress. I checked a few pages with comments and they were all posted in the proper amount and approved with an ability to change the status if needed!

It is clear now that all posts were imported. I am unable yet to say would all comments be properly imported but I don't see a reason if one works fine why the rest would not. So will consider that also done perfectly.

In regards to the Home page title, it changed again (before it was Home and now is imported a the first page alphabetically added. Because of that it looses its fields and comments (this happened when I imported the XML chunks). Will try now to import the whole file but I doubt that this would happen again. For sure I can preserve the content of the page imported to the Home by creating and adding all the necessary fields to it and reimporting, but just reporting that something is still to be fixed. If a need be, I can provide you with the slices of XML.

  • Like 1

Share this post


Link to post
Share on other sites

Ok. Tried with the whole file and for the testing purposes inside the Home I created a child page WordPress inside of which I created another child calling it Import. Pointed the module to the Import page as parent and thought that this way the Home would not change but would the Wordpress. Well I was wrong - the Home title changed again to the same so for some reason the import grabs a page (probably the first) and imports it to Home.

On top of that, if I try to import the whole file, then some of the pages were not imported (those that were originaly as Wordpress pages). Categories list is empty too.

After the import in both cases I noticed another error:

 SQLSTATE[42S22]: Column not found: 1054 Unknown column 'post' in 'field list' along with the message of successful import. So for now importing the XML in chunks works much better and properly includes categories as well as users etc. Even if I will have to recreate the page that is imported in the Home, I am still totally OK but just reporting to make sure others won't be able to move on.

  • Like 1

Share this post


Link to post
Share on other sites

@MilenKo  - thanks for all those details. I have pushed a new version which fixes that eq() error.

I finally did a full import of your file and I think I now know why you are having troubles. Your XML is for the entire Wordpress site. I have only ever used this to migrate blog posts. I would suggest that you create an XML export of just posts (I think that's an option). I think that will probably take care of the homepage renaming issue, as well as the categories and tags problems. It's all about the pages being relative to the correct parent and when you import pages and posts at once, it messes that up. Obviously this should be fixed, but I think that for the moment if you do this, you'll get a complete import of all posts - fingers crossed :)

  • Like 1

Share this post


Link to post
Share on other sites

Adrian, you are 100% right and how come I did not think of that earlier. I will export now only the posts, but I am not sure if I do that will the export include the comments too. Will test it right away and see what is happening. One thing is sure though - nobody would care if the name of the old website was imported or not as those settings will have to be changed to apply with PW and new theme. 

Crossing my fingers and trying it now with the latest version of the Migrator.

Share this post


Link to post
Share on other sites

Hello @adrian . I installed your updated module and tested to import ONLY THE POSTS export. To be clear, I still needed to convert all custom type posts to the standard Wordpress posts as otherwise it only creates a few empty pages and that is all.

I tried to start the import this time with the entire XML so as expected I got an error:

SQLSTATE[HY000]: General error: 2006 MySQL server has gone away  (in other words it means that the import took quite a time and server gives up). 

To help the server "survive" I decided to edit the my.cnf changing the wait_timeout from 60 to 6000 and this time it worked fine.

Unfortunately the issue still exists with the Home page being renamed and unfortunately it is a page that has some comments in it too. It appears that it does not just import to the parent of the page pointed to the module but straight to home.

Another thing I just noticed is that the comments date is not preserved, so most of the comments appear to be posted with a timestamp of the import. It might not be such a big issue, however it won't make much sense to have the page added 3-4 years ago and containing 50-100 comments within the last XX minutes.

In other words if the comments timestamp can be fixed and the page that imports into the home properly inserted to the parent, it will be all perfect!

 

Share this post


Link to post
Share on other sites

Thanks @MilenKo - I'll take a look at the comment timestamp issue tomorrow. Could you please PM me the new XML with just the posts. I might also look at working with the full one - we'll see.

How did you go with Categories with just the posts in the XML?

Share this post


Link to post
Share on other sites

@adrian I did not care much of the categories as I am planning on completely redoing most of the post information, pictures, tags, categories etc. I noticed that after moving the custom type posts to the standard WP post all the posts got to Uncategorized (the default WP category). During the import I do see the category created and pages have the Uncategorized as their category so my guess is that if everything was OK with the posts and categories, they will be imported properly.

One other thing I just noticed is that the tags are also not imported and the field that was automatically created by the module is not allowing to add new tags (need to change a few settings to allow it). Again, reaching to this level is waaay more than I have ever expected so I am sure others would follow switching from WP to PW and enjoying the simplicity and elegance of the code (not to mention the support of the community!)

I am PM-ing you the XML and thanking you in advance for looking into it.

Share this post


Link to post
Share on other sites
1 minute ago, MilenKo said:

Again, reaching to this level is waaay more than I have ever expected so I am sure others would follow switching from WP to PW and enjoying the simplicity and elegance of the code (not to mention the support of the community!)

Thanks, but I am certain this used to work fully, so I would like to get it back to that state.

I have the comment date sorted out, but am working on a couple of other things and have to head out shortly, so probably won't post until tomorrow, so will look again at categories and tags.

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By louisstephens
      I have been following @bernhard's great tutorial on creating a Dashboard and everything has very easy to understand so far. However, I need to add a select field that a template is using (so users can create a new page from the dashboard), but I am really at a complete lost on how to achieve this. I have been googling, but can't really find anything on the matter.
      $field = $this->modules->get('InputfieldSelect'); $field->name = "Select Manufacturer"; $field->findPagesSelector = "template=basic-page"; //not needed for select field $field->labelFieldName = "title"; $field->name = "Manufacturer Select"; $field->columnWidth = 34; $fieldset->add($field); I have the above code, which was originally an InputfieldPage, but since this is an options field, I have changed it. Has anyone actually gotten a select from a template (and its' values) in a module?
    • By Lex Sanchez
      Video or Social Post Embed
      Based on the TextformatterVideoEmbed module developed by Ryan Cramer, we have added the possibility to embed publications of the main social networks Facebook, Twitter and Instagram.
      ProcessWire Textformatter module that enables translation of YouTube, Vimeo, Instagram, Facebook, Twitter or Issuu URLs to full embed codes, resulting in a viewable video or social post in textarea fields you apply it to.
      How to install
      Download or Clone from Github: https://github.com/lexsanchez/VideoOrSocialPostEmbed
      Copy the VideoOrSocialPostEmbed.module file to your /site/modules/ directory (or place it in /site/modules/VideoOrSocialPostEmbed/).
      Click check for new modules in ProcessWire Admin Modules screen. Click install for the module labeled: "Video or Social Post Embed".
      Now you will be on the module config screen. Please make note of the config options and set as you see fit.
      How to use
      Edit your body field in Setup > Fields (or whatever field(s) you will be placing videos in). On the details tab, find the Text Formatters field and select "Video or Social Post Embed". Save.
      Edit a page using the field you edited and paste in YouTube, Vimeo, Facebook, Twitter, Instagram and/or Issuu URLs each on their own paragraph.
      Example
      How it might look in your editor (like TinyMCE):
      How it works
      This module uses YouTube, Vimeo, Instagram, Facebook, Twitter and Issuu oEmbed services to generate the embed codes populated in your content. After these services are queried the first time, the embed code is cached so that it doesn't need to be pulled again.
      Configuration
      You may want to update the max width and max height settings on the module's configuration screen. You should make these consistent with what is supported by your site design.
      If you change these max width / max height settings you may also want to check the box to clear cache, so that YouTube/Vimeo/Facebook/Twitter/Instagram/Issuu oembed services will generate new embed codes for you.
      Using with Markdown, Textile or other LML
      This text formatter is looking for a YouTube, Vimeo, Instagram, Facebook, Twitter or Issuu video URL surrounded by paragraph tags. As a result, if you are using Markdown or Textile (or something else like it) you want that text formatter to run before this one. That ensures that the expected paragraph tags will be present when VideoOrSocialPostEmbed runs. You can control the order that text formatters are run in by drag/drop sorting in the field editor.
      Copyright 2018 by Ryan Cramer / Updated by Lex Sanchez
    • By kixe
      Email Verification
      This module provides functions to validate email adresses and hosts. The module generates a textfile for blacklisted mailhosts (trashmail), which will be always up to date.
      Download
      https://modules.processwire.com/modules/email-verification/
      API
      // get module $mailcheck = $modules->get('EmailVerification'); // return bool/ string - automatted update of blacklist file $mailcheck->blacklisted(email|domain) // return bool - validate a top level domain, checks against IANA list $mailcheck->validTLD(tld) // return array of punycoded TLDs - cyclic updated, data pulled from IANA $mailcheck->getTLDs(cycle=2592000) // return bool - checks syntax converts to punycode $mailcheck->validDomainName(domain); // return bool - checks punycode encoded syntax $mailcheck->validHostName(host); // return bool - checks syntax and accessibility $mailcheck->validHost(email|domain) // add a single value to blacklist $mailcheck->addToBlacklist(email|domain) USAGE
      $mailcheck = $modules->get('EmailCheck'); $email = 'susi@trashmail.com'; if($mailcheck->blacklisted($email)) echo 'Email Provider not allowed'; if(!$mailcheck->validHost($email)) echo 'Mailhost not available'; Example blacklist file:  blacklist.txt
       
    • By teppo
      Code Blocks Textformatter is a tiny Textformatter module I cooked up to add support for code blocks to text/textarea/RTE fields on some of the sites I work with. Unlike a full-blown Markdown Textformatter – which is something that we already have in the core – this module simply adds support for fenced and inline code blocks.
      The syntax is based on the GitHub code block documentation, so please refer to that for additional instructions. The README at GitHub also includes some basic examples. As with any Textformatter, in order to enable this one, install it and enable it via field settings. Note that there's no syntax highlighting built in (at least for the time being), so use a tool of your choice for that – personally I prefer Prism.js.
      Since this module doesn't use a Markdown tool behind the scenes, but rather some home baked regular expressions, there's always the possibility that I've missed something – but please let me know if you use this module and run into any issues. On the other hand this module should be relatively fast and unobtrusive, as there are no unnecessary bits of code to run 
      GitHub repository: https://github.com/teppokoivula/TextformatterCodeBlocks Modules directory: http://modules.processwire.com/modules/textformatter-code-blocks/