Module Migrator

[MilenKo]

Ok. I installed 2.8.5 with the ProcessMigrator that you suggested + MigratorWordpress. It seemed to surpass the initial error but now after uploading the wordpress XML file is showing me this error:

 Can't save page 1: /: Selected homepage template cannot be used because it does not define access.

After that, going back to the home screen I only see two pages created (Categories & Tags).

I noticed that the error above is showing only when I use replace/overwrite. If I choose Append, it was showing that all 571 posts were processed properly but in the pages I did not see them.

I decided to try and convert the posts to pages (with comments) and this time I was able to see all the pages created. Now the issue is that I do not see the comments so I guess I should redo the import but setup a template with all the fields created in advance and have them added to a template that would be used for the imported pages. On top of that I did not have installed the comments field module so it seems like it might work once I rethink the whole process again.

Do you have any idea how to make replace/overwrite work without the error saving the page?

P.S. It looks like if I use Append, there are no comments attached to the page even though I have the module now installed and created a template with all fields that were initially created during the Append. So I need to figure out how to define the access for the selected template and try again.

[adrian]

@MilenKo - I don't think that error has anything to do with Migrator. Please google this for the access issue: 

site:processwire.com "Selected homepage template cannot be used because it does not define access"

Once that is sorted, I think everything else should work fine. I have only ever imported WP posts into a subpage, like "blog", but importing into home should also be fine.

Append vs Replace/Overwrite should both work fine, but if you are importing to a new install, I would go with append.

I would suggest reducing your xml to just a few posts while you try different things - just so each test is much faster.

BTW, I have never needed to change the post type for normal WP blog post type pages - they just worked as is.

Late here, so won't be able to respond again until tomorrow, so good luck!

[MilenKo]

Adrian, thanks for the points. The need to switch the post type is that the themes I used were using custom post types so most of the converters are made to just look into the standard type - posts or else, but not news (for example). Switching the post type is not an issue and is semi automatic (semi, as I had to do around 100 posts at once to avoid the server timeout.)

I will google the error and see where would that lead. Even if the pages are not properly inserted, but the comments are assigned to a title, that would fully satisfy my needs. Again, I have planned the post cleanup no matter of the CMS but PW is now my top prefference.

[adrian]

@MilenKo - if you fix that error and posts are still not being imported fully, PM me your XML file (if you can) and I'll take a look - it should work fine.

[MilenKo]

I will Adrian. The posts are imported correctly but the comments are not there. This is the only issue. I have a feeling that I might have not set up properly the comments field or something else. There are no issues with sending you the XML file  but I would first try to recover the issue myself using your tool and other infos and in case I am failing, will use the Joker

[MilenKo]

@adrian Looks like I had some progress by creating a simple template matching most of the field names etcl with the wordpress one. Installed the FieldTypeComments and imported the XML. For an unknown reason the name/content of my home page changed to one of the pages from the import as well as the comments are imported in repeating one commment several times (ex, Comment1 x10, Comment 2 x10 etc.) I know I need to go over every page and make the fields properly so that won't be an issue even if I have to work over all comments in the page, but in the admin of the comments I just see them with Pending status and I am unable to just delete the repeating ones.

[adrian]

@MilenKo - I don't know why this is proving so problematic for you. I haven't ever imported a WP blog into the PW homepage - always into /blog/, but it has always worked perfectly and I have never had to create any fields ahead of time, including the comment fields. The pending status is a known issue. I had been waiting on the response to this: https://github.com/ryancramerdesign/ProcessWire/issues/1034, which ended up here: https://github.com/processwire/processwire-issues/issues/88 - it looks like it can be solved now.

 

[MilenKo]

@adrian, I did not import the WP blog directly to homepage but created a page called Import and setup the module to point to it. Somehow I ended up with the home page changed to one of the WP Posts, and inside the include folder I got all the pages with the comments. If you feel like having some time, I would send you the XML archive to try and see if it works for you. Thanks in advance

 

[adrian]

@MilenKo - sure, send it through - I'll try to test later today.

[MilenKo]

Done

[MilenKo]

Btw, thinking of the conversion from WP to PW, it seems like I need to export from the PW profile the pages, images, comments etc. to the theme profile as well so the fun will be on MAX but it worth every effort knowing how many years I have tried to work the things out with WP and how for a really short time I got my first theme (thanks to you and others).

[adrian]

Hi @MilenKo - thanks for the XML file - it was very helpful.

I have committed a new version of Migrator to Github that fixes the duplicate comments issue and also now properly sets the approved status of comments.

Note that I didn't test your complete file (because it would take quite a long time I expect). I cut it down to 5 posts, but with those everything is now working as expected. I didn't see the issue with the homepage being renamed, but maybe there is one post in there that is doing something strange. Could you please test with a limited set to make sure everything is working at your end. And then try with all and let me know about the homepage issue. If you could narrow it down to a particular post then I should be able to fix.

It's probably a good idea to start with a completely fresh PW install (after all your previous attempts, something might be messed up by now) - you can even let Migrator install the comments module and set up the field etc. 

Thanks,
Adrian

[MilenKo]

@adrian Thank you very much for the update and I am glad I helped a bit in the process.

I will try to import a few posts first and see how it goes after which I will try to import the whole database. If the posts are imported properly but only one changes the homepage - I can totally live with this, but I understand your point of having that fixed so that it does not affect somebody else using the module.

To import the WP to PW I am definitely using the latest PW 3.X with a blank profile. After I do that, I will just need to import my existing profile I have almost finished. But let's do this step by step and at the end we will have a nice and smooth approach.

Will let you know as soon as finish both tests.

[adrian]

Just now, MilenKo said:

To import the WP to PW I am definitely using the latest PW 3.X with a blank profile.

Oh yes, I should mention that I did my testing with 3.0.55, but it should still work in 2.x

Thanks for helping to narrow down what might be causing that homepage change. I think it will be a matter of doing chunks of posts (maybe 20 at a time, or something like that) and then narrowing down from there. There must be something weird

[MilenKo]

Oh, I should be the one thanking you.

I was thinking about the post import - would it be possible to do that limitting via the API and some coding versus cutting the posts manually? Saying the import goes in a step of 20, 50 etc. and keeps on doing it until the last one is processed. This way you might import a bigger DB without the need of messing around with the XML.

[adrian]

Just now, MilenKo said:

I was thinking about the post import - would it be possible to do that limitting via the API and some coding versus cutting the posts manually? Saying the import goes in a step of 20, 50 etc. and keeps on doing it until the last one is processed. This way you might import a bigger DB without the need of messing around with the XML.

It would be possible (and probably a good idea) to add this functionality to Migrator, but it's beyond what I have time for at the moment I'm afraid.

[MilenKo]

Huh, it is true - ideas we could have plenty, but there are only 24 hours in 1 day and they are never enough

Thank you again for being so helpful...

[MilenKo]

Hello @adrian I tested the importer and got an error. Here are the steps I took and the error to just check if I did everything right or missed something:

1. I cut the whole WordPress file (14-15Mb size)  in 6 pieces of 2Mb to make it much smaller and reduce the load etc. as per your instructions. To reduce the size of the file I used a simple app called wxrsplitter that I know worked for WP before.

2. Installed fresh PW 3.0.55 with blank profile and only added ProcessMigrator & MigratorWordpress (I downloaded latest update for ProcessMigrator from the git source but MigratorWordpress was kept the same as I did not se any updated files).

3. Created a page Wordpress and assigned it as parent at the Import page.

4. Selected to import user details.

5. Selected to Import created/modified dates.

6. Choose to automatically download and install missing filetypes.

7. As data source used WordPress Migrator and pointed to the first chunk.

The error I got after the import is:

Error: Call to a member function eq() on a non-object (line 1704 of C:\OpenServer\domains\recipemagician.pw\site\assets\cache\FileCompiler\site\modules\ProcessMigrator\ProcessMigrator.module) 

This error message was shown because: you are logged in as a Superuser. Error has been logged.

If I refresh the page, then it continued and finished with the informing message of how many pages were imported successfully. The same thing is applied for the rest of the files or if I use the file at once (which by obvious reasons takes more time to import but does not fail other than the error).

So I pushed the whole file for the test and was able to import 559 posts as child pages of Wordpress. I checked a few pages with comments and they were all posted in the proper amount and approved with an ability to change the status if needed!

It is clear now that all posts were imported. I am unable yet to say would all comments be properly imported but I don't see a reason if one works fine why the rest would not. So will consider that also done perfectly.

In regards to the Home page title, it changed again (before it was Home and now is imported a the first page alphabetically added. Because of that it looses its fields and comments (this happened when I imported the XML chunks). Will try now to import the whole file but I doubt that this would happen again. For sure I can preserve the content of the page imported to the Home by creating and adding all the necessary fields to it and reimporting, but just reporting that something is still to be fixed. If a need be, I can provide you with the slices of XML.

[MilenKo]

Ok. Tried with the whole file and for the testing purposes inside the Home I created a child page WordPress inside of which I created another child calling it Import. Pointed the module to the Import page as parent and thought that this way the Home would not change but would the Wordpress. Well I was wrong - the Home title changed again to the same so for some reason the import grabs a page (probably the first) and imports it to Home.

On top of that, if I try to import the whole file, then some of the pages were not imported (those that were originaly as Wordpress pages). Categories list is empty too.

After the import in both cases I noticed another error:

 SQLSTATE[42S22]: Column not found: 1054 Unknown column 'post' in 'field list' along with the message of successful import. So for now importing the XML in chunks works much better and properly includes categories as well as users etc. Even if I will have to recreate the page that is imported in the Home, I am still totally OK but just reporting to make sure others won't be able to move on.

[adrian]

@MilenKo  - thanks for all those details. I have pushed a new version which fixes that eq() error.

I finally did a full import of your file and I think I now know why you are having troubles. Your XML is for the entire Wordpress site. I have only ever used this to migrate blog posts. I would suggest that you create an XML export of just posts (I think that's an option). I think that will probably take care of the homepage renaming issue, as well as the categories and tags problems. It's all about the pages being relative to the correct parent and when you import pages and posts at once, it messes that up. Obviously this should be fixed, but I think that for the moment if you do this, you'll get a complete import of all posts - fingers crossed

[MilenKo]

Adrian, you are 100% right and how come I did not think of that earlier. I will export now only the posts, but I am not sure if I do that will the export include the comments too. Will test it right away and see what is happening. One thing is sure though - nobody would care if the name of the old website was imported or not as those settings will have to be changed to apply with PW and new theme. 

Crossing my fingers and trying it now with the latest version of the Migrator.

[MilenKo]

Hello @adrian . I installed your updated module and tested to import ONLY THE POSTS export. To be clear, I still needed to convert all custom type posts to the standard Wordpress posts as otherwise it only creates a few empty pages and that is all.

I tried to start the import this time with the entire XML so as expected I got an error:

SQLSTATE[HY000]: General error: 2006 MySQL server has gone away  (in other words it means that the import took quite a time and server gives up). 

To help the server "survive" I decided to edit the my.cnf changing the wait_timeout from 60 to 6000 and this time it worked fine.

Unfortunately the issue still exists with the Home page being renamed and unfortunately it is a page that has some comments in it too. It appears that it does not just import to the parent of the page pointed to the module but straight to home.

Another thing I just noticed is that the comments date is not preserved, so most of the comments appear to be posted with a timestamp of the import. It might not be such a big issue, however it won't make much sense to have the page added 3-4 years ago and containing 50-100 comments within the last XX minutes.

In other words if the comments timestamp can be fixed and the page that imports into the home properly inserted to the parent, it will be all perfect!

 

[adrian]

Thanks @MilenKo - I'll take a look at the comment timestamp issue tomorrow. Could you please PM me the new XML with just the posts. I might also look at working with the full one - we'll see.

How did you go with Categories with just the posts in the XML?

[MilenKo]

@adrian I did not care much of the categories as I am planning on completely redoing most of the post information, pictures, tags, categories etc. I noticed that after moving the custom type posts to the standard WP post all the posts got to Uncategorized (the default WP category). During the import I do see the category created and pages have the Uncategorized as their category so my guess is that if everything was OK with the posts and categories, they will be imported properly.

One other thing I just noticed is that the tags are also not imported and the field that was automatically created by the module is not allowing to add new tags (need to change a few settings to allow it). Again, reaching to this level is waaay more than I have ever expected so I am sure others would follow switching from WP to PW and enjoying the simplicity and elegance of the code (not to mention the support of the community!)

I am PM-ing you the XML and thanking you in advance for looking into it.

[adrian]

1 minute ago, MilenKo said:

Again, reaching to this level is waaay more than I have ever expected so I am sure others would follow switching from WP to PW and enjoying the simplicity and elegance of the code (not to mention the support of the community!)

Thanks, but I am certain this used to work fully, so I would like to get it back to that state.

I have the comment date sorted out, but am working on a couple of other things and have to head out shortly, so probably won't post until tomorrow, so will look again at categories and tags.