kongondo

Pages Export (export PW 2.x - import into PW 3.x)

Recommended Posts

Posted (edited)

Pages Export

This module is for specifically exporting ProcessWire 2.x sites for later importing into ProcessWire 3.x.

Github: Project Page

Modules Directory: https://modules.processwire.com/modules/process-pages-export/

Credits: Ryan Cramer

Background

As I make my modules ProcessWire 3.x-compatible only, I've had the need to re-create/mirror their ProcessWire 2.x test sites in respective ProcessWire 3.x sites. These ProcessWire 3.x sites (one for each module) were already in place and I didn't feel like re-doing them by exporting/importing site profiles. I also like working with JSON rather than other export formats. So, I decided to write a custom pages export/import script for moving the ProcessWire 2.x sites to their respective ProcessWire 3.x counterpart sites. I'd just finished the export side of things when I came across a post in the forums that reminded me that ProcessWire 3.x already boasts a pages export/import feature, although some of it is still in development. Great! I like its API (PagesExportImport.php) and GUI (ProcessPagesExportImport.module) so no need to re-invent the wheel. 

I still had the small problem of making sure my JSON export would be compatible with the JSON input that the ProcessWire 3.x import expects. No need to re-invent the wheel, again! I ditched my custom script and instead ported the export functionalities of ProcessWire 3.x Pages Export/Import for use in ProcessWire 2.2 - 2.7, specifically to help migrate older sites to ProcessWire 3.x.

Compatibility

The module and class have been tested and work in ProcessWire 2.2, 2.3, 2.4, 2,5, 2.6 and 2.7.  The module is currently tagged as 'in development' until Pages Import feature of ProcessWire 3.x is released as stable. Nonetheless, I have not encountered any issues so far in either the export or the ProcessWire 3.x import. I think Ryan is waiting until he can support more complex field types before tagging the ProcessWire 3.x Pages Export/Import as production-ready.

This is not a ProcessWire 3.x module and will never be. It has no such need :). Just in case you forget and try to install it in a ProcessWire 3.x site, the module will throw a WireException(). I will also not be porting the ProcessWire 3.x import functionality for use in ProcessWire 2.x. That will defeat the purpose here; to move sites to ProcessWire 3.x and not the other way round.

Supported Fields

  • All non-complex fields such as integer, text, textarea, etc
  • Page fields
  • Repeaters
  • File and Image fields

I think these cover most needs. Note: not yet tested with Multilingual fields.

Technical

To ensure exports will be compatible with ProcessWire 3.x Pages Import, where necessary, the module borrows (and amends as needed) methods from ProcessWire 3.x for use in ProcessWire 2.x. For instance, ProcessWire 3.x Pages Export/Import uses the new(-ish) $file functions found in WireFileTools. Rather than copy and include such files, the module only borrowed and amended needed methods. These are listed below.

PagesExport.php

  • From /wire/core/Functions.php: wireInstanceOf(), wireClassName() and wireClassParents()
  • From /wire/core/Fieldtype.php: getImportValueOptions() and getDatabaseSchema()
  • From /wire/core/WireFileTools.php: zip(), chmod() and mkdir()
  • From /wire/core/WireHttp.php: sendFile
  • From /wire/modules/Fieldtype/FieldtypeFile.module: exportValue() and exportDescription()
  • From /wire/modules/Fieldtype/FieldtypeImage.module: exportValue()
  • From /wire/modules/Fieldtype/FieldtypePage.module: exportValue() and exportValuePage()
  • From /wire/modules/Fieldtype/FieldtypeRepeater.module: exportValue()
  • From /wire/core/Fieldtype/WireTempDir.php: create(), createName() and getTempDir()
  • All the export methods from the /wire/core/PagesExportImport.php class

ProcessPagesExport.module

  • All the export methods from /wire/modules/process/ProcessPagesExportImport/ProcessPagesExportImport.module

Newer methods such as $this->wire() will gracefully degrade to the older wire() function, ensuring smooth and uniform operation in ProcessWire 2.2  - 2.7.

Use

This module and class is for supersusers only and has only 1 aim; to export ProcessWire 2.x sites ready for importing into ProcessWire 3.x sites. You can either install (like any other module) and use the process module (ProcessPagesExport.module) or skip the install and just include and use the class (PagesExport.php) to export your sites.

Both the module (Export GUI) and API require that you are logged in as a supersuser before you can use them. The PagesExport class has a gateway method and option not found in the original class (PagesExportImport). The method export() allows access to the three export methods in the original class, i.e. pagesToArray(), exportJSON() and exportZip(). See example usage below.

GUI/Process Module

On install, the module will create a new admin page Export Pages. Please note that unlike the original code, this page is created directly under /admin/ and not /admin/pages/. Click on Export Pages to start.

Nothing much has changed from the original ProcessPagesExportImport.

In older ProcessWire versions where InputfieldSelector was not available, the module will instead present you with a text input to specify a valid (for that version of ProcessWire!) selector for finding pages. The other two methods for adding pages (add pages manually or add by parent) are still available.

Custom JS ensures older installs without showIf functionality still get the inputfield dependency treatment.

API

export($items, $options) 

PageArray $items: The PageArray to export.

Array $options: In addition to the options in the original class that you can pass to pagesToArray(), there are two more options here. mode to specify the export type (array or json or zip) and fieldNamesExclude, to specify fields to exclude from the export. Note that the original class option fieldNames still works. It's for specifying the fields to include in the export.

 

// API USAGE

// get and include PagesExport class
/* @note: you'll need to include the path differently if you are using the
 class directly without installing the Process module
*/
$path = $config->paths->ProcessPagesExport . 'PagesExport.php';
require_once($path);

// create new instance of the class
$siteExport = new PagesExport();

// find items to export
/* 
	a. export whole site! (minus admin and children)
	careful! in some cases, better to export in batches
*/
//$items = $pages->get('/')->find('has_parent!=2');

// export a batch of pages
$items = $pages->find('template=basic-page|computer');

/* you could also use these methods directly
#$data = $siteExport->pagesToArray($items);
#$data = $siteExport->exportJSON($items);
#$data = $siteExport->exportZIP($items);
*/
$options = array(
	// @kongondo addition: export to screen as 'array' or 'json' OR export to zip
	// zip will be saved in /site/assets/backups/PagesExport/ 
    'mode' => 'array',// array or json or ZIP
	// export only these field names, when specified
    'fieldNames' => array('images', 'files', 'multi_pages'),
	// @kongondo addition: exclude fields from export. Here we exclude 'body' field
	'fieldNamesExclude' => array('body'), 
);
// get the export
$data = $siteExport->export($items, $options);

if(is_array($data)) {
    echo '<pre> EXPORTED SITE USING pagesToArray ';
    print_r($data);
    echo '</pre>';	
}
// JSON export
else echo $data;

Screenshots

See also the links to Ryan's blog posts above.

ProcessWire 2.2

5ab398144be01_PagesExport-001-PW2.2.thumb.png.501eae0c5ca448cb005c96177b356f2b.png

 

ProcessWire 2.4

5ab3983dd1753_PagesExport-002-PW2.4.thumb.png.4f748e07e1c2dcd0814bd529a6e94668.png

 

ProcessWire 2.5

5ab3984db5a6e_PagesExport-003-PW2.5.thumb.png.58b2fb9ebd4cdeb4ac6a3cd007002c92.png

 

ProcessWire 2.7

5ab39858de23b_PagesExport-004-PW2.7.thumb.png.eecf9eecb468b1b7dce647007b3c0153.png

 

Video Demo

(Sorry, long and boring) 

Demo shows various exports from ProcessWire 2.x and their importing into ProcessWire 3.x. Remember the old Skyscrapers site profile? See how a whole Skyscrapers site gets exported from a ProcessWire 2.7.3 site and imported into a ProcessWire 3.x starting from here.

 

Edited by kongondo
Link to modules directory
  • Like 7
  • Thanks 1

Share this post


Link to post
Share on other sites

Did you find some problems with upgrading sites from 2.x to 3.x? Just wondering why you would export/import pages rather than duplicate the site then upgrade to PW3?

Share this post


Link to post
Share on other sites
3 hours ago, Robin S said:

Did you find some problems with upgrading sites from 2.x to 3.x? Just wondering why you would export/import pages rather than duplicate the site then upgrade to PW3?

I needed clean installs free from some modules and pages in the 2.x sites. I didn't want those carrying over into the new install and then uninstalling/deleting them. 

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By bernhard
      WHY?
      This module was built to fill the gap between simple $pages->find() operations and complex SQL queries.
      The problem with $pages->find() is that it loads all pages into memory and that can be a problem when querying multiple thousands of pages. Even $pages->findMany() loads all pages into memory and therefore is a lot slower than regular SQL.
      The problem with SQL on the other hand is, that the queries are quite complex to build. All fields are separate tables, some repeatable fields use multiple rows for their content that belong to only one single page, you always need to check for the page status (which is not necessary on regular find() operations and therefore nobody is used to that).
      In short: It is far too much work to efficiently and easily get an array of data based on PW pages and fields and I need that a lot for my RockGrid module to build all kinds of tabular data.

      Basic Usage

       
      Docs & Download
      https://modules.processwire.com/modules/rock-finder/
      https://gitlab.com/baumrock/RockFinder/tree/master
       
      Changelog
      180817, v1.0.6, support for joining multiple finders 180810, v1.0.5, basic support for options fields 180528, v1.0.4, add custom select statement option 180516, change sql query method, bump version to 1.0.0 180515, multilang bugfix 180513, beta release <180513, preview/discussion took place here: https://processwire.com/talk/topic/18983-rocksqlfinder-highly-efficient-and-flexible-sql-finder-module/
    • By OLSA
      Hello for all,
      ConfigurationForm fieldtype module is one my experiment from 2016.
      Main target to build this module was to store multiple setup and configuration values in just 1 field and avoid to use 1 db table to store just single "number of items on page", or another db table to store "layout type" etc. Thanks to JSON formatted storage this module can help you to reduce number of PW native fields in project, save DB space, and reduce number of queries at front-end.
      Install and setup:
      Download (at the bottom ), unzip and install like any other PW module (site/modules/...). Create some filed using this type of field (ConfigurationForm Fieldtype) Go to field setup Input tab and drag some subfields to container area (demo). Set "Name" and other params for subfields Save and place field to templates ("Action tab") How to use it:
      In my case, I use it to store setup and configurations values, but also for contact details, small content blocks... (eg. "widgets").
      Basic usage example:
      ConfigForm fieldtype "setup" has subfields:
      "limit", type select, option values: 5, 10, 15, 20
      "sort", type select, option values: "-date", "date",  "-sort", "sort"
      // get page children (items) $limit = isset($page->setup->limit) ? $page->setup->limit : 10; $sort = isset($page->setup->sort) ? $page->setup->sort : '-sort'; $items = $page->children("limit=$limit, sort=$sort");  
      Screenshots:


       
      Notes:
      Provide option to search inside subfields Provide multilanguage inputs for text and textarea field types Provide option for different field layout per-template basis Do not place/use field type "Button" or "File input" because it won't works. Please read README file for more details and examples Module use JSON format to store values. Text and textarea field types are multilanguage compatible, but please note that main target for this module was to store setup values and small content blocks and save DB space. Search part inside JSON is still a relatively new in MySQL (>=5.77) and that's on you how and for what to use this module.
      Thanks:
      Initial point for this fieldtype was jQuery plugin FormBuiled and thanks to Kevin Chappel for this plugin.
      In field type "link" I use javascript part from @marcostoll module and thanks to him for that part.
      Download:
      FieldtypeConfigForm.zip
      Edit: 14. August 2018. please delete/uninstall previously downloaded zip
      Regards.
         
    • By bernhard
      @Sergio asked about the pdf creation process in the showcase thread about my 360° feedback/survey tool and so I went ahead and set my little pdf helper module to public.
      Description from PW Weekly:
       
      Modules Directory: https://modules.processwire.com/modules/rock-pdf/
      Download & Docs: https://gitlab.com/baumrock/RockPdf
       
      You can combine it easily with RockReplacer: 
      See also a little showcase of the RockPdf module in this thread:
       
    • By Thomas Diroll
      Hi guys I'm relatively new to PW and just finished developing a page for a client. I was able to include all necessary functionality using the core fieldtypes but now I it seems that I need to extend them with a custom one. What I need is a simple button, that copies the absolute url (frontend not PW-backend) of the page which is currently edited to the clipboard. As this feature is only needed inside a specific template, I tend to use a custom fieldtype which provides this feature. I've been looking inside the core modules code (eg. FieldtypeCheckbox.module) but I don't really get the structure of it and how its rendered to the admin page. I also didn't find a lot of tutorials covering custom fieldtypes.
      Maybe some of you could give me some tips on how to write a basic custom fieldtype that renders a button which copies the value of
      page->httpUrl() to the clipboard using JS. Thanks!
    • By bernhard
      Some of you might have followed the development of this module here: https://processwire.com/talk/topic/15524-previewdiscussion-rockdatatables/ . It is the successor of "RockDataTables" and requires RockFinder to get the data for the grid easily and efficiently. It uses the open source part of agGrid for grid rendering.
       
      WHY?
      ProcessWire is awesome for creating all kinds of custom backend applications, but where it is not so awesome in my opinion is when it comes to listing this data. Of course we have the built in page lister and we have ListerPro, but none of that solutions is capable of properly displaying large amounts of data, for example lists of revenues, aggregations, quick and easy sorts by the user, instant filter and those kind of features. RockGrid to the rescue 😉 
       
      Features/Highlights:
      100k+ rows Instant (client side) filter, search, sort (different sort based on data type, eg "lower/greater than" for numbers, "contains" for strings) extendable via plugins (available plugins at the moment: fullscreen, csv export, reload, batch-processing of data, column sum/statistics, row selection) all the agGrid features (cell renderers, cell styling, pagination, column grouping etc) vanilla javascript, backend and frontend support (though not all plugins are working on the frontend yet and I don't plan to support it as long as I don't need it myself)  
      Limitations:
      While there is an option to retrieve data via AJAX the actual processing of the grid (displaying, filtering, sorting) is done on the client side, meaning that you can get into troubles when handling really large datasets of several thousands of rows. agGrid should be one of the most performant grid options in the world (see the official example page with a 100k row example) and does a lot to prevent problems (such as virtual row rendering), but you should always have this limitation in mind as this is a major difference to the available lister options that do not have this limitation.
      Currently it only supports AdminThemeUikit and I don't plan to support any other admin theme.
       
      Download: https://gitlab.com/baumrock/RockGrid
      Installation: https://gitlab.com/baumrock/RockGrid/wikis/Installation
      Quikckstart: https://gitlab.com/baumrock/RockGrid/wikis/quickstart
      Further instructions: https://gitlab.com/baumrock/RockGrid/wikis/quickstart#further-instructions
       
      Module status: alpha, License: MIT
      Note that every installation and uninstallation sends an anonymous google analytics event to my google analytics account. If you don't want that feel free to remove the appropriate lines of code before installation/uninstallation.
       
      Contribute:
      You can contribute to the development of this and other modules or just say thank you by
      testing, reporting issues and making PRs at gitlab liking this post buying me a drink: paypal.me/baumrock/5 liking my facebook page: facebook.com/baumrock hiring me for pw work: baumrock.com  
      Support: Please note that this module might not be as easy and plug&play as many other modules. It needs a good understanding of agGrid (and JavaScript in general) and it likely needs some looks into the code to get all the options. Please understand that I can not provide free support for every request here in the forum. I try to answer all questions that might also help others or that might improve the module but for individual requests I offer paid support for 60€ per hour.
       
      Changelog
      180730 support subdir installations 180711 bugfix (naming issue) 180630 alpha realease  
      Use Cases / Examples:
      Colored grid cells, Icons, Links etc. The Grid also has a "batcher" feature built in that helps communicating with the server via AJAX and managing resource intensive tasks in batches:

      Filters, PW panel links and instant reload on panel close:

      You can combine the grid with a chart library like I did with the (outdated) RockDataTables module: