hellomoto

PW equivalent of WP All Import Pro?

Recommended Posts

Would anyone here be able and inspired to develop a ProcessWire equivalent to the WordPress plugin WP All Import Pro? or to help me do so?

This is what I'm envisioning...

  • Upon installation the module creates an admin page titled "Import & Update". On the module config page you can specify allowed templates to run this on, otherwise allowing any.
  • Include the following PHP libraries: hQuery for web scraping, Csv for CSV handling, and Parser for XML.
  • Create template "import-update".
  • On the "Import & Update" page, a list of current import/updaters will be displayed (0 initially), each with corresponding links to "edit" or "run".

When you "Add New", this be the "import-update" template (with all module-specific fields tagged "impupd"):

  • title
  • destination (req.): parent, template
  • source (req.):
    • type (web, csv, xml)
    • location (url, file, text)
      • if web: opt. index URL & link selector, + paginator selector
      • if csv: opt. ignore 1st row
      • if xml: req. individual item node xpath
  • actions (check):
    • import (if none matching UID)
    • update (if matching UID & field values differ)
  • save() [req. here in flow]
  • map (repeater):
    • input (select fields from specified template to affect)
    • intake (corresponding DOM selectors / CSV col. letters/headers / xpath per field) (req.)
  • UID (unique ID; field reference to compare against, from selected input fields) (req.)
  • Lazy Cron interval

Scripts can be run via the import-update template; keep logs; show preview (iframe/ajax) for manual runs. ...

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By ryanC
      Hi, I have created a new install of Processwire 3.0.98, (local machine) and I am unable to install any modules from within Processwire itself. I go to Modules/Site/Add New/ then under "Module Class Name" I enter the name. The module appears, I hit "Download and Install", but I always get an error. For example: 
      "File could not be downloaded (https://github.com/adrianbj/TracyDebugger/archive/master.zip) 2: fsockopen(): SSL operation failed with code 1. OpenSSL Error messages: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version, 2: fsockopen(): Failed to enable crypto, 2: fsockopen(): unable to connect to ssl://github.com:443 (Unknown error) (tried: curl, fopen, socket)"
      I have been able to track down some of the modules and just upload the zip file manually, but this is happening with every module that I had previously installed in another installation with no problems. Any ideas? 
      Thanks!
    • By gebeer
      Hi,
      just wanted to share something I came across while working on an import module for XML data from a web service. The XML I got was not huge, but still, loading around 3.5 MB of XML with 250+ large child nodes into memory at once with simplexml_load_file() and then looping over it had significant impact on performance.
      I searched for a solution and found this great article about how to parse large XML files.
      It basically explains how to utilize the native XMLReader class together with SimpleXMLElement to handle such situations in a memory efficient way.
      After implementing it I got a significant improval on perceived performance. No comparison in numbers to share here as I'm a bit short on time.
    • By rareyush
      i am receiving and error whenever I try to run my processwire on localhost,

       
       
      sql code
       
      -- -- Table structure for table `field_fieldset_meta_end` -- CREATE TABLE `field_fieldset_meta_end` ( `pages_id` int(10) UNSIGNED NOT NULL, `data` int(11) NOT NULL ) ENGINE=MyISAM DEFAULT CHARSET=utf8; -- -------------------------------------------------------- -- -- Table structure for table `field_fieldset_meta_END` -- CREATE TABLE `field_fieldset_meta_END` ( `pages_id` int(10) UNSIGNED NOT NULL, `data` int(11) NOT NULL ) ENGINE=MyISAM DEFAULT CHARSET=utf8; anyone ?
      whenever I make a new database and upload it there, database get imported without errors.
    • By benbyf
      Hello, having trouble getting PW to save my user or field in cron hook.
      It seems to report the same field info each time unchanged. It should be counting down in seconds.
      public function init() { // set to every30Seconds in settings $this->defaultInterval = $this->cron_check; // add hooks to CRON $this->addHookAfter("LazyCron::{$this->defaultInterval}", $this, 'checkUserSubscription'); } public function ___checkUserSubscription(HookEvent $e){ // seconds since last lazycron $seconds = $e->arguments[0]; $log = wire('log'); $field = $this->exclusiveRole; // users with specific role and field $users = $this->users->find("roles=$field"); foreach ($users as $user) { // get stored time $currentTime = $user->get($this->fieldName); // if time not null if($currentTime){ // evalute stored time minus time past $resultTime = intval($currentTime) - intval($seconds); $log->message("result ".$resultTime); // save user page $user->set($this->fieldName, $resultTime); $this->users->save($user); } } }  
    • By kongondo
      Pages Export
      This module is for specifically exporting ProcessWire 2.x sites for later importing into ProcessWire 3.x.
      Github: Project Page
      Modules Directory: https://modules.processwire.com/modules/process-pages-export/
      Credits: Ryan Cramer
      Background
      As I make my modules ProcessWire 3.x-compatible only, I've had the need to re-create/mirror their ProcessWire 2.x test sites in respective ProcessWire 3.x sites. These ProcessWire 3.x sites (one for each module) were already in place and I didn't feel like re-doing them by exporting/importing site profiles. I also like working with JSON rather than other export formats. So, I decided to write a custom pages export/import script for moving the ProcessWire 2.x sites to their respective ProcessWire 3.x counterpart sites. I'd just finished the export side of things when I came across a post in the forums that reminded me that ProcessWire 3.x already boasts a pages export/import feature, although some of it is still in development. Great! I like its API (PagesExportImport.php) and GUI (ProcessPagesExportImport.module) so no need to re-invent the wheel. 
      I still had the small problem of making sure my JSON export would be compatible with the JSON input that the ProcessWire 3.x import expects. No need to re-invent the wheel, again! I ditched my custom script and instead ported the export functionalities of ProcessWire 3.x Pages Export/Import for use in ProcessWire 2.2 - 2.7, specifically to help migrate older sites to ProcessWire 3.x.
      Compatibility
      The module and class have been tested and work in ProcessWire 2.2, 2.3, 2.4, 2,5, 2.6 and 2.7.  The module is currently tagged as 'in development' until Pages Import feature of ProcessWire 3.x is released as stable. Nonetheless, I have not encountered any issues so far in either the export or the ProcessWire 3.x import. I think Ryan is waiting until he can support more complex field types before tagging the ProcessWire 3.x Pages Export/Import as production-ready.
      This is not a ProcessWire 3.x module and will never be. It has no such need . Just in case you forget and try to install it in a ProcessWire 3.x site, the module will throw a WireException(). I will also not be porting the ProcessWire 3.x import functionality for use in ProcessWire 2.x. That will defeat the purpose here; to move sites to ProcessWire 3.x and not the other way round.
      Supported Fields
      All non-complex fields such as integer, text, textarea, etc Page fields Repeaters File and Image fields I think these cover most needs. Note: not yet tested with Multilingual fields.
      Technical
      To ensure exports will be compatible with ProcessWire 3.x Pages Import, where necessary, the module borrows (and amends as needed) methods from ProcessWire 3.x for use in ProcessWire 2.x. For instance, ProcessWire 3.x Pages Export/Import uses the new(-ish) $file functions found in WireFileTools. Rather than copy and include such files, the module only borrowed and amended needed methods. These are listed below.
      PagesExport.php
      From /wire/core/Functions.php: wireInstanceOf(), wireClassName() and wireClassParents() From /wire/core/Fieldtype.php: getImportValueOptions() and getDatabaseSchema() From /wire/core/WireFileTools.php: zip(), chmod() and mkdir() From /wire/core/WireHttp.php: sendFile From /wire/modules/Fieldtype/FieldtypeFile.module: exportValue() and exportDescription() From /wire/modules/Fieldtype/FieldtypeImage.module: exportValue() From /wire/modules/Fieldtype/FieldtypePage.module: exportValue() and exportValuePage() From /wire/modules/Fieldtype/FieldtypeRepeater.module: exportValue() From /wire/core/Fieldtype/WireTempDir.php: create(), createName() and getTempDir() All the export methods from the /wire/core/PagesExportImport.php class ProcessPagesExport.module
      All the export methods from /wire/modules/process/ProcessPagesExportImport/ProcessPagesExportImport.module Newer methods such as $this->wire() will gracefully degrade to the older wire() function, ensuring smooth and uniform operation in ProcessWire 2.2  - 2.7.
      Use
      This module and class is for supersusers only and has only 1 aim; to export ProcessWire 2.x sites ready for importing into ProcessWire 3.x sites. You can either install (like any other module) and use the process module (ProcessPagesExport.module) or skip the install and just include and use the class (PagesExport.php) to export your sites.
      Both the module (Export GUI) and API require that you are logged in as a supersuser before you can use them. The PagesExport class has a gateway method and option not found in the original class (PagesExportImport). The method export() allows access to the three export methods in the original class, i.e. pagesToArray(), exportJSON() and exportZip(). See example usage below.
      GUI/Process Module
      On install, the module will create a new admin page Export Pages. Please note that unlike the original code, this page is created directly under /admin/ and not /admin/pages/. Click on Export Pages to start.
      Nothing much has changed from the original ProcessPagesExportImport.
      In older ProcessWire versions where InputfieldSelector was not available, the module will instead present you with a text input to specify a valid (for that version of ProcessWire!) selector for finding pages. The other two methods for adding pages (add pages manually or add by parent) are still available.
      Custom JS ensures older installs without showIf functionality still get the inputfield dependency treatment.
      API
      export($items, $options) 
      PageArray $items: The PageArray to export.
      Array $options: In addition to the options in the original class that you can pass to pagesToArray(), there are two more options here. mode to specify the export type (array or json or zip) and fieldNamesExclude, to specify fields to exclude from the export. Note that the original class option fieldNames still works. It's for specifying the fields to include in the export.
       
      // API USAGE // get and include PagesExport class /* @note: you'll need to include the path differently if you are using the class directly without installing the Process module */ $path = $config->paths->ProcessPagesExport . 'PagesExport.php'; require_once($path); // create new instance of the class $siteExport = new PagesExport(); // find items to export /* a. export whole site! (minus admin and children) careful! in some cases, better to export in batches */ //$items = $pages->get('/')->find('has_parent!=2'); // export a batch of pages $items = $pages->find('template=basic-page|computer'); /* you could also use these methods directly #$data = $siteExport->pagesToArray($items); #$data = $siteExport->exportJSON($items); #$data = $siteExport->exportZIP($items); */ $options = array( // @kongondo addition: export to screen as 'array' or 'json' OR export to zip // zip will be saved in /site/assets/backups/PagesExport/ 'mode' => 'array',// array or json or ZIP // export only these field names, when specified 'fieldNames' => array('images', 'files', 'multi_pages'), // @kongondo addition: exclude fields from export. Here we exclude 'body' field 'fieldNamesExclude' => array('body'), ); // get the export $data = $siteExport->export($items, $options); if(is_array($data)) { echo '<pre> EXPORTED SITE USING pagesToArray '; print_r($data); echo '</pre>'; } // JSON export else echo $data; Screenshots
      See also the links to Ryan's blog posts above.
      ProcessWire 2.2

       
      ProcessWire 2.4

       
      ProcessWire 2.5

       
      ProcessWire 2.7

       
      Video Demo
      (Sorry, long and boring) 
      Demo shows various exports from ProcessWire 2.x and their importing into ProcessWire 3.x. Remember the old Skyscrapers site profile? See how a whole Skyscrapers site gets exported from a ProcessWire 2.7.3 site and imported into a ProcessWire 3.x starting from here.