ProcessWire modules for importing and handling large data sets.

DataSet

It is a set of ProcessWire modules for importing, manipulating and displaying large (50k+ entries) data sets.
The software was developed for the [Mikes-dictionary] and other Digital Humanities projects.

Main features


  • import data from CSV and XML sources
  • user configurable input <-> field mappings
  • on-the-fly field data composition
  • supports downloading external resources (files, images)
  • purge, extend or overwrite existing data (PW pages and their fields)
  • handle page references and option fields
  • fairly low resource requirements (uses Tasker to execute long-running jobs)
  • and many more (filtering, limits, default values etc.)

How to use it


See the wiki.

Important notice


This module is under development.
It is now considered fairly stable but things may be broken and the internal API may change at any time.

History


The first version was created in 2017 to import a large XML dataset into ProcessWire pages.
The CSV import sub-module was created in 2018. It was tested to import large dataset containing 200k+ entries and many kinds of references between them.
The CSV + PDF import was developed in 2019 to create a complete digital library using a single CSV upload.

License


The "github-version" of the software is licensed under MPL 2.0.

Install and use modules at your own risk. Always have a site and database backup before installing new modules.

Twitter updates

  • In this week’s blog post we’ll take a brief look at a powerful new ProFields module for ProcessWire that’s just around the corner—the Combo field: More
    4 December 2020
  • There’s a new modules directory on the ProcessWire site now up and running. In this post we’ll cover a few details about what’s changed and what’s new—More
    20 November 2020
  • ProcessWire 3.0.168 core updates — More
    26 October 2020

Latest news

  • ProcessWire Weekly #343
    In the 343rd issue of ProcessWire Weekly we'll introduce ProFields Combo and Payment Stripe Elements, as well as the alpha version of Padloper 2. Read on!
    Weekly.pw / 5 December 2020
  • About the new ProcessWire ProFields Combo Field
    This week we'll take a brief look at a powerful new ProFields module for ProcessWire that's just around the corner—the Combo field.
    Blog / 4 December 2020
  • Subscribe to weekly ProcessWire news

“Indeed, if ProcessWire can be considered as a CMS in its own right, it also offers all the advantages of a CMF (Content Management Framework). Unlike other solutions, the programmer is not forced to follow the proposed model and can integrate his/her ways of doing things.” —Guy Verville, Spiria Digital Inc.