module Introducing Data Migrator – Import SQL/CSV/JSON/XML into ProcessWire

[Mikel]

Hey, everyone, here at frameless we frequently work with clients who already have a website but aren't happy with it and want us to rebuild it from scratch. Whenever possible, we use ProcessWire for new web projects – no surprise there, given the flexibility and clean API we all love.

For smaller sites, migrating content is usually straightforward – a bit of copy/paste and you're done. But for larger projects with hundreds or thousands of records across multiple database tables, this quickly becomes tedious and error-prone.

Over the years, we've written various import scripts and parsers to handle these migrations. We finally decided to clean them up and package everything into a proper module that we'd like to share with the community.

Introducing: Data Migrator
Data Migrator is a Process module that imports external data (SQL dumps, CSV, JSON, XML) directly into ProcessWire's page structure – including automatic creation of templates, fields, and even PHP template files.

Key Features

• Multi-format support – Import from .sql, .csv, .json, and .xml files
• Automatic type detection – Recognizes emails, URLs, dates, booleans, integers, etc. and maps them to appropriate ProcessWire fieldtypes
• SQL schema parsing – Extracts column types from CREATE TABLE statements for better field mapping
• Foreign Key handling – Detects FK relationships and sorts tables by dependency order
• Dry Run mode – Preview exactly what will be created before committing anything
• Full Rollback – Undo an entire migration with one click (removes all created pages, templates, and fields)
• Template file generation – Automatically creates ready-to-use .php template files in /site/templates/

How it works

• Upload your data file (SQL dump, CSV, JSON, or XML)
• Review the analysis – the module shows detected tables, columns, suggested fieldtypes, and sample values
• Fine-tune if needed – override fieldtypes via dropdown, configure FK relationships
• Run a Dry Run to preview all changes
• Execute the migration – templates, fields, parent pages, and data pages are created automatically
• If something's wrong – hit Rollback to cleanly undo everything

Requirements
ProcessWire 3.0.0+
PHP 7.4+

Links
GitHub: github.com/frameless-at/ProcessDataMigrator
Modules Directory: /modules/process-data-migrator/

We've been using the methods and classes bundled in this module internally for a while now and it has saved us a lot of time on migration projects. We hope it's useful for others facing similar challenges. Feedback, bug reports, and feature requests are welcome!

Cheers,
Mike

[Ivan Gretsky]

Wow! Looks super cool! 🎉

Well, reads) Could you please add some screenshots or a video? 

And a questions right away (I have not install or use the module yet).  Are Repeater and Repeater Matrix fields handled somehow? We've got content builder builder based on Repeater Matrix.

[Mikel]

4 hours ago, Ivan Gretsky said:

Wow! Looks super cool! 🎉Well, reads) Could you please add some screenshots or a video? 

And a questions right away (I have not install or use the module yet).  Are Repeater and Repeater Matrix fields handled somehow? We've got content builder builder based on Repeater Matrix.

Thanks, Ivan. Screenshots are included in the Readme file on Github and the module directory (when module is approved).
Regarding Repeater (Matrix) Fields: As the module handles external (non ProcessWire) data structures, what exactly do you mean by „handled“? It is not possible to map data to an existing template/field structure, if that´s what you are asking for. 

[Ivan Gretsky]

7 hours ago, Mikel said:

It is not possible to map data to an existing template/field structure, if that´s what you are asking for.

Hmm. I took that for granted. Though that existing data structure could be used. Visually mapped to. Sorry for asking questions without doing my part to investigate first.

[eelkenet]

Hi @Mikel, thank you for sharing this module! I am currently going through first attempts to use this module.
In my case I am trying to get (a database dump of) a WordPress site into PW. 

During my process I bumped into a couple of things, you may or may not be aware of:

1. There is no warning if the uploaded file is too big, for instance due to upload_max_filesize being set too low. 
2. If the module cannot figure out a Suggested Title Field for a table, there is no way to define this yourself and the dry run will fail early (in MappingEngine::createMapping).
3. Also there is no way to select another column as source for the title field.  
4. I think the limits don't work as intended (at least when importing SQL). When setting the sample_size to 100, and maximum_rows to 0, it will still only import 100 rows of data per table. 

Again, thanks for setting this all up!

---

(PS, I kept running into issues while attempting to modify your module, so for now I have moved on to building a WP-scraper that leveraged the built-in rest-api.)

 

[Mikel]

Hi, @eelkenet, thanks for the findings! We updated the module to version 1.1.2 that addresses all 4 points: 

1. Specific error messages for upload_max_filesize, UPLOAD_ERR_PARTIAL, etc.
2. TheMappingEngine now has multiple fallback strategies
3. title field selection via Dropdown in UI for manual selection
4. All parsers now use max_rows for data limiting

Good luck with WP import, that one can be a bit tricky 😉 due to WPs "unique" structure.

Cheers, Mike