Jump to content
ryan

Module: Import Pages from CSV file

Recommended Posts

Here is a new module for ProcessWire 2.1 that imports pages from a CSV file. By default it will create new pages from data in the CSV file, but you can also configure it to modify existing pages too (existing pages that have the same title).

Please give it a try and let me know how it works for you and if you run into any issues with it. This module is something I've had in the works for awhile, and regularly use on various projects, so figured I should clean it up a bit and release it. Also attached are a couple screenshots from it.

How to Install:

1. Download from: https://github.com/r.../ImportPagesCSV

2. Place the file ImportPagesCSV.module in your /site/modules/ directory.

3. In ProcessWire admin, click to 'Modules' and 'Check for new modules'.

4. Click 'install' next to the 'Import Pages CSV' module (under heading 'Import').

Following that, you'll see a new menu option for this module on your Admin > Setup menu.

Supported field types for importing:*

  • PageTitle
  • Text
  • Textarea (including normal or TinyMCE)
  • Integer
  • Float
  • Email
  • URL
  • Checkbox (single)

*I'll be adding support for multi-value, page-reference and file-based Fieldtypes in a future version.

post-1-132614278648_thumb.png

post-1-13261427868_thumb.png

  • Like 9

Share this post


Link to post
Share on other sites

Very cool module! Thanks a lot Ryan! I'll have a look and try out soon.

Share this post


Link to post
Share on other sites

Looks great and will definitely come in use! Will test and learn from the code later on.

Share this post


Link to post
Share on other sites

Looking great! Seems like a very useful module. Maybe even something to include by default, because most if not all projects i can think of require you to do some importing.

I have made a reference to this thread/module over here: http://processwire.com/talk/index.php/topic,24.0.html

Because my code reading skills suck i do have a question: In 'Step 2', how do you come up with the list of fields found in the csv? I'm guessing you need to have a header line in your csv file?

Share this post


Link to post
Share on other sites

Yep, it requires header row: "The list of field names must be provided as the first row in the CSV file. "

Share this post


Link to post
Share on other sites

Yep, it requires header row: "The list of field names must be provided as the first row in the CSV file. "

stupid, i totally missed that 

Share this post


Link to post
Share on other sites

I could modify it to not require a header row. I guess I just figured every CSV file I've ever come across had one.

Share this post


Link to post
Share on other sites

Usually they have. And since that works by uploading a file then anyone can add those titles to the file.

Share this post


Link to post
Share on other sites

I could modify it to not require a header row. I guess I just figured every CSV file I've ever come across had one.

I think it's fine to require the header row, without it there will be no way to do Stap 2 in it's current form.

If a header row is missing you can easily add it, because you have control over the file you choose to use.

Share this post


Link to post
Share on other sites

Ryan,

How did you know I would need something like this today?

I came to this forum to ask a question about creating pages from a feed and here it is... (ok, I was focused on an xml feed, but CSV will work as well I guess.)

Thanks!

/Jasper

Share this post


Link to post
Share on other sites

Hi!

I just started testing this module and it works great to import a product feed. I love it!

Only one minor thing I noticed is images: my productfeed contains URLs to an image, but I can't select the imagefields in the import module.

Is there a reason for this? I tried to manually add a URL into the field_image and it worked great, it saved a copy (or multiple copies in different sizes) of the image on my server.

/Jasper

Share this post


Link to post
Share on other sites

Hi!

I just started testing this module and it works great to import a product feed. I love it!

Only one minor thing I noticed is images: my productfeed contains URLs to an image, but I can't select the imagefields in the import module.

Is there a reason for this? I tried to manually add a URL into the field_image and it worked great, it saved a copy (or multiple copies in different sizes) of the image on my server.

/Jasper

The following is written in Ryan's topicstart:

Supported field types for importing:*

    * PageTitle

    * Text

    * Textarea (including normal or TinyMCE)

    * Integer

    * Float

    * Email

    * URL

    * Checkbox (single)

*I'll be adding support for multi-value, page-reference and file-based Fieldtypes in a future version.

So i guess you'll have to wait till a future version (dunno if Ryan has this planned anytime soon), or you could have a look at the code and possibly add something yourself.

Share this post


Link to post
Share on other sites

Oops, I missed that the imagefield wasn't listed.  Sorry about that. :-\

I am not sure if my coding skills are sufficient to add the image field import myself. But it's of course worth a try.  :)

A workaround could of course be using the URL field type until Ryan releases a future version.

/Jasper

Share this post


Link to post
Share on other sites

I don't think a multi-image field will work, but a single image field may very well work if you want to try it.

To try it, backup your PW database and site first. If it doesn't work, you want to be able to restore to where you were. Though, chances are you won't have to do any kind of restoring files, but you can never be too safe.

Next, make sure you are dealing with a single image field. Edit your field (in Setup > Fields) and make sure it's "max number of files" is set to "1".

Next edit the ImportPagesCSV.module file and locate this (near the top, and add the lines indicated at the bottom).

<?php
        protected $fieldtypes = array(
                'FieldtypePageTitle',
                'FieldtypeText',
                'FieldtypeTextarea',
                'FieldtypeInteger',
                'FieldtypeFloat',
                'FieldtypeEmail',
                'FieldtypeURL',
                'FieldtypeCheckbox',
                'FieldtypeFile',  // add this line
                'FieldtypeImage', // add this line
                );

Save, and try it out. Let us know if it worked?

Share this post


Link to post
Share on other sites

Save, and try it out. Let us know if it worked?

It didn't really work. It looks like the page need to be created before an image can be added.

I received the following error:

TemplateFile: New page '/shs/verblijf-in-stockholm/0/' must be saved before files can be accessed from it
#0 C:\xampp\htdocs\shs\wire\core\PagefilesManager.php(133): PagefilesManager->path()
#1 C:\xampp\htdocs\shs\wire\core\PagefilesManager.php(43): PagefilesManager->createPath()
#2 C:\xampp\htdocs\shs\wire\core\PagefilesManager.php(32): PagefilesManager->init(Object(Page))
#3 C:\xampp\htdocs\shs\wire\core\Page.php(1132): PagefilesManager->__construct(Object(Page))
#4 C:\xampp\htdocs\shs\wire\core\Pagefiles.php(63): Page->filesManager()
#5 C:\xampp\htdocs\shs\wire\core\Pagefiles.php(47): Pagefiles->setPage(Object(Page))
#6 C:\xampp\htdocs\shs\wire\modules\Fieldtype\FieldtypeImage.module(33): Pagefiles->__construct(Object(Page))
#7 C:\xampp\htdocs\shs\wire\core\Fieldtype.php(289): FieldtypeImage->getBlankValue(Object(Page), Object(Field))
#8 C:\xampp\htdocs\shs\wire\core\Page.php(523): Fieldtype->getDefaultValue(Object(Page), Object(Field))
#9 C:\xampp\htdocs\shs\wire\core\Page.php(467): Page->getFieldValue('hotel_image')
#10 C:\xampp\htdocs\shs\wire\core\Page.php(364): Page->get('hotel_image')
#11 C:\xampp\htdocs\shs\wire\core\Page.php(308): Page->setFieldValue('hotel_image', 'http://images.t...', true)
#12 C:\xampp\htdocs\shs\site\modules\ImportPagesCSV.module(385): Page->set('hotel_image', 'http://images.t...')
#13 C:\xampp\htdocs\shs\site\modules\ImportPagesCSV.module(346): ImportPagesCSV->importPage(Array, Object(InputfieldForm))
#14 C:\xampp\htdocs\shs\site\modules\ImportPagesCSV.module(126): ImportPagesCSV->processForm2(Object(InputfieldForm))
#15 [internal function]: ImportPagesCSV->___executeFields()
#16 C:\xampp\htdocs\shs\wire\core\Wire.php(267): call_user_func_array(Array, Array)
#17 C:\xampp\htdocs\shs\wire\core\Wire.php(229): Wire->runHooks('executeFields', Array)
#18 C:\xampp\htdocs\shs\wire\core\ProcessController.php(194): Wire->__call('executeFields', Array)
#19 C:\xampp\htdocs\shs\wire\core\ProcessController.php(194): ImportPagesCSV->executeFields()
#20 [internal function]: ProcessController->___execute()
#21 C:\xampp\htdocs\shs\wire\core\Wire.php(267): call_user_func_array(Array, Array)
#22 C:\xampp\htdocs\shs\wire\core\Wire.php(229): Wire->runHooks('execute', Array)
#23 C:\xampp\htdocs\shs\wire\core\admin.php(42): Wire->__call('execute', Array)
#24 C:\xampp\htdocs\shs\wire\core\admin.php(42): ProcessController->execute()
#25 C:\xampp\htdocs\shs\wire\templates-admin\controller.php(13): require('C:\xampp\htdocs...')
#26 C:\xampp\htdocs\shs\site\templates\admin.php(13): require('C:\xampp\htdocs...')
#27 C:\xampp\htdocs\shs\wire\core\TemplateFile.php(88): require('C:\xampp\htdocs...')
#28 [internal function]: TemplateFile->___render()
#29 C:\xampp\htdocs\shs\wire\core\Wire.php(267): call_user_func_array(Array, Array)
#30 C:\xampp\htdocs\shs\wire\core\Wire.php(229): Wire->runHooks('render', Array)
#31 C:\xampp\htdocs\shs\wire\modules\PageRender.module(194): Wire->__call('render', Array)
#32 C:\xampp\htdocs\shs\wire\modules\PageRender.module(194): TemplateFile->render()
#33 [internal function]: PageRender->___renderPage(Object(HookEvent))
#34 C:\xampp\htdocs\shs\wire\core\Wire.php(267): call_user_func_array(Array, Array)
#35 C:\xampp\htdocs\shs\wire\core\Wire.php(229): Wire->runHooks('renderPage', Array)
#36 C:\xampp\htdocs\shs\wire\core\Wire.php(289): Wire->__call('renderPage', Array)
#37 C:\xampp\htdocs\shs\wire\core\Wire.php(289): PageRender->renderPage(Object(HookEvent))
#38 C:\xampp\htdocs\shs\wire\core\Wire.php(229): Wire->runHooks('render', Array)
#39 C:\xampp\htdocs\shs\wire\modules\Process\ProcessPageView.module(73): Wire->__call('render', Array)
#40 C:\xampp\htdocs\shs\wire\modules\Process\ProcessPageView.module(73): Page->render()
#41 [internal function]: ProcessPageView->___execute()
#42 C:\xampp\htdocs\shs\wire\core\Wire.php(267): call_user_func_array(Array, Array)
#43 C:\xampp\htdocs\shs\wire\core\Wire.php(229): Wire->runHooks('execute', Array)
#44 C:\xampp\htdocs\shs\index.php(170): Wire->__call('execute', Array)
#45 C:\xampp\htdocs\shs\index.php(170): ProcessPageView->execute()
#46 {main}

/Jasper

Share this post


Link to post
Share on other sites

Sorry, I neglected to think about that before (how pages have to be created before you can add images to them). One possible way around it is to try to first import without the image field, then import the same spreadsheet again but with the image field (updating the existing pages that were created). You'd need to choose the option to "modify existing page." But it's possible that might work.

Share this post


Link to post
Share on other sites

I tried that as well, but it didn't work.  :(

Even when the pages exist, the import module will not add the images.

The error/warning is exactly the same some in my previous reply.

/Jasper

Share this post


Link to post
Share on other sites

Jasper, I didn't want to leave you empty handed, especially after you've tried this a few times and my suggestions didn't work. Here's an updated version of the ImportPagesCSV module that supports file and image filed importing. It supports both single and multi-file fields, so there aren't any limitations in that area.

https://github.com/ryancramerdesign/ImportPagesCSV

To import a multi-file field, place each filename or URL on it's own line in your spreadsheet, OR separate each by a pipe "|", OR separate each by a tab (you decide) – basically, you are delimiting the filenames/URLs within the field. In my own tests, I used the pipe "|" to separate the URLs and all seemed to work well. Of course, if there is only one image/file, you don't need anything other than the filename or URL (no delimiter necessary).

I ended up changing quite a bit of code, so please let me know if you run into any error messages or anything – it may not be perfect yet, but hopefully close.

  • Like 4

Share this post


Link to post
Share on other sites

Wow, that's great Ryan. Thank you!

I am going to try it later today and will tell you how it worked.

/Jasper

Share this post


Link to post
Share on other sites

It works great. I hoped to be able to add one image, but suddenly I can add even more.

Thanks Ryan

(I owe you a beer  :D)

Share this post


Link to post
Share on other sites

I have found the import CSV module to be really useful, and has become a big part of my site building workflow with processwire. It's much easier to collect and manage lots of data in excel / numbers and then directly import into processwire. This is also great for updating content with a re-import.

I think data journalists and data hackers will find this sort of functionality very useful for doing quick data visualisation mashups (especially when you add on the friendly pw api).

I think content strategists will like this workflow as most of them use excel for collecting and organising site content at the start of projects.

And of course the clients will like it - this will be a nice way to start a new project - once basic site structure is agreed, we can give the client an excel template (or google doc) and tell them to start adding content. Wouldn't it be great to start a project with actual real content!

I hope you continue adding support for other pw fields like page-references and dates (I use pages extensively for category management).

Could this module also work in the opposite direction and export the data to CSV?

Then we just need to work out a way to import / export fields and templates :)  as much as I admire the processwire interface, I still think managing data / settings / fields is much quicker and easier in text files, at least at the beginning  of a project.

Processwire - a CMS that gets out of the way.

  • Like 1

Share this post


Link to post
Share on other sites

This module looks like exactly what I need for migrating a couple of larger sites to PW. Would it be possible to hack this for adding users as well?

Thanks,

Stephen

Share this post


Link to post
Share on other sites

Thanks for the message mjmurphy–glad that you like this module. I use it quite a bit myself too. I will definitely be adding support for more fieldtypes to it. Actually, I think dates and page references are the only two that we don't support yet. Adding dates will be easy, but the page references a little more complex. However, I need the page references functional in the near future so will likely be adding both of those types soon.

An ExportCSV module is also planned for sure.

Then we just need to work out a way to import / export fields and templates

This won't be a module, it is already planned for the core (likely in 2.3/2.4). This feature was in PW1, but just hasn't made it into PW2 yet. But it's a very useful thing to have I agree.

This module looks like exactly what I need for migrating a couple of larger sites to PW. Would it be possible to hack this for adding users as well?

I think it should work now, though I haven't tried. Although it looks like I need to add support for FieldtypePassword (another one I missed) if you want to import passwords. Should be easy to add though–added to my list.

  • Like 1

Share this post


Link to post
Share on other sites

I want to import some users. So I chose "user" as "Template to use for imported pages" and "Users" as "parent page". But if I'm uploading the .csv and have to choose the right fields I can't choose "title". The only option is "email". And if I submit without a title there will be an error saying "Unable to import page because it has no required 'title' field or it is blank."

So how can I import users?

Greets,

Nico

Share this post


Link to post
Share on other sites

I haven't actually tried to import users yet. But it should be possible. Go ahead and add the 'title' field to your user template and populate it with something (email address?) to see if that makes the import happy. I'll plain to make the import a little smarter in this regard on the next update.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Mike Rockett
      Jumplinks for ProcessWire
      Release: 1.5.56
      Composer: rockett/jumplinks
      Jumplinks is an enhanced version of the original ProcessRedirects by Antti Peisa.
      The Process module manages your permanent and temporary redirects (we'll call these "jumplinks" from now on, unless in reference to redirects from another module), useful for when you're migrating over to ProcessWire from another system/platform. Each jumplink supports wildcards, shortening the time needed to create them.
      Unlike similar modules for other platforms, wildcards in Jumplinks are much easier to work with, as Regular Expressions are not fully exposed. Instead, parameters wrapped in curly braces are used - these are described in the documentation.
      Under Development: 2.0, to be powered by FastRoute
      As of version 1.5.0, Jumplinks requires at least ProcessWire 2.6.1 to run.
      View on GitLab
      Download via the Modules Directory
      Read the docs
      Features
      The most prominent features include:
      Basic jumplinks (from one fixed route to another) Parameter-based wildcards with "Smart" equivalents Mapping Collections (for converting ID-based routes to their named-equivalents without the need to create multiple jumplinks) Destination Selectors (for finding and redirecting to pages containing legacy location information) Timed Activation (activate and/or deactivate jumplinks at specific times) 404-Monitor (for creating jumplinks based on 404 hits) Additionally, the following features may come in handy:
      Stale jumplink management Legacy domain support for slow migrations An importer (from CSV or ProcessRedirects) Feedback & Feature Requests
      I’d love to know what you think of this module. Please provide some feedback on the module as a whole, or even regarding smaller things that make it whole. Also, please feel free to submit feature requests and their use-cases.
      Note: Features requested so far have been added to the to-do list, and will be added to 2.0, and not the current dev/master branches.
      Open Source

      Jumplinks is an open-source project, and is free to use. In fact, Jumplinks will always be open-source, and will always remain free to use. Forever. If you would like to support the development of Jumplinks, please consider making a small donation via PayPal.
      Enjoy! :)
    • By BitPoet
      As threatened in Ryan's announcement for 3.0.139, I built a little module for sliding toggles as a replacement for checkboxes. Styling of the input is CSS3 only (with all the usual caveats about older browsers), no JS necessary, and may still be a bit "rough around the edges", so to speak, since I didn't have much time for testing on different devices or brushing things up enough so I'd feel comfortable pushing it to the module directory. But here's the link to the GitHub repo for now:
      InputfieldSlideToggle
      Fieldtype and Inputfield that implements smartphone-style toggles as replacement for checkbox inputs. The visualization is CSS-only, no additional JS necessary.
      Status
      Still very alpha, use with caution!
      Features / Field Settings
      Size
      You can render the toggles in four different sizes: small, medium, large and extra large.
      Off Color
      Currently, "unchecked" toggles can be displayed either in grey (default) or red.
      On Color
      "Checked" toggles can be rendered in one of these colors: blue (default), black, green, grey, orange or red.
      Screenshots

      Some examples with checkbox label


      View all Size and Color Combinations
      Small toggles Medium toggles Big toggles Extra big toggles  









    • By Orkun
      Hi Guys
      I needed to add extended functionalities for the InputfieldDatetime Module (module is from processwire version 2.7.3) because of a Request of Customer.
      So I duplicated the module and placed it under /site/modules/.
      I have added 3 new Settings to the InputfieldDatetime Module.
      1. Day Restriction - Restrict different days based on weekdays selection (e.g. saturday, sunday) - WORKING

       
      2. Time Slots - Define Time slots based on custom Integer Value (max is 60 for 1 hour) - WORKING

       
      3. Time Range Rules per Weekday - Define a minTime and MaxTime per Weekday (e.g. Opening Hours of a Restaurant) - NOT WORKING PROPERLY

       
      The Problem
      Time Slots and Day Restriction working fine so far. But the Time Range Rules per Weekday doesn't work right.
      What should happen is, that when you click on a date, it should update the minTime and maxTime of the Time Select.
      But the change on the select only happens if you select a date 2 times or when you select a date 1 time and then close the datepicker and reopen it again.
      The time select doesn't get change when you select a date 1 time and don't close the picker.
      Here is the whole extended InputfieldDatetime Module.
      The Files that I have changed:
      InputfieldDatetime.module InputfieldDatetime.js jquery-ui-timepicker-addon.js (https://trentrichardson.com/examples/timepicker/) - updated it to the newest version, because minTime and maxTime Option was only available in the new version  
      Thats the Part of the JS that is not working correctly:
      if(datetimerules && datetimerules.length){ options.onSelect = function(date, inst) { var day = $(this).datetimepicker("getDate").getDay(); day = day.toString(); var mintime = $(this).attr('data-weekday'+day+'-mintime'); var maxtime = $(this).attr('data-weekday'+day+'-maxtime'); console.log("weekday: "+day); console.log("minTime: "+mintime); console.log("maxTime: "+maxtime); var optionsAll = $(this).datetimepicker( "option", "all" ); optionsAll.minTime = mintime; optionsAll.maxTime = maxtime; $(this).datetimepicker('destroy'); $(this).datetimepicker(optionsAll); $(this).datetimepicker('refresh'); //$.datepicker._selectDate($(this).attr("id"),date); //$.datepicker._base_getDateDatepicker(); // var inst = $.datepicker._getInst($(this)); // $.datepicker._updateDatepicker(inst); /*$(this).datetimepicker('destroy'); InputfieldDatetimeDatepicker($(this), mintime, maxtime); $(this).datetimepicker('refresh'); */ // $(this).datetimepicker('option', {minTime: mintime, maxTime: maxtime}); } } Can you have a look and find out what the Problem is?
      InputfieldDatetime.zip
       
      Kind Regards
      Orkun
    • By teppo
      This module tracks changes, additions, removals etc. of public (as in "not under admin") pages of your site. Like it's name says, it doesn't attempt to be a version control system or anything like that - just a log of what's happened.
      At the moment it's still a work in progress and will most likely be a victim of many ruthless this-won't-work-let's-try-that-instead cycles, but I believe I've nailed basic functionality well enough to post it here.. so, once again, I'll be happy to hear any comments you folks can provide
      https://modules.processwire.com/modules/process-changelog/
      https://github.com/teppokoivula/ProcessChangelog
      How does it work?
      Exactly like it's (sort of) predecessor, Process Changelog actually consists of two modules: Process Changelog and Process Changelog Hooks. Hooks module exists only to serve main module by hooking into various functions within Pages class, collecting data of performed operations, refining it and keeping up a log of events in it's own custom database table (process_changelog.) Visible part is managed by Process Changelog, which provides users a (relatively) pretty view of the contents of said log table.
      How do you use it?
      When installed this module adds new page called Changelog under Admin > Setup which provides you with a table view of collected data and basic filtering tools See attached screenshots to get a general idea about what that page should look like after a while.
      For detailed installation instructions etc. see README.md.
       


    • By Gadgetto
      Status update links (inside this thread) for SnipWire development will be always posted here:
      2019-08-08
      2019-06-15
      2019-06-02
      2019-05-25
      If you are interested, you can test the current state of development:
      https://github.com/gadgetto/SnipWire
      Please note that the software is not yet intended for use in a production system (alpha version).
      If you like, you can also submit feature requests and suggestions for improvement. I also accept pull requests.
      ---- INITIAL POST FROM 2019-05-25 ----
      I wanted to let you know that I am currently working on a new ProcessWire module that fully integrates the Snipcart Shopping Cart System into ProcessWire. (this is a customer project, so I had to postpone the development of my other module GroupMailer).
      The new module SnipWire offers full integration of the Snipcart Shopping Cart System into ProcessWire.
      Here are some highlights:
      simple setup with (optional) pre-installed templates, product fields, sample products (quasi a complete shop system to get started immediately) store dashboard with all data from the snipcart system (no change to the snipcart dashboard itself required) Integrated REST API for controlling and querying snipcart data webhooks to trigger events from Snipcart (new order, new customer, etc.) multi currency support self-defined/configurable tax rates etc. Development is already well advanced and I plan to release the module in the next 2-3 months.
      I'm not sure yet if this will be a "Pro" module or if it will be made available for free.
      I would be grateful for suggestions and hints!
      (please have a look at the screenshots to get an idea what I'm talking about)
       




×
×
  • Create New...