Jump to content

Module: Import Pages from CSV file


ryan

Recommended Posts

Here is a new module for ProcessWire 2.1 that imports pages from a CSV file. By default it will create new pages from data in the CSV file, but you can also configure it to modify existing pages too (existing pages that have the same title).

Please give it a try and let me know how it works for you and if you run into any issues with it. This module is something I've had in the works for awhile, and regularly use on various projects, so figured I should clean it up a bit and release it. Also attached are a couple screenshots from it.

How to Install:

1. Download from: https://github.com/r.../ImportPagesCSV

2. Place the file ImportPagesCSV.module in your /site/modules/ directory.

3. In ProcessWire admin, click to 'Modules' and 'Check for new modules'.

4. Click 'install' next to the 'Import Pages CSV' module (under heading 'Import').

Following that, you'll see a new menu option for this module on your Admin > Setup menu.

Supported field types for importing:*

  • PageTitle
  • Text
  • Textarea (including normal or TinyMCE)
  • Integer
  • Float
  • Email
  • URL
  • Checkbox (single)

*I'll be adding support for multi-value, page-reference and file-based Fieldtypes in a future version.

post-1-132614278648_thumb.png

post-1-13261427868_thumb.png

  • Like 9
Link to comment
Share on other sites

Looking great! Seems like a very useful module. Maybe even something to include by default, because most if not all projects i can think of require you to do some importing.

I have made a reference to this thread/module over here: http://processwire.com/talk/index.php/topic,24.0.html

Because my code reading skills suck i do have a question: In 'Step 2', how do you come up with the list of fields found in the csv? I'm guessing you need to have a header line in your csv file?

Link to comment
Share on other sites

I could modify it to not require a header row. I guess I just figured every CSV file I've ever come across had one.

I think it's fine to require the header row, without it there will be no way to do Stap 2 in it's current form.

If a header row is missing you can easily add it, because you have control over the file you choose to use.

Link to comment
Share on other sites

Ryan,

How did you know I would need something like this today?

I came to this forum to ask a question about creating pages from a feed and here it is... (ok, I was focused on an xml feed, but CSV will work as well I guess.)

Thanks!

/Jasper

Link to comment
Share on other sites

  • 2 weeks later...

Hi!

I just started testing this module and it works great to import a product feed. I love it!

Only one minor thing I noticed is images: my productfeed contains URLs to an image, but I can't select the imagefields in the import module.

Is there a reason for this? I tried to manually add a URL into the field_image and it worked great, it saved a copy (or multiple copies in different sizes) of the image on my server.

/Jasper

Link to comment
Share on other sites

Hi!

I just started testing this module and it works great to import a product feed. I love it!

Only one minor thing I noticed is images: my productfeed contains URLs to an image, but I can't select the imagefields in the import module.

Is there a reason for this? I tried to manually add a URL into the field_image and it worked great, it saved a copy (or multiple copies in different sizes) of the image on my server.

/Jasper

The following is written in Ryan's topicstart:

Supported field types for importing:*

    * PageTitle

    * Text

    * Textarea (including normal or TinyMCE)

    * Integer

    * Float

    * Email

    * URL

    * Checkbox (single)

*I'll be adding support for multi-value, page-reference and file-based Fieldtypes in a future version.

So i guess you'll have to wait till a future version (dunno if Ryan has this planned anytime soon), or you could have a look at the code and possibly add something yourself.

Link to comment
Share on other sites

Oops, I missed that the imagefield wasn't listed.  Sorry about that. :-\

I am not sure if my coding skills are sufficient to add the image field import myself. But it's of course worth a try.  :)

A workaround could of course be using the URL field type until Ryan releases a future version.

/Jasper

Link to comment
Share on other sites

I don't think a multi-image field will work, but a single image field may very well work if you want to try it.

To try it, backup your PW database and site first. If it doesn't work, you want to be able to restore to where you were. Though, chances are you won't have to do any kind of restoring files, but you can never be too safe.

Next, make sure you are dealing with a single image field. Edit your field (in Setup > Fields) and make sure it's "max number of files" is set to "1".

Next edit the ImportPagesCSV.module file and locate this (near the top, and add the lines indicated at the bottom).

<?php
        protected $fieldtypes = array(
                'FieldtypePageTitle',
                'FieldtypeText',
                'FieldtypeTextarea',
                'FieldtypeInteger',
                'FieldtypeFloat',
                'FieldtypeEmail',
                'FieldtypeURL',
                'FieldtypeCheckbox',
                'FieldtypeFile',  // add this line
                'FieldtypeImage', // add this line
                );

Save, and try it out. Let us know if it worked?

Link to comment
Share on other sites

Save, and try it out. Let us know if it worked?

It didn't really work. It looks like the page need to be created before an image can be added.

I received the following error:

TemplateFile: New page '/shs/verblijf-in-stockholm/0/' must be saved before files can be accessed from it
#0 C:\xampp\htdocs\shs\wire\core\PagefilesManager.php(133): PagefilesManager->path()
#1 C:\xampp\htdocs\shs\wire\core\PagefilesManager.php(43): PagefilesManager->createPath()
#2 C:\xampp\htdocs\shs\wire\core\PagefilesManager.php(32): PagefilesManager->init(Object(Page))
#3 C:\xampp\htdocs\shs\wire\core\Page.php(1132): PagefilesManager->__construct(Object(Page))
#4 C:\xampp\htdocs\shs\wire\core\Pagefiles.php(63): Page->filesManager()
#5 C:\xampp\htdocs\shs\wire\core\Pagefiles.php(47): Pagefiles->setPage(Object(Page))
#6 C:\xampp\htdocs\shs\wire\modules\Fieldtype\FieldtypeImage.module(33): Pagefiles->__construct(Object(Page))
#7 C:\xampp\htdocs\shs\wire\core\Fieldtype.php(289): FieldtypeImage->getBlankValue(Object(Page), Object(Field))
#8 C:\xampp\htdocs\shs\wire\core\Page.php(523): Fieldtype->getDefaultValue(Object(Page), Object(Field))
#9 C:\xampp\htdocs\shs\wire\core\Page.php(467): Page->getFieldValue('hotel_image')
#10 C:\xampp\htdocs\shs\wire\core\Page.php(364): Page->get('hotel_image')
#11 C:\xampp\htdocs\shs\wire\core\Page.php(308): Page->setFieldValue('hotel_image', 'http://images.t...', true)
#12 C:\xampp\htdocs\shs\site\modules\ImportPagesCSV.module(385): Page->set('hotel_image', 'http://images.t...')
#13 C:\xampp\htdocs\shs\site\modules\ImportPagesCSV.module(346): ImportPagesCSV->importPage(Array, Object(InputfieldForm))
#14 C:\xampp\htdocs\shs\site\modules\ImportPagesCSV.module(126): ImportPagesCSV->processForm2(Object(InputfieldForm))
#15 [internal function]: ImportPagesCSV->___executeFields()
#16 C:\xampp\htdocs\shs\wire\core\Wire.php(267): call_user_func_array(Array, Array)
#17 C:\xampp\htdocs\shs\wire\core\Wire.php(229): Wire->runHooks('executeFields', Array)
#18 C:\xampp\htdocs\shs\wire\core\ProcessController.php(194): Wire->__call('executeFields', Array)
#19 C:\xampp\htdocs\shs\wire\core\ProcessController.php(194): ImportPagesCSV->executeFields()
#20 [internal function]: ProcessController->___execute()
#21 C:\xampp\htdocs\shs\wire\core\Wire.php(267): call_user_func_array(Array, Array)
#22 C:\xampp\htdocs\shs\wire\core\Wire.php(229): Wire->runHooks('execute', Array)
#23 C:\xampp\htdocs\shs\wire\core\admin.php(42): Wire->__call('execute', Array)
#24 C:\xampp\htdocs\shs\wire\core\admin.php(42): ProcessController->execute()
#25 C:\xampp\htdocs\shs\wire\templates-admin\controller.php(13): require('C:\xampp\htdocs...')
#26 C:\xampp\htdocs\shs\site\templates\admin.php(13): require('C:\xampp\htdocs...')
#27 C:\xampp\htdocs\shs\wire\core\TemplateFile.php(88): require('C:\xampp\htdocs...')
#28 [internal function]: TemplateFile->___render()
#29 C:\xampp\htdocs\shs\wire\core\Wire.php(267): call_user_func_array(Array, Array)
#30 C:\xampp\htdocs\shs\wire\core\Wire.php(229): Wire->runHooks('render', Array)
#31 C:\xampp\htdocs\shs\wire\modules\PageRender.module(194): Wire->__call('render', Array)
#32 C:\xampp\htdocs\shs\wire\modules\PageRender.module(194): TemplateFile->render()
#33 [internal function]: PageRender->___renderPage(Object(HookEvent))
#34 C:\xampp\htdocs\shs\wire\core\Wire.php(267): call_user_func_array(Array, Array)
#35 C:\xampp\htdocs\shs\wire\core\Wire.php(229): Wire->runHooks('renderPage', Array)
#36 C:\xampp\htdocs\shs\wire\core\Wire.php(289): Wire->__call('renderPage', Array)
#37 C:\xampp\htdocs\shs\wire\core\Wire.php(289): PageRender->renderPage(Object(HookEvent))
#38 C:\xampp\htdocs\shs\wire\core\Wire.php(229): Wire->runHooks('render', Array)
#39 C:\xampp\htdocs\shs\wire\modules\Process\ProcessPageView.module(73): Wire->__call('render', Array)
#40 C:\xampp\htdocs\shs\wire\modules\Process\ProcessPageView.module(73): Page->render()
#41 [internal function]: ProcessPageView->___execute()
#42 C:\xampp\htdocs\shs\wire\core\Wire.php(267): call_user_func_array(Array, Array)
#43 C:\xampp\htdocs\shs\wire\core\Wire.php(229): Wire->runHooks('execute', Array)
#44 C:\xampp\htdocs\shs\index.php(170): Wire->__call('execute', Array)
#45 C:\xampp\htdocs\shs\index.php(170): ProcessPageView->execute()
#46 {main}

/Jasper

Link to comment
Share on other sites

Sorry, I neglected to think about that before (how pages have to be created before you can add images to them). One possible way around it is to try to first import without the image field, then import the same spreadsheet again but with the image field (updating the existing pages that were created). You'd need to choose the option to "modify existing page." But it's possible that might work.

Link to comment
Share on other sites

Jasper, I didn't want to leave you empty handed, especially after you've tried this a few times and my suggestions didn't work. Here's an updated version of the ImportPagesCSV module that supports file and image filed importing. It supports both single and multi-file fields, so there aren't any limitations in that area.

https://github.com/ryancramerdesign/ImportPagesCSV

To import a multi-file field, place each filename or URL on it's own line in your spreadsheet, OR separate each by a pipe "|", OR separate each by a tab (you decide) – basically, you are delimiting the filenames/URLs within the field. In my own tests, I used the pipe "|" to separate the URLs and all seemed to work well. Of course, if there is only one image/file, you don't need anything other than the filename or URL (no delimiter necessary).

I ended up changing quite a bit of code, so please let me know if you run into any error messages or anything – it may not be perfect yet, but hopefully close.

  • Like 4
Link to comment
Share on other sites

  • 1 month later...

I have found the import CSV module to be really useful, and has become a big part of my site building workflow with processwire. It's much easier to collect and manage lots of data in excel / numbers and then directly import into processwire. This is also great for updating content with a re-import.

I think data journalists and data hackers will find this sort of functionality very useful for doing quick data visualisation mashups (especially when you add on the friendly pw api).

I think content strategists will like this workflow as most of them use excel for collecting and organising site content at the start of projects.

And of course the clients will like it - this will be a nice way to start a new project - once basic site structure is agreed, we can give the client an excel template (or google doc) and tell them to start adding content. Wouldn't it be great to start a project with actual real content!

I hope you continue adding support for other pw fields like page-references and dates (I use pages extensively for category management).

Could this module also work in the opposite direction and export the data to CSV?

Then we just need to work out a way to import / export fields and templates :)  as much as I admire the processwire interface, I still think managing data / settings / fields is much quicker and easier in text files, at least at the beginning  of a project.

Processwire - a CMS that gets out of the way.

  • Like 1
Link to comment
Share on other sites

Thanks for the message mjmurphy–glad that you like this module. I use it quite a bit myself too. I will definitely be adding support for more fieldtypes to it. Actually, I think dates and page references are the only two that we don't support yet. Adding dates will be easy, but the page references a little more complex. However, I need the page references functional in the near future so will likely be adding both of those types soon.

An ExportCSV module is also planned for sure.

Then we just need to work out a way to import / export fields and templates

This won't be a module, it is already planned for the core (likely in 2.3/2.4). This feature was in PW1, but just hasn't made it into PW2 yet. But it's a very useful thing to have I agree.

This module looks like exactly what I need for migrating a couple of larger sites to PW. Would it be possible to hack this for adding users as well?

I think it should work now, though I haven't tried. Although it looks like I need to add support for FieldtypePassword (another one I missed) if you want to import passwords. Should be easy to add though–added to my list.

  • Like 1
Link to comment
Share on other sites

  • 2 weeks later...

I want to import some users. So I chose "user" as "Template to use for imported pages" and "Users" as "parent page". But if I'm uploading the .csv and have to choose the right fields I can't choose "title". The only option is "email". And if I submit without a title there will be an error saying "Unable to import page because it has no required 'title' field or it is blank."

So how can I import users?

Greets,

Nico

Link to comment
Share on other sites

I haven't actually tried to import users yet. But it should be possible. Go ahead and add the 'title' field to your user template and populate it with something (email address?) to see if that makes the import happy. I'll plain to make the import a little smarter in this regard on the next update.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Robin S
      This module lets you add some custom menu items to the main admin menu, and you can set the dropdown links dynamically in a hook if needed.
      Sidenote: the module config uses some repeatable/sortable rows for the child link settings, similar to the ProFields Table interface. The data gets saved as JSON in a hidden textarea field. Might be interesting to other module developers?
      Custom Admin Menus
      Adds up to three custom menu items with optional dropdowns to the main admin menu.
      The menu items can link to admin pages, front-end pages, or pages on external websites.
      The links can be set to open in a new browser tab, and child links in the dropdown can be given an icon.
      Requires ProcessWire v3.0.178 or newer.
      Screenshots
      Example of menu items

      Module config for the menus

      Link list shown when parent menu item is not given a URL

      Advanced
      Setting child menu items dynamically
      If needed you can set the child menu items dynamically using a hook.
      Example:
      $wire->addHookAfter('CustomAdminMenus::getMenuChildren', function(HookEvent $event) { // The menu number is the first argument $menu_number = $event->arguments(0); if($menu_number === 1) { $colours = $event->wire()->pages->findRaw('template=colour', ['title', 'url', 'page_icon']); $children = []; foreach($colours as $colour) { // Each child item should be an array with the following keys $children[] = [ 'icon' => $colour['page_icon'], 'label' => $colour['title'], 'url' => $colour['url'], 'newtab' => false, ]; } $event->return = $children; } }); Create multiple levels of flyout menus
      It's also possible to create multiple levels of flyout submenus using a hook.

      For each level a submenu can be defined in a "children" item. Example:
      $wire->addHookAfter('CustomAdminMenus::getMenuChildren', function(HookEvent $event) { // The menu number is the first argument $menu_number = $event->arguments(0); if($menu_number === 1) { $children = [ [ 'icon' => 'adjust', 'label' => 'One', 'url' => '/one/', 'newtab' => false, ], [ 'icon' => 'anchor', 'label' => 'Two', 'url' => '/two/', 'newtab' => false, 'children' => [ [ 'icon' => 'child', 'label' => 'Red', 'url' => '/red/', 'newtab' => false, ], [ 'icon' => 'bullhorn', 'label' => 'Green', 'url' => '/green/', 'newtab' => false, 'children' => [ [ 'icon' => 'wifi', 'label' => 'Small', 'url' => '/small/', 'newtab' => true, ], [ 'icon' => 'codepen', 'label' => 'Medium', 'url' => '/medium/', 'newtab' => false, ], [ 'icon' => 'cogs', 'label' => 'Large', 'url' => '/large/', 'newtab' => false, ], ] ], [ 'icon' => 'futbol-o', 'label' => 'Blue', 'url' => '/blue/', 'newtab' => true, ], ] ], [ 'icon' => 'hand-o-left', 'label' => 'Three', 'url' => '/three/', 'newtab' => false, ], ]; $event->return = $children; } }); Showing/hiding menus according to user role
      You can determine which menu items can be seen by a role by checking the user's role in the hook.
      For example, if a user has or lacks a role you could include different child menu items in the hook return value. Or if you want to conditionally hide a custom menu altogether you can set the return value to false. Example:
      $wire->addHookAfter('CustomAdminMenus::getMenuChildren', function(HookEvent $event) { // The menu number is the first argument $menu_number = $event->arguments(0); $user = $event->wire()->user; // For custom menu number 1... if($menu_number === 1) { // ...if user does not have some particular role... if(!$user->hasRole('foo')) { // ...do not show the menu $event->return = false; } } });  
      https://github.com/Toutouwai/CustomAdminMenus
      https://processwire.com/modules/custom-admin-menus/
    • By tcnet
      This module for ProcessWire sends a notification email for each failed login attempt. Similar modules exists already in the module directory of ProcessWire. However, this module is designed to notify, even if specified user doesn't exist.
      Settings
      The settings for this module are located in the menu Modules=>Configure=>LoginFailNotifier.
      Notification email
      Specifies the email address to which the notification emails should be sent.
        Email subject
      Specifies the subject line for the notification email.
        Post variables
      Specifies the $_POST variables to be included in the notification email. Each variable must be separated by a comma. For example: login_name,login_pass
        Server variables
      Specifies the $_SERVER variables to be included in the notification email. Each variable must be separated by a comma. For example: REMOTE_ADDR,HTTP_USER_AGENT
      Link to ProcessWire module directory:
      https://processwire.com/modules/login-fail-notifier/
      Link to github.com:
      https://github.com/techcnet/LoginFailNotifier
    • By Fokke
      ProcessWire 3.x markup module for rendering meta tags in HTML document head section. Note that this module is not a full-blown SEO solution, but rather a simple tool for rendering meta tags based on module configuration. Adding custom meta tags is also supported.
      Built-in meta tags
      The following meta tags are supported out-of-the-box:
      Document title consisting of page title and site name Character set Canonical Viewport Description Keywords Hreflang tags Open Graph og:title og:site_name og:type og:url og:description og:image og:image:width og:image:height Twitter meta tags twitter:card twitter:site twitter:creator twitter:title twitter:description twitter:image Facebook meta tags fb:app_id The full documentation with configurable options can be found here: https://github.com/Fokke-/MarkupMetadata
       
      Requirements:
      ProcessWire>=3.0.0 PHP >=7.1 Installation using Composer
      composer require fokke/markup-metadata Manual installation
      Download latest version from https://github.com/Fokke-/MarkupMetadata/archive/master.zip Extract module files to site/modules/MarkupMetadata directory.
    • By m.sieber
      ITRK-Service for ProcessWire
      Module for the automated transfer of imprint, data protection declaration and terms and conditions from IT-Recht Kanzlei to your ProcessWire installation
      What is ITRK Service for ProcessWire?
      ITRK-Service for ProcessWire is a free module for ProcessWire CMS. It provides an interface to the update service of IT-Recht Kanzlei, via which the legal texts of your online presence are automatically updated. In this way, the texts remain legally secure and warning-proof in the long term. Imprint, data protection declaration, revocation and general terms and conditions are currently supported.
      You can find our documentation (in german language) here: https://www.pupit.de/itrk-service-for-processwire/dokumentation/

      Download: https://www.pupit.de/itrk-service-for-processwire/
      Github: https://github.com/pupit-de/pwItrkServiceConnector
    • By LuisM
      Symprowire is a PHP MVC Framework based and built on Symfony using ProcessWire 3.x as DBAL and Service-Provider
      It acts as a Drop-In Replacement Module to handle the Request/Response outside the ProcessWire Admin. Even tough Symfony or any other mature MVC Framework could be intimidating at first, Symprowire tries to abstract Configuration and Symfony Internals away as much as possible to give you a quick start and lift the heavy work for you.
      The main Goal is to give an easy path to follow an MVC Approach during development with ProcessWire and open up the available eco-system.
      You can find the GitHub Repo and more Information here: https://github.com/Luis85/symprowire
      Documentation
      The Symprowire Wiki https://github.com/Luis85/symprowire/wiki How to create a simple Blog with Symprowire https://github.com/Luis85/symprowire/wiki/Symprowire-Blog-Tutorial Last Update
      16.07.2021 // RC 1 v0.6.0 centralized ProcessWire access trough out the Application by wrapping to a Service https://github.com/Luis85/symprowire/releases/tag/v0.6.0-rc-1 Requirements
      PHP ^7.4 Fresh ProcessWire ^3.0.181 with a Blank Profile Composer 2 (v1 should work, not recommended) The usual Symfony Requirements Features
      Twig Dependency Injection Monolog for Symprowire Support for .env YAML Configuration Symfony Console and Console Commands Symfony Webprofiler Full ProcessWire access inside your Controller and Services Webpack Encore support Caveats
      Symfony is no small Framework and will come with a price in terms of Memory Usage and added Overhead. To give you a taste I installed Tracy Debugger alongside to compare ProcessWire profiling with the included Symfony Webprofiler

      So in a fresh install Symprowire would atleast add another 2MB of Memory usage and around 40ms in response time, should be less in production due to the added overhead of the Webprofiler in dev env
       
×
×
  • Create New...