Jump to content
ryan

Module: Import Pages from CSV file

Recommended Posts

Nik has the best posts!

Another thing I wanted to add is that it's also very easy just to use API code to do these types of imports. 90% of the time I use API code to script imports with shell scripts, versus using the import pages CSV module. Though part of that is that I use the API every day and there's not a lot of figuring out to do. But if you want to post a sample of the data you are importing, I'd be happy to respond with a sample of how you might import it from the API.

Share this post


Link to post
Share on other sites

Thanks Ryan, although that's quite an overstatement what you're saying. But I'll take it. ;)

It's always nice to get feedback, especially positive one coming from the Guru Himself.

  • Like 1

Share this post


Link to post
Share on other sites

It works great. I hoped to be able to add one image, but suddenly I can add even more.

I'm not sure of the format in which to enter the 'image' column... I saw apeisa put a full url to his image column. Would this module then download the image from the link and put it in into the assets folder? or would the system always load that image from the mentioned url (ie, setting the url field equal to any, possibly external, link.)

If it does set to url field to the specified URL, then it may be a good opportunity to use a cdn for images, although I doubt it will work well that way if using the width() or height() resizers..

otherwise, I'm assuming the image is downloaded and saved to the assets folder.

Am I way off?

Share this post


Link to post
Share on other sites

otherwise, I'm assuming the image is downloaded and saved to the assets folder.

Your assumption is correct. Images are downloaded during import process.

  • Like 1

Share this post


Link to post
Share on other sites

Yup, that part of the process does make any conversion simpler :)

As long as that image is directly accessible via a browser then the importer can fetch it and put it into ProcessWire. Same with files and the file field.

The only thing you would have to watch in either case is that there is always the slim possibility you could get an unexpected file type coming from the source site. I don't think it's a problem as PW file fields only accept the file types you tell them to, but I just wasn't sure whether it would throw an error with the API so it might be worth putting in an additional check to see if it is a valid type.

Share this post


Link to post
Share on other sites

Hello! Is there any way to modify the "Created" field with this module? 

I have a CSV file where I store the dates of my old blog in a field, and the new entries to the blog are sorted with the "-created" selector in processwire. What I want to do is import the old articles of my non-processwire site to my new processwire website but using the date from the CSV to overwrite the "created" field so that it appears that it was created in that date and time.

Is there any way to do this?

Share this post


Link to post
Share on other sites

I have a situation right now, I'm not sure how unique it is or how easy it would be to solve. I have a group that runs a carshow that maintains their registration list on a custom built database application. They currently have the ability to export a .dbf file and import that directly into their website so registrants can confirm their paperwork is done.

I'm redoing the website and now I'm struggling to figure out how I'll be able to give them that same option. I'm not sure how different a .dbf is to a .csv so is it possible to customize the csv import module to allow a .dbf?

Are their any other options? Certainly there's software that allows for a .dbf to be converted to .csv but that's extra steps I'd prefer not to have to impose on them. I could also convince them to try to get the original database application changed to export .csv instead of .dbf. That's a last resort though.

Any suggestions would be appreciated.

Share this post


Link to post
Share on other sites

 I'm redoing the website and now I'm struggling to figure out how I'll be able to give them that same option. I'm not sure how different a .dbf is to a .csv so is it possible to customize the csv import module to allow a .dbf?

Are their any other options? ...

I'm not totally sure but it seems that you can read the dBase file directly with PHP, so you will be able to read the records from the file and use the API to import them.

http://php.net/manual/en/ref.dbase.php

EDIT: here is a guy who has converted dBase to CSV: http://www.dev-zero.de/php/mit-php-dbase-zu-csv-konvertieren.html

He uses a dbase class from another guy: http://www.dev-zero.de/downloads/dbf_class.php

and the code to do the convert is short:

<?php
include("dbf_class.php");          // include the class
 
$dbaseDB = ‘DB.dbf’;         
$dbf = new dbf_class($dbaseDB);    // open the dBase file
 
$csvFile = ‘file.csv’;             // create a CSV file
$csv = fopen($csvFile,"w");
 
                                   // get record count, loop through and convert to csv
$num_rec=$dbf->dbf_num_rec;
fputcsv($csv, array_keys($dbf->getRowAssoc(0)));
 
for($i=0; $i<$num_rec; $i++) {
    fputcsv($csv, $dbf->getRow($i));
}
 
fclose($csv);          // ready! now use Ryans CSV-Importer 

But want to note that I only have googled that and have not tested / used this by myself, so- no warranties ;-)

Edited by horst
  • Like 3

Share this post


Link to post
Share on other sites

Hi! Great module, thanks for it.

I think it can help me to keep catalog up to date. But I need to generate page name from another field than Title (Many titles are similar, but "article number" field is unique). Is it possible somehow?

Share this post


Link to post
Share on other sites

Is there a possibility to make the module available for an editor? If I move it out of the "admin" tree - it is available, but he has to click "view" to see the module. Would it be possible to leave it in the admin tree and make it visible in the top navigation for him?

Share this post


Link to post
Share on other sites

Is there a possibility to make the module available for an editor? If I move it out of the "admin" tree - it is available, but he has to click "view" to see the module. Would it be possible to leave it in the admin tree and make it visible in the top navigation for him?

I'm not sure what you want but the module can already be reached via the top navigation by 'Setup->Import Pages From CSV'

You could probably also make some easy adjustments to the module file before installing it to make the page go wherever you want.

https://github.com/ryancramerdesign/ImportPagesCSV/blob/master/ImportPagesCSV.module#L548 and further, i guess change the $parent variable.

Share this post


Link to post
Share on other sites

Yes, I know it is accessible through the "Setup" - but this part of the navigation is not allowed for a normal editor, just for a superuser.

I would like to make it available for a non-superuser in the top navigation ;-)

Share this post


Link to post
Share on other sites

Of course, stupid me..i will try to come up with a solution. Anyways i was wondering how you were able to move the page out of the admin tree, because this shouldn't be possible; the Import Pages From CSV page has the template 'admin' and this template only allows parents with admin template.

Share this post


Link to post
Share on other sites

Yes, you have to change the children settings for that template once to move the page, then set it back to the standard settings.

And I tried a second possibility: I copied the admin template and gave that other permissions. But somehow I couldn't manage it to make the page viewable in the top navigation for a non-superuser...

Share this post


Link to post
Share on other sites

Ok, got it - and I have to say once again: "Thanks, Soma!"

Here the thread and the post with the solution:

http://processwire.com/talk/topic/1272-new-page-nav-in-admin/?p=11276

The important part is the setting of the moduls permisson:

public static function getModuleInfo() {
		return array(
			'title' => 'Import Tabelle als CSV', 
			'version' => 103, 
			'summary' => 'Import CSV files to create ProcessWire pages.',
			'singular' => true, 
			'autoload' => false, 
			'permission' => 'page-edit'
			);

In this way the modul is editable by a non-superuser and can be accessed through the top navigation.

  • Like 2

Share this post


Link to post
Share on other sites

Just a quick question, is it possible that this could work with a password field (for uploading users). Thanks.

Share this post


Link to post
Share on other sites

Nik, your FieldtypePage tip with the id's as the reference worked perfectly. Add one more day that you are my hero :).

Share this post


Link to post
Share on other sites

Just wanted to say Thank You for this great module. I am working on a quite large project at the moment for which this module already has helped a lot.

  • Like 1

Share this post


Link to post
Share on other sites

Greetings,

I'll jump in as well...

Not too long ago, I used this module to run the initial import for a TV station project, where I needed to get 550+ pages from an old Joomla site.  After getting the data into the right csv format from the database, it was unbelievably smooth and easy to populate the pages.  I kept thinking, "Something will certainly go wrong with this," but it didn't!  I got all the existing Joomla pages into ProcessWire perfectly.  After that, the client used ProcessWire to create new pages.

My next goal with this -- build a front-end way for clients to do the imports themselves.

By the way, I was prompted to post this after reading and responding to this post: http://processwire.com/talk/topic/3917-importing-users-and-content-from-joomla/

Thanks Ryan!

Matthew

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Gadgetto
      SnipWire - Snipcart integration for ProcessWire
      Snipcart is a powerful 3rd party, developer-first HTML/JavaScript shopping cart platform. SnipWire is the missing link between Snipcart and the content management framework ProcessWire.
      With SnipWire, you can quickly turn any ProcessWire site into a Snipcart online shop. The SnipWire plugin helps you to get your store up and running in no time. Detailed knowledge of the Snipcart system is not required.
      SnipWire is free and open source licensed under Mozilla Public License 2.0! A lot of work and effort has gone into development. It would be nice if you could donate an amount to support further development:

      Status update links (inside this thread) for SnipWire development
      2020-04-06 -- SnipWire 0.8.6 (beta) released! Adds support for Snipcart subscriptions and also fixes some problems 2020-03-21 -- SnipWire 0.8.5 (beta) released! Improves SnipWires webhooks interface and provides some other fixes and additions 2020-03-03 -- SnipWire 0.8.4 (beta) released! Improves compatibility for Windows based Systems. 2020-03-01 -- SnipWire 0.8.3 (beta) released! The installation and uninstallation process has been heavily revised. 2020-02-08 -- SnipWire 0.8.2 (beta) released! Added a feature to change the cart and catalogue currency by GET, POST or SESSION param 2020-02-03 -- SnipWire 0.8.1 (beta) released! All custom classes moved into their own namespaces. 2020-02-01 -- SnipWire is now available via ProcessWire's module directory! 2020-01-30 -- SnipWire 0.8.0 (beta) first public release! (module just submitted to the PW modules directory) 2020-01-28 -- added Custom Order Fields feature (first SnipWire release version is near!) 2020-01-21 -- Snipcart v3 - when will the new cart system be implemented? 2020-01-19 -- integrated taxes provider finished (+ very flexible shipping taxes handling) 2020-01-14 -- new date range picker, discount editor, order notifiactions, order statuses, and more ... 2019-11-15 -- orders filter, order details, download + resend invoices, refunds 2019-10-18 -- list filters, REST API improvements, new docs platform, and more ... 2019-08-08 -- dashboard interface, currency selector, managing Orders, Customers and Products, Added a WireTabs, refinded caching behavior 2019-06-15 -- taxes provider, shop templates update, multiCURL implementation, and more ... 2019-06-02 -- FieldtypeSnipWireTaxSelector 2019-05-25 -- SnipWire will be free and open source Plugin Key Features
      Fast and simple store setup Full integration of the Snipcart dashboard into the ProcessWire backend (no need to leave the ProcessWire admin area) Browse and manage orders, customers, discounts, abandoned carts, and more Multi currency support Custom order and cart fields Process refunds and send customer notifications from within the ProcessWire backend Process Abandoned Carts + sending messages to customers from within the ProcessWire backend Complete Snipcart webhooks integration (all events are hookable via ProcessWire hooks) Integrated taxes provider (which is more flexible then Snipcart own provider) Useful Links
      SnipWire in PW modules directory SnipWire Docs (please note that the documentation is a work in progress) SnipWire @GitHub (feature requests and suggestions for improvement are welcome - I also accept pull requests) Snipcart Website  
      ---- INITIAL POST FROM 2019-05-25 ----
       
    • By bernhard
      #######################
      Please use the new RockFinder2
      #######################
      WHY?
      This module was built to fill the gap between simple $pages->find() operations and complex SQL queries.
      The problem with $pages->find() is that it loads all pages into memory and that can be a problem when querying multiple thousands of pages. Even $pages->findMany() loads all pages into memory and therefore is a lot slower than regular SQL.
      The problem with SQL on the other hand is, that the queries are quite complex to build. All fields are separate tables, some repeatable fields use multiple rows for their content that belong to only one single page, you always need to check for the page status (which is not necessary on regular find() operations and therefore nobody is used to that).
      In short: It is far too much work to efficiently and easily get an array of data based on PW pages and fields and I need that a lot for my RockGrid module to build all kinds of tabular data.

      Basic Usage

       
      Docs & Download
      https://modules.processwire.com/modules/rock-finder/
      https://github.com/BernhardBaumrock/RockFinder
       
      Changelog
      180817, v1.0.6, support for joining multiple finders 180810, v1.0.5, basic support for options fields 180528, v1.0.4, add custom select statement option 180516, change sql query method, bump version to 1.0.0 180515, multilang bugfix 180513, beta release <180513, preview/discussion took place here: https://processwire.com/talk/topic/18983-rocksqlfinder-highly-efficient-and-flexible-sql-finder-module/
    • By MoritzLost
      TrelloWire
      This is a module that allows you to automatically create Trello cards for ProcessWire pages and update them when the pages are updated. This allows you to setup connected workflows. Card properties and change handling behaviour can be customized through the extensive module configuration. Every action the module performs is hookable, so you can modify when and how cards are created as much as you need to. The module also contains an API-component that makes it easy to make requests to the Trello API and build your own connected ProcessWire-Trello workflows.
      Features
      All the things the module can do for you without any custom code: Create a new card on Trello whenever a page is added or published (you can select applicable templates). Configure the target board, target list, name and description for new cards. Add default labels and checklists to the card. Update the card whenever the page is updated (optional). When the status of the card changes (published / unpublished, hidden / unhidden, trashed / restored or deleted), move the card to a different list or archive or delete it (configurable). You can extend this through hooks in many ways: Modifiy when and how cards are created. Modify the card properties (Target board & list, title, description, et c.) before they are sent to Trello. Create your own workflows by utilizing an API helper class with many convenient utility methods to access the Trello API directly. Feedback & Future Plans
      Let me know what you think! In particular:
      If you find any bugs report them here or on Github, I'll try to fix them. This module was born out of a use-case for a client project where we manage new form submissions through Trello. I'm not sure how many use-cases there are for this module. If you do use it, tell me about it! The Trello API is pretty extensive, I'll try to add some more helper methods to the TrelloWireApi class (let me know if you need anything in particular). I'll think about how the module can support different workflows that include Twig – talk to me if you have a use-case! Next steps could be a dashboard to manage pages that are connected to a Trello card, or a new section in the settings tab to manage the Trello connection. But it depends on whether there is any interest in this 🙂 Links
      Repository on Github Complete module documentation (getting started, configuration & API documentation) [Module directory pending approval] Module configuration

    • By MoritzLost
      Process Cache Control
      This module provides a simple solution to clearing all your cache layers at once, and an extensible interface to perform various cache-related actions.
      The simple motivation behind this module was that I was tired of manually clearing caches in several places after deploying a change on a live site. The basic purpose of this module is a simple Clear all caches link in the Setup menu which clears out all caches, no matter where they hide. You can customize what exactly the module does through it's configuration menu:
      Expire or delete all cache entries in the database, or selectively clear caches by namespace ($cache API) Clear the the template render cache. Clear out specific folders inside your site's cache directory (/site/assets/cache) Clear the ProCache page render cache (if your site is using ProCache) Refresh version strings for static assets to bust client-side browser caches (this requires some setup, see the full documentation for details). This is the basic function of the module. However, you can also add different cache management action through the API and execute them through the module's interface. For this advanced usage, the module provides:
      An interface to see all available cache actions and execute them. A system log and logging output on the module page to see verify what the module is doing. A CacheControlTools class with utility functions to clear out different caches. An API to add cache actions, execute them programmatically and even modify the default action. Permission management, allowing you granular control over which user roles can execute which actions. The complete documentation can be found in the module's README.
      Plans for improvements
      If there is some interest in this, I plan to expand this to a more general cache management solution. I particular, I would like to add additional cache actions. Some ideas that came to mind:
      Warming up the template render cache for publicly accessible pages. Removing all active user sessions. Let me know if you have more suggestions!
      Links
      https://github.com/MoritzLost/ProcessCacheControl ProcessCacheControl in the Module directory CHANGELOG in the repository Screenshots


    • By Macrura
      PrevNextTabs Module
      Github: https://github.com/outflux3/PrevNextTabs
      Processwire helper modules for adding page navigation within the editor.
      Overview
      This is a very simple module that adds Previous and Next links inline with the tabs on the page editor. Hovering over the tab shows the title of the previous or next page (using the admin's built in jqueryUI tooltips.)
      Usage
      This module is typically used during development where you or your editors need to traverse through pages for the purpose of proofing, flagging and/or commenting. Rather than returning to the page tree or lister, they can navigate with these links.
      Warnings
      If you are using PW version 2.6.1 or later, the system will prevent you from leaving the page if you have unsaved edits.
      For earlier versions, to avoid accidentally losing changes made to a page that might occur if a user accidentally clicks on one of these, make sure to have the Form Save Reminder module installed.
      http://modules.processwire.com/modules/prev-next-tabs/
×
×
  • Create New...