Jump to content
ryan

Module: Import Pages from CSV file

Recommended Posts

Hey Ryan,

Nice trick with using: $this->page->filesManager()->path();

I'll make use of that instead of the pagefileSecure check I was using in one of my modules.

Should this be listed in the cheatsheet? Seems pretty useful to me!

  • Like 1

Share this post


Link to post
Share on other sites

Since the title field creates the page name automatically and represents the url structure of the page, is there a way to define both when importing? When importing, you must declare a title field. My title fields may contain a word with an apostrophe in it.  For example, "it's" when converted to the url would be it-s. The apostrophe is replaced with a hyphen. Is there a way to just trim the apostrophe so the url would be "its"?

Thank in advance for any suggestions. 

Share this post


Link to post
Share on other sites

Not currently with this module. Though you could write a quick file to do it for you. Place the following in your web root where ProcessWire's index.php file is, perhaps in a file called fix-names.php. Then load it in your browser. 

<?php
include("./index.php"); // bootstrap PW
$items = wire("pages")->find("template=your-template"); 
foreach($items as $item) {
  $name = preg_replace('/-s(-|$)/', '', $item->name);
  if($name != $item->name) {
    echo "<p>Changing $item->name to $name</p>";
    $item->name = $name;
    // $item->save(); 
  }
}

Uncomment that $item->save(); after you run it the first time to make sure it's doing what you want. I wrote the above here in the forum and haven't tested, so it may need tweaks. 

  • Like 1

Share this post


Link to post
Share on other sites

This is awesome Ryan! I actually used your snippet below and modified it to do what I needed and it worked perfectly, Just took me longer to create the temp field :)..

foreach($mypages as $p) {
    $p->of(false);
    $p->image->description = $p->temp;
    $p->save();
}

Your above snippet is definitely a time saver and has been added to my library. It works perfectly when I tested it. I just had to modify it slightly to meet my needs.

I would just like to say your willingness to bend over backwards to assist others in this forum is a true judgement of your character. You and several others go above and beyond to help others learn with examples. One in particular I would like to mention who helped me way beyond my expectations was Nik. Just like you, he took the time to break things down with examples and even explanations. How you both manage to find the time, I do not know. To you, Nik and all the others who help us learn everyday, I thank you sincerely for helping us. I am confident I am speaking for others as well and say that ya'll truly make a difference to those of us following in your paths.

  • Like 8

Share this post


Link to post
Share on other sites

Any progress with import page (reference) fields via this module?

I need to import categories to a page reference field.

Great module! :)

Works fine for other fields.

Share this post


Link to post
Share on other sites

Thank You for helping me with my problem  :)

It was so simple to get it work... I also found Niks post ;)

Imported all data in two steps. First simple imported the categories and ignored all other fields. After the categories added a second import works fine! Thanks.

Share this post


Link to post
Share on other sites

Hi!

I'm reading Nik's post (http://processwire.com/talk/topic/383-module-import-pages-from-csv-file/page-4#entry21476) but I cannot get my Pages to 'link'.

I'm using this line in my csv:

Ocky Docky,425,22-09-2012,,Zaterdagrit,André Ockerman,90,,,,,,,,,1306|1041|1037

The pages that I'm trying to link are 'user' templates. Selecting is absolutely ok (even works with the page's name!)

So the 'pipe' character and multi-select doesn't work :-(

Please help me out here.

Share this post


Link to post
Share on other sites

You added 'FieldtypePage' to the $fieldtypes array correct? What field type does the 'user' template have? Have you tried to connect a single page id?

  • Like 1

Share this post


Link to post
Share on other sites

Goodmorning,

Yes I added the 'FieldtypePage' correctly.

After extensive testing I found out that piping id's works with the mulitple field, but not with piping the 'name' or 'title'.

The last two things work when importing a single item in this field.

So my question becomes:

How can I alter the module to support the 'title' field when multiple values are seperated by a 'pipe' (|)?

Edit:

I looked into it, and I think the core fieldtype is not adapted to my use case. Some code should be there where it's currently missing:

https://github.com/ryancramerdesign/ProcessWire/blob/master/wire/modules/Fieldtype/FieldtypePage.module#L319

Maybe there is a quick solution to this?

Share this post


Link to post
Share on other sites

Hi Ryan,

I'm trying to import a list of retailer locations into processwire using your module.

The retailer stores all got the same name which contains an "umlaut" (ö). 

In the options I chose "Make the name unique and import new page". The Records are imported correctly but the title is cut after the umlaut (every page title is "xxx" instead of "xxxöyyy").

Do you have any suggestions how to fix this?

  • Like 1

Share this post


Link to post
Share on other sites

Hi Felix, be sure utf-8 is selected when saving your csv sheet. That should do the trick for you. 

  • Like 5

Share this post


Link to post
Share on other sites

Doh': I thought the files already utf8 encoded. I was wrong. Shame on me! :)

Thanks a lot!

Share this post


Link to post
Share on other sites

So finally, with help of others including Joss, I managed to get a select to work...

Other people were also mentioning this module. And it's probably what I need considering I need to have select options for at least 30plus values.

But how to set up the cvs file since I only need 1 e.g. title?

Share this post


Link to post
Share on other sites

Every page you want to create on a new line like this:

this is a page

and this

and an other

the fourth

we continue

let's make

a few more

and again

a page

this we can use as title

etc.

...

..

.

Save this and import it.

Share this post


Link to post
Share on other sites

that did only one insert: the last one.

Do I need to specify each line like this:

text1,

text2,

etc?

Solved:

I had to specify the header first, like:

title

text1

text2

text3

etc

  • Like 1

Share this post


Link to post
Share on other sites

that did only one insert: the last one.

Do I need to specify each line like this:

text1,

text2,

etc?

Yes. CSV (comma-separated values) needs a delimiter. This can be a comma, tab, semicolon or more. I haven't used this module for a while, but i think you can (or may need to) add a very first record as a header containing a list of field names. Furthermore, read the options the module gives. So the CSV something like this:

title,myfield1,myfield2
titleA,myfield1_value,myfield2_value
titleB,myfield1_value,myfield2_value
titleC,myfield1_value,myfield2_value

Share this post


Link to post
Share on other sites

Hi Ryan (and others), thank you for this great module. I'm looking to transfer an old ModX Revo site that is currently using latin-1 encoding. My understanding from the thread here:

https://processwire.com/talk/topic/1735-get-data-from-a-latin-1-encoded-db-and-insert-it-into-pw/

is that the following code:

$newPage->title = iconv("ISO-8859-1", "UTF-8", $row['HEADLINE']);

should convert to utf-8 first. Does your import module take this in to account or will I have to somehow add this in to your module code? Or should I just somehow create a separate php file that will cycle through my CSV and do this first, then use your module to import?

I have approximately 12,000 pages of varying types to import, including user accounts.

Thanks,

K.

Share this post


Link to post
Share on other sites

Hi @formulate, welcome to the forums!

If I understand right, you first create csv-files from the iso-8859-1 site, then you import them into PW with the csv-importer module?

Why not simply convert the whole iso-files to utf-8 files before using them for the import?

------------------

There is a case study from a (lets call it 'more customized') import scenario with a W**dPr*ss site. I have done something similar with a DB from another system (Gallery2). Maybe this is of interest for you? If you do know the DB structure of the old system good enough, so that you can query the right things in the right order, you may be able to setup a single script that does the whole import in greater chunks.

Importer scripts are easy but powerfull. If you are interested we can point you to other examples or assist in building yours (if you know enough of the old system structure and how to query it).

  • Like 1

Share this post


Link to post
Share on other sites

Or if you have the skills, perhaps you'd be interested in creating an plug-in for Migrator, like Nico's WP migrator. We have a lot of ModX refugees here, so I am sure it would get lots of use. I don't know anything about ModX, but I'd be happy to help in any way I can.

  • Like 2

Share this post


Link to post
Share on other sites

@adrian: good call!

Share this post


Link to post
Share on other sites

Or if you have the skills, perhaps you'd be interested in creating an plug-in for Migrator, like Nico's WP migrator. We have a lot of ModX refugees here, so I am sure it would get lots of use. I don't know anything about ModX, but I'd be happy to help in any way I can.

While my understanding of some aspects of ModX is top-shelf, my database knowledge and experience is woeful. I can get around in phpMyAdmin and understand ModX's table structure, but writing custom SQL queries, etc. is something I've never done. Trying to create a plug-in for Migrator seems like an unachievable task for me. I think I'm going to have enough trouble as it is just trying to get my database converted properly.

Thanks for the responses.

Share this post


Link to post
Share on other sites
This module was written before multi-language support in PW, and hasn't yet been updated to support multi language title/text/textarea fields

Would it be very difficult to create a new version supporting multi-language support?

Or are there any other tools I've missed? (i.e. I know how to create pages / populate fields via API, and also how to edit alternative languages, but it would be nice to have an automated import CSV tool for multilang sites). Perhaps a slight variation: An import tool for already existing pages, that inserts / updates one or multiple alternative language versions? If we'd declare the page ID as mandatory in the CSV, I guess this wouldn't be rocket science?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Gadgetto
      SnipWire - Snipcart integration for ProcessWire
      Snipcart is a powerful 3rd party, developer-first HTML/JavaScript shopping cart platform. SnipWire is the missing link between Snipcart and the content management framework ProcessWire.
      With SnipWire, you can quickly turn any ProcessWire site into a Snipcart online shop. The SnipWire plugin helps you to get your store up and running in no time. Detailed knowledge of the Snipcart system is not required.
      SnipWire is free and open source licensed under Mozilla Public License 2.0! A lot of work and effort has gone into development. It would be nice if you could donate an amount to support further development:

      Status update links (inside this thread) for SnipWire development
      2020-07-03 -- SnipWire 0.8.7 (beta) released! Fixes some small bugs and adds an indicator for TEST mode 2020-04-06 -- SnipWire 0.8.6 (beta) released! Adds support for Snipcart subscriptions and also fixes some problems 2020-03-21 -- SnipWire 0.8.5 (beta) released! Improves SnipWires webhooks interface and provides some other fixes and additions 2020-03-03 -- SnipWire 0.8.4 (beta) released! Improves compatibility for Windows based Systems. 2020-03-01 -- SnipWire 0.8.3 (beta) released! The installation and uninstallation process has been heavily revised. 2020-02-08 -- SnipWire 0.8.2 (beta) released! Added a feature to change the cart and catalogue currency by GET, POST or SESSION param 2020-02-03 -- SnipWire 0.8.1 (beta) released! All custom classes moved into their own namespaces. 2020-02-01 -- SnipWire is now available via ProcessWire's module directory! 2020-01-30 -- SnipWire 0.8.0 (beta) first public release! (module just submitted to the PW modules directory) 2020-01-28 -- added Custom Order Fields feature (first SnipWire release version is near!) 2020-01-21 -- Snipcart v3 - when will the new cart system be implemented? 2020-01-19 -- integrated taxes provider finished (+ very flexible shipping taxes handling) 2020-01-14 -- new date range picker, discount editor, order notifiactions, order statuses, and more ... 2019-11-15 -- orders filter, order details, download + resend invoices, refunds 2019-10-18 -- list filters, REST API improvements, new docs platform, and more ... 2019-08-08 -- dashboard interface, currency selector, managing Orders, Customers and Products, Added a WireTabs, refinded caching behavior 2019-06-15 -- taxes provider, shop templates update, multiCURL implementation, and more ... 2019-06-02 -- FieldtypeSnipWireTaxSelector 2019-05-25 -- SnipWire will be free and open source Plugin Key Features
      Fast and simple store setup Full integration of the Snipcart dashboard into the ProcessWire backend (no need to leave the ProcessWire admin area) Browse and manage orders, customers, discounts, abandoned carts, and more Multi currency support Custom order and cart fields Process refunds and send customer notifications from within the ProcessWire backend Process Abandoned Carts + sending messages to customers from within the ProcessWire backend Complete Snipcart webhooks integration (all events are hookable via ProcessWire hooks) Integrated taxes provider (which is more flexible then Snipcart own provider) Useful Links
      SnipWire in PW modules directory SnipWire Docs (please note that the documentation is a work in progress) SnipWire @GitHub (feature requests and suggestions for improvement are welcome - I also accept pull requests) Snipcart Website  

       
      ---- INITIAL POST FROM 2019-05-25 ----
       
    • By Sten
      Hello
      Till now I hacked something with the twig template but it works no more with new PW versions so I look forward to create a module. I am working on a site in multiple languages : French, English, Italian, German, Spanish, Portuguese, Hebrew, Russian. The new posts are entered in any language with a field for language. Till now, I got twig files to get the translations with constants defined for each part of the pages.
      So I'd like to create a module to include theses files added according to the url /fr/en/...
      Have you some observations to do before I begin about the direction to take ?
      Thank you
    • By ukyo
      Mystique Module for ProcessWire CMS/CMF
      Github repo : https://github.com/trk/Mystique
      Mystique module allow you to create dynamic fields and store dynamic fields data on database by using a config file.
      Requirements
      ProcessWire 3.0 or newer PHP 7.0 or newer FieldtypeMystique InputfieldMystique Installation
      Install the module from the modules directory:
      Via Composer:
      composer require trk/mystique Via git clone:
      cd your-processwire-project-folder/ cd site/modules/ git clone https://github.com/trk/Mystique.git Module in live reaction with your Mystique config file
      This mean if you remove a field from your config file, field will be removed from edit screen. As you see on youtube video.
      Using Mystique with your module or use different configs path, autoload need to be true for modules
      Default configs path is site/templates/configs/, and your config file name need to start with Mystique. and need to end with .php extension.
      Adding custom path not supporting anymore !
      // Add your custom path inside your module class`init` function, didn't tested outside public function init() { $path = __DIR__ . DIRECTORY_SEPARATOR . 'configs' . DIRECTORY_SEPARATOR; Mystique::add($path); } Mystique module will search site/modules/**/configs/Mystique.*.php and site/templates/Mystique.*.php paths for Mystique config files.
      All config files need to return a PHP ARRAY like examples.
      Usage almost same with ProcessWire Inputfield Api, only difference is set and showIf usage like on example.
      <?php namespace ProcessWire; /** * Resource : testing-mystique */ return [ 'title' => __('Testing Mystique'), 'fields' => [ 'text_field' => [ 'label' => __('You can use short named types'), 'description' => __('In file showIf working like example'), 'notes' => __('Also you can use $input->set() method'), 'type' => 'text', 'showIf' => [ 'another_text' => "=''" ], 'set' => [ 'showCount' => InputfieldText::showCountChars, 'maxlength' => 255 ], 'attr' => [ 'attr-foo' => 'bar', 'attr-bar' => 'foo' ] ], 'another_text' => [ 'label' => __('Another text field (default type is text)') ] ] ]; Example:
      site/templates/configs/Mystique.seo-fields.php <?php namespace ProcessWire; /** * Resource : seo-fields */ return [ 'title' => __('Seo fields'), 'fields' => [ 'window_title' => [ 'label' => __('Window title'), 'type' => Mystique::TEXT, // or InputfieldText 'useLanguages' => true, 'attr' => [ 'placeholder' => __('Enter a window title') ] ], 'navigation_title' => [ 'label' => __('Navigation title'), 'type' => Mystique::TEXT, // or InputfieldText 'useLanguages' => true, 'showIf' => [ 'window_title' => "!=''" ], 'attr' => [ 'placeholder' => __('Enter a navigation title') ] ], 'description' => [ 'label' => __('Description for search engines'), 'type' => Mystique::TEXTAREA, 'useLanguages' => true ], 'page_tpye' => [ 'label' => __('Type'), 'type' => Mystique::SELECT, 'options' => [ 'basic' => __('Basic page'), 'gallery' => __('Gallery'), 'blog' => __('Blog') ] ], 'show_on_nav' => [ 'label' => __('Display this page on navigation'), 'type' => Mystique::CHECKBOX ] ] ]; Searching data on Mystique field is limited. Because, Mystique saving data to database in json format. When you make search for Mystique field, operator not important. Operator will be changed with %= operator.
      Search example
      $navigationPages = pages()->find('my_mystique_field.show_on_nav=1'); $navigationPages = pages()->find('my_mystique_field.page_tpye=gallery');
    • By Robin S
      This is a module I made as an experiment a while ago and never got around to releasing publicly. At the time it was prompted by discussions around using Repeater fields for "page builder" purposes, where the depth feature could possibly be used for elements that would be nested inside other elements. I thought it would be useful to enforce some depth rules and translate the depth data into a multi-dimensional array structure.
      I'm not using this module anywhere myself but maybe it's useful to somebody.
      Repeater Depth Helper
      This module does two things relating to Repeater fields that have the "Item depth" option enabled:
      It enforces some depth rules for Repeater fields on save. Those rules are:
      The first item must have a depth of zero. Each item depth must not be more than one greater than previous item depth. It provides a RepeaterPageArray::getDepthStructure helper method that returns a nested depth structure for a Repeater field value.
      Helper method
      The module adds a RepeaterPageArray::getDepthStructure method that returns a multi-dimensional array where the key is the page ID and the value is an array of nested "child" items, or null if there are no nested children.
      Example

      The module doesn't make any assumptions about how you might want to use the depth structure array, but here is a way you might use it to output a nested unordered list.
      // Output a nested unordered list from a depth structure array function outputNestedList($depth_structure, $repeater_items) { $out = "<ul>"; foreach($depth_structure as $page_id => $nested_children) { $out .= "<li>" . $repeater_items->get("id=$page_id")->title; // Go recursive if there are nested children if(is_array($nested_children)) $out .= outputNestedList($nested_children, $repeater_items); $out .= "</li>"; } $out .= "</ul>"; return $out; } $repeater_items = $page->my_repeater; $depth_structure = $repeater_items->getDepthStructure(); echo outputNestedList($depth_structure, $repeater_items);
       
      https://github.com/Toutouwai/RepeaterDepthHelper
      https://modules.processwire.com/modules/repeater-depth-helper/
    • By MoritzLost
      Cacheable Placeholders
      This module allows you to have pieces of dynamic content inside cached output. This aims to solve the common problem of having a mostly cacheable site, but with pieces of dynamic output here and there.  Consider this simple example, where you want to output a custom greeting to the current user:
      <h1>Good morning, <?= ucfirst($user->name) ?></h1> This snippet means you can't use the template cache (at least for logged-in users), because each user has a different name. Even if 99% of your output is static, you can only cache the pieces that you know won't include this personal greeting. A more common example would be CSRF tokens for HTML forms - those need to be unique by definition, so you can't cache the form wholesale.
      This module solves this problem by introducing cacheable placeholders - small placeholder tokens that get replaced during every request. The replacement is done inside a Page::render hook so it runs during every request, even if the response is served from the template cache. So you can use something like this:
      <h1>Good morning, {{{greeting}}}</h1> Replacement tokens are defined with a callback function that produces the appropriate output and added to the module through a simple hook:
      // site/ready.php wire()->addHookAfter('CachePlaceholders::getTokens', function (HookEvent $e) { $tokens = $e->return; $tokens['greeting'] = [ 'callback' => function (array $tokenData) { return ucfirst(wire('user')->name); } ]; $e->return = $tokens; }); Tokens can also include parameters that are parsed and passed to the callback function. There are more fully annotated examples and step-by-step instructions in the README on Github!
      Features
      A simple and fast token parser that calls the appropriate callback and runs automatically. Tokens may include multiple named or positional parameters, as well as multi-value parameters. A manual mode that allows you to replace tokens in custom pieces of cached content (useful if you're using the $cache API). Some built-in tokens for common use-cases: CSRF-Tokens, replacing values from superglobals and producing random hexadecimal strings. The token format is completely customizable, all delimiters can be changed to avoid collisions with existing tag parsers or template languages. Links
      Github Repository & documentation Module directory If you are interested in learning more, the README is very extensive, with more usage examples, code samples and usage instructions!
×
×
  • Create New...