joey102030

Export pages to csv

21 posts in this topic

Not that I know of, but you can export your database tables (e.g. using phpMyAdmin) as CSV....Not sure if this will help you since I don't know your ultimate aim with the export :-)

Edited for clarity...

Edited by kongondo

Share this post


Link to post
Share on other sites

Too simple to be a module, consider a script like this:

$array = $pages->find("template=basic-page")->explode(function($item){
    return array(
        'id'=> $item->id,
        'title' => $item->title
        );
});

$fp = fopen('file.csv', 'w');
foreach ($array as $fields) fputcsv($fp, $fields);
fclose($fp);
 

Note, $pagearray->explode() used here is only available in 2.4 (2.3 dev)

http://cheatsheet.processwire.com/pagearray-wirearray/getting-items/a-explode/

And the anonymous functions requires php >= 5.3 http://php.net/manual/de/functions.anonymous.php

12 people like this

Share this post


Link to post
Share on other sites

No just a example. Of course a module to create csv would be possible and cool, but some work required to make it flexible. ;)

Share this post


Link to post
Share on other sites

Thanks for the replies

I think this could work as a module.

You would need to configure it at a template level with the fields you want to output, then in the list of page actions it could say something like 'export children as csv' (if there are any children)

Share this post


Link to post
Share on other sites

thanks Soma! i was working on an export function sort of like this yesterday - this will help a lot!

Share this post


Link to post
Share on other sites

OK I've had a play around with this.

I created a new module which extends ProcessPageSearch, but instead of outputting the results to screen they are outputted in .csv format to the browser as a downloadable file.

All parameters are accepted in the query string, so the download could be a full URL linked from anywhere in the admin area (ie a custom page action)

I'm new to module creation, so I'm not sure if extending a core module like ProcessPageSearch is a good idea as it could change in future releases.

Maybe someone more experience could advise?

Share this post


Link to post
Share on other sites

It's true that ProcessPageSearch may change down the road, though no specific plans on that at present. But if you are worried about that, rather than extending it (in the PHP class sense) you could just copy ProcessPageSearch to another module, and modify it to make your own. The PW API behavior doesn't change, so when you make something where that is the dependency (as opposed to the implementation of class that's already an endpoint) then it's a safer bet. 

1 person likes this

Share this post


Link to post
Share on other sites

Thanks Ryan

That sounds sensible.

I've expressed my love for ProcessPageSearch in these forums before... but seriously there is tonnes of potential to utilise the output in various ways, including datatables, charts, reports etc.

I will get the csv output module I've created into a usable state ASAP. 

1 person likes this

Share this post


Link to post
Share on other sites
I've expressed my love for ProcessPageSearch in these forums before... but seriously there is tonnes of potential to utilise the output in various ways, including datatables, charts, reports etc.

That was the hope with the JSON API we put into it. Two modules that use it are the InputfieldPageListAutocomplete and ServicePages modules. Though of those two, only InputfieldPageAutocomplete uses it in the way originally intended, as ServicePages essentially translates ProcessPageSearch for front-end use. 

Share this post


Link to post
Share on other sites

Soma has an excellent script above that uses the new WireArray::explode() function to bulk-export pages to CSV. But the script requires that you hard-code any custom fields for your pages. Is it possible to get the field names and field values dynamically (meaning, it is not necessary to hard-code each field key and value)?

Example: (to illustrate my point -- this code snippet does not work at present)

$array = $pages->find("template=foo")->explode(function($items){
    foreach ($items as $key => $item) {
        $arr[$key] = $item->$key;
    }
    return $arr;
    # Or, perhaps just cast using (array):
    # return (array) $items;
});

$fp = fopen('export.csv', 'w');
$i = 0;
foreach ($array as $fields) {
   // Add headings to the first line.
    if($i==0) fputcsv($fp, array_keys($fields), "\t");
    fputcsv($fp, $fields, "\t");
    $i++;
}
fclose($fp);

Note: Page fields are exported with their ID ("1001"), in a future version the script could look up these Page IDs.

See also:

1 person likes this

Share this post


Link to post
Share on other sites

@SwimToWin: to get all fields from a page (template) you can use a snippet like this:

foreach ($page->template->fieldgroup as $field) {

But you have to check what types of field you can export to csv, I think. I'm not sure but I think so. Or maybe I'm wrong.

If you want to check for fieldtypes you do it like

if($field->type instanceof FieldtypeFile)  // or FieldtypeText, or ...
1 person likes this

Share this post


Link to post
Share on other sites

$page->fields is better. Does $page->template->fieldgroup work? Thought more like $page->template->fields or fieldgroup->fields.

Share this post


Link to post
Share on other sites

Ok, I have read in the cheatSheet:

$template->fieldgroup = Get or set a template's Fieldgroup. Can also be used to iterate a template's fields.

$template->fields = Syntactical alias for $template->fieldgroup. Use whatever makes more sense for your code readability.

$page->fields = All the Fields assigned to this page (via it's template, same as $page->template->fields). Returns a FieldsArray.

So, this is all the same! And therefor I will use $page->fields in future. (It's more close for me. Ok, for Ryan it is more close to use $template->fieldgroup because in his thinking it is obvious that that is the part what gets invoked at the end) :)

Edited by horst
1 person likes this

Share this post


Link to post
Share on other sites

Some cross-linking and gratuitous self promotion :)

I have just implemented highly configurable CSV exporting (admin and API) into Batch Child Editor:

https://processwire.com/talk/topic/6102-batch-child-editor/page-2#entry95855

It does lots of work for you, including supporting ProFields Textareas and Multiplier, along with outputting Page field title/name, rather than ID.

Export settings can be configured by the developer and/or the site editors. 

It might be useful for others searching for CSV export and finding this thread.

2 people like this

Share this post


Link to post
Share on other sites
On 5/26/2015 at 3:55 PM, adrian said:

Some cross-linking and gratuitous self promotion :)

I have just implemented highly configurable CSV exporting (admin and API) into Batch Child Editor:

https://processwire.com/talk/topic/6102-batch-child-editor/page-2#entry95855

It does lots of work for you, including supporting ProFields Textareas and Multiplier, along with outputting Page field title/name, rather than ID.

Export settings can be configured by the developer and/or the site editors. 

It might be useful for others searching for CSV export and finding this thread.

Something I couldn't find or notice in your code Adrian, is there a way to filter the child pages you want as CSV by a selector?

I was looking to use the exportCsv method to export pages stored under a PageTable.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By hellomoto
      Would anyone here be able and inspired to develop a ProcessWire equivalent to the WordPress plugin WP All Import Pro? or to help me do so?
      This is what I'm envisioning...
      Upon installation the module creates an admin page titled "Import & Update". On the module config page you can specify allowed templates to run this on, otherwise allowing any. Include the following PHP libraries: hQuery for web scraping, Csv for CSV handling, and Parser for XML. Create template "import-update". On the "Import & Update" page, a list of current import/updaters will be displayed (0 initially), each with corresponding links to "edit" or "run". When you "Add New", this be the "import-update" template (with all module-specific fields tagged "impupd"):
      title destination (req.): parent, template source (req.): type (web, csv, xml) location (url, file, text) if web: opt. index URL & link selector, + paginator selector if csv: opt. ignore 1st row if xml: req. individual item node xpath actions (check): import (if none matching UID) update (if matching UID & field values differ) save() [req. here in flow] map (repeater): input (select fields from specified template to affect) intake (corresponding DOM selectors / CSV col. letters/headers / xpath per field) (req.) UID (unique ID; field reference to compare against, from selected input fields) (req.) Lazy Cron interval Scripts can be run via the import-update template; keep logs; show preview (iframe/ajax) for manual runs. ...
    • By benbyf
      Hi!
      I'm trying to export a large site but the export profile  module keeps timing out due to the size of the installation its trying to export (around 1500 pages - half with body content). Is there any work around or is it worth simply duplicating the site and files without using teh export profile module?
      Thanks
    • By electricarts
      Hallo,
      i'm searching for a way to import entries from a css file. I have a csv (or txt) file with some comma-separated data and want to import just 1 entry into a list (text or textarea field) in the PW (3.0.33) backend. 
      I tried the "Import Pages from csv files" module. I'm not sure if this is the right solution. The file can't be imported, because after Step 2 and starting the import, the "Unable to import page because it has no required 'title' field or it is blank" error message appears. I'm doing something wrong or is using this module the wrong way of importing csv entries?
      Mario
    • By iNoize
      Hello, 
      i have some uderdata that i need to import to mailchimp or something else. 
      How can i create with the api an xls or cv file ? This is simple data like name surname email an newsletterstatus. 
       
    • By sjohnson
      [Posted this in another post, but realized it's doesn't really pertain to it, so I've moved it to this new topic -- again sorry]
      Hi Ryan (and everyone else!),
      Just starting using PW and I'm loving it... I've got a site build that requires a bunch of CSV loading and would like some help with API Importing, as I am running into some glitches with page fields.
      So far I have a bunch of CSVs that work fine with my php code, but I have one that crashes, unless I put in the pageID.
      I think it may be something larger as I have tried using both BatchChildEditor & ImportPagesCSV modules with a modified CSV, and both get the same error as my code.
      So, here's my code, it does the following:
      Reads a page field to get CSV filename Reads 1st row of file to figure out what template and parent to use Reads 2nd row for field titles Processes the remaining rows as data, it also echoes out what it is doing -- not elegant, but it's only for my use //    If no filename exit     if ($wire->page->getUnformatted('section_slogan') == '') throw new Wire404Exception(); //    File Pointer         $myfile = "http://" . $config->httpHost . $config->urls->templates . "loaders/" . $wire->page->getUnformatted('csvfilename');     $content = "<p>Using Loader File '{$myfile}'</p><hr/>"; //    Read The Data File     if (($handle = fopen("$myfile", "r")) !== FALSE) {     //    Get the Setup Info         if (($data = fgetcsv($handle, 0, ",")) !== FALSE) {             $myTemplate = $templates->get("$data[0]");             $myParent = $wire->pages->get("path='$data[1]'");             if (($myTemplate->id == 'NullPage') || ($myParent->id == 'NullPage')){                 throw new Wire404Exception();                             }         } else {             throw new Wire404Exception();         }     //    Setup the parent         $newItem = new Page();         $newItem->template = $myTemplate;     //    Tell User what's going on:             $content .= "<p>Adding New Pages based on the '{$newItem->template->name}' template to parent '{$myParent->title}' path: '{$myParent->path}'</p><hr/>";     //    List out the fields in the template         $num = count($newItem->fields);         $content .= "<p> $num fields in template</p><ol>";         foreach($newItem->fields as $itemfd) {             $content .= "<li>" . $itemfd->name . "</li>";         }         $content .= "</ol><hr/>";     //    Read header record and match-up fields to template         $fieldmatchup = '';         if (($data = fgetcsv($handle, 0, ",")) !== FALSE) {             $num = count($data);             for ($c=0; $c < $num; $c++) {                 foreach($newItem->fields as $itemfd) {                     if($data[$c] == $itemfd->name) {                         $fieldmatchup[$itemfd->name] = $c;                     }                 }             }                     }     //    Show the File's Field Array         $content .= "<p>Display the Field Array</p><ul>";         foreach($fieldmatchup as $item_key => $item_value) {             $content .= "<li>['" . $item_key . "'] = '" . $item_value . "'</li>";         }         $content .= "</ul><hr/><h4>Reading File</h4><hr/>";     //    Now add the data         $row = 0;         $newItem = '';         while (($data = fgetcsv($handle, 0, ",")) !== FALSE) {             $num = count($data)-1;             $row++;             $mySel = "parent=" . $myParent->id . ", title='" . $data[$fieldmatchup['title']] . "'";             $dupPage = $wire->pages->get($mySel);             if($dupPage->id == 'NullPage') {                 $content .= "<p>newItem (" . $mySel . ")<br/><b>-- Creating from File</b>.</p>\n";             //    Setup blank WireArray based on a template                 $newItem = new Page();                 $newItem->template = $myTemplate;                 $newItem->title = $data[$fieldmatchup['title']];                 $newItem->parent = $myParent;                 $newItem->save();             }             else {                 $content .= "<p>Duplicate Found (" . $mySel . ")<br/><b>-- Updating from File</b>.</p>\n";                 $newItem = $dupPage;             }         $content .= "<p> {$num} fields in record {$row}: <br /></p>\n";             $newItem->of(false);             foreach($newItem->fields as $itemfd) {                 $key = $itemfd->name;                 if ($key != 'title') {                     $value = $data[$fieldmatchup[$key]];                     $content .= "[{$key}] = [{$value}]<br />\n";                     $newItem->set($key, $value);                 }             }             $newItem->save();             $content .= "<b>-- Saved: </b> <a target='_blank' href='" . $newItem->editUrl . "'>" . $newItem->title . " [" . $newItem->id . "]</a><hr/>";         }         fclose($handle);     }     $content .= "<b>{$row} records added/updated</b>";     And here's a sample of data that works: (associated_stat & special_category are page fields)
      skill,"/systems/sol/" title,associated_stat,multiplier,special,special_category,body Alien Archeology,Int,4,,,"Skill description" Alien Tech,Tech,4,,,"Skill description" Alien Weapons,Ref,4,1,Weapons,"Skill description" And the file that's giving me issues:(pc_role_category, associated_book, sa_skill are page fields, career_skills is a multiple page field)
      cp-role,"/systems/sol/" title,pc_role_category,associated_book,page_no,verified_via,sa_skill,career_skills,body "Merc","Combat Related","Book Name","8","Book Review","Combat Zen","Athletics|Alien Tech|Drive|Shoot","Role Description" It's the associated_book field that's giving me the issue, if I blank it or put in the PageID it works, but with anything else it errors out with this:
      Fatal Error Call to a member function __unset() on boolean search Source File: ...\core\wire\modules\Fieldtype\FieldtypePage.module:439 431: if($value instanceof Page) { 432: // ok 433: } else if($value instanceof PageArray) { 434: $value = $value->first(); 435: } else if(is_string($value) || is_int($value)) { 436: $value = $this->sanitizeValueString($page, $field, $value); 437: if($value instanceof PageArray) $value = $value->first(); 438: if($value->_FieldtypePage_remove === $value->id) { 439: $value->__unset('_FieldtypePage_remove'); 440: $value = null; // remove item 441: } 442: } So, looking at this field, and the associated page template, everything is setup the same as all the other page fields, except that the page template in question (books) has a field that references a page field (associated_system) that I'm not even referencing, so I'm not sure if that's the culprit or not, but that is the only thing that separated this template from the others is this custom label code and that the pages are outside of the parent:

      And yes, I've removed the custom label and it still has the same error.
      With my luck it's something simple, but I can't see it... Any help would be appreciated