Jump to content
adrian

Profields Table CSV Importer / Exporter

Recommended Posts

Hi @awebcreature - I have just tested with the contents of your CSV but am not seeing the problem here. Based on the line of the error, it looks like you have "Convert comma decimals to dots" checked. But none of your entries have any commas or dot decimals. Did you change the "Columns separated with" to semicolon to match your data?

Can you perhaps send me a screenshot of your settings for the import, along with the Details tab of the table field so I can see how the table fields have been defined. Also, a copy of the CSV might be good, just to make sure I am dealing with exactly the same data.

Also, can you dump $value before that error so I can see what is in the array?

Share this post


Link to post
Share on other sites

Thanks for all those details. It's really late and I need to get to sleep so I will look at this in the morning, but in the meantime, I see that you have Tracy installed, so you can just insert this line:

bd($value);

just after this line: https://github.com/adrianbj/TableCsvImportExport/blob/master/TableCsvImportExport.module#L359

This will populate the dumps panel in Tracy - please post the content here.

There must be something strange going on, because you don't have the convert decimals option checked, but it is being called anyway. Could you also add:

bd($convertDecimals);

just after this line: https://github.com/adrianbj/TableCsvImportExport/blob/master/TableCsvImportExport.module#L274

I just noticed I have an incorrect "==" which should be "===" here: https://github.com/adrianbj/TableCsvImportExport/blob/master/TableCsvImportExport.module#L339 - this shouldn't cause the problem - more of a reminder for me to change this tomorrow.

Share this post


Link to post
Share on other sites

Is there any chance this site is online and I could possibly get access to take a look?

Are all 63 errors shown in that screenshot coming from this module? Could you maybe at least show me the contents of that panel?

  • Like 1

Share this post


Link to post
Share on other sites

These errors are : 

42×PHP Notice: Array to string conversion in .../TableCsvImportExport/TableCsvImportExport.module:340

21×PHP Notice: Indirect modification of overloaded property TableRow::$Array has no effect in .../TableCsvImportExport/TableCsvImportExport.module:340

error on each element  from .csv file

Tomorrow this PW will be online.

Thanks again!

Share this post


Link to post
Share on other sites

... and i will eat my mouse!

On the my local machine i have PHP version 7. When I change it to 5.6.10 - no errors anymore. Read and import cvs file correctly into table. 

Thanks for your support Andrian! 

I hope that situation will be in plus to your future development of that useful module.

Thanks again! 

 

  • Like 1

Share this post


Link to post
Share on other sites
2 minutes ago, awebcreature said:

... and i will eat my mouse!

On the my local machine i have PHP version 7. When I change it to 5.6.10 - no errors anymore. Read and import cvs file correctly into table. 

Thanks for your support Andrian! 

I hope that situation will be in plus to your future development of that useful module.

Thanks again! 

 

Thanks for reporting that - I was struggling to figure out why you were getting those errors. I guess it's getting to the point that we all need to have a PHP7 testing environment set up so we can fix these sorts of things. 

I'll see what I can do about getting it 7 compatible fairly soon.

  • Like 1

Share this post


Link to post
Share on other sites

Nice - I am a HomeBrew user, so that looks like it should make things very simple - cheers!

Share this post


Link to post
Share on other sites

First of all: thank you very much for the great module.  This is so convenient to have!!!!

Just a small feedback: I managed to shoot my table down. Maybe to many entries (250) maybe and invalid file. No big deal, I have backups and the full database saved as CSV.

I got warnings from PW about not properly escaped things (don't remember exactly the wording). And, yes, I found some single and double quotes in table cells from the import file (actually an exported and then re-imported CSV file), After deleting those chars, the import worked.

If this is possible, maybe a warning about illegal chars or a proper escape during import would be very useful?

Also in my situation it would help if there was an "import & replace" feature. So I can do another import that is properly replacing a malfunctioning previous one. Right now the import was added. (Now my table has double entries - and maybe even double IDs because I cannot select and delete an entry any more, it re-appears after a save)

A replace would be a convenient way to clean this mess up quickly. 

Share this post


Link to post
Share on other sites

Hi @ceberlin - glad you are finding it convenient. Sorry there was a problem with your upload though. It shouldn't be a number of entries issue - I have imported thousands of entries without a problem (although field Table itself is slow to render with that many until Ryan adds pagination - hopefully soon).

Any chance you could send me the file in it's original form so I can test and add some checks for illegal characters etc. Just to confirm - was the exported file exported using this module? Sounds like I might have an issue there.

BTW - what version of PHP and PW are you running?

There is already an overwrite option - you just need to enable it:

Screen Shot 2016-06-18 at 3.45.19 PM.png

  • Like 1

Share this post


Link to post
Share on other sites

Hi @adrian -  thank you for your quick response.

1. Oh I see, I overlooked that feature, Brilliant!!!!!!!

(In the explanation, of you replace the word "overwrite" with "replace", it maybe gets clearer? Also i.m.h.o. it would be good to prepent the word "Caution: " maybe?)

 

2. My problem now with the import is that after the import I cannot save the page any longer (any changes/updates, regardless in what field, are ignored on save). No errors get recorded. Looks like something is timing out or confusing the database storage script.

This is the page I am working at: /tour-list/

The fields in use are a "single page" selector, a date field and some tiny text fields.

I send you PM with the data I am trying to load. And you can also have access to anything (it's a fan community website only), if you need to investigate further.

 

 

 

Share this post


Link to post
Share on other sites

SOLVED!!!!

OK, here is, what I discovered, and a workaround:

  • if you have errors on upload (database error, "not properly escaped"), look for single quotes ' in your file.
  • If you cannot change and save the page any more (you click save and the page tries to save but nothing gets actually saved any more - and no php- or js error is given) try this:
    in PHP set "max_input_vars = 5000" as a workaround.

max_input_vars = 5000 is unusual (there is a reason why the limit is normally 1000), so there is a possible bug in the pro table field which I will report now accordingly.

 

 

  • Like 1

Share this post


Link to post
Share on other sites
On 17.6.2016 at 7:44 PM, awebcreature said:

... and i will eat my mouse!

On the my local machine i have PHP version 7. When I change it to 5.6.10 - no errors anymore. Read and import cvs file correctly into table. 

Thanks for your support Andrian! 

I hope that situation will be in plus to your future development of that useful module.

Thanks again! 

 

Hi 

I also stumbled across this problem. It looks like the new variable handling of php7 is responsible for that...

I found a solution wich works on php 5.6 and 7.

Just posted an github issue with explanation...
https://github.com/adrianbj/TableCsvImportExport/issues/1

greetings, tom

 

  • Like 3

Share this post


Link to post
Share on other sites

Thanks @gingebaker - I have applied your curly brace fixes. Please let me know if you have any further  troubles.

  • Like 3

Share this post


Link to post
Share on other sites

Hi everyone,

I have spent quite a bit of time today getting this module working with the new pagination functionality in the Table field.

I have tested it in a variety of import (append and overwrite) and export scenarios with standard as well as page fields (radios, checkbox, selects, etc) and it seems to be working well.

The trouble is that this new version will only with with the new version of the Table field. It will work even if pagination is not enabled, but it won't work with older Table field versions.

I would really appreciate it if you guys could test and let me know how it works for you. I'd like to get this pushed to Github fairly soon, but it's such a major change that I would like some additional testing first.

Thanks!

 

PS - this update also add support for line breaks in textarea fields - I added the same CSV parser that I recently added to my BCE module.

Edited by adrian
Remove attachment because it's now on Github
  • Like 2

Share this post


Link to post
Share on other sites

Given the underwhelming response I am going to assume that not many people are using this module, so I am just going to update the Github repo with the new version. I used it a LOT yesterday and it seems to be working great with the new Table pagination feature. 

I have removed 2.7 and below from the compatibility list for this module. It is now only 3.x (or 2.8). If you need the old version that support 2.7 and below, you can grab it from this commit: https://github.com/adrianbj/TableCsvImportExport/tree/b83df4d13fd18fcb6c4bc8ac467e6ae35cba9711

Hope some of you out there will find the new version useful.

  • Like 3

Share this post


Link to post
Share on other sites

Latest version supports:

  1. Column selection and ordering for CSV export.
  2. Exported rows also respect the Find filters that have been applied.

Should be very handy for clients to be able to extract relevant info from large table datasets.

Screen Shot 2016-10-03 at 9.03.17 AM.png

  • Like 4

Share this post


Link to post
Share on other sites

Great!
(this is an overwhelming response :))

  • Like 2

Share this post


Link to post
Share on other sites

Hi!

I am getting an error on export:

Uncaught Error: Call to a member function getColumns() on null in \site\assets\cache\FileCompiler\site\modules\TableCsvImportExport\ProcessTableCsvExport.module:108

Stack trace:
#0 \site\assets\cache\FileCompiler\site\modules\TableCsvImportExport\ProcessTableCsvExport.module(62): ProcessTableCsvExport->exportCsv()
#1 \wire\core\Wire.php(374): ProcessTableCsvExport->___execute()
#2 \wire\core\WireHooks.php(698): ProcessWire\Wire->_callMethod('___execute', Array)
#3 \wire\core\Wire.php(402): ProcessWire\WireHooks->runHooks(Object(ProcessTableCsvExport), 'execute', Array)
#4 \wire\core\ProcessController.php(244): ProcessWire\Wire->__call('execute', Array)
#5 \wire\core\Wire.php(374): ProcessWire\ProcessController->___execute()
#6 \wire\core\WireHooks.php(698): ProcessW in <b>\site\assets\cache\FileCompiler\site\modules\TableCsvImportExport\ProcessTableCsvExport.module</b> on line <b>108</b><br />

testing on:

PW 3.0.52 @ php7.1
PW 3.0.48 @ php5.6.12

UPD: ProFields Table v0.1.5

Share this post


Link to post
Share on other sites
18 minutes ago, k07n said:

@adrian I have v15

Ok, I can reproduce here. I'll commit the fix shortly, but in the meantime, replace line 107 with this:

        $rows = $p->$actualFieldName(ltrim($this->wire('input')->get->filter, ',')."limit=".$totalRows);

 

  • Like 1

Share this post


Link to post
Share on other sites

@adrian one more thing:

nothing happens on this:

   $modules->get('ProcessTableCsvExport'); // load module
   $pages->get('/clients/')->client_emails->exportCsv(',', '"', 'csv', '|', true);

 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By BitPoet
      Here's a little text formatter module that redirects all external links found in a textarea (e.g. for hit counting) to a given URL.
      TextformatterExternalRedirect
      GitHub repo
      As TextFormatters do, this module modifies the contents of the textarea (HTML) at rendering time. It prepends the given URL to the link address for external links (http and https), optionally makes sure that the link is opened in a new window and allows you to add css classes to such links.
      All options can be configured in the module's configuration settings.
      Usage:
      Download the zip archive and extract it to site/modules, the rename the top folder to TextformatterExternalRedirect Go to the backend and select "Modules" -> "Refresh", then click "Install" for "External Link Redirect" Click "Settings" and configure the module
      Go into the field configuration for the field(s) you want this formatter to apply to and add it from the dropdown
      Done, now any external links in the configured fields will have the configured settings applied Feel free to leave feedback and post and questions you have here.
       
    • By teppo
      Hey folks!
      Took a couple of late nights, but managed to turn this old gist of mine into a proper module. The name is SearchEngine, and currently it provides support for indexing page contents (into a hidden textarea field created automatically), and also includes a helper feature ("Finder") for querying said contents. No fancy features like stemming here yet, but something along those lines might be added later if it seems useful (and if I find a decent implementation to integrate).
      Though the API and selector engine make it really easy to create site search pages, I pretty much always end up duplicating the same features from site to site. Also – since it takes a bit of extra time – it's tempting to skip over some accessibility related things, and leave features like text highlighting out. Overall I think it makes sense to bundle all that into a module, which can then be reused over and over again 🙂
      Note: markup generation is not yet built into the module, which is why the examples below use PageArray::render() method to produce a simple list of results. This will be added later on, as a part of the same module or a separate Markup module. There's also no fancy JS API or anything like that (yet).
      This is an early release, so be kind – I got the find feature working last night (or perhaps this morning), and some final tweaks and updates were made just an hour ago 😅
      GitHub repository: https://github.com/teppokoivula/SearchEngine Modules directory: https://modules.processwire.com/modules/search-engine/ Demo: https://wireframe-framework.com/search/ Usage
      Install SearchEngine module. Note: the module will automatically create an index field install time, so be sure to define a custom field (via site config) before installation if you don't want it to be called "search_index". You can change the field name later as well, but you'll have to update the "index_field" option in site config or module settings (in Admin) after renaming it.
      Add the site search index field to templates you want to make searchable. Use selectors to query values in site search index. Note: you can use any operator for your selectors, you will likely find the '=' and '%=' operators most useful here. You can read more about selector operators from ProcessWire's documentation.
      Options
      By default the module will create a search index field called 'search_index' and store values from Page fields title, headline, summary, and body to said index field when a page is saved. You can modify this behaviour (field name and/or indexed page fields) either via the Module config screen in the PocessWire Admin, or by defining $config->SearchEngine array in your site config file or other applicable location:
      $config->SearchEngine = [ 'index_field' => 'search_index', 'indexed_fields' => [ 'title', 'headline', 'summary', 'body', ], 'prefixes' => [ 'link' => 'link:', ], 'find_args' => [ 'limit' => 25, 'sort' => 'sort', 'operator' => '%=', 'query_param' => null, 'selector_extra' => '', ], ]; You can access the search index field just like any other ProcessWire field with selectors:
      if ($q = $sanitizer->selectorValue($input->get->q)) { $results = $pages->find('search_index%=' . $query_string . ', limit=25'); echo $results->render(); echo $results->renderPager(); } Alternatively you can delegate the find operation to the SearchEngine module as well:
      $query = $modules->get('SearchEngine')->find($input->get->q); echo $query->resultsString; // alias for $query->results->render() echo $query->pager; // alias for $query->results->renderPager() Requirements
      ProcessWire >= 3.0.112 PHP >= 7.1.0 Note: later versions of the module may require Composer, or alternatively some additional features may require installing via Composer. This is still under consideration – so far there's nothing here that would really depend on it, but advanced features like stemming most likely would.
      Installing
      It's the usual thing: download or clone the SearchEngine directory into your /site/modules/ directory and install via Admin. Alternatively you can install SearchEngine with Composer by executing composer require teppokoivula/search-engine in your site directory.
    • By teppo
      MarkupMenu is a markup module for generating menu trees. When provided a root page as a starting point, it generates a navigation tree (by default as a HTML "<ul>" element wrapped by a "<nav>" element) from that point onwards. If you've also provided it with current (active) page, the menu will be rendered accordingly, with current item highlighted and items rendered up to that item and its children (unless you disable the "collapsed" option, in which case the full page tree will be rendered instead).
      Modules directory: https://modules.processwire.com/modules/markup-menu/ GitHub repository: https://github.com/teppokoivula/MarkupMenu Usage
      As a markup module, MarkupMenu is intended for front-end use, but you can of course use it in a module as well. Typically you'll only need the render() method, which takes an array of options as its only argument:
      echo $modules->get('MarkupMenu')->render([ 'root_page' => $pages->get(1), 'current_page' => $page, ]); Note: if you omit root_page, site root page is used by default. If you omit current_page, the menu will be rendered, but current (active) page won't be highlighted etc.
      A slightly more complex example, based on what I'm using on one of my own sites to render a (single-level) top menu:
      echo $modules->get('MarkupMenu')->render([ 'current_page' => $page, 'templates' => [ 'nav' => '<nav class="{classes} menu--{menu_class_modifier}" aria-label="{aria_label}">%s</nav>', 'item_current' => '<a class="menu__item menu__item--current" href="{item.url}" tabindex="0" aria-label="Current page: {item.title}">{item.title}</a>', ], 'placeholders' => [ 'menu_class_modifier' => 'top', 'aria_label' => 'Main navigation', ], 'include' => [ 'root_page' => true, ], 'exclude' => [ 'level_greater_than' => 1, ], ]); Note: some things you see above may not be entirely sensible, such as the use of {menu_class_modifier} and {aria_label} placeholders. On the actual site the "nav" template is defined in site config, so I can define just these parts on a case-by-case basis while actual nav markup is maintained in one place.
      Please check out the README file for available render options. I'd very much prefer not to keep this list up to date in multiple places. Basically there are settings for defining "templates" for different parts of the menu (list, item, etc.), include array for defining rules for including in the menu and exclude array for the opposite effect, classes and placeholders arrays for overriding default classes and injecting custom placeholders, etc. 🙂
      MarkupMenu vs. MarkupSimpleNavigation
      TL;DR: this is another take on the same concept. There are many similarities, but also some differences – especially when it comes to the supported options and syntax. If you're currently using MarkupSimpleNavigation then there's probably no reason to switch over.
      I'd be surprised if someone didn't draw lines between this module and Soma's awesome MarkupSimpleNavigation. Simply put I've been using MSN (...) for years, and it's been great – but there are some issues with it, particularly in the markup generation area, and it also does some things in a way that doesn't quite work for me – the xtemplates thing being one of these. In some ways less about features, and more about style, I guess 🙂
      Anyhow, in MarkupMenu I've tried to correct those little hiccups, modernise the default markup, and allow for more flexibility with placeholder variables and additional / different options. MarkupMenu was built for ProcessWire 3.0.112+ and with PHP 7.1+ in mind, it's installable with Composer, and I have a few additional ideas (such as conditional placeholders) still on my todo list.
      One more small(ish) difference is that MarkupMenu supports overriding default options via $config->MarkupMenu. I find myself redefining the default markup for every site, which until now meant that each site had a wrapper function for MarkupSimpleNavigation (to avoid code / config repetition), and this way I've been able to leave that out 🙂
      Requirements
      ProcessWire >= 3.0.112 PHP >= 7.1.0 If you're working on an earlier version of ProcessWire or PHP, use MarkupSimpleNavigation instead.
    • By teppo
      Hey folks!
      I'm happy to finally introduce a project I've been working on for quite a while now: it's called Wireframe, and it is an output framework for ProcessWire.
      Note that I'm posting this in the module development area, maily because this project is still in rather early stage. I've built a couple of sites with it myself, and parts of the codebase have been powering some pretty big and complex sites for many years now, but this should still be considered a soft launch 🙂
      --
      Long story short, Wireframe is a module that provides the "backbone" for building sites (and apps) with ProcessWire using an MVC (or perhaps MVVM – one of those three or four letter abbreviations anyway) inspired methodology. You could say that it's an output strategy, but I prefer the term "output framework" since in my mind the word "strategy" means something less tangible. A way of doing things, rather than a tool that actually does things.
      Wireframe (the module) provides a basic implementation for some familiar MVC concepts, such as Controllers and a View layer – the latter of which consists of layouts, partials, and template-specific views. There's no "model" layer, since in this context ProcessWire is the model. As a module Wireframe is actually quite simple – not even nearly the biggest one I've built – but there's still quite a bit of stuff to "get", so I've put together a demo & documentation site for it at https://wireframe-framework.com/.
      In addition to the core module, I'm also working on a couple of site profiles based on it. My current idea is actually to keep the module very light-weight, and implement most of the "opinionated" stuff in site profiles and/or companion modules. For an example MarkupMenu (which I released a while ago) was developed as one of those "companion modules" when I needed a menu module to use on the site profiles.
      Currently there are two public site profiles based on Wireframe:
      site-wireframe-docs is the demo&docs site mentioned above, just with placeholder content replaced with placeholder content. It's not a particularly complex site, but I believe it's still a pretty nice way to dig into the Wireframe module. site-wireframe-boilerplate is a boilerplate (or starter) site profile based on the docs site. This is still very much a work in progress, but essentially I'm trying to build a flexible yet full-featured starter profile you can just grab and start building upon. There will be a proper build process for resources, it will include most of the basic features one tends to need from site to site, etc. --
      Requirements and getting started:
      Wireframe can be installed just like any ProcessWire module. Just clone or download it to your site/modules/ directory and install. It doesn't, though, do a whole lot of stuff on itself – please check out the documentation site for a step-by-step guide on setting up the directory structure, adding the "bootstrap file", etc. You may find it easier to install one of the site profiles mentioned above, but note that this process involves the use of Composer. In the case of the site profiles you can install ProcessWire as usual and download or clone the site profile directory into your setup, but after that you should run "composer install" to get all the dependencies – including the Wireframe module – in place. Hard requirements for Wireframe are ProcessWire 3.0.112 and PHP 7.1+. The codebase is authored with current PHP versions in mind, and while running it on 7.0 may be possible, anything below that definitely won't work. A feature I added just today to the Wireframe module is that in case ProcessWire has write access to your site/templates/ directory, you can use the module settings screen to create the expected directories automatically. Currently that's all, and the module won't – for an example – create Controllers or layouts for you, so you should check out the site profiles for examples on these. (I'm probably going to include some additional helper features in the near future.)
      --
      This project is loosely based on an earlier project called pw-mvc, i.e. the main concepts (such as Controllers and the View layer) are very similar. That being said, Wireframe is a major upgrade in terms of both functionality and architecture: namespaces and autoloader support are now baked in, the codebase requires PHP 7, Controllers are classes extending \Wireframe\Controller (instead of regular "flat" PHP files), implementation based on a module instead of a collection of drop-in files, etc.
      While Wireframe is indeed still in a relatively early stage (0.3.0 was launched today, in case version numbers matter) for the most part I'm happy with the way it works, and likely won't change it too drastically anytime soon – so feel free to give it a try, and if you do, please let me know how it went. I will continue building upon this project, and I am also constantly working on various side projects, such as the site profiles and a few unannounced helper modules.
      I should probably add that while Wireframe is not hard to use, it is more geared towards those interested in "software development" type methodology. With future updates to the module, the site profiles, and the docs I hope to lower the learning curve, but certain level of "developer focus" will remain. Although of course the optimal outcome would be if I could use this project to lure more folks towards that end of the spectrum... 🙂
      --
      Please let me know what you think – and thanks in advance!
    • By Robin S
      After forgetting the class name of the wonderful AdminPageFieldEditLinks module for what feels like the 100th time I decided I needed to give my failing memory a helping hand...
      Autocomplete Module Class Name
      Provides class name autocomplete suggestions for the "Add Module From Directory" and "Add Module From URL" fields at Modules > New.
      Requires ProcessWire >= v3.0.16.
      Screencast

      Installation
      Install the Autocomplete Module Class Name module.
      Configuration
      Add Module From Directory
      Choose the type of autocomplete suggestions list: "Module class names from directory" or "Custom list of module class names". The latter could be useful if you regularly install some modules and would prefer a shorter list of autocomplete suggestions. The list of class names in the modules directory is generated when the Autocomplete Module Class Name module is installed. It doesn't update automatically (because the retrieval of the class names is quite slow), but you can use the button underneath when you want to retrieve an updated list of class names from the directory. Add Module From URL
      If you want to see autocomplete suggestions for the "Add Module From URL" field then enter them in the following format:
      [autocomplete suggestion] > [module ZIP url]
      Example: RepeaterImages > https://github.com/Toutouwai/RepeaterImages/archive/master.zip Awesomplete options
      The "fuzzy search" option uses custom filter and item functions for Awesomplete so that the characters you type just have to exist in the autocomplete suggestion item and occur after preceding matches but do not need to be contiguous. Uncheck this option if you prefer the standard Awesomplete matching. Custom settings for Awesomplete can be entered in the "Awesomplete options" field if needed. See the Awesomplete documentation for more information.  
      https://github.com/Toutouwai/AutocompleteModuleClassName
      https://modules.processwire.com/modules/autocomplete-module-class-name/
×
×
  • Create New...