Profields Table CSV Importer / Exporter

[adrian]

Hi @awebcreature - I have just tested with the contents of your CSV but am not seeing the problem here. Based on the line of the error, it looks like you have "Convert comma decimals to dots" checked. But none of your entries have any commas or dot decimals. Did you change the "Columns separated with" to semicolon to match your data?

Can you perhaps send me a screenshot of your settings for the import, along with the Details tab of the table field so I can see how the table fields have been defined. Also, a copy of the CSV might be good, just to make sure I am dealing with exactly the same data.

Also, can you dump $value before that error so I can see what is in the array?

[awebcreature]

Hello , Adrian! 

Thanks for your fast response! 

That are my details: 

https://yadi.sk/i/UAbWy1rLsXHwU   -  ScreenShot of fields in Table Field

https://yadi.sk/i/yRGhFd8wsXJ4A    - ScreenShot of importing CSV file

https://yadi.sk/d/MvepdRvosXJ77  - The CSV file. 

I'm sorry, but how to dump $value? 

Thanks again! 

[adrian]

Thanks for all those details. It's really late and I need to get to sleep so I will look at this in the morning, but in the meantime, I see that you have Tracy installed, so you can just insert this line:

bd($value);

just after this line: https://github.com/adrianbj/TableCsvImportExport/blob/master/TableCsvImportExport.module#L359

This will populate the dumps panel in Tracy - please post the content here.

There must be something strange going on, because you don't have the convert decimals option checked, but it is being called anyway. Could you also add:

bd($convertDecimals);

just after this line: https://github.com/adrianbj/TableCsvImportExport/blob/master/TableCsvImportExport.module#L274

I just noticed I have an incorrect "==" which should be "===" here: https://github.com/adrianbj/TableCsvImportExport/blob/master/TableCsvImportExport.module#L339 - this shouldn't cause the problem - more of a reminder for me to change this tomorrow.

[awebcreature]

Hello Adrian! 

That is the result after putting dumps in the source code:

https://yadi.sk/i/hq-vi1_SsYmyK

[adrian]

Is there any chance this site is online and I could possibly get access to take a look?

Are all 63 errors shown in that screenshot coming from this module? Could you maybe at least show me the contents of that panel?

[awebcreature]

These errors are : 

42×PHP Notice: Array to string conversion in .../TableCsvImportExport/TableCsvImportExport.module:340

21×PHP Notice: Indirect modification of overloaded property TableRow::$Array has no effect in .../TableCsvImportExport/TableCsvImportExport.module:340

error on each element  from .csv file

Tomorrow this PW will be online.

Thanks again!

[awebcreature]

... and i will eat my mouse!

On the my local machine i have PHP version 7. When I change it to 5.6.10 - no errors anymore. Read and import cvs file correctly into table. 

Thanks for your support Andrian! 

I hope that situation will be in plus to your future development of that useful module.

Thanks again! 

 

[adrian]

2 minutes ago, awebcreature said:

... and i will eat my mouse!

On the my local machine i have PHP version 7. When I change it to 5.6.10 - no errors anymore. Read and import cvs file correctly into table. 

Thanks for your support Andrian! 

I hope that situation will be in plus to your future development of that useful module.

Thanks again! 

 

Thanks for reporting that - I was struggling to figure out why you were getting those errors. I guess it's getting to the point that we all need to have a PHP7 testing environment set up so we can fix these sorts of things. 

I'll see what I can do about getting it 7 compatible fairly soon.

[LostKobrakai]

I'm not sure how your local setup does look like, but I really like this one: https://github.com/wilmoore/php-version Just switch around php versions in no time

[adrian]

Nice - I am a HomeBrew user, so that looks like it should make things very simple - cheers!

[ceberlin]

First of all: thank you very much for the great module.  This is so convenient to have!!!!

Just a small feedback: I managed to shoot my table down. Maybe to many entries (250) maybe and invalid file. No big deal, I have backups and the full database saved as CSV.

I got warnings from PW about not properly escaped things (don't remember exactly the wording). And, yes, I found some single and double quotes in table cells from the import file (actually an exported and then re-imported CSV file), After deleting those chars, the import worked.

If this is possible, maybe a warning about illegal chars or a proper escape during import would be very useful?

Also in my situation it would help if there was an "import & replace" feature. So I can do another import that is properly replacing a malfunctioning previous one. Right now the import was added. (Now my table has double entries - and maybe even double IDs because I cannot select and delete an entry any more, it re-appears after a save)

A replace would be a convenient way to clean this mess up quickly. 

[adrian]

Hi @ceberlin - glad you are finding it convenient. Sorry there was a problem with your upload though. It shouldn't be a number of entries issue - I have imported thousands of entries without a problem (although field Table itself is slow to render with that many until Ryan adds pagination - hopefully soon).

Any chance you could send me the file in it's original form so I can test and add some checks for illegal characters etc. Just to confirm - was the exported file exported using this module? Sounds like I might have an issue there.

BTW - what version of PHP and PW are you running?

There is already an overwrite option - you just need to enable it:

[ceberlin]

Hi @adrian -  thank you for your quick response.

1. Oh I see, I overlooked that feature, Brilliant!!!!!!!

(In the explanation, of you replace the word "overwrite" with "replace", it maybe gets clearer? Also i.m.h.o. it would be good to prepent the word "Caution: " maybe?)

 

2. My problem now with the import is that after the import I cannot save the page any longer (any changes/updates, regardless in what field, are ignored on save). No errors get recorded. Looks like something is timing out or confusing the database storage script.

This is the page I am working at: /tour-list/

The fields in use are a "single page" selector, a date field and some tiny text fields.

I send you PM with the data I am trying to load. And you can also have access to anything (it's a fan community website only), if you need to investigate further.

 

 

 

[ceberlin]

SOLVED!!!!

OK, here is, what I discovered, and a workaround:

• if you have errors on upload (database error, "not properly escaped"), look for single quotes ' in your file.
• If you cannot change and save the page any more (you click save and the page tries to save but nothing gets actually saved any more - and no php- or js error is given) try this:
  in PHP set "max_input_vars = 5000" as a workaround.

max_input_vars = 5000 is unusual (there is a reason why the limit is normally 1000), so there is a possible bug in the pro table field which I will report now accordingly.

 

 

[gingebaker]

On 17.6.2016 at 7:44 PM, awebcreature said:

... and i will eat my mouse!

On the my local machine i have PHP version 7. When I change it to 5.6.10 - no errors anymore. Read and import cvs file correctly into table. 

Thanks for your support Andrian! 

I hope that situation will be in plus to your future development of that useful module.

Thanks again! 

 

Hi 

I also stumbled across this problem. It looks like the new variable handling of php7 is responsible for that...

I found a solution wich works on php 5.6 and 7.

Just posted an github issue with explanation...
https://github.com/adrianbj/TableCsvImportExport/issues/1

greetings, tom

 

[adrian]

Thanks @gingebaker - I have applied your curly brace fixes. Please let me know if you have any further  troubles.

[adrian]

Hi everyone,

I have spent quite a bit of time today getting this module working with the new pagination functionality in the Table field.

I have tested it in a variety of import (append and overwrite) and export scenarios with standard as well as page fields (radios, checkbox, selects, etc) and it seems to be working well.

The trouble is that this new version will only with with the new version of the Table field. It will work even if pagination is not enabled, but it won't work with older Table field versions.

I would really appreciate it if you guys could test and let me know how it works for you. I'd like to get this pushed to Github fairly soon, but it's such a major change that I would like some additional testing first.

Thanks!

 

PS - this update also add support for line breaks in textarea fields - I added the same CSV parser that I recently added to my BCE module.

Edited September 27, 2016 by adrian
Remove attachment because it's now on Github

[adrian]

Given the underwhelming response I am going to assume that not many people are using this module, so I am just going to update the Github repo with the new version. I used it a LOT yesterday and it seems to be working great with the new Table pagination feature. 

I have removed 2.7 and below from the compatibility list for this module. It is now only 3.x (or 2.8). If you need the old version that support 2.7 and below, you can grab it from this commit: https://github.com/adrianbj/TableCsvImportExport/tree/b83df4d13fd18fcb6c4bc8ac467e6ae35cba9711

Hope some of you out there will find the new version useful.

[adrian]

Latest version supports:

1. Column selection and ordering for CSV export.
2. Exported rows also respect the Find filters that have been applied.

Should be very handy for clients to be able to extract relevant info from large table datasets.

[tpr]

Great!
(this is an overwhelming response :))

[k07n]

Hi!

I am getting an error on export:

Uncaught Error: Call to a member function getColumns() on null in \site\assets\cache\FileCompiler\site\modules\TableCsvImportExport\ProcessTableCsvExport.module:108

Stack trace:
#0 \site\assets\cache\FileCompiler\site\modules\TableCsvImportExport\ProcessTableCsvExport.module(62): ProcessTableCsvExport->exportCsv()
#1 \wire\core\Wire.php(374): ProcessTableCsvExport->___execute()
#2 \wire\core\WireHooks.php(698): ProcessWire\Wire->_callMethod('___execute', Array)
#3 \wire\core\Wire.php(402): ProcessWire\WireHooks->runHooks(Object(ProcessTableCsvExport), 'execute', Array)
#4 \wire\core\ProcessController.php(244): ProcessWire\Wire->__call('execute', Array)
#5 \wire\core\Wire.php(374): ProcessWire\ProcessController->___execute()
#6 \wire\core\WireHooks.php(698): ProcessW in <b>\site\assets\cache\FileCompiler\site\modules\TableCsvImportExport\ProcessTableCsvExport.module</b> on line <b>108</b><br />

testing on:

PW 3.0.52 @ php7.1
PW 3.0.48 @ php5.6.12

UPD: ProFields Table v0.1.5

[adrian]

@k07n - what version of Table are you using? It requires v15

If you need support for an older version, grab this commit: https://github.com/adrianbj/TableCsvImportExport/tree/f1d6704af7d8bbd2dc1dda62f76a70d55cc09a7f although I would recommend upgrading Table instead.

[k07n]

@adrian I have v15

[adrian]

18 minutes ago, k07n said:

@adrian I have v15

Ok, I can reproduce here. I'll commit the fix shortly, but in the meantime, replace line 107 with this:

        $rows = $p->$actualFieldName(ltrim($this->wire('input')->get->filter, ',')."limit=".$totalRows);

 

[k07n]

@adrian one more thing:

nothing happens on this:

   $modules->get('ProcessTableCsvExport'); // load module
   $pages->get('/clients/')->client_emails->exportCsv(',', '"', 'csv', '|', true);