kongondo

Module: Matrix Fieldtype & Inputfield

Recommended Posts

Thanks Adrian!

  1. Modal: CSS: Let me start with this one. I hate CSS...there I said it!  >:(  :-X  :P  :lol: . OK, I don't hate it that much...But this drove me nuts for hours! OK, so PHP has its quirkiness too but goodness, CSS! OK, maybe am not just good at it and debugging it is a pain. The modal is a jQuery UI thing and overriding its defaults caused me more pain than I want to go through again :-)...So, if anyone can send me a PR, please do. Am serious, LoL. It's either this or I go back to browser default alert box (FF's is nice btw, but Chrome, urggh!).
  2. New Rows: Maybe there's two ways around this. In an earlier post I was wondering whether we need to use pages for Row and Column headers/labels (e.g. the Nitrogen, 1987 in your screenshot). In this option, we would let the first column and first row be the row and column headers respectively. That would mean, instead of using those columns (in the db) to store references to page IDs, we would use them to store Row and Column Labels. Alternatively, we could let the user choose whether they want labels to be existing pages or if they want to import custom strings as Row/Column headers.  That would make for easier appending new rows since it wouldn't need the existence of a page. The second option here is to automatically create new row and column pages on import by reading the row/label titles (or checking if that page exists and add it to the matrix if it does). The problem here is if auto-creating new pages, what templates will they use and what page(s) will be their parents.
  3. Ignore first column/row: Makes sense, will look into it -  if you can please submit a separate GitHub feature request, thanks
  4. Checking rows means they will be exported: Good idea, will look into it - if you can please submit a separate GitHub feature request, thanks
  5. Shift+Click: - ditto, thanks
  6. Natural/Normal way of structuring CSV data: I think you misunderstood me/I wasn't clear. This is what I meant. Consider the following data, including the row and column headers as it would appear in Excel:

    A                           B      C     D     E     F    G

1 Peter 35

2 Peter 80Kg

3 Peter $50,000

4 Peter Married

5 Peter Volvo

Mary etc as above

As you can see, the first 5 rows are all about Peter. If you get data like that, normally you would change it so that all of Peter's data is on one row, with the columns each storing one property like so:

            A             B             C             D                           E              F        G

1          Age       Weight   Salary    Marital Status   Vehicle

2 Peter 35         80 kg     $50,000   Married                Volvo

3 Mary  etc....

So, whilst example 1 would have better worked for finding out what was 'null', it is not the normal way of organising data. Anyway, it's not important now, since you meant something totally different by 'append'.

  • Like 1

Share this post


Link to post
Share on other sites

I hate CSS too - not my strong point, that's for sure! Given that the PW core uses the brower's alert box, I wouldn't bother struggling with it too much. It might be nice if PW had it's own consistent alert (like the pw-modal that we have for modals), but I don't think you should put too much effort into overriding the browser in this case.

New Rows: Maybe there's two ways around this. In an earlier post I was wondering whether we need to use pages for Row and Column headers/labels (e.g. the Nitrogen, 1987 in your screenshot). In this option, we would let the first column and first row be the row and column headers respectively. That would mean, instead of using those columns (in the db) to store references to page IDs, we would use them to store Row and Column Labels. Alternatively, we could let the user choose whether they want labels to be existing pages or if they want to import custom strings as Row/Column headers.  That would make for easier appending new rows since it wouldn't need the existence of a page. The second option here is to automatically create new row and column pages on import by reading the row/label titles (or checking if that page exists and add it to the matrix if it does). The problem here is if auto-creating new pages, what templates will they use and what page(s) will be their parents.

For my purposes, I don't think I would want what you are suggesting because an incorrectly prepared CSV file could damage my site by adding additional years and indicators (nitrogen etc). The indicators are actual pages that have their own content, so I think it is important that site editors make a conscious decision first to add a new indicator which will then automatically appear in the Matrix field. This is the reason I am using Matrix and not Table - the automatic population and new indicators and years.

Natural/Normal way of structuring CSV data: I think you misunderstood me/I wasn't clear. This is what I meant. Consider the following data, including the row and column headers as it would appear in Excel:

Yes, I didn't see your initial point - sorry, but I also think that trying to deal with that is getting too complicated :)

Share this post


Link to post
Share on other sites

I was talking about appending new rows. In my current use case, each year there will be a new "year" row that will appear in the matrix. I would like users to potentially be able to import a CSV of the data for that row. Of course importing for one row may not be worth the effort, but in some cases I can have up to 20 columns, so it might just be worth it. In that case, I would want "import" to just append the new row to the blank year that is in the matrix table.

Reading this line again, if we are not CREATING new rows per se (i.e. not creating new row/column pages) but appending starting from the first AVAILABLE BLANK row, that is doable and is how I was thinking originally, before I managed to confuse myself trying to figure out what exactly you meant by append :-). In that case, the users will have to ensure that a blank row exists before attempting an 'append import'. If at least one blank row exists, that row will be populated and any other subsequent blank rows as well as long as there are still other rows of data in the imported CSV. As usual, any extraneous rows and columns will be discarded (nothing to import to). Hmm, need to see which is easier; using PHP to find the first blank row or using jQuery/JavaScript to read the matrix client-side and get the first blank row and send that with the submitted form :-). Happy days. As for your users creating new blank rows every year, am sure you can work that out :-).

Edited by kongondo
  • Like 1

Share this post


Link to post
Share on other sites

Ah yes - perhaps I should have said: "populate blank rows" vs "overwrite all existing rows", instead of append/overwrite.

Sorry for all the confusion, but I think we are on the same wavelength now :)

Yes - creating a new blank row for each year is easy - I have a "Years" parent page and they simply create a new year under that as required - well actually it is done using Batch Child Editor and the child pages are hidden to avoid confusion.

Thanks again!

Share this post


Link to post
Share on other sites

Just a quick thought on the "Note: Empty values will be saved" message - I am not sure that this is helpful to site editors, and may even be confusing!

Maybe it is something that doesn't need to be displayed, or maybe just for superusers?

Thanks!

Share this post


Link to post
Share on other sites

OK..What about an overwrite versus append message?

I think it is important that the user knows what is about to happen regarding appending vs overwriting for sure. Are you going to make the behaviour configurable in the import section settings?

Share this post


Link to post
Share on other sites

Append vs Overwrite is configurable at both field (we set a default import mode [superuser]) and page (import section settings [page editor]; here we override the default) level. If no default has been set by superuser, it defaults to append. In addition to the note above the matrix table, if in append mode, the last blank row is highlighted, just a little visual reminder about what's about to happen if they hit save. The append note also tells the user that if a last blank row happens to have other non-blank rows beneath it, the imported values will be saved starting from the last blank row and overwrite any rows below it. If they want to avoid this, users need to make sure that the last blank row is always the last matrix table row. Thought about a pop-up warning the user instead (that there are populated rows beneath the last blank row) but didn't pursue the idea, putting the onus on the editor instead. It would mean getting the pop-up if they hit save, even for other reasons (e.g saving other fields on the page). Alternatively, we could have a separate 'save matrix' button, but that is probably complicating things. 

Share this post


Link to post
Share on other sites

Append vs Overwrite is configurable at both field (we set a default import mode [superuser]) and page (import section settings [page editor]; here we override the default) level. If no default has been set by superuser, it defaults to append. In addition to the note above the matrix table, if in append mode, the last blank row is highlighted, just a little visual reminder about what's about to happen if they hit save. The append note also tells the user that if a last blank row happens to have other non-blank rows beneath it, the imported values will be saved starting from the last blank row and overwrite any rows below it. If they want to avoid this, users need to make sure that the last blank row is always the last matrix table row. Thought about a pop-up warning the user instead (that there are populated rows beneath the last blank row) but didn't pursue the idea, putting the onus on the editor instead. It would mean getting the pop-up if they hit save, even for other reasons (e.g saving other fields on the page). Alternatively, we could have a separate 'save matrix' button, but that is probably complicating things. 

Are these new settings? I am running the latest from the dev branch and don't see these settings - am I being dumb? It's been a long week!

post-985-0-86645100-1446630559_thumb.png

post-985-0-36726200-1446630626_thumb.png

  • Like 1

Share this post


Link to post
Share on other sites

Also, what about an API method to create data - it would be nice to make it easier to create an import routine from an existing database.

You mean something more elaborate than this (which has always been possible btw)?

$p = $pages->get(1091);
$p->of(false);
$matrix = new Matrix();
$matrix->row = 1060;
$matrix->column = 1073;
$matrix->value = "This is my value";
$p->results->add($matrix);
$p->save('results');
$p->of(true);

  • Like 2

Share this post


Link to post
Share on other sites

Yeah, I was talking about the ability to import a CSV via the API - perhaps like I did with the TableCSVImportExport:

http://modules.processwire.com/modules/table-csv-import-export/

$modules->get('TableCsvImportExport'); // load module // data, delimiter, enclosure, convert decimals, ignore first row, multiple values separator, append or overwrite 
$page->fields->tablefield->importCsv($csvData, ',', '"', false, true, ',', 'append');

Share this post


Link to post
Share on other sites

Update version 1.0.4 (Dev only for now)
 
Note: Haven't had time to update README
 
Changes

  1. Import Mode for file/copy-pasted CSV: Two options, 'append' and 'overwrite'. Default setting set in Field's Details tab. The setting can be overridden at page-edit level. Default setting is append (sorry guys, you will have to inform your clients of the changes. The prospect of data loss inherent in the overwrite mode outweighed the inconvenience of re-teaching your clients about the new default append mode).
    1. Append: In this mode, the module will look for the last available empty/blank row in the matrix. If it finds one, imported rows will be inserted from that point forward. Any subsequent rows (i.e. rows beneath that 'last blank row') will be overwritten. Any rows above the 'last blank row' will have their values preserved (untouched). When in append mode, the 'last blank row' will always be highlighted. The modules (jQuery) monitors any live changes to the table rows (typing and clear data) and the highlight will shift accordingly if the 'last blank row' changes. If there is no 'last blank row' and you attempt to save in this mode, you will get an PW error message and your older values will be preserved in the matrix. Used wisely, append can actually mimic an insert. Btw, for an empty matrix, append doesn't make much sense since data will only be inserted in the very last row of the matrix :-).
    2. Overwrite: What was the default behaviour; incoming values overwrite existing values.
  2. Ignore first row and/or column: Default set independently at Field setting (i.e. can ignore one but not the other if you wish). Default is ignore both first row and column. Both settings can be independently by-passed at page-edit level. In real world examples, first rows usually store column headers/labels and first columns usually store row headers/labels.
  3. Export checked rows: You can now export only the rows you want by checking them. If nothing is checked and you attempt to export you will get a JS error alert. The 'set range to export' setting became redundant and I removed it.
  4. Shift + Click enables the selection of a range of rows. The checkbox for toggling select all still works.
  5. Lots of code re-factoring

Thanks to @adrian for suggesting all of the above features ^-^.

Please test and let me know.

I think I have given this module enough love for now. MenuBuilder has refused to talk to me until the situation improves. So, for now, there's a feature freeze on this one and let's see if I can get MenuBuilder to smile again  :lol:  :P .

post-894-0-72496200-1446755440_thumb.png

Edited by kongondo
Added screenshot
  • Like 3

Share this post


Link to post
Share on other sites

Beautiful work!

I think I might have said this just recently to some of your other new features, but it is true - this has become one of my favourite field types. 

Now go spend some time on some of those commercial modules you have cooking :)

  • Like 1

Share this post


Link to post
Share on other sites

I have been hooked on it! And I've learnt some fun jQuery stuff along the way. Btw, didn't have time to test with non-superuser account + thoroughly check if 'append' caused any performance issues..Seemed there was a slight (milliseconds) delay when I added the feature. Maybe I just need some sleep  :P .

Share this post


Link to post
Share on other sites

Hey kongondo - small bug for you. When editing a page in a modal window - eg through Lister, the Export Data button is ending up at the bottom of the modal.

post-985-0-50390800-1446848568_thumb.png

The PW modal code grabs some buttons and forces them to be in the footer, but I am not sure why "export data" is ending up there, but "clear data" isn't - likely a class that has been applied to the export button - but I haven't investigated.

Thanks!

Share this post


Link to post
Share on other sites

Hey kongondo,

I have noticed an issue when using unpublished or hidden pages.

Here is my scenario. I am using this custom php code as the row selector:

return $pages->find("parent=/report-cards/{$page->parent->parent->name}/years/, sort=sort, include=all");

The "include=all" ensures that even though the latest year is unpublished (we are currently adding data and it's not ready to be live yet), we still see that year as a row in the matrix table. 

The problem is this line: https://github.com/kongondo/FieldtypeMatrix/blob/master/InputfieldMatrix.module#L439

//remove matrix-rows that are no longer needed i.e. 'unpublished', 'hidden', 'deleted' or 'trashed'
if(!$rp || $rp->is(Page::statusUnpublished) || $rp->isHidden() || $rp->isTrash()) continue;

Even though data is saved to the db, that line prevents it from being displayed in the matrix table - you enter and save, but it appears blank.

For my needs I have commented this line and it seems to be working fine. I am not sure of the best fix - perhaps this line is simply not necessary or maybe it needs to be an option?

The same issue occurs with columns as well.

Please let me know if I haven't explained properly or if you have any other questions.

Thanks!

Share this post


Link to post
Share on other sites

Hi adrian,

I think the original idea was to avoid displaying and saving data that was 'no longer there/invalid', aka, that page is not reachable. But maybe I should have limited this to trashed pages only. I like your idea about making this configurable. Definitely I'll still exclude trashed pages but will make displaying of 'unpublished' and 'hidden' rows and columns configurable (not sure as one choice or two options [i.e. the page statuses as choices])  but separately for rows and columns .The default (for backward compatibility) will be to exclude such pages. Hope it makes sense (typing quickly...)...

Edited by kongondo
typos, etc
  • Like 1

Share this post


Link to post
Share on other sites

Just announcing that I've tested and can confirm that Matrix is compatible with ProcessWire 2.8.x and ProcessWire 3.x

  • Like 4

Share this post


Link to post
Share on other sites
2 hours ago, kongondo said:

Just announcing that I've tested and can confirm that Matrix is compatible with ProcessWire 2.8.x and ProcessWire 3.x

Just to add to your testing, I can confirm that I have been using this awesome module with 3.x for probably 6 months now without any problems!

  • Like 4

Share this post


Link to post
Share on other sites

Hey @kongondo - when adding via the API, eg:

$matrix = new Matrix();
$matrix->row = 1060;
$matrix->column = 1073;
$matrix->value = "This is my value";
$p->results->add($matrix);
$p->save('results');

is there a recommended way to overwrite existing rows? At the moment doing this when the cells are already populated results in duplicated entries in the "field_matrixfield" db table. Obviously only one set is shown when editing the field on a page in the backend, but I am worried about consistency of what will be returned (backend and frontend) and of course just wasted db space and data integrity. I am thinking that the easiest option might be to just remove the entries manually with a raw SQL statement before adding the matrix row.

Any thoughts?

Thanks!

Share this post


Link to post
Share on other sites

Hi @adrian,

Good catch. Yes, raw SQL would be the way to go. I'm still mulling and testing most efficient way to go about this. 

 

  • Like 1

Share this post


Link to post
Share on other sites

Hey @kongondo - I just went to export for the first time and noticed that all I get is the first column entitled "Row Label". Is there something I am missing, or is this a bug?

I need this pretty urgently (actually immediately), so I am going to take a look and see what I can figure out, but in case you happy to be online and have an idea for a solution, or something I am missing, that would be great :)

Share this post


Link to post
Share on other sites

Ok, I figured it out and have a quick fix in place. It was a multi-language issue.

This is my quick fix - to get the default row and column titles, rather than returning a title object.

			//prepare export values
			foreach($rows as $row) {
				$rowTitle = $row->title->getLanguageValue("default");
				//only export selected rows
				if(!in_array($row->id, $rowsChecked)) continue;

				foreach($columns as $column) {
					$columnTitle = $column->title->getLanguageValue("default");
					//get each matrix value at given coordinates (WireArray)
					$v = $values->get("rowLabel=$rowTitle, columnLabel=$columnTitle");
					$value = $v ? $v->value : '';//force blank values export
					$exportMatrix[$row->id][$this->_('Row Label')] = $rowTitle;
					$exportMatrix[$row->id][$columnTitle] = $value;
				}
			}

Not sure if you want to implement just like this, or whether you think it's worth adding a more complete ML solution.

PS - what about changing "Row Label" to the title of the parent page of the first column. In my case I have a page branch of Years to control this column. It would be much nicer if this read "Years" or "Year". Would that work, or are there more things to consider?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By OLSA
      Hello for all,
      ConfigurationForm fieldtype module is one my experiment from 2016.
      Main target to build this module was to store multiple setup and configuration values in just 1 field and avoid to use 1 db table to store just single "number of items on page", or another db table to store "layout type" etc. Thanks to JSON formatted storage this module can help you to reduce number of PW native fields in project, save DB space, and reduce number of queries at front-end.
      Install and setup:
      Download (at the bottom ), unzip and install like any other PW module (site/modules/...). Create some filed using this type of field (ConfigurationForm Fieldtype) Go to field setup Input tab and drag some subfields to container area (demo). Set "Name" and other params for subfields Save and place field to templates ("Action tab") How to use it:
      In my case, I use it to store setup and configurations values, but also for contact details, small content blocks... (eg. "widgets").
      Basic usage example:
      ConfigForm fieldtype "setup" has subfields:
      "limit", type select, option values: 5, 10, 15, 20
      "sort", type select, option values: "-date", "date",  "-sort", "sort"
      // get page children (items) $limit = isset($page->setup->limit) ? $page->setup->limit : 10; $sort = isset($page->setup->sort) ? $page->setup->sort : '-sort'; $items = $page->children("limit=$limit, sort=$sort");  
      Screenshots:


       
      Notes:
      Provide option to search inside subfields Provide multilanguage inputs for text and textarea field types Provide option for different field layout per-template basis Do not place/use field type "Button" or "File input" because it won't works. Please read README file for more details and examples Module use JSON format to store values. Text and textarea field types are multilanguage compatible, but please note that main target for this module was to store setup values and small content blocks and save DB space. Search part inside JSON is still a relatively new in MySQL (>=5.77) and that's on you how and for what to use this module.
      Thanks:
      Initial point for this fieldtype was jQuery plugin FormBuiled and thanks to Kevin Chappel for this plugin.
      In field type "link" I use javascript part from @marcostoll module and thanks to him for that part.
      Download:
      FieldtypeConfigForm.zip
      Edit: 14. August 2018. please delete/uninstall previously downloaded zip
      Regards.
         
    • By bernhard
      @Sergio asked about the pdf creation process in the showcase thread about my 360° feedback/survey tool and so I went ahead and set my little pdf helper module to public.
      Description from PW Weekly:
       
      Modules Directory: https://modules.processwire.com/modules/rock-pdf/
      Download & Docs: https://gitlab.com/baumrock/RockPdf
       
      You can combine it easily with RockReplacer: 
      See also a little showcase of the RockPdf module in this thread:
       
    • By bernhard
      WHY?
      This module was built to fill the gap between simple $pages->find() operations and complex SQL queries.
      The problem with $pages->find() is that it loads all pages into memory and that can be a problem when querying multiple thousands of pages. Even $pages->findMany() loads all pages into memory and therefore is a lot slower than regular SQL.
      The problem with SQL on the other hand is, that the queries are quite complex to build. All fields are separate tables, some repeatable fields use multiple rows for their content that belong to only one single page, you always need to check for the page status (which is not necessary on regular find() operations and therefore nobody is used to that).
      In short: It is far too much work to efficiently and easily get an array of data based on PW pages and fields and I need that a lot for my RockGrid module to build all kinds of tabular data.

      Basic Usage

       
      Docs & Download
      https://modules.processwire.com/modules/rock-finder/
      https://gitlab.com/baumrock/RockFinder/tree/master
       
      Changelog
      180810, v1.0.5, basic support for options fields 180528, v1.0.4, add custom select statement option 180516, change sql query method, bump version to 1.0.0 180515, multilang bugfix 180513, beta release <180513, preview/discussion took place here: https://processwire.com/talk/topic/18983-rocksqlfinder-highly-efficient-and-flexible-sql-finder-module/
    • By Thomas Diroll
      Hi guys I'm relatively new to PW and just finished developing a page for a client. I was able to include all necessary functionality using the core fieldtypes but now I it seems that I need to extend them with a custom one. What I need is a simple button, that copies the absolute url (frontend not PW-backend) of the page which is currently edited to the clipboard. As this feature is only needed inside a specific template, I tend to use a custom fieldtype which provides this feature. I've been looking inside the core modules code (eg. FieldtypeCheckbox.module) but I don't really get the structure of it and how its rendered to the admin page. I also didn't find a lot of tutorials covering custom fieldtypes.
      Maybe some of you could give me some tips on how to write a basic custom fieldtype that renders a button which copies the value of
      page->httpUrl() to the clipboard using JS. Thanks!
    • By bernhard
      Some of you might have followed the development of this module here: https://processwire.com/talk/topic/15524-previewdiscussion-rockdatatables/ . It is the successor of "RockDataTables" and requires RockFinder to get the data for the grid easily and efficiently. It uses the open source part of agGrid for grid rendering.
       
      WHY?
      ProcessWire is awesome for creating all kinds of custom backend applications, but where it is not so awesome in my opinion is when it comes to listing this data. Of course we have the built in page lister and we have ListerPro, but none of that solutions is capable of properly displaying large amounts of data, for example lists of revenues, aggregations, quick and easy sorts by the user, instant filter and those kind of features. RockGrid to the rescue 😉 
       
      Features/Highlights:
      100k+ rows Instant (client side) filter, search, sort (different sort based on data type, eg "lower/greater than" for numbers, "contains" for strings) extendable via plugins (available plugins at the moment: fullscreen, csv export, reload, batch-processing of data, column sum/statistics, row selection) all the agGrid features (cell renderers, cell styling, pagination, column grouping etc) vanilla javascript, backend and frontend support (though not all plugins are working on the frontend yet and I don't plan to support it as long as I don't need it myself)  
      Limitations:
      While there is an option to retrieve data via AJAX the actual processing of the grid (displaying, filtering, sorting) is done on the client side, meaning that you can get into troubles when handling really large datasets of several thousands of rows. agGrid should be one of the most performant grid options in the world (see the official example page with a 100k row example) and does a lot to prevent problems (such as virtual row rendering), but you should always have this limitation in mind as this is a major difference to the available lister options that do not have this limitation.
      Currently it only supports AdminThemeUikit and I don't plan to support any other admin theme.
       
      Download: https://gitlab.com/baumrock/RockGrid
      Installation: https://gitlab.com/baumrock/RockGrid/wikis/Installation
      Quikckstart: https://gitlab.com/baumrock/RockGrid/wikis/quickstart
      Further instructions: https://gitlab.com/baumrock/RockGrid/wikis/quickstart#further-instructions
       
      Module status: alpha, License: MIT
      Note that every installation and uninstallation sends an anonymous google analytics event to my google analytics account. If you don't want that feel free to remove the appropriate lines of code before installation/uninstallation.
       
      Contribute:
      You can contribute to the development of this and other modules or just say thank you by
      testing, reporting issues and making PRs at gitlab liking this post buying me a drink: paypal.me/baumrock/5 liking my facebook page: facebook.com/baumrock hiring me for pw work: baumrock.com  
      Support: Please note that this module might not be as easy and plug&play as many other modules. It needs a good understanding of agGrid (and JavaScript in general) and it likely needs some looks into the code to get all the options. Please understand that I can not provide free support for every request here in the forum. I try to answer all questions that might also help others or that might improve the module but for individual requests I offer paid support for 60€ per hour.
       
      Changelog
      180730 support subdir installations 180711 bugfix (naming issue) 180630 alpha realease  
      Use Cases / Examples:
      Colored grid cells, Icons, Links etc. The Grid also has a "batcher" feature built in that helps communicating with the server via AJAX and managing resource intensive tasks in batches:

      Filters, PW panel links and instant reload on panel close:

      You can combine the grid with a chart library like I did with the (outdated) RockDataTables module: