Preview/Discussion: RockDataTables

[bernhard]

DEPRECATED - this module will not see any updates. I'm short before releasing RockGrid as commercial module. If you are interested contact me via PM

this is a preview of a module that i'm working on for quite a long time. I needed it for an intranet office management application that i'm still working on. It makes it very easy to create very customized Datatables using the awesome jquery datatables plugin (www.datatables.net)

Download - sorry, removed as it changes too frequently; closed alpha - contact me if you think you can contribute

 

 

Remarks:

The module is intended to be used on the backend. Loading styles is at the moment only done via adding files to the $config->styles/scripts arrays. Also the communicaton to javascript is done via the $config->js() method that is built into the admin and would have to be implemented manually on frontend use. But it would not be difficult of course

Installation:

Nothing special here, just download + install

 

edit: removed some parts, because i made a complete rewrite of the module (and may not have been the last one)!

[...] removed

Why i created this module:

• of course i know @Soma s module but i needed a lot more features and the newer datatables version. also i like to define all the columns as objects and have everything on one place.
• lister & markupadmindatatable: nice for basic tables but lacks of features to modify the appearance of the cell values (like rendering icons, background colors and so on)
• datatables provides a great frontend API for filtering, showing/hiding columns, getting data, modifying it...
• it also plays well together with frontend charts like google chart api in this case:
  

 

todo / roadmap:

• reload only one row
• add filters to all columns (in future also dropdowns, smaller than, bigger than, Regex, ...)
• make it possible to add table on frontend pages
• make buttons look like pw buttons
• make it possible to set settings globally and only for one table
• provide easy way of colorbars (percentage, red/green), maybe at different positions (left, top, right, bottom)
• provide easy way of adding action items (edit, show, link etc - visible or onhover)
• make own layout for tables (topleft, topright, bottom etc to make it easy to create extensions and show messages)
• privide way of simple javascript plugins (like I already have for row sums etc)
• provide easy way of handling actions for all selected items (delete selected, update selected...)
• provide easy way of reloading data (--> easy when using ajax source)
• easy way of showing/hiding columns
• excel/csv/clipboard export
• GUI for table setup
• processmodule to show different tables (lister replacement)

 

[bernhard]

need to setup tables via custom sql queries?

PHP

// init datatables module
    $dt2 = $modules->get('RockDataTables2');

// setup columns
    $sql = 'SELECT * FROM invoicetable';
    $dt2->setupColsBySql($sql);

// setup table
    $dt2->id = 'dt_finanzbuch';
    $dt2->js('/site/modules/ProcessRockFinance/dt_finanzbuch.js');
    
    $f = $modules->get('InputfieldMarkup');
    $f->label = 'Tabelle';
    $f->value = $dt2->render();
    $form->add($f);

// ##################################
// ajax request -> return data
// non-ajax -> render form + table
// ##################################
if($config->ajax) {
    echo $dt2->getJSON($dt2->getDataBySql($sql));
    die();
}

JS

$(document).ready(function() {
    // setup variables
        var opt = ProcessWire.config.dt_finanzbuch; // options from backend
        var colDefs = []; // column definitions

    // custom column definitions here

    // load default column definitions
        colDefs = colDefs.concat(dtGetDefaultDefs(opt));

    // initialise table
        $('#dt_finanzbuch').DataTable({
            ajax: { url: './book/', type: 'post' },
            columnDefs: colDefs,
            pageLength: 10,
            footerCallback: columnSums,
        });
});

Result

As you can see the sum function is not ready. it shows euros for my usecase...

[kathep]

Very nice! Do you think it is feasible to use this - or a future version - on the front end?

[bernhard]

hi kathep,

sure it is also usable in the frontend. but it is no click-click solution. the module just provides some helpers to define the columns like i showed in the example. to make the code more clear to read, write and maintain. you would just have to include the right scripts (i'm currently loading all the plugins via cdn) and you would be fine. maybe you would also have to do some styling. but all of that is - at least for now - not intended to be handled by my module.

hope that answer helps you

[Macrura]

plus DataTables are so easy to setup on the front end, you really don't need a module... just make your table, and then load the datatables scripts and init; the instructions are all there on the datatables site, and it is really well documented..

[bernhard]

current version has properly formatted currency fields (atm only euro) and supports colorbars and column sums (also from selected rows)

had to remove this video, sorry

[bernhard]

did a total rewrite of the module and will release it the next weeks when i'm done

 

[szabesz]

Hi @bernhard Thanks for putting all the work into sharing this module. What came to my mind was that I do not really like providing code in admin input fields, except when it is a few lines of code. So wouldn't it be possible (optionally maybe) to provide paths to files instead?

[bernhard]

Already done

The fields are there for really simple tables and the files for more complex ones. The module will include all files related to a field automatically (PHP, js and CSS).

[szabesz]

Oh I see, banging on open doors again  Thanks for the info!

[bernhard]

making progress on this but there's still a lot to do until it is releasable as a module (proper documentation mostly)...

sneak peak what's easily doable: a todo-app

definition is as easy as that:

	/**
	 * show table of todos
	 */
	public function executeTodos() {
		
    // create form
      $form = modules('InputfieldForm');
      $form->action = './';

      // add new project
        $b = modules('InputfieldButton');
        $b->attr("id+name", "addTodo");
        $b->addClass("ui-priority-primary pw-panel pw-panel-reload");
        $b->value = __("Neues Todo");
        $b->attr('data-href', pages(2)->url . 'page/add/?parent_id=' . pages('template=todos'));
        $b->icon = "plus";

        $f = modules('InputfieldMarkup');
        $f->value = $b->render();
        $form->add($f);

      // table
        $t = modules('InputfieldRockDatatables');
        $t->attr('id+name', 'manageTodos');

        $t->rows = pages('template=todo'); // define the table's source rows with one selector
        $t->ajax = 1; // set table mode to ajax loading making it possible to reload data via a simple $table.ajax.reload();

      [...]

        // remaining
        $col = new dtCol;
        $col->name = "remaining";
        $col->title = 'Tage';
        $col->className = 'minwidth';
        $col->data = function($page) {
          if(!$page->deadline) return '';

          $now = new \DateTime('now');
          $then = new \DateTime();
          $then->setTimestamp($page->deadline);
          $days = $now->diff($then)->format('%R%a');

          $color = config()->colors->lightred;
          if($days > 0) $color = config()->colors->lightorange;
          if($days > 7) $color = config()->colors->lightgreen;

          $obj = new \stdClass();
          $obj->display = $days;
          $obj->colorBars = [ [1, $color] ];

          return $obj;
        };
        $t->cols->add($col);

      [...]

      // add field to form

        $f = modules('InputfieldMarkup');
        $f->value = $t->render();
        $form->add($f);
    
    $out = $form->render();
    return $out;
	}

 

[bernhard]

just to show that this module can render any kind of field (seems that this was not clear enough: https://processwire.com/talk/topic/16929-choose-custom-repeater-template/?do=findComment&comment=148803 )

a simple column showing images:

// pic
$col = new dtCol;
$col->name = "pic";
$col->title = 'pic';
$col->className = 'minwidth';
$col->data = function($page) {
  if(!$page->pics->count()) return '';
  return $page->pics->each(function($img) {
    return "<img src='{$img->maxSize(200,200)->url}' alt=''><br>";
  });
};
$t->cols->add($col);

so you can do whatever you want and whatever the API + PHP can do...

the module even supports a plugin-system where we can easily build cell-renderers that handle common tasks and make rendering even easier.

this for example is the js file of the column filter that shows a filter input on top of each column:

/**
 * this plugin creates filter inputs for every column
 * 
 * #### usage ####
 * just set your table's settings: { colfilter: true }
 * 
 * #### issues ####
 * the column filter breaks the table's column alignment on window resize
 * 
 * #### roadmap ####
 * make it possible to have different kind of inputs like range sliders and dropdowns
 * 
 */

/**
 * make sure that the tables searching setting is set to true
 */
$(document).on('beforeInit.rdt', '.RockDatatable', function(e, dt) {
  if(!dt.settings.colfilter) return;

  $table = $(dt.el);
  if(dt.settings.searching == false) console.log('overwriting setting "searching" to true for table ' + $table.attr('id'));
  dt.settings.searching = true;
});

/**
 * add colfilter row to table header
 */
$(document).on('afterInit.rdt', '.RockDatatable', function(e, dt) {
  if(!dt.settings.colfilter) return;

  var $table = $(e.target); // table dom element
  var table = $table.DataTable();
  var $thead = table.table().header();

  // prepare the new header row
  var $tr = $("<tr role='row' class='colfilter' />");
  $.each(dt.columnNames, function() {
    $tr.append($('<th><input type="text" class="colfilter"></th>'));
  });

  // add it to the table header
  $tr.prependTo($thead);
});

/**
 * handle filterinputs
 */
colfiltertimer = 0;
$(document).on('keypress keyup change', 'input.colfilter', function(e) {
  var $input = $(this);
  var $tr = $input.closest('tr');
  var $th = $input.closest('th');
  // datatables splits the table in 2 divs when scrollX is used! thats why we need to find the table via the wrapper
  var table = $input.closest('.dataTables_wrapper').find('table.RockDatatable').DataTable();
  var regex = false;

  // disable enter form submit
  var keyCode = e.keyCode || e.which;
  if (keyCode === 13) { 
    e.preventDefault();
    return false;
  }

  // delay search while typing
  clearTimeout(colfiltertimer);
  colfiltertimer = setTimeout(function() {
    table.columns($th.index()+':visIdx').search($input.val(), regex, false).draw();
  }, 500);
});

this may look complex in the beginning but once we have such a plugin we only have to set one option of the table and the filter inputs appear

 

[bernhard]

datatable connected to the pagefield

updating the pageautocomplete field seemed a little tricky in the beginning but the final approach was quite easy:

// handle buttonclicks
$('#addcompetences').click(function(e) {
  e.preventDefault();
  var table = $table.DataTable(); // datatable instance
  var $field = $('#wrap_Inputfield_competences'); // asm select field
  var $template = $field.find('li.itemTemplate'); // selected page template
  var $ol = $template.closest('ol');
  
  // get selected competences from table
  var selectedtable = $.map(
    table.rows({selected:true}).data(),
    function(val, i) { return [[ val.id.sort, val.title + ' (' + val.cluster.filter + ')' ]]; }
  );

  // update ASM
  // clone template and populate values
  $.each(selectedtable, function(i,val) {
    // check double
    if($ol.find('.itemValue:contains("' + val[0] + '")').length) return;

    // add clone
    var $new = $template.clone().appendTo($template.parent());
    $new.find('.itemValue').text(val[0]);
    $new.find('.itemLabel').text(val[1]);
    $new.removeClass('itemTemplate');
  });
  
  // add state changed class
  $field.addClass('InputfieldStateChanged');

  // update original input
  InputfieldPageAutocomplete.rebuildInput($ol);
  
  return false;
});

 

[Macrura]

very nice - a good alternative to other page select options out there, and intuitive for the user;

[Macrura]

looks like i could use this module pretty soon on a project - LMK if you need someone to test it...

[bernhard]

PM

[bernhard]

another example how versatile and extendable this module is: you can write your own plugins to show custom column statistics. here i'm showing max, average and minimum values of some collected feedbacks. X means no answer and is excluded from the stats:

plugincode is quite simple once you know the basics of datatables api:

$(document).on('update.rdt', 'div.colstats, div.colstatsselected', function(e, table, col) {
  var $div = $(e.target);
  var $table = $(table);
  var table = $table.DataTable();

  // get table settings
  var id = $table.attr('id').replace('RockDatatable_','');
  var dt = ProcessWire.config.RockDatatables[id];
  var colname = dt.columnNames[col.index()];

  var selector = {search: 'applied'};
  if($div.hasClass('colstatsselected')) selector.selected = true;

  // get data
  var data = table
    .rows(selector)
    .data()
    .filter(function(row) {
      // return true if it is a number
      // don't count empty cells or "noanswer" cells that have an X
      return row[colname]*1;
    })
    .pluck(colname);

  // get sum + rows
  var sum = data.sum();
  var rows = data.length;

  // early exit?
  if(!sum || !rows) {
    $div.html('');
    return;
  }

  // add stats for each column
  var avg = sum / rows;
  $div.html(
    ' ↑ ' + Math.max.apply(null, data) + 
    ' ø ' + avg.toFixed(2) +
    ' ↓ ' + Math.min.apply(null, data)
  );
});

[adrian]

Hey @bernhard - I love what I see here, but I am curious how the data is stored? I assume it links to a custom db table that you create manually? 

As an aside in my pre-PW life I used SigmaGrid (which was ahead of its time) in the admin of my custom CMS to handle display/editing of purchases from an online store, so I am excited to play around with this when it's ready.

[bernhard]

hi adrian,

data is stored just as normal pw pages. nothing fancy here. this makes it very easy to use! the current setup is that you need 3 files to define your field:

// yourfield_data.php
<?php
// return all relations that belong to this project
return $pages->find(
  "template=feedback". // find all relations
  ",has_parent=$page" // project must be the current project
);

// yourfield_columns.php
<?php namespace ProcessWire;
// sample column showing the ID of the page
$col = new dtCol();
$col->name = "id";
$col->title = 'Columns ID';
$col->data = function($page) {
  return $page->id;
};
$this->cols->add($col);

// yourfield_js.js (optional, but in most cases necessary to do advanced stuff)
var settings = {
  select: true,
  pageLength: 100,
};

11 minutes ago, adrian said:

I am excited to play around with this when it's ready.

I'm also very excited to take this further. I cannot release it at the current stage though because major changes are not only likely to come but almost planned

 

i have to work more on performance. I'm thinking about some kind of caching technique and maybe also providing AJAX pagination.

some things like column statistics would only work on the client side when all the data is loaded, though. but i know that datatables can handle several thousands of rows quite efficiently so i think i'll find a way to work with those amounts of data.

[dragan]

Any news on this? A beta version maybe under the Christmas tree?

 

 

[bernhard]

1 hour ago, dragan said:

Any news on this? A beta version maybe under the Christmas tree?

Hi dragan,

unfortunately not. I need to do a complete rewrite of the module. As i need it for all my projects it is on top of my todolist for the next year. I'm still not sure if I should release it free or as a pro module... But I hope I can release something around march 2018...

[szabesz]

4 hours ago, bernhard said:

I'm still not sure if I should release it free or as a pro module...

Maybe you want to consider other ways to get some funding. A quick idea is "croudfounding". I'm not suggesting using such a service because as far as I know they cost money, so maybe a dedicated PW forum thread under the topic Beer Garden will do. We send you money the cheapest way possible (bank transfer?) and you update the first post of the thread announcing who sent what. You can assign a random ID to each donator and that way only you and the donator can identify it (in order to keep this info private.)

[bernhard]

got some very good news on this (it feels like christmas for myself, because this made me some headache over the last months and the solution is quite simple and solves a lot of problems).

 

I did some performance tests on a table with more than 10.000 rows and got the following results here on my local test environment (laragon + i7 @ 2ghz).

• rendering the table with javascript test-data works instantly (using deferrender option, did not try without)
• rendering the table with data coming from pw pages and 5 different fields takes around 16s
• single-language setup without caching takes around 5s to load
• multi-language setup without caching around 8s
• with markup cache enabled 180ms

the screenshot shows a multi language table with 10.000 rows and 12 fields that loaded in 9,1s when building the cache.

thats some really nice results, because it will make the setup even easier and it will even work with multilanguage wich I didn't know how to tackle before.

my goal always was to stay on the client side, because there you have all the power of datatables: you have a powerful API to filter, sort etc the table and this plays perfectly well together with other clientside libraries like chart.js; my testcase showed that the search and sorting was also very snappy so I think it's definitely doable!

It will be the first project after my master thesis and I may have something ready around 03/2018

On 9.12.2017 at 7:50 PM, szabesz said:

Maybe you want to consider other ways to get some funding. A quick idea is "croudfounding".

I don't think that's a proper way to do this. I think there are only the 2 options: Release it for free (donations are always possible and welcome of course) without any support, documentation etc. or release it as pro module and provide proper support and docs.

maybe @kongondo or @apeisa can share some of their experiences with us (or at least with me in a PM)?

[dragan]

1 hour ago, bernhard said:

the screenshot shows a multi language table with 10.000 rows and 12 fields that loaded in 9,1s when building the cache.

Does that mean: 10k PW-pages with 5 fields each? Multi-lang = how many? 2, 3, more?

1 hour ago, bernhard said:

my goal always was to stay on the client side

I don't quite understand this part: Does that mean your tests are done in a PW-module, and previously you only used it in the frontend?

In any case, that's good news.

re: pro/free

I agree. And (if it's reasonably priced) I'd be happy to buy a pro version, even if I wasn't sure if I would have a specific need in a client project right now.

re: cache mentioned above

Are you talking about caching the 10k pages? Or do you save the queried results (JSON?) as a file first, and then use this saved version as DT data-source?
I use the latter method for a custom search that features autosuggest/-complete + fuzzy search, where the generated JSON is around 500-600kb. It speeds up the whole thing tremendously.

[bernhard]

Ok guys, I got some REALLY nice results today

@dragan sorry, I was unclear in my previous post... I had to leave to a christmas party

So here are some explanations and the new results:

• Table with 10.000 rows without cache: 400ms (see the screenshot)
• Table with 10.000 rows with cache: 200ms

The key was to get the data directly from the database. Of course I knew before that this possibility exists, but I didn't know how to solve the problem of multilanguage and returning complex data (like $page->parent->title for one column) and still keep the setup of a table simple and straightforward. Now I found a great way and I'm really eager to start working on this  

What I meant by "stay on the client side" was, that I want all the data get transferred to the client and then rendered by datatables. The other option would be to use ajax pagination and load only junks of the data to the client. But having all the data on the client is a huge benefit for manipulating, filtering, sorting, charting etc.; and you where right about my "cache" wording: I'm talking about a cached string holding all the data for the datatable. So if the cache exists it just loads the string - if not it creates the string from the database and then loads the data into the datatables.

Any wishes/ideas for features that I should think of when developing the next version of the module?

...see the feature list in the first post