bernhard

Preview: RockDataTables

Recommended Posts

hi everybody,

this is a preview of a module that i'm working on for quite a long time. I needed it for an intranet office management application that i'm still working on. It makes it very easy to create very customized Datatables using the awesome jquery datatables plugin (www.datatables.net)

Download - sorry, removed as it changes too frequently; closed alpha - contact me if you think you can contribute

 

 

Remarks:

The module is intended to be used on the backend. Loading styles is at the moment only done via adding files to the $config->styles/scripts arrays. Also the communicaton to javascript is done via the $config->js() method that is built into the admin and would have to be implemented manually on frontend use. But it would not be difficult of course :)

Installation:

Nothing special here, just download + install

 

edit: removed some parts, because i made a complete rewrite of the module (and may not have been the last one)!

[...] removed

Why i created this module:

  • of course i know @Soma s module but i needed a lot more features and the newer datatables version. also i like to define all the columns as objects and have everything on one place.
  • lister & markupadmindatatable: nice for basic tables but lacks of features to modify the appearance of the cell values (like rendering icons, background colors and so on)
  • datatables provides a great frontend API for filtering, showing/hiding columns, getting data, modifying it...
  • it also plays well together with frontend charts like google chart api in this case:
    googlechart.gif

 

todo / roadmap:

  • reload only one row
  • add filters to all columns (in future also dropdowns, smaller than, bigger than, Regex, ...)
  • make it possible to add table on frontend pages
  • make buttons look like pw buttons
  • make it possible to set settings globally and only for one table
  • provide easy way of colorbars (percentage, red/green), maybe at different positions (left, top, right, bottom)
  • provide easy way of adding action items (edit, show, link etc - visible or onhover)
  • make own layout for tables (topleft, topright, bottom etc to make it easy to create extensions and show messages)
  • privide way of simple javascript plugins (like I already have for row sums etc)
  • provide easy way of handling actions for all selected items (delete selected, update selected...)
  • provide easy way of reloading data (--> easy when using ajax source)
  • easy way of showing/hiding columns
  • excel/csv/clipboard export
  • GUI for table setup
  • processmodule to show different tables (lister replacement)

 

  • Like 18

Share this post


Link to post
Share on other sites

need to setup tables via custom sql queries?

PHP

// init datatables module
    $dt2 = $modules->get('RockDataTables2');

// setup columns
    $sql = 'SELECT * FROM invoicetable';
    $dt2->setupColsBySql($sql);

// setup table
    $dt2->id = 'dt_finanzbuch';
    $dt2->js('/site/modules/ProcessRockFinance/dt_finanzbuch.js');
    
    $f = $modules->get('InputfieldMarkup');
    $f->label = 'Tabelle';
    $f->value = $dt2->render();
    $form->add($f);

// ##################################
// ajax request -> return data
// non-ajax -> render form + table
// ##################################
if($config->ajax) {
    echo $dt2->getJSON($dt2->getDataBySql($sql));
    die();
}

JS

$(document).ready(function() {
    // setup variables
        var opt = ProcessWire.config.dt_finanzbuch; // options from backend
        var colDefs = []; // column definitions

    // custom column definitions here

    // load default column definitions
        colDefs = colDefs.concat(dtGetDefaultDefs(opt));

    // initialise table
        $('#dt_finanzbuch').DataTable({
            ajax: { url: './book/', type: 'post' },
            columnDefs: colDefs,
            pageLength: 10,
            footerCallback: columnSums,
        });
});

Result

2017-02-16 17_52_11-.png

As you can see the sum function is not ready. it shows euros for my usecase...

  • Like 5

Share this post


Link to post
Share on other sites

Very nice! Do you think it is feasible to use this - or a future version - on the front end?

  • Like 1

Share this post


Link to post
Share on other sites

hi kathep,

sure it is also usable in the frontend. but it is no click-click solution. the module just provides some helpers to define the columns like i showed in the example. to make the code more clear to read, write and maintain. you would just have to include the right scripts (i'm currently loading all the plugins via cdn) and you would be fine. maybe you would also have to do some styling. but all of that is - at least for now - not intended to be handled by my module.

hope that answer helps you :)

  • Like 1

Share this post


Link to post
Share on other sites

plus DataTables are so easy to setup on the front end, you really don't need a module... just make your table, and then load the datatables scripts and init; the instructions are all there on the datatables site, and it is really well documented..

  • Like 1

Share this post


Link to post
Share on other sites

current version has properly formatted currency fields (atm only euro) and supports colorbars and column sums (also from selected rows)

had to remove this video, sorry

  • Like 2

Share this post


Link to post
Share on other sites

did a total rewrite of the module and will release it the next weeks when i'm done

 

  • Like 10

Share this post


Link to post
Share on other sites

Hi @bernhard Thanks for putting all the work into sharing this module. What came to my mind was that I do not really like providing code in admin input fields, except when it is a few lines of code. So wouldn't it be possible (optionally maybe) to provide paths to files instead?

  • Like 1

Share this post


Link to post
Share on other sites

Already done ;)

The fields are there for really simple tables and the files for more complex ones. The module will include all files related to a field automatically (PHP, js and CSS).

  • Like 1

Share this post


Link to post
Share on other sites

Oh I see, banging on open doors again :) Thanks for the info!

Share this post


Link to post
Share on other sites

making progress on this but there's still a lot to do until it is releasable as a module (proper documentation mostly)...

sneak peak what's easily doable: a todo-app

5979eafe2b221_2017-07-2715_25_26-BaumrockAdminProcessWirebaumrock_dev.thumb.png.5ea6b9a66bc50536acc38be52edbeba4.png

definition is as easy as that:

	/**
	 * show table of todos
	 */
	public function executeTodos() {
		
    // create form
      $form = modules('InputfieldForm');
      $form->action = './';

      // add new project
        $b = modules('InputfieldButton');
        $b->attr("id+name", "addTodo");
        $b->addClass("ui-priority-primary pw-panel pw-panel-reload");
        $b->value = __("Neues Todo");
        $b->attr('data-href', pages(2)->url . 'page/add/?parent_id=' . pages('template=todos'));
        $b->icon = "plus";

        $f = modules('InputfieldMarkup');
        $f->value = $b->render();
        $form->add($f);

      // table
        $t = modules('InputfieldRockDatatables');
        $t->attr('id+name', 'manageTodos');

        $t->rows = pages('template=todo'); // define the table's source rows with one selector
        $t->ajax = 1; // set table mode to ajax loading making it possible to reload data via a simple $table.ajax.reload();

      [...]

        // remaining
        $col = new dtCol;
        $col->name = "remaining";
        $col->title = 'Tage';
        $col->className = 'minwidth';
        $col->data = function($page) {
          if(!$page->deadline) return '';

          $now = new \DateTime('now');
          $then = new \DateTime();
          $then->setTimestamp($page->deadline);
          $days = $now->diff($then)->format('%R%a');

          $color = config()->colors->lightred;
          if($days > 0) $color = config()->colors->lightorange;
          if($days > 7) $color = config()->colors->lightgreen;

          $obj = new \stdClass();
          $obj->display = $days;
          $obj->colorBars = [ [1, $color] ];

          return $obj;
        };
        $t->cols->add($col);

      [...]

      // add field to form

        $f = modules('InputfieldMarkup');
        $f->value = $t->render();
        $form->add($f);
    
    $out = $form->render();
    return $out;
	}

 

  • Like 7

Share this post


Link to post
Share on other sites

just to show that this module can render any kind of field (seems that this was not clear enough: https://processwire.com/talk/topic/16929-choose-custom-repeater-template/?do=findComment&comment=148803 )

a simple column showing images:

// pic
$col = new dtCol;
$col->name = "pic";
$col->title = 'pic';
$col->className = 'minwidth';
$col->data = function($page) {
  if(!$page->pics->count()) return '';
  return $page->pics->each(function($img) {
    return "<img src='{$img->maxSize(200,200)->url}' alt=''><br>";
  });
};
$t->cols->add($col);

2017-08-08 11_06_16-BaumrockAdmin • ProcessWire • baumrock.dev.png

so you can do whatever you want and whatever the API + PHP can do...

the module even supports a plugin-system where we can easily build cell-renderers that handle common tasks and make rendering even easier.

this for example is the js file of the column filter that shows a filter input on top of each column:

/**
 * this plugin creates filter inputs for every column
 * 
 * #### usage ####
 * just set your table's settings: { colfilter: true }
 * 
 * #### issues ####
 * the column filter breaks the table's column alignment on window resize
 * 
 * #### roadmap ####
 * make it possible to have different kind of inputs like range sliders and dropdowns
 * 
 */

/**
 * make sure that the tables searching setting is set to true
 */
$(document).on('beforeInit.rdt', '.RockDatatable', function(e, dt) {
  if(!dt.settings.colfilter) return;

  $table = $(dt.el);
  if(dt.settings.searching == false) console.log('overwriting setting "searching" to true for table ' + $table.attr('id'));
  dt.settings.searching = true;
});

/**
 * add colfilter row to table header
 */
$(document).on('afterInit.rdt', '.RockDatatable', function(e, dt) {
  if(!dt.settings.colfilter) return;

  var $table = $(e.target); // table dom element
  var table = $table.DataTable();
  var $thead = table.table().header();

  // prepare the new header row
  var $tr = $("<tr role='row' class='colfilter' />");
  $.each(dt.columnNames, function() {
    $tr.append($('<th><input type="text" class="colfilter"></th>'));
  });

  // add it to the table header
  $tr.prependTo($thead);
});

/**
 * handle filterinputs
 */
colfiltertimer = 0;
$(document).on('keypress keyup change', 'input.colfilter', function(e) {
  var $input = $(this);
  var $tr = $input.closest('tr');
  var $th = $input.closest('th');
  // datatables splits the table in 2 divs when scrollX is used! thats why we need to find the table via the wrapper
  var table = $input.closest('.dataTables_wrapper').find('table.RockDatatable').DataTable();
  var regex = false;

  // disable enter form submit
  var keyCode = e.keyCode || e.which;
  if (keyCode === 13) { 
    e.preventDefault();
    return false;
  }

  // delay search while typing
  clearTimeout(colfiltertimer);
  colfiltertimer = setTimeout(function() {
    table.columns($th.index()+':visIdx').search($input.val(), regex, false).draw();
  }, 500);
});

this may look complex in the beginning but once we have such a plugin we only have to set one option of the table and the filter inputs appear :)

 

  • Like 2

Share this post


Link to post
Share on other sites

datatable connected to the pagefield :)

asmdemo.thumb.gif.8856771cdd6bcbc3a7bc2b487fe2caa3.gif

updating the pageautocomplete field seemed a little tricky in the beginning but the final approach was quite easy:

// handle buttonclicks
$('#addcompetences').click(function(e) {
  e.preventDefault();
  var table = $table.DataTable(); // datatable instance
  var $field = $('#wrap_Inputfield_competences'); // asm select field
  var $template = $field.find('li.itemTemplate'); // selected page template
  var $ol = $template.closest('ol');
  
  // get selected competences from table
  var selectedtable = $.map(
    table.rows({selected:true}).data(),
    function(val, i) { return [[ val.id.sort, val.title + ' (' + val.cluster.filter + ')' ]]; }
  );

  // update ASM
  // clone template and populate values
  $.each(selectedtable, function(i,val) {
    // check double
    if($ol.find('.itemValue:contains("' + val[0] + '")').length) return;

    // add clone
    var $new = $template.clone().appendTo($template.parent());
    $new.find('.itemValue').text(val[0]);
    $new.find('.itemLabel').text(val[1]);
    $new.removeClass('itemTemplate');
  });
  
  // add state changed class
  $field.addClass('InputfieldStateChanged');

  // update original input
  InputfieldPageAutocomplete.rebuildInput($ol);
  
  return false;
});

 

  • Like 3

Share this post


Link to post
Share on other sites

very nice - a good alternative to other page select options out there, and intuitive for the user;

  • Like 1

Share this post


Link to post
Share on other sites

looks like i could use this module pretty soon on a project - LMK if you need someone to test it...

Share this post


Link to post
Share on other sites

another example how versatile and extendable this module is: you can write your own plugins to show custom column statistics. here i'm showing max, average and minimum values of some collected feedbacks. X means no answer and is excluded from the stats:

colstats.thumb.gif.21d09aa4282454f3e33ea9b216d3be4d.gif

plugincode is quite simple once you know the basics of datatables api:

$(document).on('update.rdt', 'div.colstats, div.colstatsselected', function(e, table, col) {
  var $div = $(e.target);
  var $table = $(table);
  var table = $table.DataTable();

  // get table settings
  var id = $table.attr('id').replace('RockDatatable_','');
  var dt = ProcessWire.config.RockDatatables[id];
  var colname = dt.columnNames[col.index()];

  var selector = {search: 'applied'};
  if($div.hasClass('colstatsselected')) selector.selected = true;

  // get data
  var data = table
    .rows(selector)
    .data()
    .filter(function(row) {
      // return true if it is a number
      // don't count empty cells or "noanswer" cells that have an X
      return row[colname]*1;
    })
    .pluck(colname);

  // get sum + rows
  var sum = data.sum();
  var rows = data.length;

  // early exit?
  if(!sum || !rows) {
    $div.html('');
    return;
  }

  // add stats for each column
  var avg = sum / rows;
  $div.html(
    ' ↑ ' + Math.max.apply(null, data) + 
    ' ø ' + avg.toFixed(2) +
    ' ↓ ' + Math.min.apply(null, data)
  );
});

:)

  • Like 4

Share this post


Link to post
Share on other sites

Hey @bernhard - I love what I see here, but I am curious how the data is stored? I assume it links to a custom db table that you create manually? 

As an aside in my pre-PW life I used SigmaGrid (which was ahead of its time) in the admin of my custom CMS to handle display/editing of purchases from an online store, so I am excited to play around with this when it's ready.

Share this post


Link to post
Share on other sites

hi adrian,

data is stored just as normal pw pages. nothing fancy here. this makes it very easy to use! the current setup is that you need 3 files to define your field:

// yourfield_data.php
<?php
// return all relations that belong to this project
return $pages->find(
  "template=feedback". // find all relations
  ",has_parent=$page" // project must be the current project
);
// yourfield_columns.php
<?php namespace ProcessWire;
// sample column showing the ID of the page
$col = new dtCol();
$col->name = "id";
$col->title = 'Columns ID';
$col->data = function($page) {
  return $page->id;
};
$this->cols->add($col);
// yourfield_js.js (optional, but in most cases necessary to do advanced stuff)
var settings = {
  select: true,
  pageLength: 100,
};
11 minutes ago, adrian said:

I am excited to play around with this when it's ready.

I'm also very excited to take this further. I cannot release it at the current stage though because major changes are not only likely to come but almost planned :)

 

i have to work more on performance. I'm thinking about some kind of caching technique and maybe also providing AJAX pagination.

some things like column statistics would only work on the client side when all the data is loaded, though. but i know that datatables can handle several thousands of rows quite efficiently so i think i'll find a way to work with those amounts of data.

  • Like 3

Share this post


Link to post
Share on other sites

Any news on this? A beta version maybe under the Christmas tree? :-)

 

 

  • Like 2

Share this post


Link to post
Share on other sites
1 hour ago, dragan said:

Any news on this? A beta version maybe under the Christmas tree? :-)

Hi dragan,

unfortunately not. I need to do a complete rewrite of the module. As i need it for all my projects it is on top of my todolist for the next year. I'm still not sure if I should release it free or as a pro module... But I hope I can release something around march 2018...

  • Like 2
  • Sad 1

Share this post


Link to post
Share on other sites
4 hours ago, bernhard said:

I'm still not sure if I should release it free or as a pro module...

Maybe you want to consider other ways to get some funding. A quick idea is "croudfounding". I'm not suggesting using such a service because as far as I know they cost money, so maybe a dedicated PW forum thread under the topic Beer Garden will do. We send you money the cheapest way possible (bank transfer?) and you update the first post of the thread announcing who sent what. You can assign a random ID to each donator and that way only you and the donator can identify it (in order to keep this info private.)

  • Like 2

Share this post


Link to post
Share on other sites

got some very good news on this (it feels like christmas for myself, because this made me some headache over the last months and the solution is quite simple and solves a lot of problems).

screencapture-datatables-to-de-test6-1513092836163.thumb.png.ad29f6fcf69094ddfdc1ab9d9047df75.png

 

I did some performance tests on a table with more than 10.000 rows and got the following results here on my local test environment (laragon + i7 @ 2ghz).

  • rendering the table with javascript test-data works instantly (using deferrender option, did not try without)
  • rendering the table with data coming from pw pages and 5 different fields takes around 16s
  • single-language setup without caching takes around 5s to load
  • multi-language setup without caching around 8s
  • with markup cache enabled 180ms :)

the screenshot shows a multi language table with 10.000 rows and 12 fields that loaded in 9,1s when building the cache.

thats some really nice results, because it will make the setup even easier and it will even work with multilanguage wich I didn't know how to tackle before.

my goal always was to stay on the client side, because there you have all the power of datatables: you have a powerful API to filter, sort etc the table and this plays perfectly well together with other clientside libraries like chart.js; my testcase showed that the search and sorting was also very snappy so I think it's definitely doable!

It will be the first project after my master thesis and I may have something ready around 03/2018

On 9.12.2017 at 7:50 PM, szabesz said:

Maybe you want to consider other ways to get some funding. A quick idea is "croudfounding".

I don't think that's a proper way to do this. I think there are only the 2 options: Release it for free (donations are always possible and welcome of course) without any support, documentation etc. or release it as pro module and provide proper support and docs.

maybe @kongondo or @apeisa can share some of their experiences with us (or at least with me in a PM)? :)

  • Like 6

Share this post


Link to post
Share on other sites
1 hour ago, bernhard said:

the screenshot shows a multi language table with 10.000 rows and 12 fields that loaded in 9,1s when building the cache.

Does that mean: 10k PW-pages with 5 fields each? Multi-lang = how many? 2, 3, more?

1 hour ago, bernhard said:

my goal always was to stay on the client side

I don't quite understand this part: Does that mean your tests are done in a PW-module, and previously you only used it in the frontend?

In any case, that's good news.

re: pro/free

I agree. And (if it's reasonably priced) I'd be happy to buy a pro version, even if I wasn't sure if I would have a specific need in a client project right now.

re: cache mentioned above

Are you talking about caching the 10k pages? Or do you save the queried results (JSON?) as a file first, and then use this saved version as DT data-source?
I use the latter method for a custom search that features autosuggest/-complete + fuzzy search, where the generated JSON is around 500-600kb. It speeds up the whole thing tremendously.

Share this post


Link to post
Share on other sites

Ok guys, I got some REALLY nice results today :):):)

screencapture-datatables-to-test10-1513163338292.thumb.png.f5d0bd18d2ab498fad3f68febea73790.png

@dragan sorry, I was unclear in my previous post... I had to leave to a christmas party ;)

So here are some explanations and the new results:

  • Table with 10.000 rows without cache: 400ms (see the screenshot)
  • Table with 10.000 rows with cache: 200ms

The key was to get the data directly from the database. Of course I knew before that this possibility exists, but I didn't know how to solve the problem of multilanguage and returning complex data (like $page->parent->title for one column) and still keep the setup of a table simple and straightforward. Now I found a great way and I'm really eager to start working on this :) 

What I meant by "stay on the client side" was, that I want all the data get transferred to the client and then rendered by datatables. The other option would be to use ajax pagination and load only junks of the data to the client. But having all the data on the client is a huge benefit for manipulating, filtering, sorting, charting etc.; and you where right about my "cache" wording: I'm talking about a cached string holding all the data for the datatable. So if the cache exists it just loads the string - if not it creates the string from the database and then loads the data into the datatables.

Any wishes/ideas for features that I should think of when developing the next version of the module?

...see the feature list in the first post :)

  • Like 5

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By kixe
      Email Verification
      This module provides functions to validate email adresses and hosts. The module generates a textfile for blacklisted mailhosts (trashmail), which will be always up to date.
      Download
      https://modules.processwire.com/modules/email-verification/
      API
      // get module $mailcheck = $modules->get('EmailVerification'); // return bool/ string - automatted update of blacklist file $mailcheck->blacklisted(email|domain) // return bool - validate a top level domain, checks against IANA list $mailcheck->validTLD(tld) // return array of punycoded TLDs - cyclic updated, data pulled from IANA $mailcheck->getTLDs(cycle=2592000) // return bool - checks syntax converts to punycode $mailcheck->validDomainName(domain); // return bool - checks punycode encoded syntax $mailcheck->validHostName(host); // return bool - checks syntax and accessibility $mailcheck->validHost(email|domain) // add a single value to blacklist $mailcheck->addToBlacklist(email|domain) USAGE
      $mailcheck = $modules->get('EmailCheck'); $email = 'susi@trashmail.com'; if($mailcheck->blacklisted($email)) echo 'Email Provider not allowed'; if(!$mailcheck->validHost($email)) echo 'Mailhost not available'; Example blacklist file:  blacklist.txt
       
    • By teppo
      Code Blocks Textformatter is a tiny Textformatter module I cooked up to add support for code blocks to text/textarea/RTE fields on some of the sites I work with. Unlike a full-blown Markdown Textformatter – which is something that we already have in the core – this module simply adds support for fenced and inline code blocks.
      The syntax is based on the GitHub code block documentation, so please refer to that for additional instructions. The README at GitHub also includes some basic examples. As with any Textformatter, in order to enable this one, install it and enable it via field settings. Note that there's no syntax highlighting built in (at least for the time being), so use a tool of your choice for that – personally I prefer Prism.js.
      Since this module doesn't use a Markdown tool behind the scenes, but rather some home baked regular expressions, there's always the possibility that I've missed something – but please let me know if you use this module and run into any issues. On the other hand this module should be relatively fast and unobtrusive, as there are no unnecessary bits of code to run 
      GitHub repository: https://github.com/teppokoivula/TextformatterCodeBlocks Modules directory: http://modules.processwire.com/modules/textformatter-code-blocks/
    • By theo
      Here is my first processwire module (beta).
      https://github.com/theo222/ImagePickerField
      A lot of "first times". First time Github, Markdown and of course PW Modules.
      So be gentle please.
      I hope it works.

    • By horst
      Image Animated GIF  v 2.0.2
       
      Module for PW 2.5 stable or newer, but is obsolete for PW Versions greater then 3.0.89 (... read more ...)
       
      This module helps with resizing and cropping of animated GIFs when using the GD-Library.
      The GD-Library does not support this.


      This module is completely based upon the fantastic work of

         László Zsidi (http://www.gifs.hu/, builts the initial classes)
         xurei (https://github.com/xurei/GIFDecoder_optimized, enhanced the classes)


      I have ported it to a PW module and it works with core imagefields, together with Pia and with CropImagefields that uses the new naming scheme since PW 2.5.
       
      ------------------------------------------------------------------------------------------------------------------------------------------
      read more in this post about the rewrite from version 1.0.0 to version 2.0.0
      ------------------------------------------------------------------------------------------------------------------------------------------
      You can find it in the modules directory: https://modules.processwire.com/modules/image-animated-gif/
      and on Github: https://github.com/horst-n/ImageAnimatedGif
      ------
      A preview can be found here