Jump to content

DataSet import modules


mtwebit

Recommended Posts

 After installing Dataset module I get this error:

Module reported error during install (DataSet): SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'label' cannot be null

When I click an Import link nothing happens. Tasks page is empty

I tried to do a simple import for testing using this config

Any ideas? Where can I see error log?

JSON{
  "name": "Testing the import",
  "input": {
    "type": "csv",
    "delimiter": ",",
    "header": 1,
    "limit": 10
  },
  "fieldmappings": {
    "title": 1
  },
  "pages": {
    "template": "basic-page",
    "selector": "title=@title"
  }
}

 

Link to comment
Share on other sites

23 hours ago, double said:

 After installing Dataset module I get this error:


Module reported error during install (DataSet): SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'label' cannot be null

 

Thanks for reporting this. Probably a core compatibility issue. Fixed now.
Pull the updated module from github or delete from your site/modules and reinstall it.

  • Like 1
Link to comment
Share on other sites

  • 3 weeks later...

I'm not sure what's gone wrong, but I had this module working perfectly earlier in the day , and I imported all but one data table I needed from CSV, but now it fails to run and my browser shows a 500 server error when I attempt to run it.

although I updated to the latest Processwire Dev version to address something else, and I'm not sure if that has broken it?

I tried copying the latest version of this and the tasker module from Github, as the modules directory doesn't seem to be up to date with the latest, and I wondered if there was a compatibility issue with the latest build of Processwire, but that didn't fix it.

Chrome debug tools show a resource like so: admin/page/tasks/api//?cmd=run&id=4820

Opening this resource directly in the browser says that the task has already been run and won't run again.

Manually updating the task page and setting 'task is running' to zero triggers an error 

Tracy Debugger shows : Call to undefined function ProcessWire\pcntl_signal() (/site/modules/Tasker/Tasker.module.php:697)

According to some places in PHP docs, pcntl functions shouldn't be called by a web server like Apache, so I'm not sure if that is an issue?

The key thing is the module was working just fine, then suddenly stopped working, with import configurations that had previously worked.

Any ideas?

[Edit] - I found out the problem: I'm running on a VPS with Plesk and PHP as an FPM App doesn't include pcntl but PHP as FastCGI does. The site was one that hadn't been updated in a while and was still running on PHP 7.3 so I updated to 7.4 and in the process, without thinking, changed it to FPM, as most of my other sites are, but this disabled pcntl support.  Something to be aware of for anyone else running Plesk.

Link to comment
Share on other sites

@Kiwi Chris I think it may be the file compiler as it seems to be translating the module so that PHP is looking for pcntl_signal() in the processwire namespace, hence the line from tracy: "ProcessWire\pcntl_signal()".

It's also telling you where that line occurs. I've not used Tasker before, but I'm guessing you can edit that line and wherever you see "pcntl_signal()" being called, just prepend it with a backslash "\" character to put it in the global namespace.

Doing so may reveal more errors about pcntl functions - but if that's the case, we know we are on the right track and you can do the same to them as they appear.

  • Like 1
Link to comment
Share on other sites

Although LazyCron and web-based task execution are supported, tasks should be invoked by command line tools (e.g. Unix cron), not from the the Web server environment.
See the wiki.
Tasker uses process control functions for checking signals (e.g. an execution timer) which should be safe to use in a webserver environment but the code also monitors the SIGINT and SIGTERM interrupts that should be moved to the command line specific part. And it seems I need to check the availability of that function.

Thanks for pointing this out.

  • Like 1
Link to comment
Share on other sites

I recently did an import of about 15K records with these modules, and they worked well, however there was one undesirable side effect of storing the log to a standard Processwire text_area field, log_messages.

Processwire creates a full text index on text_area and text field types.

I had a few issues with field mappings and defining unique identifiers the first couple of times I ran the import, so just used Tracy Debugger to delete the pages, fix the import definition, and run the import again.

Unfortunately, over my efforts, the full text index on the field log_messages grew to over 2GB in size, even though the imported CSV file itself was only about 1MB.

It took me a bit to figure out why the database had suddenly grown from a few MB to several GB, and once I located the problem, I ran Optimise Table in mySQLAdmin, which cleaned things up and got things back down to a normal size.

I suspect in this case, the log_messages field probably doesn't need to have a full text index on it, although I guess if someone has a large number of import pages, maybe there is a case for them to be searchable?
If there isn't a need for the log_messages field to be searchable, then perhaps a custom text fieldtype without a full text index might make both the import run more quickly if the field is repeatedly updated during import, and also avoid the issue of creating an enormous index file.

I've made a post to the Wishlist and roadmap thread, suggesting an option for textarea fields to have indexing disabled if not required, as this would resolve the issue.

 

Edited by Kiwi Chris
Post to wishlist
  • Thanks 1
Link to comment
Share on other sites

  • 3 weeks later...
On 12/23/2020 at 7:40 AM, Kiwi Chris said:

I suspect in this case, the log_messages field probably doesn't need to have a full text index on it, although I guess if someone has a large number of import pages, maybe there is a case for them to be searchable?

Actually, the log_messages field is created and used by Tasker not DataSet.
I don't see any easy solution to handle this problem. If PW core gets noindex support then I'll use that.
The best thing you can do atm is to turn off profiling and debugging in Tasker in a production system (this is independent from PW settings).
It is also a good practice to delete tasks after they are finished and create new ones when needed.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Similar Content

    • By monollonom
      (once again I was surprised to see a work of mine pop up in the newsletter, this time without even listing the module on PW modules website 😅. Thx @teppo !)
      FieldtypeQRCode
      Github: https://github.com/romaincazier/FieldtypeQRCode
      Modules directory: https://processwire.com/modules/fieldtype-qrcode/
      A simple fieldtype generating a QR Code from the public URL of the page, and more.
      Using the PHP library QR Code Generator by Kazuhiko Arase.

      Options
      In the field’s Details tab you can change between .gif or .svg formats. If you select .svg you will have the option to directly output the markup instead of a base64 image. SVG is the default.
      You can also change what is used to generate the QR code and even have several sources. The accepted sources (separated by a comma) are: httpUrl, editUrl, or the name of any text/URL/file/image field.
      If LanguageSupport is installed the compatible sources (httpUrl, text field, ...) will return as many QR codes as there are languages. Note however that when outputting on the front-end, only the languages visible to the user will be generated.
      Formatting
      Unformatted value
      When using $page->getUnformatted("qrcode_field") it returns an array with the following structure:
      [ [ "label" => string, // label used in the admin "qr" => string, // the qrcode image "source" => string, // the source, as defined in the configuration "text" => string // and the text used to generate the qrcode ], ... ] Formatted value
      The formatted value is an <img>/<svg> (or several right next to each other). There is no other markup.
      Should you need the same markup as in the admin you could use:
      $field = $fields->get("qrcode_field"); $field->type->markupValue($page, $field, $page->getUnformatted("qrcode_field")); But it’s a bit cumbersome, plus you need to import the FieldtypeQRCode's css/js. Best is to make your own markup using the unformatted value.
      Static QR code generator
      You can call FieldtypeQRCode::generateQRCode to generate any QR code you want. Its arguments are:
      string $text bool $svg Generate the QR code as svg instead of gif ? (default=true) bool $markup If svg, output its markup instead of a base64 ? (default=false) Hooks
      Please have a look at the source code for more details about the hookable functions.
      Examples
      $wire->addHookAfter("FieldtypeQRCode::getQRText", function($event) { $page = $event->arguments("page"); $event->return = $page->title; // or could be: $event->return = "Your custom text"; }) $wire->addHookAfter("FieldtypeQRCode::generateQRCodes", function($event) { $qrcodes = $event->return; // keep everything except the QR codes generated from editUrl foreach($qrcodes as $key => &$qrcode) { if($qrcode["source"] === "editUrl") { unset($qrcodes[$key]); } } unset($qrcode); $event->return = $qrcodes; })
    • By Sebi
      AppApiFile adds the /file endpoint to the AppApi routes definition. Makes it possible to query files via the api. 
      This module relies on the base module AppApi, which must be installed before AppApiFile can do its work.
      Features
      You can access all files that are uploaded at any ProcessWire page. Call api/file/route/in/pagetree?file=test.jpg to access a page via its route in the page tree. Alternatively you can call api/file/4242?file=test.jpg (e.g.,) to access a page by its id. The module will make sure that the page is accessible by the active user.
      The GET-param "file" defines the basename of the file which you want to get.
      The following GET-params (optional) can be used to manipulate an image:
      width height maxwidth maxheight cropX cropY Use GET-Param format=base64 to receive the file in base64 format.
    • By MarkE
      This fieldtype and inputfield bundle was built for storing measurement values within a field, rendering them in a variety of formats and converting them to other units or otherwise modifying them via the API.
      The API consists of a number of predefined functions, some of which include...
      render() for rendering the measurement object, valueAs() for converting the value to another unit value, convertTo() for converting the whole measurement object to different units, and add() and subtract() for for modifying the stored value by the value (converted as required) in another measurement. In the admin the inputfield includes a checkbox (which can be optionally disabled) for converting values on page save. For an example if a value was typed in as centimeters, the unit was changed to metres, and the page saved with this checkbox selected, said value would be automatically converted so that e.g. 170 cm becomes 1.7 m.

      A simple length field using Fieldtype Measurement and Inputfield Measurement.
      Combination units (e.g. feet and inches) are also supported.
      Please note that this module is 'proof of concept' at the moment - there are limited units available and quite a lot of code tidying to do. More units will be added shortly.
      See the GitHub at https://github.com/MetaTunes/FieldtypeMeasurement for full details and updates.
    • By tcnet
      File Manager for ProcessWire is a module to manager files and folders from the CMS backend. It supports creating, deleting, renaming, packing, unpacking, uploading, downloading and editing of files and folders. The integrated code editor ACE supports highlighting of all common programming languages.
      https://github.com/techcnet/ProcessFileManager

      Warning
      This module is probably the most powerful module. You might destroy your processwire installation if you don't exactly know what you doing. Be careful and use it at your own risk!
      ACE code editor
      This module uses ACE code editor available from: https://github.com/ajaxorg/ace

      Dragscroll
      This module uses the JavaScript dragscroll available from: http://github.com/asvd/dragscroll. Dragscroll adds the ability to drag the table horizontally with the mouse pointer.
      PHP File Manager
      This module uses a modified version of PHP File Manager available from: https://github.com/alexantr/filemanager
       
    • By tcnet
      This module implements the website live chat service from tawk.to. Actually the module doesn't have to do much. It just need to inserted a few lines of JavaScript just before the closing body tag </body> on each side. However, the module offers additional options to display the widget only on certain pages.
      Create an account
      Visit https://www.tawk.to and create an account. It's free! At some point you will reach a page where you can copy the required JavaScript-code.

      Open the module settings and paste the JavaScript-code into the field as shown below. Click "Submit" and that's all.

      Open the module settings
      The settings for this module are located int the menu Modules=>Configure=>LiveChatTawkTo.

       
×
×
  • Create New...