Input validation in back-end fields

[gcaplan]

Hi folks

Excited to find Processwire - a much needed breath of fresh thinking!

The only downer I've spotted so far is the apparent lack of any way to constrain data entry in the admin area fields. For example to limit an integer between 1 an 32.

I realise that this is a CMS and not a RDBMS, but the other option on my short-list, Symphony, seems to offer more here.

Am I missing something? Or is there a practical workaround (I'm fluent in PHP & regexes and have basic JQuery)?

Any advice much appreciated.

[Soma]

Edit: oh hi and welcome gcplan, glad you found us!

I think you'll have no problems to make a custom integer fieldtype/inputfieldtype module, that would check for a certain range and output an error when saved. It's possible to create field setting you would be able with 2 integer fields to enter a upper-lower bound.

You could even implement jquery script to check on runtime before saving.

Jut have a look at how the PW integer,float,text fieldtype/inputfieldtypes are done.

I think you'll have no problem if you're fluent in php. PW makes it very easy to do what ever you like in that regard.

[apeisa]

FieldtypeInteger actually has this commented function:

/* TO ADD BACK LATER
public function ___getConfigInputfields(Field $field) {
 $inputfields = parent::___getConfigInputfields($field);
 $field = $this->modules->get('InputfieldInteger');
 $field->setAttribute('name', 'minValue');
 $field->label = 'Minimum Allowed Value';
 $field->setAttribute('value', $field->minValue);
 $field->setAttribute('size', 20);
 $field->description = 'The smallest number allowed by this field.';
 $inputfields->append($field);
 $field = $this->modules->get('InputfieldInteger');
 $field->setAttribute('name', 'maxValue');
 $field->label = 'Maximum Allowed Value';
 $field->setAttribute('value', $field->maxValue);
 $field->setAttribute('size', 20);
 $field->description = 'The largest number allowed by this field (may not exceed ' . PHP_INT_MAX . ")";
 $inputfields->append($field);
 return $inputfields;
}
*/

[gcaplan]

Thanks guys

Rolling my own fields isn't ideal, but the other advantages are compelling enought that it's worth considering.

I come from the RDBMS world, so the idea of allowing uncontrained imput is rather scary. You folks have a lot of hands on experience with CMS projects - do you really find that there's no need for validation? Symphony does offer some facilities - basic, but better than nothing.

A quick fix would be to add a couple of fields to the advanced tab of the field definitions (where relevant) where you could add a regex and an error message. Basic but useful?

[ryan]

I like your idea of a regex, that would be easy to implement and very powerful. I agree on the importance of input limits. Though the need for that level of specificity in limits is pretty rare in my own projects. And this is the first time it's come up here. As a result, it's not been at the top of the priority list, but it's something that Fieldtypes are built to support (and already do in many instances). Actually, I think the number fields are probably one of the few that don't yet have many options for setting boundaries. It's not that the boundaries aren't useful, it's that I'm very cautious about enforcing any rules that could put a page in a state where one couldn't save. If I have an integer field that I know has to be between 1 and 3, then I'm usually enforcing that in my template code, where the need for the boundary exists. Longer term, I still think it's better to have it in the admin. Now that it's come up (via your message) that builds momentum for doing it sooner rather than later. Beyond a min/max in the integer and float fields, are there any other fields you think specifically additional validations would be worthwhile? I do like the idea of adding the regex option to the single line text field, though I'm not sure how many will use it (beyond us).

[Soma]

How about a slider you could configure to go from 1-3

Maybe different types of input slider with ranges would make a cool new inputfieldtype like my colorpicker. I will consider maybe doing one soon.

[apeisa]

Little OT: just realized that InputfieldInteger doesn't allow negative values. Not sure why? It saves negative values ok, but sanitizeValue changes them to positive. So in database I have -10 but edit field shows 10. Maybe a setting for this?

[ryan]

I think it's time for some upgrades to InputfieldInteger. I'm adding this to my list.

[gcaplan]

Ryan

Great to hear from the developer!

I'm very cautious about enforcing any rules that could put a page in a state where one couldn't save.

Don't really follow you here, as surely the idea is to prevent users from saving bad data? So long as you provide clear error messages how is this a problem? With jQuery it's pretty easy to provide the message as soon as users tab out of the field.

I'm usually enforcing that in my template code, where the need for the boundary exists. Longer term, I still think it's better to have it in the admin. Now that it's come up (via your message) that builds momentum for doing it sooner rather than later. Beyond a min/max in the integer and float fields, are there any other fields you think specifically additional validations would be worthwhile?

Don't understand much about PW yet, but doesn't doing this in the template mean that issues will only show up at runtime?

I suggested the regex idea as a quick and dirty fix, but obviously it's going to restrict use of the feature to sad geeks like us. Plus, on thinking it through, a single field for an error message would hit problems with your back-end internationalisation.

Years ago I developed a flexible declarative approach and it has served me well. Some of the modern PHP frameworks use a similar idea. You develop an easily extended utility library of validations, and each validation has an error string in all active languages. The string can use template-like placeholders for parameters: "Number should be between {1} and {2}." Then you develop a simple declarative syntax for setting validations and any relevant parameters. Here's a couple of examples:

is_greater_than,10

is_between,1,365

is_alphanumeric;is_max_length,256

As a catchall, you can offer access to regexes:

matches_pattern,

^\$(\d{1,3}(\,\d{3})*|(\d+))(\.\d{2})?$,"Must be a valid USD price"

Another useful feature, I find, is filters that clean up input. These are run before the validations and ensure consistent presentation.

For example, for a city or postal code you might use:

to_uppercase

For a field in which you are entering, say, architects, you can tidy up capitalisation with:

to_pretty_name

If input isn't entirely trusted, you can offer a range of sanitation filters to eliminate cross-site scripting etc. This would open up the possibility of using PW for, say, social content management.

The actual plumbing for this is just a few lines of code, and there are plenty validation and filter libraries to plunder. You'd be getting a lot of functionality for a small investment! I guess you'd tuck this out of the way on the advanced tab. For types with limited validation options such as numeric fields you could improve usability by offering the validation options as fields on the form instead of declarations. For text fields, the declarative string would be the most flexible, I think. You could offer popup help so users could check their options without leaving the form.

Personally, I would find this pretty useful. Why not help content creators achieve consistency and correctness?

[ryan]

It really depends on the fieldtype, but for text fieldtypes the idea is to be flexible. There is no bad data with trusted input, just bad output. If you don't want to allow HTML for instance, then you would select textformatters such as 'strip tags' and 'html entities'. But we avoid modifying what the user enters as we don't want to limit PW to one use over another. Specific validations are for specific fieldtypes. PW's basic fieldtypes are meant to be open-ended, as much as possible. For specific data needs, the intention is that you would use (or create) a specific fieldtype rather than expecting it from a general purpose one. Taking that zip code example, that's a pretty specific input -- if we are taking a zip code as input, we'd ideally use a zip code fieldtype so that it knows how to validate zip codes and doesn't take up any more storage space in the DB than what is needed for a zip code. However, I'm not at all against adding more validation capabilities to the basic fieldtypes, and like the examples in your approach. It certainly can't hurt anything so long as it's optional, so will plan to continue adding more validation possibilities to the basic fieldtypes.

[MatthewSchenker]

Greetings,

I'm now turning to validation for my ProcessWire sites and this seems like a good discussion on the matter.

...I'm very cautious about enforcing any rules that could put a page in a state where one couldn't save. If I have an integer field that I know has to be between 1 and 3, then I'm usually enforcing that in my template code, where the need for the boundary exists.

If the user is a fellow admin, I can see that validation is less of a worry. But for sites where I am allowing submissions from a wider audience, I definitely think the priority shifts from "make sure the page can be saved" to "make sure the data is correct."

I'm fine doing validation in the templates, as Ryan suggested. But is this for front-end forms (i.e., not for the admin interface)?

Thanks,

Matthew

[ryan]

Quite a bit of validation options have been added to the Inputfield types since Form Builder was introduced. For instance, InputfieldInteger now supports min/max values and HTML5 number type. Text fields now support regex validation. Most of this is in the dev branch though.

[MatthewSchenker]

Hey Ryan,
It's always very reassuring to see your name and face in a discussion!

I'm very glad to know that these validation elements are in the development branch. I'm going to start testing/using that one some more!

I guess this is one more reason to be excited about the next version.

Thanks,
Matthew

[Hari KT]

Hey,

I am wondering if PW itself have some sort of validation. Not regarding the FieldType. I was having a custom registration form and want to validate the username, email, password etc. If a validator is there I can use the inbuilt one, else I wish to got with the Aura.Filter which is independent.

[Martijn Geerts]

All inputfields are sanitized, except for password.

If you want to sanitze and not really validate. ProcessWire has the $sanitizer.

I do use the $sanitizer->email($value) sanitizer as validator to, as it blanks out invalid adresses.

if (!$sanitizer->email('bweugh')) {
    echo "Oops I'm something else";
}

[Soma]

What I use in front end from is doing my own validation when needed, some fields are already validated like password inputfield or email inputfield.

On a register form I have something like this for example, combined with jquery validation for client side validation...

if($input->post->register) {

    // processes form, validates and add error messages to fields
    $regform->processInput($input->post);

    // check email unique if any
    if($sanitizer->email($regform->get("email")->value) != '') {
        if(!$helper->isUniqueUserEmail($regform->get("email")->value)){
            $regform->email->error(__("Diese E-mail wird bereits verwendet."));
        }
    }

    // check username format only a-z0-9-_.
    $username = $sanitizer->pageName($input->post->username, Sanitizer::translate);
    if($username != $input->post->username){
        $input->post->username = '';
        $regform->username->error(__('Benutzername ist nicht im richtigen Format.'));
    }

    // check if username ($user->name) unique
    if(!$helper->isUniqueUsername($username)){
        $regform->username->error(__("Dieser Benutzername wird bereits verwendet."));
    }

    if(!count($regform->getErrors())){
...

I have helper functions like isUniqueUsername etc, that is at the same time used in jquery validation via ajax.

Depending on your knowledge and how fluent you are with PHP and PW, I find it easy to add my own methods or checks where needed and found use of any external validation class not needed (was using that before).

[Soma]

What you can also use, where PW does its validation is the processInput() of inputfields. You can hook into it and do you validation and add errors as needed.

$form->addHookAfter("processInput", null, "formValidation");
function formValidation($event){
    $form = $event->object;
    $email_inputfield = $form->get("email");
    if($email_inputfield->value == "spam@hotmail.com"){
        $email_inputfield->error("We got you!");
    }
}

// direct on the inputfield type
wire()->addHookAfter("InputfieldText::processInput", null, "validatText");
function validateText($event){
    $inputfield = $event->object;
    if($inputfield->name == "username"){
       ...
    }
}

And there's also the HTML5 regex pattern that can be used.

[Hari KT]

Sorry @Martijn Geerts , @Soma ,

Not sure how I missed these topics. I thought I didn't got any reply. My bad.

By the way what I was doing is like some custom input fields for username , password.

User Name : <input type="text" name="username" />
Password : <input type="password" name="password" />

And I was going to register the user via the api . So basically I am not sure whether it is good idea to create a form via PW api form and render . Or make use of the api and the validation via some validation library .

Thank you.

[spackmat]

It's an old thread, but I came along this with my first PW steps. I found a syntax of conditions, when a field is required or not and I'm wondering, why this neat stuff is not available for simple validation tasks. For example to ensure that some date is after one or several others.

In my first case this is an optional "publicly available end" field that must be after "publicly available start" and after "shown publication date". The selectors used in the templates later depend on those constraints to find the correct pages. Basically I want to enforce my users to input consistent data. Another usecase is an image field, where I want to validate the images count (min 1, max 5 images). In that case I could write a notice to the users that more than five images are ignored, but a validation would be the better solution from my point of view.

Am I missing something? In my other life I develop Symfony-based applications with a lot of (soft/noticing and hard/failing) constraints between input fields and Im clear about the point, that a CMS cannot provide such complex constraints on its own (without using custom field types). But some basic constraints should be usable out of the box and there is already a system, which is limited to decide, if a field is required.

Greets,
Gregor

[gmclelland]

Hi @spackmat, welcome to the forums.  Some fields have basic validations that can be configured in the field's settings.  In more advanced cases like you mentioned, it would be better to handle those with hooks to add your own validation.

[LostKobrakai]

Afaik ryan's position on hard constraints in form validation is that they don't work and you should expect things to be off. Like what happens if you change a input field from min. 1 to min. 2 images. Suddenly all your pages are in an invalid state and there's no form validation, which can save you from that.

[spackmat]

Thanks for your quick answers!

@gmclelland the concept of hooks in the backend sounds interesting, I'll have a look into that later.

@LostKobrakai I know about this view, but I strongly disagree with it. From my perspective, there are two layers of handling invalid data. The one at input time, where UX and software ergonomics say that systems should give valuable feedback to the users in case of invalid or problematic input (and try to fix minor problems automatically, but also with feedback). And the one at computation/output time, where the system should handle all data fault tolerant. If I had to choose only one, it would be handling invalid data at output time (at least for the case, where the validation rules fail or change). But there is no restriction on that, so my best case is to handle invalid input on both sides. So Ryan's point is completely valid: Nobody wants systems, where some invalid data breaks the whole page's output. But that doesn't invalidate arguments towards input-validation, which is another issue.

And about silent fixing: A system cannot decide, what a user wanted. For the example of end date before start date: Did the user just mixed the fields up? Or is one of the dates simply mistaken? In the first case, a simple fix would be helpful, but in the second case in the end, the user did one mistake and the silent fix made it worse and now both fields store data, that the user didn't want to input. A system then has to ask the user what his/her intended input was and therefor input validation ist a must, since output fixing cannot ask questions.

I was wondering, why there is an engine using expressions to decide, if a field is required or shown at all depending on other fields values, but not if its own value is valid. It would be a great addition to make this engine available for simple validation tasks.

[LostKobrakai]

5 minutes ago, spackmat said:

I was wondering, why there is an engine using expressions to decide, if a field is required or shown at all depending on other fields values, but not if its own value is valid.

Because it's (loosely) based on the selectors engine you select pages by and because it was never intended to serve bigger validation purposes, but more of a convenience tool to quickly show/hide things based on some external data. Having that it was a quick thing to add "required if" as another option besides "show if". So all in all: historical reasons.

10 minutes ago, spackmat said:

It would be a great addition to make this engine available for simple validation tasks.

Real validation needs to run on the server side, and while show if and required if is checked against on the server side the main functionality it added at the time is the client side in showing/hiding fields and or showing/hiding the required asterisk. So mixing both is in my opinion a ill fit without changes to implementation.

[spackmat]

7 minutes ago, LostKobrakai said:

Because it's (loosely) based on the selectors engine you select pages by and because it was never intended to serve bigger validation purposes, but more of a convenience tool to quickly show/hide things based on some external data. Having that it was a quick thing to add "required if" as another option besides "show if". So all in all: historical reasons.

Real validation needs to run on the server side, and while show if and required if is checked against on the server side the main functionality it added at the time is the client side in showing/hiding fields and or showing/hiding the required asterisk. So mixing both is in my opinion a ill fit without changes to implementation.

Again, thanks for your quick and valuable feedback! I think I understand the case. Which doesn't solve the missing input validation, so I'll have a look at those hooks.

How do others solve this? My users, including me in an editor-role, aren't that good and input invalid data from time to time that must be handled well.

P.S. we are replacing a really bad implemented Sharepoint-based website with a better one based on Processwire. One of the pain points of the Sharepoint system is that it has a lot of pitfalls where things can go really wrong and almost no valuable feedback for our editors on this. So our replacement has to be much better in that point, because we promised it. And it has to be as convenient for the editors as the old TYPO3/TemplaVoilà based site was, that we had before the Sharepoint site and that worked well for 10 years.

[LostKobrakai]

Hooks are the way to go for validating business requirements.