AutoContent generator

[adrian]

Hi everyone,
 
This is an early (but mostly functional) version of a module that I am experimenting with for automatically populating all empty fields on a page.
 
It is designed for use during site development and should be uninstalled once the site is live.
 
https://github.com/adrianbj/AutoContent (anyone have an idea for a better name?)
 
It currently supports the following field types:

• Text - outputs "This is an example Field Label", where "Field Label" is the label for the field
• Textarea (RTE and plain) - configurable content from http://loripsum.net/ and embedded images from http://lorempixel.com
• Datetime - honors output formatting and returns random date/time with configurable min/max datetime
• Integer - honors min and max settings
• Float - honors min, max, precision settings
• URL - returns a random URL
• Email - returns a random email address
• Image - you can specify image category (people, nature, business, etc), and the range for min/max number of images to generate.
• Page - creates runtime selected child page content for output - not well tested yet.
• MapMarker - not well tested yet!

I will be adding support for Profields and other selected 3rd party fieldtypes shortly.
 
Highly configurable content is provided by:

http://loripsum.net/

http://lorempixel.com/

https://github.com/fzaninotto/Faker
 

Generated content can be localized to your region - very handy for addresses, people's names, phone numbers etc:

Many fieldtypes are configurable via the Input tab on the field's config settings, eg:

Textarea:
 

 
Text:
 

 
Images:
 

Datetime:

 
In general I am designing this to work with minimal/no configuration so you can install and have all fields on all pages on your site immediately populated for testing layout and styling without the need for manually adding dummy content.
 
There is also a dummy page batch creator and deletor (automatically tracks and deletes just the dummy pages) available from the module config settings page.
 

 
Please let me know if you think you will find this useful - trying to get an idea of how much time to put into support for additional field types.

[LostKobrakai]

The faker api is also really nice to generate dummy data. See the "hosted microservice" section in the readme: https://github.com/marak/faker.js

[adrian]

The faker api is also really nice to generate dummy data. See the "hosted microservice" section in the readme: https://github.com/marak/faker.js

Thanks for that - should come in very handy, especially for the address components of the MapMarker fieldtype.

[Martijn Geerts]

How can I double like your POST Adrian ?

[Raymond Geerts]

About a week or so ago i was thinking, it would be nice to have a module that can add dummy content

Maybe an early feature request (if its not already in your module).

An auto page generator, where one can set:

- the amount of pages to create

- pick a parent page

- pick a template

For example:

- 15

- /news/

- news-article

Then it will create 15 news articles with Lorem ipsum text and dummy images

[tpr]

How can I double like your POST Adrian ?

Ask Reno Tom

[adrian]

About a week or so ago i was thinking, it would be nice to have a module that can add dummy content

Maybe an early feature request (if its not already in your module).

An auto page generator, where one can set:

- the amount of pages to create

- pick a parent page

- pick a template

For example:

- 15

- /news/

- news-article

Then it will create 15 news articles with Lorem ipsum text and dummy images

Thanks for the thoughts Raymond!

Are you are talking about dummy pages that are not actually in the database, but are only rendered on the front-end? That sounds intriguing to me, and I think might be a cool addition to this module - will have to think more about how best to implement.

 

Or, are you talking about adding real pages? I think it might be confusing the role of this module to get into generating pages. I think perhaps what I would recommend is using the "Add" mode of BatchChildEditor to create the pages, then if you have AutoContent installed, the dummy content will be added when the pages are rendered. I could potentially add an "Auto Add" mode to BatchChildEditor which created the x number of pages with random names to make this even quicker. The other reason I wouldn't want to add pages with AutoContent is the subsequent need to delete once you no longer need the dummy pages. BatchChildEditor makes this quick and easy.

[adrian]

The faker api is also really nice to generate dummy data. See the "hosted microservice" section in the readme: https://github.com/marak/faker.js

@LostKobrakai - have you used the microservice? I started setting up support for the MapMarker fieldtype, but the response time is just too long. I have tried both curl and get_file_contents. Even calling it directly in the browser is slow - check out: http://faker.hook.io/?property=address.streetAddress&locale=en

And I would have to call at a minimum three times for streetAddress, city, country

I think I am less worried about getting different random data - if the API call is going to be slow, then I think I would rather keep things fast and just define standard default automatic values.

Any better experience at your end? Or ideas for another service?

[LostKobrakai]

I haven't used it by myself, but I though I've seen it in a few laracast videos, as it seems like I falsely mixed this (https://github.com/fzaninotto/Faker) up with faker.js, which I already knew at that time.

[adrian]

I have just committed a new version that makes use of the Faker library (https://github.com/fzaninotto/Faker) - thanks to LostKobrakai for pointing that one out. I have included the library with this module for the speed benefits, compared to calling the API.

Faker allows for automatically generating a wide variety of custom content targeted to be relevant to each particular field. The module attempts to match each field name to a Faker property automatically, but you can also link fields to specific ones if needed. For example, if you have a field named: "last_name" it will match the Faker "lastName" property and it will output things like "Smith", "Jones" etc.

I have also added a module wide config option for setting the locale. This will determine the content that is generated. For example, people's names, street addresses, etc will be relevant to the locale.

At the moment I am still using the loremipsum.net API for textarea content because I really like all the styling/content options which are not available in Faker, but this may change in a future iteration of this module.

I may also add Faker to Integer/Float etc fields to allow for more flexibility.

There is also initial support for the MapMarker field.

Please test this new version and let me know if you have any suggestions.

[teppo]

Or, are you talking about adding real pages? I think it might be confusing the role of this module to get into generating pages. I think perhaps what I would recommend is using the "Add" mode of BatchChildEditor to create the pages, then if you have AutoContent installed, the dummy content will be added when the pages are rendered. I could potentially add an "Auto Add" mode to BatchChildEditor which created the x number of pages with random names to make this even quicker. The other reason I wouldn't want to add pages with AutoContent is the subsequent need to delete once you no longer need the dummy pages. BatchChildEditor makes this quick and easy.

It's true that this might be out of the scope of this project, but it is a great idea. Nothing beats real content when working on a site, but when that's not available yet (or you're not working on an actual site but a module or something like that) it would be cool to be able to easily mock up a given amount of (real) pages, and later remove them just as easily.

 

This would be particularly helpful for testing UIs, performance, and scalability.

Just saying, not trying to put any pressure on you Adrian..

[adrian]

It's true that this might be out of the scope of this project, but it is a great idea. Nothing beats real content when working on a site, but when that's not available yet (or you're not working on an actual site but a module or something like that) it would be cool to be able to easily mock up a given amount of (real) pages, and later remove them just as easily.

 

This would be particularly helpful for testing UIs, performance, and scalability.

Just saying, not trying to put any pressure on you Adrian..

No worries - not taking suggestions as pressure at all

I think the idea of automatic page generation - I am just not sure the best approach yet. Maybe it's a simple matter of maintaining an array of page IDs that are created so that I can offer a "Cleanup all automatically created pages" button, and obviously also run this cleanup when the module is uninstalled.

Now to determine the best way to create the pages - should the "create tool" be added to the settings tab of each page? Or maybe it should be part of the main module config settings and do as Raymond suggested - choose parent, template, and number of pages to create.

Maybe it's actually pretty simple to handle.

Any further thoughts before I dive in?

[Raymond Geerts]

Thanks for the thoughts Raymond!

Are you are talking about dummy pages that are not actually in the database, but are only rendered on the front-end? That sounds intriguing to me, and I think might be a cool addition to this module - will have to think more about how best to implement.

 

Or, are you talking about adding real pages? I think it might be confusing the role of this module to get into generating pages. I think perhaps what I would recommend is using the "Add" mode of BatchChildEditor to create the pages, then if you have AutoContent installed, the dummy content will be added when the pages are rendered. I could potentially add an "Auto Add" mode to BatchChildEditor which created the x number of pages with random names to make this even quicker. The other reason I wouldn't want to add pages with AutoContent is the subsequent need to delete once you no longer need the dummy pages. BatchChildEditor makes this quick and easy.

As Teppo mentioned, Nothing beats real pages to work with when creating a new website. For example for setting up (news) lists, with pagination, prev/next buttons on a detail page, front-end search and that kind of features in a site require at least some content to be available.

As with dummy i just ment real (temporary) pages to work with. As you point out, these temporary pages should also be able to easily deleted, for example with the module GUI and on uninstalling the module.

Altough if this AutoContent module can work with the BatchChildEditor, that might just do the job too and be enough to quickly add and remove some temporary pages.

[adrian]

Ok guys - the module now supports batch creation of testing pages. There is a one click option to delete all these pages once you are done. This is also called when the module is uninstalled, although thinking about this some more, I might need to check to see if there are any fields with actual content before deleting as I can see someone repurposing one/several of these "testing" pages for actual content during development. I'll look into this more tomorrow.

EDIT: I just added "locked" status to all auto generated pages - hopefully this is enough protection against someone trying to add real content to these pages - any thoughts? Maybe I should check to see if the locked status has been removed when deleting the pages? That way superusers could remove the lock and edit if they really wanted to and know that the page wouldn't be deleted - any thoughts?

[adrian]

teppo, you mentioned that it would be cool to be able to automatically generate pages to easily test scalability, so this question is directed mostly to you!

With the new automatic test pages feature, I am storing the IDs of the created pages in the module's data field - this is so I can delete them all as requested.

The module data field a MYSQL text field with a limit of 65,535 bytes. Based on my calculations, with UTF-8 numeric characters (0-9) being one byte each and PW page IDs having a minimum of 4 characters, eg: 1001, the most I will be able to store is 16383 page IDs. Of course when you get to that many pages, the page IDs will contain 5 characters, eg, 12005, which brings the number down to 13107 pages. 

So, is 13000 pages enough for most of your scalability testing, or can you envision wanting to test more than that? I am guessing it will be fine, but thought I should ask

On another note, I just implemented my idea for only deleting auto generated pages if their status is still locked, which means it is now possible to unlock a page and edit it with real content and know that it won't be deleted when you delete all auto pages. I am still not convinced that using the locked status is the best approach here - definitely open to ideas!

[adrian]

Just committed a new version which supports Page fields.

Like all other field types (except textArea which has other options), check the Faker property under the Auto Content Options for the page field. If there is no automatic match, or no manually entered property, it will fallback to 1 to 3 word lorem ipsum titles.

This became pretty complex pretty quickly because I wanted to support output of all fields from the faked page fields, so please test and let me know if you have any problems.

[adrian]

Another small update that swaps out placeholder.it for lorempixel which means you can now have real images and you can even configure from their categories to tailor the images to your site:

• Abstract
• Animals
• Business
• Cats (although I almost want to remove this option)
• City
• Fashion
• Food
• Nature
• Nightlife
• People
• Sports
• Technics
• Transport

I also fixed the image output - I had forgotten to return a pageimage, rather than a path.

EDIT: because this now returns a pageimage, the images are copied to the page's files folder so I have added a cleanup routine to remove all images when the module is uninstalled. I thought I had previously created a pageimage at runtime without it being added to the assets/files folders, but for reason I don't seem to be able to do that - did I imagine that?

[adrian]

I thought I had better make some improvements after teppo's very generous, but slightly premature publicity in PW weekly this morning

I really wanted the images working better, so you can now specify how many images to randomly generate. Also, images are now cached so even though the first view of a page may be a little slow (while images are grabbed), subsequents loads will be as fast as normal.

I will be plugging away at this module as I find inspiration for new fieldtype support, and snippets of time when procrastinating, but please let me know if you have any specific requirements for auto content generation.

[teppo]

I thought I had better make some improvements after teppo's very generous, but slightly premature publicity in PW weekly this morning

Sorry for being a bit hasty there! The module seemed mature enough to be mentioned, and I've personally found that often good stuff gets buried for way too long just because it's not "fully polished" yet. Early announcement combined with a little warning tends to work out just fine.. and it does also have the added benefit of getting feedback at a stage where making changes doesn't necessarily mean hours and hours work being wasted

[adrian]

Absolutely no apology necessary - I am honored that you featured it and completely agree that early attention and testing is a great benefit to myself and any potential users, so thank you for including it and for PW weekly and everything else you do for PW!

[adrian]

New update to support random images within RTE textarea fields.

You can specify whether to include images, and the positions that they might be placed in.

You can reload the page several times and the images will change positions/alignment randomly.

I would very much like to hear suggestions from anyone using this if they have any ideas for improvements - do you want more configuration options - eg size of the embedded images, captions, etc?

[netcarver]

Hi Adrian,

Thank-you for this module - I've been looking forward to trying it out for over a week and now have my chance. I just installed it in a PW 2.7.3 site and have given it a cursory try. Unfortunately, it seems to conflict with horst's WireQueue module.

I have several queues set up and they all use the WireQueueFile queue type. When I installed AutoContent, the WireQueue children become inaccessible in the page tree, and via the API. Removing your module restored everything to working order, so I enabled Tracy and re-installed AutoContent to find out why.

In the end, it turned out to be the auto-content generator getting called for the fields in the WireQueue templates.  I changed the AutoContent hook code to the following and all started working with my site again.

    public function autoContent($event) {
        $p = $event->arguments(0);
        if($p->template == "admin") return;
        if($p->template == "wire-queue-tools") return;   // Added for WireQueue compatibility
        if($p->template == "wire-queue-storage") return; // ditto
        $field = $event->arguments("field");
        $value = $event->arguments("value");

        //if not empty then display available real content
        if(!is_object($event->return) && $event->return != '') return;
        if(is_object($event->return) && count($event->return) > 0) return;
        //if MapMarker set to default then exit
        if(isset($event->return->lat) && $event->return->lat != '0.000000') return;

        $event->return = $this->generateContent($p, $field, $value);
    }

Not sure what the best approach to handling this kind of needed exclusion might be, perhaps a multi-select of templates to be excluded from auto-content generation?

Now, on with the trial of this module!

[adrian]

Hey Steve - thanks for your interest in this module and thanks for figuring out the issues with WireQueue - as you mention it will probably happen with others too so I need a way to exclude as necessary, or maybe it can be automatically dealt with - I'll take a look.

The one main issue still remaining with this module is the slowdown when a template includes page fields - at the moment the module generates content for all fields from the templates of these page fields - this can take some time. I need to figure out a better way to do this. Connected to this is the delay from calling the http://loripsum.net API. Even though I make use of Faker locally for most things, the RTE fields still use http://loripsum.net because of their random text/image options. I would like to recreate this functionality directly with the Faker API as it would make it much quicker.

Otherwise I think it is working well, although I would still like to add support for Profields etc.

[adrian]

I just committed a new version - it now ignores pages under admin and trash. Before it just excluded pages with the "admin" template. I think this should take care of the problem with WireQueue. 

I also fixed a bug with images being blank in RTEs when you haven't manually specified the max number of images for the images field that is used.

[Joynal]

Hi guys, I just installed and tested this module. I'm trying to generate page with dummy data. AutoContent module only fills page title field and other fields remain empty. Any idea? Thanks in advance.