Revamped Modules install interface

[adrian]

As first discussed here: https://processwire.com/talk/topic/20596-new-post-new-pw-website-ready/?do=findComment&comment=178781

Here is the next version of the searching interface. You can see how it automatically populates the module class name in the field from the matches based on what you have typed. This is basically ready to go if Ryan is willing to incorporate.

Now we just need to figure out a browsing/filtering interface to make it easy to discover modules base on categories and keywords.

[dotnetic]

I hope to come up with a mockup/prototype the next few days/weeks. Here are my thoughts:

We can use @Soma Module Manager as a starting base for getting information about the modules overview and each individual module.

There is an API for getting information about modules available at http://modules.processwire.com/export-json/

or for a specific module like TracyDebugger http://modules.processwire.com/export-json/tracy-debugger/?apikey=pw223
sadly, we can't get the install instructions or readme via this API. Maybe we can use something in addition to get more detailed information.

Of course this information should be cached, and there should be an option to force reload the information.

My approach would be a nice looking filterable overview, which also can be sorted like on the modules directory

Each module should have an "Install" button and a detail button which opens a panel/popup that shows extended information about this module (for example the readme)

Filtering could be very simple but I am in love with filtering like isotope or mixitup. Luckily Uikit also comes with a component for similar filtering: https://getuikit.com/docs/filter

We also should stick to default AdminThemeUikit styling, so it integrates nicely into PW and maybe one day replaces the old (actual) installer.

[adrian]

5 minutes ago, jmartsch said:

There is an API for getting information about modules available at http://modules.processwire.com/export-json/

Yep, already using that to populate the autocomplete entries in the screencast in the first post. I am caching the results already so we can use this cached data for the autocomplete/search shown above as well as to populate all the other data for modules in the proposed browsing/filtering interface.

8 minutes ago, jmartsch said:

sadly, we can't get the install instructions

Do we really need install instructions given that this will install the module when selected?

Looking forward to getting this all going!

 

[dotnetic]

2 minutes ago, adrian said:

Do we really need install instructions given that this will install the module when selected?

 

No we don't need that, but often there is additional info in the README, which also includes install instructions along with extended information about the module.

Mainly I want to get that additional information

[dotnetic]

@adrian It would also be nice if we could install multiple modules at once like you do with your https://github.com/adrianbj/ProcessModuleToolkit

So we have to be able to select multiple modules, then click install, and magic happens ?

[dotnetic]

First work in progress version ?

Built with vue.js for fast DOM updates. Needs to be AJAXified and more details. So maybe wait a little bit with feedback. I know there is lots of information missing.

OK, now I go to sleep, because it's 4:30 am and I am invited to a birthday today.

[Pete]

Hehe so much for not having much time ? Since you guys are a million miles faster than I am what I'll try to do today and tomorrow is an interface mockup for how I think it could look with categories and thus favourites idea so we've got a few ideas to pull from.

In terms of instructions for modules - they're hit and miss as to whether they're on a GitHub readme.md file or entered into the Modules directory itself, but I think the code on the Modules directory can easily be tweaked to pull from either source.

I also don't think that there are so many modules that if we thought there was some other essential functionality that was required (like something that HAD to be read before install) that there could easily be an extra field added to the DB to flag that up in a consistent manner. Sure there are a lot of modules, but for better or worse there are a lot of ways to store the config info too so maybe the Modules database itself is the only consistent way around that if we need to?

Will post more later. Got to paint a kitchen first (yay ??)

[bernhard]

Hi Adrian, this looks great, thank you! I'm sure it will save me lots of time and clicks, because I can often not remember the exact name and have to head over to google, click, click, copy, paste etc...

10 hours ago, adrian said:

Now we just need to figure out a browsing/filtering interface to make it easy to discover modules base on categories and keywords.

I don't think so. Why should we rebuild what we already have? The modules directory is exactly for that purpose. I don't see any benefit in building a new UI here.

8 hours ago, jmartsch said:

@adrian It would also be nice if we could install multiple modules at once like you do with your https://github.com/adrianbj/ProcessModuleToolkit

So we have to be able to select multiple modules, then click install, and magic happens ?

I know what you mean and I partially agree. Just FYI I'm building a new migrations module right now. And this would make such things easy. Similar to the PW KIckstart recipe approach you could easily create a module that defines which modules to download and install. And you could also define your default settings - which would not be possible with your suggested approach (or at least would need another step to implement).

[Pete]

The reason for "rebuilding" the categories layout into the module (though it's not a hard build to be honest) is that it lowers the barrier to entry further. It's just a convenience thing, but also a way to prevent new users having to go back and forth straight after installation. It's also how some other CMS's do it so it would also be intuitive for those users and just a nice bonus for the rest of us.

I guess they will need to go to the PW site anyway for things like docs, but I think installation and setup should be made as simple as possible ideally.

Nothing about the current setup is hard of course, but less clicks and easier/quicker module discovery would be a nice bonus - I'm a PW veteran and I honestly don't scour the modules directory for new modules often, but if there were a feed of recommended or new modules within PW itself that would help lazy people like me to not overlook things like Tracy ?

[dotnetic]

1 hour ago, bernhard said:

I don't see any benefit in building a new UI here

Well, the improved module manager/installler shows which modules you have installed and which can be installed or upgraded with one click (ok actually it's 2 clicks).

Why should you leave your PW installation to go to the ProcessWire modules website, of which you have to be aware of, search for a module, copy or remember the module name, go back to your ProcessWire installation, paste the module name, click on "get module info" and finally install the module?

Make it easy for everybody!
Lower the barrier for new users, and make it easier for existing users to find an install modules. 

That is one thing, that many other frameworks/CMS's have by default. Like ModX, WordPress or PrestaShop.

Right now I am making good progress with the development of the module. I will post an updated screenshot a little bit later or tomorrow. Please note, that this is actually working, it is not only a mockup.

Data is fetched and cached and then being used to display the modules with all information:

Here are some more features I am planning to integrate:

• Live-Search/Filtering
• Quick uninstallation/disabling of a module
• Top modules label, so users can see modules that are being used the most (based on the likes, sadly we have no module installation counter)

I will put it up on Github, as soon as it is in a presentable state, and you guys can improve it further. You can also write your thoughts and wishes here.

[Pete]

I'm absolutely in awe at the speed you two are working at. I feel like a snail in comparison ?

It's a separate thing so not one for right now but some similar code would be involved - the installer could do with reworking to allow easy presentation and installation of site profiles a bit like you can install a WP theme easily. You should be able to choose a site profile during installation (maybe thumbnails for each and click to popup the information about what it is and what it contains along with any caveats) and make that side of things easier too. I'm far from the only person to mention this, but I do know it's been mentioned for so many years now off and on and it lowers another barrier for entry in that people get to code that's relevant to them that little bit quicker.

I think that one change, albeit not a simple one, would lead to more people producing site profiles as the current steps involved are again more numerous than they could to be and site profiles aren't paticularly visible as they are lumped with other modules in the directory.

Also a note to say you guys probably want to make sure your tweaks to the modules interface don't show site profiles from the modules directory or that could get confusing.

[dotnetic]

There is an option to hide certain categories from listing, this could also be made configurable via module settings, but this is not my priority right now.

[adrian]

2 hours ago, jmartsch said:

with the development of the module.

Thanks for all your work on this @jmartsch - are you planning this as a third party module? I assumed we were trying to integrate these changes into the core modules installer interface, at least that's what I have done with the above autocomplete interface. I think it would also make sense this way because then we can both share the cached modue data json. On that note, are you getting the data in batches? I ask because the json feed is limited to 350 modules and we have more than that.

[dotnetic]

@adrian I am developing it as a module for now, but at a later point will replace ProcessModule with my code.

For getting the JSON data I used code from Somas Module Manager, which seems to get only the 350 items you mentioned ? We have to think of a way, to get them in batches, or @ryan has to change the limit. I did not thought about the complications yet when the limit is higher.

I got filtering working, but not in the way I want. Some things turn out to be more complicated than I thought.

Anyways, here's a lil teaser for you guys ?

[kongondo]

1 hour ago, jmartsch said:

We have to think of a way, to get them in batches, or @ryan has to change the limit. I did not thought about the complications yet when the limit is higher.

Maybe Ryan can introduce pagination in the public API ? E.g.:

../exportjson?limit=10&start=350

This is not uncommon in high-traffic public APIs. 

[bernhard]

Wouldn't it be a lot more efficient and easy to serve one json file with all modules? At the moment we have around 25x23 modules in the directory, that's 575. For example displaying those with RockGrid would be easy and fast and filters would already be there.

Not saying RockGrid is the best solution, just throwing in a concept for discussion you might not have thought of yet (meaning doing the work on the client side and not on the server). And this JSON could be served via procache, so it would need very little resources.

[dotnetic]

Yes, it would ideal to serve a JSON file with info for all modules. But thats a lot of information, so it would be good for the module installer to define, which fields should be queried, so the JSON file is minimal.

As I partied until now, I think I am not going to work on the module today. Have to get a lil bit of sleep tonight, ehm, day, because I didn't sleep last night.

I am hoping for an answer from @ryan about the limit for the JSON file.

[Pete]

I'm wondering if it needs to pull down data for all files in one hit anyway? If it starts off with categories and totals and just the first XX amount of modules listed by default it could fetch the rest as the filters/searches change surely?

It would mean a change to the code on the modules directory but I'm happy to look at that for you - maybe set up a separate feed for our tests if you only want to pull a small amount of data until someone clicks on a module for more details.

[Pete]

I don't think interactive requests are out of the question either instead of relying on static JSON on the server, but I can see the benefit in separate cached JSON files on the server for category totals and all the modules per category in the short term.

[bernhard]

In one of my RockGrids I have >3000 entries like this:

  {
    "id":"17627",
    "lists":"56708,59126,61021",
    "firstrevenue":null,
    "type":"2",
    "title":"John Doe",
    "prefix":"BA",
    "forename":"John",
    "surname":"Doe",
    "suffix":"MSc",
    "companyname":null,
    "tagsjson":"[9952]",
    "id":null,
    "short":null,
    "birthday":"xxxx-xx-xx 00:00:00",
    "mobile":"12345678",
    "tel":null,
    "fax":null,
    "email":"john@doe.com",
    "street":null,
    "zip":"1234",
    "city":"Wien",
    "country":"AUSTRIA",
    "poc":null,
    "provision":null
    "camefrom":null,
    "camefrom:title":null,
    "poc_id":null,
    "poc_short":null
  }

The json file is 194kB gzipped and the grid performs well.

RockGrid does already have a batcher feature built in. It would be easy to select modules and then click install and watch a progressbar while each of the modules is installed one by one. It would even be easy to define some kind of presets that should be installed (as it would just be an array of ids - or classnames). What could be a problem in this scenario would be the order of the modules. But I already have an idea how that could easily be solved ? RockGrid could serve as interface to select the wanted modules and then populate a ASM select field where entries would be sortable and then could be installed in exactly that order.

A category filter would be a simple double-click.

Not sure how we could treat modules that are not part of the modules directory, though.

And on the other hand RockGrid is quite a beast and while I'd love to see it being part of the core (it could be such a great extended page reference field), as long as it is not part of the core it would be quite an overhead to the current version (regarding library dependencies etc).

@Pete how hard would it be to offer a JSON that contains module data as an array of objects like shown above?
 

[{obj1},{obj2},{obj3}]

 

[Pete]

Theoretically I guess pretty easy. But what I also think it would be nice to do is to be able to click on a module and view the instructions and other readme data without leaving the PW admin which is where the data would balloon massively if it were in one large file. I guess that could be a JSON file per-module for the more detailed information as they would get called less and all of these files could of course be generated when the module gets updated in the directory.

You raise a good point on modules that aren't in the directory - I guess it's more incentive for people to add them ?

[bernhard]

Well. What if we defined an url to a readme file for every module? The initial data could be provided as one single json and the readme could be linked from there, eg:

{
  classname: "FieldtypeRockGrid",
  readme: "https://raw.githubusercontent.com/BernhardBaumrock/FieldtypeRockGrid/master/readme.md",
  ...
}

For modules in the modules directory we could serve the readme directly when the module is requested via urlsegment, eg

https://modules.processwire.com/modules/fieldtype-rock-grid/readme could serve just the raw markdown content of the module. Then we could request and parse details for modules via AJAX.

But maybe @Soma has some input for us, as he worked on such a modules manager long time ago...

[dotnetic]

@bernhard An extra field for a Readme would be an ideal solution, but then every author has to update his modules information in the modules directory. So there might be many modules without a readme, because they are not maintained anymore. The module 

 overcomes this, and tries to get the needed information automatically. So maybe code from this module can be reused for my module. Have to look into this. 

[dotnetic]

On 1/13/2019 at 8:12 AM, Pete said:

I'm wondering if it needs to pull down data for all files in one hit anyway? If it starts off with categories and totals and just the first XX amount of modules listed by default it could fetch the rest as the filters/searches change surely?

The reason why I would need data all modules is, that I want to filter and search the entries on the frontend side, instead of making a new AJAX request, because it is much faster.

There are advantages and disadvantages with frontend only filtering, but with backend filtering also. I think frontend filtering is the best way for quick searching and filtering. That is also how I do it on https://www.p-jentschura.com/de/produkte/

 

[Pete]

5 hours ago, jmartsch said:

The reason why I would need data all modules is, that I want to filter and search the entries on the frontend side, instead of making a new AJAX request, because it is much faster.

That makes sense. As others have pointed out to me as well it's not a huge amount of data at the moment either, especially if you are just grabbing the essentials to start with and then if we wanted to do anything fancier like grab the readme then that can be a request when you go to view a module's details maybe.

The modules that are on Github - it's down to the user whether they have a readme.md file so I guess that's a bit hit and miss. For non-Github modules in the directory they have an "Instructions" field we can get that data from.

I think what this might highlight during this process is how many modules there are with no instructions that are able to be fetched from the server - in which case we can get a list together, nudge the authors and tighten up the submission process - I don't mind forcing either instructions or a basic readme file - not sure what everyone else's view is on this?