Recommended Posts

Part 1 of a 2 part Module & Service Reveal.

I'm currently working on a new module: ModuleReleaseNotes that was inspired by the work I originally did on making Ryan's ProcessWireUpgrades module "release" aware. In the end, I decided to ditch the approach I was originally taking and instead work on a module that hooked in to the UpgradeConfirmation dialog and the module edit page.

Aims

My aims for this module are as follows...

  1. Make discovery of a module's changes prior to an upgrade a trivial task.
  2. Make breaking changes very obvious.
  3. Make reading of a module's support documentation post-install a trivial task.
  4. Make module authors start to think about how they can improve the change discovery process for their modules.
  5. Make sure the display of information from the module support files/commit messages doesn't introduce a vulnerability.

Looking at these in turn...

Making discovery of a module's changes prior to upgrade a trivial task.

This is done by adding a "What's changed section" to the upgrade confirmation dialog.  This section takes a best-effort approach to showing what's changed between the installed version and the updated version that's available via the module repository.

At present, it is only able to talk to github-hosted repositories in order to ask them for the release notes, the changelog file (if present) and a list of commits between the git tag that matches the installed version and the tag matching the latest version.

It will display the Release Notes (if the author is using the feature), else it will display the commits between the tags (if tagging is used by the module author) else it will show the changelog file (if present) else it will show the latest N commits on the master branch (N, of course, being configurable to your liking.)

An example of the Github Release Notes pulled in for you, taken from Mike Rockett's TextformatterTypographer Module...

typographer-003.png.38fd13d185ec06585bb1e8064a7b07ec.png

An example of a tag-to-tag commit list from the same module...

typographer-004.png.7392c1cbddca205e517922112a64f08b.png

An example of a changelog - formatted to show just the changes (formatting styles will change)...

file-editor_004.png.fb2bedfb4e846be178ffe4bc803c51f0.png

Finally, an example of a fallback list of commits - sorry Adrian :)...

recent-commits_001.png.01e9a3e67853406fe7363c0e3639a464.png

 

Making breaking changes obvious.

This is currently done by searching for a set of configurable search strings. Later versions may be able to support breaking change detection via use of Semantic Versioning - but this may require some way of signalling the use of this versioning standard on a module-by-module basis.

For now, then, you can customise the default set of change markers. Here I have added my own alias to the list of breaking change markers and the changes section of the changelog is styled accordingly (these will be improved)...

file-editor_005.png.478eb6591d8ac5df120702676b9ffa65.png

 

Make reading of a module's support documentation, post-install, a trivial task.

This is done by making some of the support files (like the README, CHANGELOG and LICENSE files) readable from the module's information/settings screen. There is an option to control the initial open/closed state of this section...

smush_001.png.c6fe574ddf846b606e9d385ad362c9c8.png

Here is Tracy's README file from within the module settings page... 

tracy-003.thumb.png.febe8b6032626ac8f1338998c15fe673.png

 

Make module authors start to think about how they can improve the change discovery process for their modules.

There are notes in each of the sections displayed on the upgrade confirmation page that help authors use each of the features...

notes_001.png.a098fb1296fae2d1a811a335d01d2c09.png

 

Make sure display of external information doesn't introduce a vulnerability.

This is an ongoing concern, and is the thing that is most likely to delay or prevent this module's release lead to this module's withdrawl should a vulnerability be found. Currently, output is formatted either via Markdown + HTML Purifier (if it was originally a Markdown file) or via htmlspecialchars() if it has come from a plaintext file.

If you discover a vulnerability, please get in contact with me via the forum PM system.

 

Ongoing...

For now, I've concentrated on integration with GitHub, as most people use that platform to host their code. I know a few people are hosting their repositories with BitBucket (PWFoo comes to mind) and some with GitLab (Mike Rockett?) and I would eventually like to have adaptor implementations for these providers (and perhaps GitKraken) - but for now, GitHub rules and the other hosts are unsupported.

 

Links

Github: ModuleReleaseNotes

PW Module Repository: Here

  • Like 21
  • Thanks 4

Share this post


Link to post
Share on other sites

Sounds amazing!

3 hours ago, netcarver said:

It will display the Release Notes (if the author is using the feature), else it will display the commits between the tags (if tagging is used by the module author) else it will show the changelog file (if present) else it will show the latest N commits on the master branch (N, of course, being configurable to your liking.)

3 hours ago, netcarver said:

There are notes in each of the sections displayed on the upgrade confirmation page that help authors use each of the features...

It would be good to get some guidance/opinions on what is the optimal way to prepare release notes. I'd rather not have to duplicate the same information in release notes, tags, changelog and detailed commit notes, but don't really want to be "told off" by the module for not doing it either. :-)

  • Like 2

Share this post


Link to post
Share on other sites

Thanks for the encouraging feedback folks!

@Robin S There's an option to turn off the timed author-auto-shock-via-email(tm) feature which should stop it telling you off too much :)

Actually, the module gives you some options here - including doing nothing more than using git to commit your code - which we all do already. But I think I take your point; I guess the module could induce some of our users to nag us about keeping the upgrade process free of yellow notes.

Regarding guidance, I think it partially comes down to your own style working and issuing releases. For example, I commit lots of small changes - which makes my commit log pretty long and difficult to review. In my case, forcing myself to either keep a concise changelog file or to git tag my releases and then use the github release-notes feature, can be better for my module users. Even if I didn't want to start using Github's Release Notes or keeping a changelog, simply learning to add a git tag on each version increment is an improvement, as it allows the length of the commit list to be automatically limited to what is relevant.

For someone like Ryan, who tends to commit larger changes at lower frequencies, just using the git-tags feature to limit the list of commits to those between your installed version and the newest version is probably all that is needed.

 

  • Like 4

Share this post


Link to post
Share on other sites

I forgot to mention that the support file list is not limit to a single readme file. If your module has more than one, they can all be listed and read. This might prove useful for people looking for a self-hosted documenting solution. As an example, here is a screenshot of the readme files from the module itself...

mrn_001.jpeg.7a9c0fbad9e39fff4e4495d212668a95.jpeg

Which allows the documentation to be split.

mrn_002.thumb.jpeg.f5af0ba6b497ca0dd33d910038d645f7.jpeg

  • Like 1

Share this post


Link to post
Share on other sites
12 hours ago, netcarver said:

Make reading of a module's support documentation post-install a trivial task

There is the case of paid modules which tend to come with manual install option only. I'm wondering how this module could somehow support those modules as well. Easy access to the "Release Notes" would be also nice to have (post-install, in the admin). 

  • Like 1

Share this post


Link to post
Share on other sites

Netcarver is da man. I want it all and I want it now! Where is the download link.... can not wait ;)

  • Like 1

Share this post


Link to post
Share on other sites

Changelog styles now improved. Here's a changelog from the update screen - no breaking changes detected...

file-editor_010.png.e8949032c93f15a7f82c75dfea4ba8fa.png

... and here it is with breaking changes ...

file-editor_012.png.9af26bea5ede59d49978f90604b36d75.png

Raw markdown also get's labelled...

file-editor_011.png.4c3edd26e2402f13ef86ec707faf9642.png

 

  • Like 7

Share this post


Link to post
Share on other sites

Version 0.7.6 is now available with the first round of reported problems fixed. Please note, the changelog for the version of the module includes the term "Breaking Changes" so if you go through the module upgrade path for this module, you'll see breaking change highlighting in action. In this case, it's OK to proceed.

mrn_003.png.44c5ae0b9f44ff892e2c68577bbe3752.png

 

Update: version 0.8.0 is available. Adds support for code highlighting in support files thanks to Abdus' TextformatterPrism module (if it happens to be installed.)

mrn_004.png.dd560735ab102e6f49349ddb717f3632.png

  • Like 4

Share this post


Link to post
Share on other sites

This is now up to version 0.9.1 and I've moved the module's status from Alpha to Beta.

New features include showing git tag points in the commit logs...

tracy-004.png.5734987340661834b064b0a5c53cad5c.png

...better detection+handling of out-of-order versions in changelogs (plus some improved styles thanks to @matjazp) and the display of the remote repository-host's API read depletion condition.

tracy-005.png.c5e4fb1d1ad4954f00bc2a24e62b7650.png

  • Like 3

Share this post


Link to post
Share on other sites
On 11/16/2017 at 7:20 AM, szabesz said:

There is the case of paid modules which tend to come with manual install option only. I'm wondering how this module could somehow support those modules as well. Easy access to the "Release Notes" would be also nice to have (post-install, in the admin). 

Although the "What's Changed" section can't be currently displayed unless there is a remote git host to query, the module's support files are readable in the module's config page post-installation as they are all pulled form the local installation, not the remote host. 

  • Like 1

Share this post


Link to post
Share on other sites
1 hour ago, netcarver said:

better detection+handling of out-of-order versions in changelogs

I don't remember how it was before, but is this what you are expecting in this scenario? I actually think it probably is the right way to go, but just wanting to make sure you had seen this issue with AOS.

5a1444862be2e_ScreenShot2017-11-21at7_16_39AM.png.c4fb00823933449eedbfe74b07f4cb14.png

  • Like 1

Share this post


Link to post
Share on other sites

@adrian,

Thanks for posting this. It's an interesting case. Here's the markdown content of the changelog for that module...

# Changelog

### 1.6.71 (2017-11-14)
- prevNextLinks: keep page sort order


### 1.6.7.2 (2017-11-19)
- ctrl+s save fix by Robin S


### 1.6.7 (2017-11-13)
- new tweak 'prevNextLinks' to Misc, based on Macrura's 'PrevNextTabs' module (thanks!)


### 1.6.6 (2017-11-07)
- UikitTweaks: login page sticky...

You can see that I'm just honouring the changelog as it has been published. ModuleReleaseNotes simply marks up the installed version and the latest available version (which is 1.6.7.2) according to what the PW module repo reports as the latest available for the module being looked at, and what the modules code in PW reports as the installed version. I'm OK with marking up the changelog - but not with re-ordering.

BTW, if anyone has a more advanced version of a module installed locally than has been publicly published via the module's repo, you'll also see a situation like the above.

I guess I could change the label shown in this case from "Latest" to "Latest Published" or something like that.

Opinions welcome.

Share this post


Link to post
Share on other sites
7 minutes ago, netcarver said:

 

I guess I could change the label shown in this case from "Latest" to "Latest Published" or something like that.

Maybe the "Latest" tag isn't needed? Perhaps all that is relevant is the "Installed" tag. The issue with AOS is a mistaken version number 1.6.71 (I think @tpr intended 1.6.7.1). The more likely scenario is what you described where you might have a version that is newer, but not published yet - that can happen relatively often.

  • Like 1

Share this post


Link to post
Share on other sites

If possible, I would like to make it visually clear which version of the module the user will be getting once they "push the button". If I leave the label off, that won't happen.

  • Like 1

Share this post


Link to post
Share on other sites
1 minute ago, netcarver said:

If possible, I would like to make it visually clear which version of the module the user will be getting once they "push the button". If I leave the label off, that won't happen.

Not sure it's quite right, but what about "Upgrade to:" as the label? Or some other way to highlight it as what you'll be upgrading to, rather than it being the latest available?

  • Like 1

Share this post


Link to post
Share on other sites

Hmm, thing is, that is the latest available. In this case, it is an accurate description. The changelog could remove the ambiguity with an edit to list things in date order.  I suppose it would be possible to detect the dates for each entry and re-order chronologically - but that would mean enforcing a date standard + edge case code for dates as well, it's not a place I want to go.

Share this post


Link to post
Share on other sites
1 minute ago, netcarver said:

Hmm, thing is, that is the latest available.

Latest available on Github (and what you'll get if you upgrade), but not actually the latest, necessarily.

I am actually confused by the changelog in AOS - the dates for 1.6.71 a 1.6.7.2 are out of order - but again, this really is an unusual situation.

Anyway, I really wouldn't worry too much about any of this - it was really just meant as an example of things to look out for - I don't really think anything needs changing.

  • Like 1

Share this post


Link to post
Share on other sites

https://semver.org/

Quotes:

"...
Given a version number MAJOR.MINOR.PATCH, increment the:

  • MAJOR version when you make incompatible API changes,
  • MINOR version when you add functionality in a backwards-compatible manner, and
  • PATCH version when you make backwards-compatible bug fixes.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.
...
How should I deal with revisions in the 0.y.z initial development phase?
The simplest thing to do is start your initial development release at 0.1.0 and then increment the minor version for each subsequent release.

How do I know when to release 1.0.0?
If your software is being used in production, it should probably already be 1.0.0. If you have a stable API on which users have come to depend, you should be 1.0.0. If you’re worrying a lot about backwards compatibility, you should probably already be 1.0.0.
..."

  • Like 2

Share this post


Link to post
Share on other sites

Version 0.10.0 is in the module repository.

This has a major behind-the-scenes refactoring to allow for remote repository adaptors (with result caching).

mrn_005.png.0ae0235c8a978c5be021f2789fc4060a.png

Github is supported fully, and I've made a start on the BitBucket adaptor...

formhelper_001.png.bf17dc97bd9ee8d9788e5f81589397c0.png

This version also fixes a couple of issues that have been reported by @adrian and @matjazp.

  • Like 5

Share this post


Link to post
Share on other sites

Just saw this when upgrading AOS. Doesn't seem right. I see that @tpr is using integer versions again, but 168 was also an integer, so not sure why it would be reporting like this.

image.thumb.png.72a0241fcb69a6ca84ab36e1f19cbd49.png

Share this post


Link to post
Share on other sites

Not sure if you can do anything about this. wireMailSMTP's changelog is not being parsed properly is it?

Also, you have a Tracy bd() call on line 702.

OT, but any reason you prefer: \TD::barDump() to bd() ?

 

image.thumb.png.200f04c061b06c06bb43bda466730e4a.png

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By dreerr
      TemplateEnginePug (formally TemplateEngineJade)
       
      This module adds Pug templates to the TemplateEngineFactory. It uses https://github.com/pug-php/pug to render templates.
      doctype html html(lang='en') head meta(http-equiv='content-type', content='text/html; charset=utf-8') title= $page->title link(rel='stylesheet', type='text/css', href=$config->urls->templates . 'styles/main.css') body include header.pug h1= $page->title if $page->editable() p: a(href=$page->editURL) Edit Project on GitHub: github.com/dreerr/TemplateEnginePug
      Project in modules directory: modules.processwire.com/modules/template-engine-pug/
       
      For common problems/features/questions about the Factory, use the TemplateEngineFactory thread.
       
    • By tpr
      ProcessNetteTester
      Run Nette Tester tests within ProcessWire admin.
      (continued from here)

      Features
      AJAX interface for running Nette Tester tests, in bulk or manually display counter, error message and execution time in a table run all tests at once or launch single tests show formatted test error messages and report PHP syntax errors stop on first failed test (optional) hide passed tests (optional) display failed/total instead passed/total (optional) re-run failed tests only (optional) auto scroll (optional) include or exclude tests based on query parameters start/stop all tests with the spacebar reset one test or all tests (ctrl+click) https://modules.processwire.com/modules/process-nette-tester/
      https://github.com/rolandtoth/ProcessNetteTester
    • By bernhard
      Some of you might have followed the development of this module here: https://processwire.com/talk/topic/15524-previewdiscussion-rockdatatables/ . It is the successor of "RockDataTables" and requires RockFinder to get the data for the grid easily and efficiently. It uses the open source part of agGrid for grid rendering.
       
      WHY?
      ProcessWire is awesome for creating all kinds of custom backend applications, but where it is not so awesome in my opinion is when it comes to listing this data. Of course we have the built in page lister and we have ListerPro, but none of that solutions is capable of properly displaying large amounts of data, for example lists of revenues, aggregations, quick and easy sorts by the user, instant filter and those kind of features. RockGrid to the rescue 😉 
       
      Features/Highlights:
      100k+ rows Instant (client side) filter, search, sort (different sort based on data type, eg "lower/greater than" for numbers, "contains" for strings) extendable via plugins (available plugins at the moment: fullscreen, csv export, reload, batch-processing of data, column sum/statistics, row selection) all the agGrid features (cell renderers, cell styling, pagination, column grouping etc) vanilla javascript, backend and frontend support (though not all plugins are working on the frontend yet and I don't plan to support it as long as I don't need it myself)  
      Limitations:
      While there is an option to retrieve data via AJAX the actual processing of the grid (displaying, filtering, sorting) is done on the client side, meaning that you can get into troubles when handling really large datasets of several thousands of rows. agGrid should be one of the most performant grid options in the world (see the official example page with a 100k row example) and does a lot to prevent problems (such as virtual row rendering), but you should always have this limitation in mind as this is a major difference to the available lister options that do not have this limitation.
      Currently it only supports AdminThemeUikit and I don't plan to support any other admin theme.
       
      Download: https://gitlab.com/baumrock/RockGrid
      Installation: https://gitlab.com/baumrock/RockGrid/wikis/Installation
      Quikckstart: https://gitlab.com/baumrock/RockGrid/wikis/quickstart
      Further instructions: https://gitlab.com/baumrock/RockGrid/wikis/quickstart#further-instructions
       
      Module status: alpha, License: MIT
      Note that every installation and uninstallation sends an anonymous google analytics event to my google analytics account. If you don't want that feel free to remove the appropriate lines of code before installation/uninstallation.
       
      Contribute:
      You can contribute to the development of this and other modules or just say thank you by
      testing, reporting issues and making PRs at gitlab liking this post buying me a drink: paypal.me/baumrock/5 liking my facebook page: facebook.com/baumrock hiring me for pw work: baumrock.com  
      Support: Please note that this module might not be as easy and plug&play as many other modules. It needs a good understanding of agGrid (and JavaScript in general) and it likely needs some looks into the code to get all the options. Please understand that I can not provide free support for every request here in the forum. I try to answer all questions that might also help others or that might improve the module but for individual requests I offer paid support for 60€ per hour.
       
      Changelog
      180711 bugfix (naming issue) 180630 alpha realease  
      Use Cases / Examples:
      Colored grid cells, Icons, Links etc. The Grid also has a "batcher" feature built in that helps communicating with the server via AJAX and managing resource intensive tasks in batches:

      Filters, PW panel links and instant reload on panel close:

      You can combine the grid with a chart library like I did with the (outdated) RockDataTables module:

    • By tpr
      Update 2018-07-09: ProcessNetteTester module is available in the Modules Directory and on GitHub.

      This is a short tutorial on how to use Nette Tester with ProcessWire.
      As you will see it's very easy to setup and use and it's perfect for testing your code's functionality. With bootstrapping ProcessWire it's also possible to check the rendered markup of pages using the API, checking page properties, etc. It's also a great tool for module developers for writing better code. 
      While there will be nothing extraordinary here that you couldn't find in Tester's docs this can serve as a good starting point.
      Prerequisites: PHP 5.6+
      01 Download Tester
      Go to https://github.com/nette/tester/releases and download the latest release (currently 2.0.2). Download from the link reading "Source code (zip)". You can use composer also if you wish.
      02 Extract Tester files
      Create a new directory in your site root called "tester". Extract the zip downloaded here, so it should look like this:
      /site /tester/src /tester/tools /tester/appveyor.yml /tester/composer.json /tester/contributing.md /tester/license.md /tester/readme.md /wire ... 03 Create directory for test files
      Add a new directory in "/tester" called "tests". Tester recognizes "*.Test.php" and "*.phpt" files in the tests directory, recursively. 
      04 Create your first test
      In the "tests" directory create a new "MyTest.php" file.
      The first test is a very simple one that bootstraps ProcessWire and checks if the Home page name is "Home". This is not the smartest test but will show you the basics.
      Add this to "/tester/tests/MyTest.php":
      <?php namespace ProcessWire; use \Tester\Assert; use \Tester\DomQuery; use \Tester\TestCase; use \Tester\Environment; require __DIR__ . '/../src/bootstrap.php'; // load Tester require __DIR__ . '/../../index.php'; // bootstrap ProcessWire Environment::setup(); class MyTest extends TestCase {     // first test (step 04)     public function testHomeTitle()     {         $expected = 'Home'; // we expect the page title to be "Home"         $actual = wire('pages')->get(1)->title; // check what's the actual title Assert::equal($expected, $actual); // check whether they are equal     }     // second test will go here (step 06)     // third test will go here (step 07) } // run testing methods (new MyTest())->run(); I've added comment placeholders for the second and third tests that we will insert later.
      05 Run Tester
      Tester can be run either from the command line or from the browser. The command line output is more verbose and colored while in the browser it's plain text only (see later).
      Running from the command line
      Navigate to the "/tester" directory in your console and execute this:
      php src/tester.php -C tests This will start "/tester/src/tester.php" and runs test files from the "/tester/tests" directory. The "-C" switch tells Tester to use the system-wide php ini file, that is required here because when bootstrapping ProcessWire you may run into errors (no php.ini file is used by default). You may load another ini file with the "-c <path>" (check the docs).
      If the title of your Home page is "Home" you should see this:

      If it's for example "Cats and Dogs", you should see this:

      Running from the browser
      First we need to create a new PHP file in ProcessWire's root, let's call it "testrunner.php". This is because ProcessWire doesn't allow to run PHP files from its "site" directory.
      The following code runs two test classes and produces a legible output. IRL you should probably iterate through directories to get test files (eg. with glob()), and of course it's better not allow tests go out to production.
      <?php ini_set('html_errors', false); header('Content-type: text/plain'); echo 'Starting tests.' . PHP_EOL; echo '--------------------------' . PHP_EOL; $file = __DIR__ . '/PATH_TO/FirstTest.php'; echo basename($file) . ' '; require $file; echo '[OK]' . PHP_EOL; $file = __DIR__ . '/PATH_TO/SecondTest.php'; echo basename($file) . ' '; require $file; echo '[OK]' . PHP_EOL; echo '--------------------------' . PHP_EOL; echo 'Tests finished.'; exit; Navigate to "DOMAIN/testrunner.php" in your browser to execute the file. If every test succeeds you should get this:

      If there are failed tests the execution stops and you can read the error message. If there were more tests (eg. ThirdTest), those won't be displayed under the failed test.

      06 DOM test
      This test will check if a page with "basic-page" template has a "h1" element. We will create the page on the fly with ProcessWire's API. To keep things simple we will add the new test as a new method to our MyTest class.
      Add this block to the MyTest class:
      public function testBasicPageHeadline() {     $p = new Page();     $p->template = 'basic-page';     $html = $p->render();     $dom = DomQuery::fromHtml($html);     Assert::true($dom->has('h1')); } This will most likely be true but of course you can check for something more specific, for example "div#main". Note that we have used the DomQuery helper here (check the "use" statement on the top of the file).
      07 Custom function test
      You will probably want to make sure your custom functions/methods will work as they should so let's write a test that demonstrates this.
      I don't want to complicate things so I'll check if the built-in "pageName" sanitizer works as expected. Add this to the myTest class:
      public function testPageNameSanitizer() {     $expected = 'hello-world';     $actual = wire('sanitizer')->pageName('Hello world!', true);     Assert::equal($expected, $actual); } This should also be true. Try to change the expected value if you are eager to see a failure message.
       
      08 Next steps
      You can add more methods to the MyTest class or create new files in the "tests" directory. Check out the range of available Assertions and other features in the docs and see how they could help you writing more fail-safe code.
      Once you make a habit of writing tests you'll see how it can assist making your code more bulletproof and stable. Remember: test early, test often 🙂
      If you find out something useful or cool with Tester make sure to share.
    • By bernhard
      @Sergio asked about the pdf creation process in the showcase thread about my 360° feedback/survey tool and so I went ahead and set my little pdf helper module to public.
      Description from PW Weekly:
       
      Download & Docs: https://gitlab.com/baumrock/RockPdf
       
      You can combine it easily with RockReplacer: 
      See also a little showcase of the RockPdf module in this thread: