Module: AIOM+ (All In One Minify) for CSS, LESS, JS and HTML

[Jonathan Dart]

Hi David, I'll try out that change you mentioned, and I appreciate the whitelist feature - thanks!

I'm trying to imagine a scenario where including a javascript/css/less file outside of the templates directory poses a security risk. How could that be exploited?

Hi Jonathan , 

yes, in this function paths are filtered to prevent directory traversal attacks. AIOM+ loads only allowed files that are located in the template folder. I think about it, in one of the upcoming versions, introduce a whitelist for asset folder.

For a workaround change the following line (749) in AllInOneMinify.module: 

$_path  = str_ireplace(array('../', './', '%2e%2e%2f', '..%2F'), '', (wire('config')->paths->templates.$_file));

to 

$_path  = wire('config')->paths->templates.$_file; 

I have not tested it and I can not recommend it too.

[David Karich]

Hi David, I'll try out that change you mentioned, and I appreciate the whitelist feature - thanks!

I'm trying to imagine a scenario where including a javascript/css/less file outside of the templates directory poses a security risk. How could that be exploited?

The easiest way is by infected third-party applications. Also on Bower or Composer, its possible, that compromised packages can be delivered. I have currently identified no scenario targeted at Process Wire, rather it is one of my personal programming rules. Just do not make it possible. Asset files belong in the template folder. This approach I have always in development, regardless of platform or environment. I have some clients who work with sensitive data, and the safety requirements are very high. Therefore, this is my natural data schizophrenia.

[David Karich]

I have the new version of AIOM+ uploaded (3.1.2). From this version it is now possible to disable the directory traversal filter. Also, the LESS parser was updated. Now compatible with the official LESS version 1.7.

[Jonathan Dart]

Would it be possible to provide an alternate less compiler for those with node available on the server? 

[David Karich]

Would it be possible to provide an alternate less compiler for those with node available on the server? 

I'm not really familiar with Node.js and I have no test environment. You're welcome to implement an alternative and send a pull request on GitHub. I think it requires only an option in the backend more and instead of PHP-based LESS parser an exec-command-line function call.

The question is rather, the detour via PHP is really faster? Because you can embed the LESS files directly into the template and can render there via LESS.js or over the Node.js package. The Node.js package has a minimization option on board.

[Jonathan Dart]

I'm not really familiar with Node.js and I have no test environment. You're welcome to implement an alternative and send a pull request on GitHub. I think it requires only an option in the backend more and instead of PHP-based LESS parser an exec-command-line function call.

The question is rather, the detour via PHP is really faster? Because you can embed the LESS files directly into the template and can render there via LESS.js or over the Node.js package. The Node.js package has a minimization option on board.

Compiling all of bootstrap with the php implementation takes 3+ seconds which is an annoyance when developing themes, with the node.js implementation it's half a second. I agree that using less in the template being a great option. I think I'll set things up that way. Thanks David

[David Karich]

Hey guys, 

i have released a new version of AIOM+. The update to version 3.1.3 includes a few improvements in the LESS parser and CSS minification. Also, I have set the status from AIOM+ to stable.

[adrian]

Hi David,

Thanks again for a great module. Just wondering if you did any experimenting with other js minification libraries? I am no expert on this, but noticed that I can get significantly smaller files (in one case 400kb down to 300kb) by minifying using http://jscompress.com/ I know that runs on nodejs, but there are other php alternatives out there, although maybe none of them are any better.

Maybe you've already been through this process and decided JSMin is the best option, but just thought I'd ask

[David Karich]

Hi David,

Thanks again for a great module. Just wondering if you did any experimenting with other js minification libraries? I am no expert on this, but noticed that I can get significantly smaller files (in one case 400kb down to 300kb) by minifying using http://jscompress.com/ I know that runs on nodejs, but there are other php alternatives out there, although maybe none of them are any better.

Maybe you've already been through this process and decided JSMin is the best option, but just thought I'd ask

Hi adrian,

for sure. JSMin is a little bit outdated and generates not the best compression result, but it is the only stable php port without any conditions to third party libraries. Some other minifications libraries destroyed the compressed js code and JSMin is the only one, with which I never had such a problem. Anyway, i'm thinking about, to integret the google closure compiler as an alternative third party service.

[adrian]

Thanks for the explanation. I am sure you've seen it, but this looks promising:

https://github.com/tedivm/JShrink

Anyway, thanks again for this module - it's a real time saver.

On an unrelated note - I did notice something yesterday when using the loadOn option. It worked fine for CSS, but it didn't seem to work properly with JS. Before you go delving too deep, I should do some more testing and see what I can find - might have been an issue at my end. I'll try again soon and let you know how it goes.

[David Karich]

Thanks for the explanation. I am sure you've seen it, but this looks promising:

https://github.com/tedivm/JShrink

Anyway, thanks again for this module - it's a real time saver.

On an unrelated note - I did notice something yesterday when using the loadOn option. It worked fine for CSS, but it didn't seem to work properly with JS. Before you go delving too deep, I should do some more testing and see what I can find - might have been an issue at my end. I'll try again soon and let you know how it goes.

Oh, no. This library has not yet come under my eyes. I'll be looking at this a bit more detail when I have more time again. Thanks for the tip.

Yes, check again and let me know, because it is the identical function that is called. If there should be a bug, CSS should not work as well. If you were able to find something, then just open a ticket on Github.

[tedivm]

Author of JShrink here (I found you guys in my web logs). I thought I'd pop in and say hello, and talk about JShrink a bit. 

I originally made it as a drop in replacement for JSMin, because JSMin is technically not open source (it has the "do no evil" license, so projects that take open source seriously, such as Debian, won't bundle projects with it). Since then I've made improvements on speed, added additional features like support for conditional comments and license preservation, and have really  put work into the test suite itself. Ryan Grove, who originally ported JSMin to PHP, has officially dropped the JSMin project and is referring people to JShrink on the Github page for it.

Yesterday, after several years of being in the "beta" phase, I launched the JShrink v1.0 to make the API official and stable. 

On a tangental note, I also maintain a caching library called Stash that may be useful to you.

[bytesource]

AIOM minifies my CSS file without any problems, but for some reason I have not been able to minify any javascript files. 

While outputing javascript the old way works:

// _init.php
$jsPath = $config->urls->templates . 'styles/js/';

// _init.php
$page->jsFiles = array(
    'jquery.min.js',
    'bootstrap.min.js',
    'jquery.fancybox.pack.js',
    'jquery.isotope.min.js',
    'jquery.sticky.js',
    'retina.js',
    'scripts.js',
);

// _done.php
foreach ($page->jsFiles as $file) {
    $fullPath = $jsPath . $file;
    echo "<script src='{$fullPath}'></script>\n";

}

// Output
// <script src='/processwire/site/templates/styles/js/jquery.min.js'></script>
// <script src='/processwire/site/templates/styles/js/bootstrap.min.js'></script>
// <script src='/processwire/site/templates/styles/js/jquery.fancybox.pack.js'></script>
// <script src='/processwire/site/templates/styles/js/jquery.isotope.min.js'></script>
// <script src='/processwire/site/templates/styles/js/jquery.sticky.js'></script>
// <script src='/processwire/site/templates/styles/js/retina.js'></script>
// <script src='/processwire/site/templates/styles/js/scripts.js'></script>

 

Using AIOM as shown below always resulted in a file with the content of

/** Generated: Thursday, 8th of May 2014, 11:36:39 PM // Powered by AIOM+ (All In One Minify) by www.flipzoom.de **/:

 

// _done.php
$jsFiles = array();

foreach ($page->jsFiles as $js) {
  $fullPath  = $jsPath . $js;
  $jsFiles[] = $fullPath;
}

$minified = AIOM::JS($jsFiles);

echo "<script src='{$minified}'></script>";

// Output 
// <script src='/processwire/site/assets/aiom/js_d41d8cd98f00b204e9800998ecf8427e_dev.js?no-cache=1399617391'></script>

I also tried to output each javascript file one by one using AIOM, but this did not work, either: 

  foreach ($page->jsFiles as $file) {
    $fullPath = $jsPath . $file;
    $minJS = AIOM::JS($fullPath);
    echo "<script src='{$minJS}'></script>\n";
  }

Output:
// <script src='/processwire/site/assets/aiom/js_d41d8cd98f00b204e9800998ecf8427e_dev.js?no-cache=1399617391'></script>
// <script src='/processwire/site/assets/aiom/js_d41d8cd98f00b204e9800998ecf8427e_dev.js?no-cache=1399617391'></script>
// <script src='/processwire/site/assets/aiom/js_d41d8cd98f00b204e9800998ecf8427e_dev.js?no-cache=1399617391'></script>
// <script src='/processwire/site/assets/aiom/js_d41d8cd98f00b204e9800998ecf8427e_dev.js?no-cache=1399617391'></script>
// <script src='/processwire/site/assets/aiom/js_d41d8cd98f00b204e9800998ecf8427e_dev.js?no-cache=1399617391'></script>
// <script src='/processwire/site/assets/aiom/js_d41d8cd98f00b204e9800998ecf8427e_dev.js?no-cache=1399617391'></script>
// <script src='/processwire/site/assets/aiom/js_d41d8cd98f00b204e9800998ecf8427e_dev.js?no-cache=1399617391'></script>

Interestingly, each time the same file is output.

I installed AIOM version 3.1.3 with Processwire 2.4

 

I am probably missing the obvious, but I haven't been able to spot it. 

 

Cheers, 

 

Stefan

[horst]

EDIT: Ah, you have it in the first line:

// _init.php
$jsPath = $config->urls->templates . 'styles/js/';

try it with changed this to:

// _init.php
$jsPath = $config->paths->templates . 'styles/js/';

-------------

I don't know if it needs the fullpath or only a url, but you are definietly provide the url and not the fullpath:

$fullPath = $jsPath . $file;   // = /proseeswire/site/assets/...

I would do a quick try with:

$_SERVER['DOCUMENT_ROOT'] . $jsPath . $file;

If this succeeds, I would change the "$_SERVER['DOCUMENT_ROOT'] . $jsPath" to some PW $config-var (e.g. $config->paths ...)

Edited May 9, 2014 by horst

[bytesource]

@horst

Thanks for your reply. 

Unfortunately, replacing $config->urls with $config->paths lead to the same result and at the same time got me 'Page not found' errors for all script files added the normal way:

// _init.php
$jsPath = $config->urls->templates . 'styles/js/';

// _init.php
$page->jsFiles = array(
    'jquery.min.js',
    'bootstrap.min.js',
    'jquery.fancybox.pack.js',
    'jquery.isotope.min.js',
    'jquery.sticky.js',
    'retina.js',
    'scripts.js',
);

// _done.php
foreach ($page->jsFiles as $file) {
    $fullPath = $jsPath . $file;
    echo "<script src='{$fullPath}'></script>\n";
}

// Output:
// <script src='/home/sovonex/Programs/rubystack-2.0.0-12/apps/processwire/htdocs/site/templates/styles/js/jquery.min.js'></script>
// <script src='/home/sovonex/Programs/rubystack-2.0.0-12/apps/processwire/htdocs/site/templates/styles/js/bootstrap.min.js'></script>
// <script src='/home/sovonex/Programs/rubystack-2.0.0-12/apps/processwire/htdocs/site/templates/styles/js/jquery.fancybox.pack.js'>//

// </script>
// <script src='/home/sovonex/Programs/rubystack-2.0.0-12/apps/processwire/htdocs/site/templates/styles/js/jquery.isotope.min.js'></script>
// <script src='/home/sovonex/Programs/rubystack-2.0.0-12/apps/processwire/htdocs/site/templates/styles/js/jquery.sticky.js'></script>
// <script src='/home/sovonex/Programs/rubystack-2.0.0-12/apps/processwire/htdocs/site/templates/styles/js/retina.js'></script>
// <script src='/home/sovonex/Programs/rubystack-2.0.0-12/apps/processwire/htdocs/site/templates/styles/js/codeslider.js'></script>
// <script src='/home/sovonex/Programs/rubystack-2.0.0-12/apps/processwire/htdocs/site/templates/styles/js/scripts.js'></script>

 

I installed Processwire via the Bitnami stack, that's where the long path names are coming from. 

 

To be fair, I am not quite sure, when to use $config->paths and when to use $config->urls most of the time.

 

As for CSS and javascript files, I chose $config->urls, because I saw it done this way on the skyscraper installation:

https://github.com/ryancramerdesign/SkyscrapersProfile/blob/master/site/templates/_out.php#L42

 

Cheers, 

 

Stefan

 

 

 

[horst]

@Stefan: unfortunately I don't know the AIOM well. It was just a thought.

I would think one mostly need the $config->urls->something if one have to provide a link to the clients browser

and need the $config->paths->something mostly to tell a php-script / function where to grap the file(s) in filesystem for preprocessing something.

[adrian]

The instructions for this module say to use the path relative to your templates directory, eg:

styles/mystyles.css or scripts/myscript.js

There is a setting in the module config:

"Allow Directory Traversal: Enable the directory traversal option to make it possible to add files from outside of the template folders. (../)"

 

which you can use if you need to access css and js files above the templates directory, but otherwise they should all be in subfolders of "templates" and linked to with a path relative to templates.

 

Make sense?

 

EDIT: not relevant to this module, but path vs url - depends on what is being done with the file being linked to. If it's a css or js file, typically you are going to want the url option. The path option is the full server path to the file which is useful for php operations on files, but no good for front-end display/calling of files. 

[bytesource]

@adrian

Using the relative path from the template folder did the trick and so the following code is working:

 

// _init.php
// Relative path from the templates folder to the javascript folder
$relativeJSPath = 'styles/js/';

// _init.php
$page->jsFiles = array(
    'jquery.min.js',
    'bootstrap.min.js',
    'jquery.fancybox.pack.js',
    'jquery.isotope.min.js',
    'jquery.sticky.js',
    'retina.js',
    'scripts.js',
);

$jsFiles = array();
foreach ($page->jsFiles as $js) {
  $fullPath  = $relativeJSPath . $js;
  $jsFiles[] = $fullPath;
}

$minified = AIOM::JS($jsFiles);

echo "<script src='{$minified}'></script>";

 

However, now the code in scripts.js, where I initialize all plugins, does not work anymore. Only when I copy its content inside script tags below the javascript file produced by AIOM does it work again. This is regardless of whether the javascript files have been minified or just been merged into a single file (developer mode). 

 

Does anyone have any idea what might have happened here?

 

Cheers, 

 

Stefan

[David Karich]

However, now the code in scripts.js, where I initialize all plugins, does not work anymore. Only when I copy its content inside script tags below the javascript file produced by AIOM does it work again. This is regardless of whether the javascript files have been minified or just been merged into a single file (developer mode). 

 

Does anyone have any idea what might have happened here?

 

Cheers, 

 

Stefan

The generated code can be incorrect if your code is not written cleanly. A missing semicolon can already generate faulty code. Please inspect your script, for example, with jslint.com.

[bytesource]

Hi David,

sorry for my late response. I missed the notification email and thought there was no new reply.

You suggestion was spot on! The problem was in fact a syntax error in my Javascript code. Once I fixed that, AIOM worked again.

Today I have one other question:

When using AIOM, the paths to background images, icons, etc. in my CSS files somehow get changed and cannot be found anymore.

Here is an example

CSS code

.some-class {
    background: url(images/sprite.png);
}

This path works

// _init.php
$cssPath = $config->urls->templates . 'styles/stylesheets/';

// _done.php
<link href="<?= $cssPath . 'main.css' ?>" rel="stylesheet">

Path: background: url(h

                                      ttp://localhost:8080/processwire/site/templates/styles/stylesheets/images/sprite.png.); 

This path does not work

<link href="<?= AIOM::CSS('styles/stylesheets/main.css') ?>" rel="stylesheet">

Path: background: url(re/htdocs/site/templates/styles/stylesheets/images/sprite.png);

Do you have any any idea what is happening here?

Cheers,

Stefan

Edited May 27, 2014 by bytesource

[bytesource]

Sorry, I copied the wrong code above, which probably made my question completely incomprehensible. 

I edited my previous comment and hope that now everything is correct. 

[bytesource]

I wanted to bring up this question again as I haven't found a solution yet.

Does anyone have any idea of why the paths inside a CSS file are not translated correctly?

[David Karich]

CSS code

.some-class {
    background: url(images/sprite.png);
}

This path works

// _init.php
$cssPath = $config->urls->templates . 'styles/stylesheets/';

// _done.php
<link href="<?= $cssPath . 'main.css' ?>" rel="stylesheet">

Path: background: url(http://localhost:8080/processwire/site/templates/styles/stylesheets/images/sprite.png.); 

How can the path be different here at once if you have only "images/sprite.png" in CSS? There is no URL rewriting or parsing with direct linked CSS.

I wanted to bring up this question again as I haven't found a solution yet.

Does anyone have any idea of why the paths inside a CSS file are not translated correctly?

Stefan, I'm sorry. I can not reproduce your problem. For me it works with your folder structure. 

[bytesource]

How can the path be different here at once if you have only "images/sprite.png" in CSS?

That is exactly what I don't understand, either. 

Maybe I should add that the file main.css is the result of using the SASS compass plugin stitching together several SASS (SCSS) files. But then again, without AIOM everything works fine, so I don't think using a CSS preprocessor is what causes the problem here. 

I guess I just have to keep looking...

Cheers, 

Stefan

[MadHatter]

Hello, is there any way for AIOM to accept a FilenameArray object for minifying or do I need to convert the object to an array manually first?