ProcessWire Pro Cache - Now Available

[jacmaes]

@onjegolders: it does feel very snappy (checking from Paris, France), and that's impressive for a rather image-heavy site as portfolios usually are. 

[Peter Falkenberg Brown]

Dear Ryan,

Can you either point me to a link where you might have already answered these questions, or could you share a bit more details about the ProCache method?

* Does ProCache convert every page into a separate html-like file, and then add an entry to the .htaccess file, for each file? I wrote a Perl/MySQL CMS years ago that published all the pages to real html files, which of course was very fast, but it wasn't caching per se. If it was one .htaccess line for each file, a large site would end up with a very large .htaccess file, which would get loaded on every page request. I'm curious about your technical method.

* How does ProCache work with dynamically updated content in iframes, like Twitter widgets, banner ads, etc? Does it ignore those third party elements as "pass-throughs"?

* Also, how does ProCache perform with sites that have hundreds of thousands of pages?

* Finally, would you say that ProCache could take the place of a highly tuned installation of Varnish?

Thanks!

Peter

[Pete]

I can answer most of those:

1. It takes the rendered page and creates a HTML file, yes. It does this when someone first visits the page rather than creating them all which makes sense so they are created as they are needed. It is caching in the sense that it completely bypasses all PHP and mySQL queries, so pages load many times faster (they appear almost instantly). The .htaccess entry is a mere 5 lines long and covers all pages in the site. It essentially says "Look here for a cached file" and if it doesn't find one it defaults to loading a normal version of the page (which will then be cached for the next person) - essentially it's as scalable as your server is in that it creates physical pages/folders as necessary in a cache directory, so I don't think you'll have any trouble.
3. Content in iframes will be unaffected as they're usually from external sources. All the other things you mention usually pull in from external sources that feed back to the page what they should be showing. The only difficulty you will have is if any of this is PHP as it will cache the PHP output, but I think all you're talking about here is usually a mixture of external javascript (for banner ads, Twitter widgets etc) and iframes.
4. 
   
5. It should perform fine with millions of pages even. Since it's not doing any queries it's just loading static HTML pages so your server will love you for taking the load off.
6. 
   
7. Varnish sounds like it would be a good addition if you need even more speed (better description here on Wikipedia than their own page) but it should be speedy with ProCache alone to be honest. Varnish seems to be more about storing frequently accessed content in memory to be quickly served, so on popular pages (assuming I understood it correctly) you will see an additional speed boost but I don't know enough about Varnish to be honest.

There are loads of settings in ProCache - default is to wipe the whole cache when you save a page which makes sense purely because that page may be referenced dozens of times elsewhere, but it is totally configurable by template or section etc.

On one website (sadly can't share the link) it is loading pages in half a second or less, making the site seem really snappy (obviously images take time to load first time for each page - no way around that).

I would say it's definitely worth the minimal investment.

[seddass]

Definitely worth the investment !

[alan]

Definitely worth the investment !

+1!

[Peter Falkenberg Brown]

Dear Pete,

Thanks for your answers!

- Upon reflection, Varnish would be good for caching the images, which don't seem to be cached by ProCache. Is that correct?

- Are the images in the page copied to that cached directory tree, or are they referenced in code, to their original location?

- How does ProCache deal with deletions of pages, or url changes? Does it delete the cached files for deleted pages? (i.e. no orphans left over)

- Is it correct to say that a ProCache page is an assembled unit of all of the code elements in that page, that were local to the system (e.g. not including 3rd party iframes)? Thus, the cached page would include all of the elements, even code that might otherwise be uncached, unless MarkupCache was used? (But not the images (i.e. the images are pulled in at run time?))

It seems like a very clever system. I'm looking forward to using it, when I have a site ready to go live.

Best regards,

Peter

[ryan]

 Upon reflection, Varnish would be good for caching the images, which don't seem to be cached by ProCache. Is that correct?

Not sure I understand the question–these are already static assets, what is there to cache? For something like images a CDN would probably be the next step, not a cache? Though note that ProcessWire does already cache image manipulations. So when you make an API call for an image at a certain size, that size gets generated only on the first call and a cached copy is returned for all later calls. 

Are the images in the page copied to that cached directory tree, or are they referenced in code, to their original location?

They stay in their original location. They are already static assets, so no need to duplicate or copy them anywhere else unless for a CDN. 

How does ProCache deal with deletions of pages, or url changes? Does it delete the cached files for deleted pages? (i.e. no orphans left over)

ProCache hooks into all ProcessWire create/save/delete actions and makes adjustments to cache files as needed. So if you delete a page or change its URL, then the cache file for that page is deleted. ProCache also comes with a screen of config options for you to fine tune this further if you'd like. 

Is it correct to say that a ProCache page is an assembled unit of all of the code elements in that page, that were local to the system (e.g. not including 3rd party iframes)? Thus, the cached page would include all of the elements, even code that might otherwise be uncached, unless MarkupCache was used? (But not the images (i.e. the images are pulled in at run time?))

ProCache contains the rendered output produced by ProcessWire. For most sites, this is essentially a static HTML file for each cached page. It is the same thing as if you viewed a page on a website in your browser and clicked File > Save, producing a static HTML file. 

[Peter Falkenberg Brown]

Dear Ryan,

Thanks for your answers! I appreciate it.

Varnish caches images and pages in RAM, so it decreases server load and increases response time dramatically.

With ProCache as the base, I think that Varnish would probably only become relevant when the server got hundreds of thousands of hits, but it's a good alternative to using a CDN.

One can use Varnish to cache things up to a certain level of millions of pages, and then use a CDN like Edgecast when the page views are too much for Varnish on a standalone server to handle.

I see ProCache in my future.

Thanks again for building such an excellent system.

Peter

[ryan]

Varnish caches images and pages in RAM, so it decreases server load and increases response time dramatically.

I suppose another way to accomplish this would be to just run ProcessWire from an SSD. Though wouldn't a servers bandwidth need to exceed the hard drive access time? I guess I figured bandwidth would be a bottleneck before the hard drive, but I don't claim to know much about server administration.

I see ProCache in my future. 

Thanks again for building such an excellent system.

Thanks for your interest in ProCache! 

[Peter Falkenberg Brown]

Dear Ryan,

Hmmm... I'll have to look into the bandwidth issue. That's a good point.

Thanks,

Peter

[Roope]

Can I use Pro Cache in the pages where I need get parameters from url to modify output?

[ryan]

Can I use Pro Cache in the pages where I need get parameters from url to modify output?

You can, but you should tell ProCache the names of your GET parameters so that it won't attempt to cache those pages. You can do this in the ProCache settings on the bypass tab. See attached screenshot. The default setting is "*" which means that GET requests with variables aren't ever cached, regardless of the name. This is a good setting to keep if you don't want to think about what the GET variable names might be. 

[k07n]

Hi!

Can anyone share nginx config for Pro Cache? Thx!

[OllieMackJames]

Hi!

Can anyone share nginx config for Pro Cache? Thx!

I second that, looking into nginx as well! edit: found it here: http://processwire.com/talk/topic/3246-nginx-directives-for-procache/

[k07n]

I second that, looking into nginx as well! edit: found it here: http://processwire.com/talk/topic/3246-nginx-directives-for-procache/

oh, I have not access to that area. How to get it?

[kongondo]

oh, I have not access to that area. How to get it?

If you bought ProCache you'd have access. Ask Ryan to give you access.

[k07n]

If you bought ProCache you'd have access. Ask Ryan to give you access.

Personal letter to Ryan, oh, I'm so excited 

[k07n]

Holy cow! Pro Cache is f#$@ng awesome!

... Hell nooo! Whare are no mates in my messengers list to share my joyfulness!

[Raymond Geerts]

First of all, I'm loving pro-cache. It make the site load so much faster its almost science fiction.

But since today we noticed that even when logged in to the manager as super user the site keeps serving the pro-cached pages.

We first have to save a page in the manager to be able to see the un-cached pages.

This gives some problems because when logged in to the manager the website has front-end edit functions, that do not show up anymore, before saving a page in the manager itself.

[teppo]

@Raymond: if you check the Bypass tab of ProCache setup page, is there anything listed in "Cookies that bypass the cache"?

By default wire_challenge should be there and this should effectively disable ProCache for logged in users. See Ryan's screenshot above for an example. If that field is empty or this cookie isn't listed, add and test again.

[Raymond Geerts]

@teppo: the field does contain the value "wire_challenge".

But your comment reminded me that i had modified the config file and set sessionChallenge and sessionFingerprint both to false when i had trouble login in to the manager just after moving the site to the live server. I have set these settings both to true again and now pro-cache works as expected again.

So it was my own mistake that caused the wire_challenge cookies to not be set anymore

[alan]

Not this threads topic but related to the comment below that made me sit up when I read 

...i had modified the config file and set sessionChallenge and sessionFingerprint both to false when i had trouble login in to the manager just after moving the site to the live server...

as for me it links back to this thread which I am intrigued to know the de facto answer to since I think it will likely stop me ever having login problems again (only had them once so far but even once sticks out like a sore thumb on a system otherwise so entirely reliable). Tks for any comment over there and sorry for barging in here.

[peterfoeng]

Hi guys,

I just purchased this great module for the website I am working on and it's fantastic I do have a question though but I am not sure what kind of caching result I should be expecting but here goes my case study:

I have a page setup like so: http://tympanus.net/Tutorials/ThumbnailGridExpandingPreview/ (this is a sample site but my setup is very similar) every thumbnail is a link and whenever the link is clicked, it will create an ajax request with jquery calling a page with a querystring prepend to the end of the url. My processwire main template is 1 main template called Default.php and many views depending on the template name (this is based on the example on the forum); so whenever Default.php capture the request, it will look into the querystring, if the querystring present it will just print the page without all the stylesheet/js/etc (see below):

<?php

  if($input->get->PW_TEMPLATE_NAME !== null) {
    $pageViewTemplate = $config->paths->templates . "views/{$page->template}.inc";
    if ( is_file($pageViewTemplate) ) {
        include($pageViewTemplate);
        return;
    }
  }
?>

<!DOCTYPE html>
<!--[if lt IE 7]>      <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]>         <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]>         <html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
  <head>
    <meta charset="utf-8">

 etc etc...

But whenever the querystring "PW_TEMPLATE_NAME" is not present then it will render the whole page with the stylesheet/js/etc (my setup as a fallback if the javascript is disabled, so people can see the page as if it is a link to a page). My question: can procache caches both of the versions: one with full template and one without the full template? as the output of the page is depending upon the querystring. Because when I try to enable the cache on this template, it only cache the one that render the whole full page, as a result whenever an ajax request is made the preview shows the whole page instead of just body copy.

I hope someone can explain how this might work or might not work This is a great module in itself Cheers

[ryan]

ProCache doesn't cache GET variables just because there would be an infinite number of possibilities to the point where it would fill up your hard drive with cache files. However, you can tell it what GET variables should bypass the cache. By default, any GET variable bypasses the cache. 

However, ProCache will cache URL segments, which can perform the same duty as GET variables. So in your case, if you enabled URL segments for your template, then you could cache both /path/to/page/ and path/to/page/TEMPLATE_NAME/. However, please be careful with any code that takes any kind of user input (especially GET variables, and URL segments to a lesser extent) and translates that to a file that gets included. This technique is full of security implications, so you've got to be especially careful.

Also see the pinned page on using URL segments with ProCache in the ProCache board (which you should now have access to).

[peterfoeng]

Hi Ryan,

Thanks for this. I have updated my code to follow the direction with urlSegment and works amazing - cant wait until we go live soon Cheers