Jump to content
recyclerobot

Processwire performance, serverload and server settings

Recommended Posts

Hi everyone,

let me first start off by expressing how much I love Processwire, I've succesfully integrated it with about 8 sites now and I'm just so in love with the simplicity (we love you ryan :D)

I was wondering how processwire scores on the performance/serverload front, how much processwires can I run on my VM with how much ram and are there any recommoned server apache settings?

currently I'm running an ubuntu with 4GB ram and 3 cores (1,6Ghz each),

I'm using php 5.1 with mod_cgi and the 8 processwire sites use about 1,6GB of RAM, having about 50k visitors/month across these sites I'm wondering how processwire will perform in case of 100k - 200k - 500k - 1M visitors, will the session system last (I noticed it's writing a cached file to the filesystem)?

also: what's the limit with the "new Page();" command, is it creating a new directory for every created page, what's the limit (the filesystem? the amout of ram?)

Share this post


Link to post
Share on other sites

Good questions RecycleRobot. This is really is going to vary from environment to environment, so my recommendations would be more general. Though awhile back Soma reported that some of his colleagues had run a stresstest called a "proxy sniffer load curves report", and the report came back quite good. But I don't know what all that entailed.

I recommend giving it 128M memory (the PHP 5.3 default) or more, in your php.ini. If you need to deal with lots of traffic, it's a good idea for your server to have a PHP opcode cache like APC or eAccelerator.

Beyond the server configuration, other performance factors depend on how you use ProcessWire. If you are doing anything heavyweight with the API towards markup generation, or just need to handle lots of traffic, it's a good idea to make use of PW's various caching options. From the template settings, you can tell it to use a cache from the 'cache' tab. Choose settings that maximize the lifetime of the cache, clearing only what you need to cleared when a page is saved. You'll also want to get familiar with the MarkupCache module, which enables you to cache individual pieces of markup in your templates. This is really handy for caching markup that may be expensive to build on every request, like larger navigation trees... or anything you want.

You'll also want to look at these best practices for long term scalability, as well as the field autojoin option. If you need to work with a lot of pages in memory at once during the same request, the $pages->uncacheAll(); function can clear memory when you need it (at the expense of having to reload any pages you try to access again).

also: what's the limit with the "new Page();" command, is it creating a new directory for every created page, what's the limit (the filesystem? the amout of ram?)

How much memory a Page takes up depends entirely upon how many fields it has and how many are loaded/populated. ProcessWire generally tries to keep Page objects as lightweight as possible, and doesn't load fields until you actually access them. As a result, you could feasibly get thousands of pages in memory if all they had were short 'title' fields. But you might only get a few hundred if those pages have their fields loaded and all in memory at once.

PW does create a new directory for each page at present. That behavior will become on-demand in the near future. But at present, this would mean your maximum quantity of pages (at least, pages that can also store files) could be limited by the file system and any hard limits present there.

I'm using php 5.1 with mod_cgi

ProcessWire is not compatible with PHP 5.1. If you are running it, I'm not sure how because 5.1 lacks some things that PW depends on. I'm guessing 5.1 was a typo, but if not, let us know as I may have missed something. I recommend running PHP 5.3 if possible to maximize performance. Also, I may be wrong about this, but I had thought that there was some overhead with running PHP as a CGI as opposed to running it as a module. If that's still the case (?), you may benefit from running PHP as an Apache module, but I'm out of area of knowledge on this one. :)

  • Like 2

Share this post


Link to post
Share on other sites

Hi Ryan,

Thanks for your answer, php 5.1 was a typo, I'm using 5.3.2 :)

I've done some tests running processwire as mod_cgi and running processwire as mod apache, both running a bare bone github install (main repo from today) and called a execution time and you're right: mod_cgi is a tiny tiny bit faster but uses alot more RAM while doing so (after 1000 refreshes, mod_cgi spiked at almost 300Mb in cycle, while mod_apache stayed at a steady 68Mb)

runtimes (after 1000 calls)

mod_cgi:
page runtime: 0.0476528 sec
mod apache:
page runtime: 0.0479234 sec

so I'm switching my server to mod apache, and looking into APC :)

Love your markupCache, I've been succcessfully using it on 2 projects now (caching large header elements)

Looking forward to the new Page > write to filesystem or not :)

again, thanks for everything!

  • Like 1

Share this post


Link to post
Share on other sites

Thanks for the testing and followup recyclerobot. Interesting about mod_cgi using up all that memory relative to mod_apache. Most servers I work on actually don't have an opcode cache, but I think it's a good thing to have for any PHP app that needs to handle large volumes of traffic. I'm running eAccelerator on this server, though don't think we needed it, but figured it can't hurt.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By LuisM
      Hi there,
      while developing a sideproject which is completly build with ProcessModules i suddenly had the urge to measure the performance of some modules 😉 as a result, say welcome to the FlowtiAppPerformance module. 
      It comes bundled with a small helper module called FlowtiModuleProfiler. In the first release, even though you could select other modules, it will track the execution of selected Site/ProcessModules. 
      This will give you the ability to gain insights how your Application behaves. 
      The Main Module itself will come with 2 Logging Options, Database or PW Logs. Select Database for Charts and Logs...well If you just want your profiles as a simple log file in PW. 
      You also could choose to dump the request profile into TracyDebugger as shown here:

      Dont wonder about my avg_sysload, somehow my laptop cant handle multiple VMs that good 😄
      Settings Screen

      Monitoring

      FlowtiLogs

      again, dont look at the sysload 😄
      I will update the Module in the future to give some filter options and aggregation, but for now it satisfies my needs. 
      I hope it is helpfull for some. 
      Module is submited to the directory and hosted at github
      https://github.com/Luis85/FlowtiAppPerformance
      Any suggestions, wishes etc. are much appreciated. 
       
      Cheers,
      Luis
    • By gebeer
      Hello all,
      wasn't sure where to put this, so it goes in General section.
      Ryan shows a hook that we can use to mirror files on demand from live server to development environment to be up to date with the files on the server without having to download complete site/assets/files folder.
      I just implemented this but had problems getting files to load from a site in development that is secured with user/password via htaccess.
      First I tried to use WireHttp setHeader method for basic authentication like this
      function mirrorFilesfromLiveServer(HookEvent $event) { $config = $event->wire('config'); $file = $event->return; if ($event->method == 'url') { // convert url to disk path $file = $config->paths->root . substr($file, strlen($config->urls->root)); } if (!file_exists($file)) { // download file from source if it doesn't exist here $src = 'http://mydomain.com/site/assets/files/'; $url = str_replace($config->paths->files, $src, $file); $http = new WireHttp(); // basic authentication $u = 'myuser'; $pw = 'mypassword'; $http->setHeader('Authorization: Basic', base64_encode("$u:$pw")); $http->download($url, $file); } } But, unfortunately this didn't work.
      So now I am using curl to do the download. My hook function now looks like this
      function mirrorFilesfromLiveServer(HookEvent $event) { $config = $event->wire('config'); $file = $event->return; if ($event->method == 'url') { // convert url to disk path $file = $config->paths->root . substr($file, strlen($config->urls->root)); } if (!file_exists($file)) { // download file from source if it doesn't exist here $src = 'http://mydomain.com/site/assets/files/'; $fp = fopen($file, 'w+'); // init file pointer $url = str_replace($config->paths->files, $src, $file); $u = 'myuser'; $pw = 'mypassword'; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_TIMEOUT, 50); // crazy high timeout just in case there are very large files curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_USERPWD, "$u:$pw"); // authentication curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC); // authentication curl_setopt($ch, CURLOPT_FILE, $fp); // give curl the file pointer so that it can write to it curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); $data = curl_exec($ch); curl_close($ch); } } Now I can load files and images from the htaccess protected development server 🙂
      If anyone knows how to get this to work with WireHttp, please let me know. Thank you.
    • By cosmicsafari
      Hi all,
      I have been asked by a client whether we can setup load balancing for their existing Processwire site.
      From my investigations on Google and within these forums, it definitely seems possible but as a newbie with a basic understanding of the subject im a bit lost.
      Does anyone know of any existing tutorials for settings up load balancing with PW?
      What items would need to be changed on their current stand alone install, is there a list of best practices worth consulting etc?
      As I understand it we would need to have some sort of copying mechanism (rsync script most likely) in order to make sure any uploaded assets are shared between the main server and the fallback ones, other than that im not sure what else would need to be ammended.
      Any thoughts/help would be greatly appreciated.
    • By Mobiletrooper
      Hey Ryan, hey friends,
      we, Mobile Trooper a digital agency based in Germany, use ProcessWire for an Enterprise-grade Intranet publishing portal which is under heavy development for over 3 years now. Over the years not only the user base grew but also the platform in general. We introduced lots and lots of features thanks to ProcessWire's absurd flexibility. We came along many CMS (or CMFs for that matter) that don't even come close to ProcessWire. Closest we came across was Locomotive (Rails-based) and Pimcore (PHP based).
      So this is not your typical ProcessWire installation in terms of size.
      Currently we count:
      140 Templates (Some have 1 page, some have >6000 pages)
      313 Fields
      ~ 15k Users (For an intranet portal? That's heavy.)
      ~ 195 431 Pages (At least that's the current AUTOINCREMENT)
       
      I think we came to a point where ProcessWire isn't as scalable anymore as it used to be. Our latest research measured over 20 seconds of load time (the time PHP spent scambling the HTML together). That's unacceptable unfortunately. We've implemented common performance strategies like:
      We're running on fat machines (DB server has 32 gigs RAM, Prod Web server has 32gigs as well. Both are running on quadcores (xeons) hosted by Azure.
      We have load balancing in place, but still, a single server needs up to 20 sec to respond to a single request averaging at around about 12 sec.
      In our research we came across pages that sent over 1000 SQL queries with lots of JOINs. This is obviously needed because of PWs architecture (a field a table) but does this slow mySQL down much? For the start page we need to get somewhere around 60-80 pages, each page needs to be queried for ~12 fields to be displayed correctly, is this too much? There are many different fields involved like multiple Page-fields which hold tags, categories etc.
      We installed Profiler Pro but it does not seem to show us the real bottleneck, it just says that everything is kinda slow and sums up to the grand total we mentioned above.
      ProCache does not help us because every user is seeing something different, so we can cache some fragments but they usually measure at around 10ms. We can't spend time optimising if we can't expect an affordable benefit. Therefore we opted against ProCache and used our own module which generates these cache fragments lazily. 
      That speeds up the whole page rendering to ~7 sec, this is acceptable compared to 20sec but still ridiculously long.
      Our page consists of mainly dynamic parts changing every 2-5 minutes. It's different across multiple users based on their location, language and other preferences.
      We also have about 120 people working on the processwire backend the whole day concurrently.
       
      What do you guys think?
      Here are my questions, hopefully we can collect these in a wiki or something because I'm sure more and more people will hit that break sooner than they hoped they would:
       
      - Should we opt for optimising the database? Since >2k per request is a lot even for a mysql server, webserver cpu is basically idling at that time.
      - Do you think at this point it makes sense to use ProcessWire as a simple REST API?
      - In your experience, what fieldtypes are expensive? Page? RepeaterMatrix?
      - Ryan, what do you consider as the primary bottleneck of processwire?
      - Is the amount of fields too much? Would it be better if we would try to reuse fields as much as possible?
      - Is there an option to hook onto ProcessWires SQL builder? So we can write custom SQL for some selectors?
       
      Thanks and lots of wishes,
      Pascal from Mobile Trooper
       
       
×
×
  • Create New...