Mobiletrooper

Hey friends, I will answer each of your questions below: @bernhard thanks for the link. Some good insights on $page->rootPage->find and "*=" selectors. Your summary sums it up perfectly, we need to load lots of different pages and fire hundreds/thousands of different complex selectors each customized per user. But a big chunk of the page is actually displaying content in a grid, so your module will come in handy although memory on our web server is not the issue currently. For concurrent edits to be honest I caused some misconception here, they work on different parts on the site concurrently but never on a single page together, so mySQL basically queues the inserts and performs them in sequence. @BitPoet we ruled that out already, latency averages at 0.500ms. We will implement better cache strategies later. The hidden JSON field is nice, this would at least reduce some joins and balances work between DB and Web. @Beluga We're giving InnoDB a shot now, thanks for the reminder. I will get back if that was the issue. @Jonathan Lahijani we're running latest PW master (= 3.0.98). We also had issues with ListerPro being unusable when querying specific fields. Site is slow in general, even without many users. We do not have debug mode enabled on prod. I think InnoDB is worth a shot at this point, also some cache strategies sound quite interesting. I will get back with more infos. Edit: InnoDB is working smooth, but performance is still mediocre. I'm checking whether or not joins are the reasons for the slowdowns we are expecting and try to fetch data from a single field. Thanks so far.

Hey Ryan, hey friends, we, Mobile Trooper a digital agency based in Germany, use ProcessWire for an Enterprise-grade Intranet publishing portal which is under heavy development for over 3 years now. Over the years not only the user base grew but also the platform in general. We introduced lots and lots of features thanks to ProcessWire's absurd flexibility. We came along many CMS (or CMFs for that matter) that don't even come close to ProcessWire. Closest we came across was Locomotive (Rails-based) and Pimcore (PHP based). So this is not your typical ProcessWire installation in terms of size. Currently we count: 140 Templates (Some have 1 page, some have >6000 pages) 313 Fields ~ 15k Users (For an intranet portal? That's heavy.) ~ 195 431 Pages (At least that's the current AUTOINCREMENT) I think we came to a point where ProcessWire isn't as scalable anymore as it used to be. Our latest research measured over 20 seconds of load time (the time PHP spent scambling the HTML together). That's unacceptable unfortunately. We've implemented common performance strategies like: We're running on fat machines (DB server has 32 gigs RAM, Prod Web server has 32gigs as well. Both are running on quadcores (xeons) hosted by Azure. We have load balancing in place, but still, a single server needs up to 20 sec to respond to a single request averaging at around about 12 sec. In our research we came across pages that sent over 1000 SQL queries with lots of JOINs. This is obviously needed because of PWs architecture (a field a table) but does this slow mySQL down much? For the start page we need to get somewhere around 60-80 pages, each page needs to be queried for ~12 fields to be displayed correctly, is this too much? There are many different fields involved like multiple Page-fields which hold tags, categories etc. We installed Profiler Pro but it does not seem to show us the real bottleneck, it just says that everything is kinda slow and sums up to the grand total we mentioned above. ProCache does not help us because every user is seeing something different, so we can cache some fragments but they usually measure at around 10ms. We can't spend time optimising if we can't expect an affordable benefit. Therefore we opted against ProCache and used our own module which generates these cache fragments lazily. That speeds up the whole page rendering to ~7 sec, this is acceptable compared to 20sec but still ridiculously long. Our page consists of mainly dynamic parts changing every 2-5 minutes. It's different across multiple users based on their location, language and other preferences. We also have about 120 people working on the processwire backend the whole day concurrently. What do you guys think? Here are my questions, hopefully we can collect these in a wiki or something because I'm sure more and more people will hit that break sooner than they hoped they would: - Should we opt for optimising the database? Since >2k per request is a lot even for a mysql server, webserver cpu is basically idling at that time. - Do you think at this point it makes sense to use ProcessWire as a simple REST API? - In your experience, what fieldtypes are expensive? Page? RepeaterMatrix? - Ryan, what do you consider as the primary bottleneck of processwire? - Is the amount of fields too much? Would it be better if we would try to reuse fields as much as possible? - Is there an option to hook onto ProcessWires SQL builder? So we can write custom SQL for some selectors? Thanks and lots of wishes, Pascal from Mobile Trooper

Hey thmsnhl, I'm using ProcessWire for a few projects at work as well as for my personal projects. Originally coming from Ruby On Rails I also encountered problems with our database synchronization when it comes to team development on a single project. Depending on the project we solved this using Field/Template Export/Import while also only working on certain parts of the page. Most projects are already finished in terms of template/field structure so there is no need to have an always up-to-date local database snapshot. If the project is not finished in terms of template/field structure we usually sit together and draft a "this-solves-everything"-kind of template/field and synchronize this to our local databases once and then just code ahead. Other frameworks solve this by using Migrations and although ProcessWire does not support Migrations in the core (as of now) you can use this module: https://modules.processwire.com/modules/migrations/ (Here are a few code samples and documentation: https://lostkobrakai.github.io/Migrations/). This module does not directly add tables and columns to the database like RoR migrations do but instead allows you to do nearly everything you can do in the Admin UI using code. You can for example create templates, move fields around, install modules, setup users etc. It can be a bit tedious at first to learn all the new APIs this module offers but the benefits, especially for large projects, speak for themselves. These migrations can be safely checked into your git repository and whenever one of your teammates encounters a problem with his out-to-date database he can simply run the migrations files. So instead of adding templates and fields on production you write them in code and deploy your changes to production, then you migrate either using the CLI utility or the admin backend. You can also integrate this into your build toolchain (like Amazon CodeStar). Check this code example I bet it feels quite familiar: <?php class Migration_2018_01_16_12_32 extends Migration { public static $description = "Add postalcode to a hotel"; public function update() { $this->insertIntoTemplate('hotel', 'postalcode', 'title'); $this->editInTemplateContext('home', 'postalcode', function($f){ $f->label = "Hotel Postalcode"; $f->columnWidth = 50; }); } public function downgrade() { $t = $this->templates->get('hotel'); $t->fieldgroup->remove('postalcode'); $t->fieldgroup->save(); } } This adds a `postalcode` field right after the title field to the hotel template. And it removes this field when you rollback your migration. This is how we tackle this problem.

We discovered a similar problems while debugging our page using Chrome Developer Tools. When you enter the mobile inspector, your user agent changes. If you then happen to use the ProcessWire Backend WITHOUT the mobile inspector, you'll get logged out. I followed the session tracelog and noticed that the fingerprint changed and thats why I got logged out. The fingerprint is dynamically created using a md5 hash with the request user agent & request_ip. Since the user agent changes in mobile inspector, your session fingerprint will be different and therefore processwire logs you out. Took me while to figure this out. Hope this helps others.