Additional fields/pages slowing down the whole system.

[NorbertH]

I got a small PW application running on a relatively small server. Recently i needed to extend the application with 20 fields and about 260 Pages.

I would expect that this possibly slows down the ProListers or pagetree where these additions are relevant , but this slows down the whole page. Even pages that have nothing to do whith it like the modules page. 

How is this possible ?     

The page rendering time gone up about 200ms from about 150ms. 

 

 

[dragan]

18 minutes ago, NorbertH said:

Recently i needed to extend the application with 20 fields and about 260 Pages.

For PW, that's really nothing.

The number of fields, templates or pages is almost irrelevant when it comes to performance. It's mainly what you do with these that can have an impact, i.e. complexity.

Do you have complex selectors running? Do you use repeaters excessively? And when you say "page rendering": frontend and backend? Or only one or the other?

e.g. what is known to slow page rendering down, is when you have lots of richtext-fields in a page (page-edit view), and lots of repeater/repeater matrix and similar fields. The browser needs some time until all that JS is doing its thing and can start render the DOM... In such cases it's worth to fine-tune field-settings.

I would try to copy the whole site somewhere else and compare speed. Maybe switch debug-mode off. Last, but certainly not least, use Tracy Debugger to analyze bottenecks.

[NorbertH]

The application is BE only , no frontend .
I added more or less just a land list whith many columns that i added. I never expected such an impact.

Debug mode has almost no impact 30 ms.  Tracy is installed , but i could need some hints how to use it to track those bottlenecks.

The only hook added just composes a field from several others.  No repeaters added . 

Concerning to ProDevTools 
The hook whith the biggest runtime was Tracy whith about 200 ms ?
Turning it of made the page significantly faster. 

Still page isn't really responsive but most time intensitive hook now takes  13 ms

Rendering of /site/templates/admin.php  still takes about 220 ms.
After Installing and adding modules it was about 60-80 ms which i feel is still very much as Applications like Dadabik come whith a rendering time of about 10 ms on this server.   

Ok turning tracy back on for further testing.

Concerning to tracy
Selector Queries (11) none of them took more than 1 ms
The PDO queries do not show how long they took... Is there a way to show this ?

Php Memmory limit is 128M
Peak of allocated memory 23.01 MB

PHP is 7.1.2
PW is 3.0.120 

 

In general it would be interesting what the factors are that slow down PW and why.

Modules installed ?
Number of Templates/Fields.?
Number of Pages?

And even on really high powered server i never had a rendering time  below 50 ms, feels kind of strange to me.   

[LostKobrakai]

1 hour ago, dragan said:

The number of fields, templates […] is almost irrelevant when it comes to performance.

That's not really true. The number of fields and templates does matter as those will be autoloaded for every request hitting processwire. It's not like 10 fields make a difference, but adding an additional 100 fields or 100 templates can indeed cause a slow down. It's different with pages though. Those are really only loaded on demand.

[NorbertH]

Why all fields or templates need to be loaded on any request ? Whats the reason , sounds unnecessary to me ?

[dragan]

@LostKobrakai I wasn't first sure whether the OP was talking about performance in FE or BE (or both), so that part of my remarks was assuming FE (of course, a tpl/page in the FE with 100 fields will certainly be slower than a page with only 10 fields...)

[adrian]

11 minutes ago, NorbertH said:

The hook whith the biggest runtime was Tracy whith about 200 ms

It would be good to know what part/panel of Tracy is causing the slowdown - I am thinking it's probably the Field List & Values section of the RequestInfo panel - I find on complex sites I tend to disable this and it can help significantly. 

[LostKobrakai]

It's part of ProcessWire's bootstrap process. I'm not exactly sure about the exact reasoning. 

@dragan FE or BE does not really matter. Fields/Templates are loaded even before anything request specific is executed.

[NorbertH]

Is there only a list of fields loaded or the actual field data.  Loading of all fields is a great bottleneck for building bigger applications  like maybe a CRM. 

[LostKobrakai]

It loads the actual data. All Field/Template classes are available once ProcessWire is bootstrapped.

[dragan]

1 hour ago, dragan said:

I would try to copy the whole site somewhere else and compare speed.

Did you try this? Either another remote server, or locally.

 

37 minutes ago, NorbertH said:

Php Memmory limit is 128M

Can you temporarily double that limit and see if that changes anything? (Drupal recommends this as the bare minimum, and Typo3's min. is 256M - I'm not comparing apples to oranges, but 128M in 2019 ain't really that much)

[NorbertH]

Double memmory has no effect (still only about 20Mb used ). Processor says 47% , so thats quite a lot.  

Copying to another a more poverfull server helps alot , moved a nother application whith same problem recently. This is a very small box so the problem comes up very early i guess. But as soon as the application starts growing drastically(200+ fields) its always the same slowdown. Although bigger server can handle 200-300 fields.    

This field/template loading would be a good thing for optimisation. I can not imagine why its necessary to load all Fields/Templates  every time.
More and more people are building applications using PW and you cannot always avoid to add fields or templates.
Maybe its possible to cache those field loads somehow or just load the fields needed ?

But ok, so if you plan on a big applications right now its necessary to save fields and templates wherever you can ?

What we have ...

Reuse fields as often as possible.
Eg. instead using one description field per dataset, generate 1 descriptions field and reuse it as often as possible.
But what do i do f if i have a webshop where each client has a invoice address and a delivery address.  So the client needs a double set for adress data?  Somwhere i read that you should combine multiple fields into one, is there an easy example for doing this somewhere ? Another idea would be making an address template and  page fields to link the reused address fields groups to the client , but there is no clean solution for editing those addresses in the client page and this ad at least a template and 2 fields.
What else can we do , and how to save templates ?      

 

[NorbertH]

So i guess i even have to reuse fieldset fields ?

[arjen]

First off, just increase raw power. Try bumping up your VPS with more power. Easy solution and this stuff is so cheap nowadays.

Upgrade the software if possible

I've had instances with 300+ fields on a template with more than 500k pages. Queries started to slow down around 200 fields. I bumped the Mariadb version to the latest, switched to INNOdb, PHP 7.2.x and it was running pretty smoother on a digitalocean medium sized VPS ($40/mo). We also had tons of runtime fields and we cached those so a page save would do the calculations instead of calculating runtime. When we eliminated those the application became really snappy with 200-300 concurrent key users   doing all kinds of imports and running reports while the 1000 regular concurrent users didn't notice lots of slowdown. Most of the slowdown nowadays come from the interface itself. We used a lot of jQuery kinda tooling inside ProcessWire admin and since the users don't have the fanciest computers, some browsers are not really capable of rendering the back-end. I guess we needed to switch to a SPA, but I'm not working on the project anymore.

58 minutes ago, NorbertH said:

Reuse fields as often as possible.

Jup, this is the key. Especially while desiging applications this is really important. 

58 minutes ago, NorbertH said:

But what do i do f if i have a webshop where each client has a invoice address and a delivery address.

Try to split up as many as you can. I don't store address information on the order, but on the client, and possible even on an address template which you can re-use. You can also use Page fields to combine all those.  To comply to GDPR you can also store the blob of address info in one or two fields on the order itself. So if a client requests to remove his data you can easily remove the accoun and associated addresses. And you still have the address data.

Another solution is to use the FieldsetPage to create on instance of all address/personal information. In the background PW creates another page containing the info. You add this field to the page once for invoice and once for the shipping info.

[NorbertH]

FieldsetPage is a good idea have to experiment with it.
As fas as i understand you can use the default saveReady hook for doing stuff with it .

Any ideas if you have multiple shipping adresses? Possibly a pagetable would do the job...

Whish we could put customized listers in those page  tabs.   

 

[NorbertH]

Hey , just found that the Tracy Panel selector shows rendering time and memory consumtion of all the panels. 

Maybe you like to see how this impacts on a slow system on my bigger server this is almost irrelevant. 

Results for pagetree:

I activated all for an overview .

But you where right the Request Info was consuming most of my time , as it was active by default. 

 

[arjen]

12 hours ago, NorbertH said:

Any ideas if you have multiple shipping adresses?

Create two templates:  "addresses" and "address". Put in a checkbox called "primary" and attach other fields (i.e. FieldtypeStreetAddress). Create a pagefield on the user template called "addresses". Now you can have as many addresses as you want, but when the user loads it won't instantly loads all the address field. Just the page field with the ID's. If you need to render it on the front-end with the primary first, you can easily sort or assign another class to highlight the primary address. Since the Page field is really powerful and we use a relational database, try to use it as much as you can. Be careful with too many joins, but for this use case it would suffice.

[adrian]

@NorbertH - thanks for the screenshot - nothing out of the ordinary there. There is no need to deactivate the RequestInfo panel, just uncheck the "Field List & Values" option from its settings:

That will reduce the size and time of that panel considerably.

The API Explorer and Captain Hook panels are naturally going to be large in size, but the time should be significantly lower the next time you load it because the content gets cached - for me it's 79ms.

Hope that helps a little.

[NorbertH]

Just, asking myself, why PW needs to load all fields and Templates at Startup. Schould be possible to store installed fileds/tamplates  in a DB and load this at startup , of possibly use some kind of autoloader? 

Is there some documentation about the Bootstrap process?
Possibly someone could point out where to find the relevant parts in the initialisation?

[LostKobrakai]

https://github.com/processwire/processwire/blob/649d2569abc10bac43e98ca98db474dd3d6603ca/wire/core/ProcessWire.php#L182-L217

This will point you to anything happening at startup. Most of the important stuff is part of ProcessWire::load, which triggers the initialization of most common api variables.

14 hours ago, NorbertH said:

Schould be possible to store installed fileds/tamplates  in a DB and load this at startup

That's exactly what is happening. On each request ProcessWire does load all fields/templates into memory from the db.

[NorbertH]

Hmmm , ok and concerning to Fields Array.php  Line 16

Quote

Per WireArray interface, only Field instances may be added

So if we expect a fully  complete  $fields array all of those fields need to be initialized right at the beginning too ?

The query itself shouldn't be so much time consuming?

Do i see this correct having too many unused ,modules / templates does a simmilar slowdown ?

 

 

[dragan]

 

On 2/20/2019 at 8:51 PM, NorbertH said:

Reuse fields as often as possible.

@NorbertH Ryan has created a few pro fields that are meant to be much more efficient, like Pro Fields Table and Textareas. Maybe you should try them out. Chances are, you can significantly reduce the number of fields, and hence, the number of DB queries. Most people just buy it for Repeater Matrix I guess, but Pro Fields has some other goodies as well...