Jump to content

Additional fields/pages slowing down the whole system.


NorbertH
 Share

Recommended Posts

I got a small PW application running on a relatively small server. Recently i needed to extend the application with 20 fields and about 260 Pages.

I would expect that this possibly slows down the ProListers or pagetree where these additions are relevant , but this slows down the whole page. Even pages that have nothing to do whith it like the modules page. 

How is this possible ?     

The page rendering time gone up about 200ms from about 150ms. 

 

 

Link to comment
Share on other sites

18 minutes ago, NorbertH said:

Recently i needed to extend the application with 20 fields and about 260 Pages.

For PW, that's really nothing.

The number of fields, templates or pages is almost irrelevant when it comes to performance. It's mainly what you do with these that can have an impact, i.e. complexity.

Do you have complex selectors running? Do you use repeaters excessively? And when you say "page rendering": frontend and backend? Or only one or the other?

e.g. what is known to slow page rendering down, is when you have lots of richtext-fields in a page (page-edit view), and lots of repeater/repeater matrix and similar fields. The browser needs some time until all that JS is doing its thing and can start render the DOM... In such cases it's worth to fine-tune field-settings.

I would try to copy the whole site somewhere else and compare speed. Maybe switch debug-mode off. Last, but certainly not least, use Tracy Debugger to analyze bottenecks.

  • Like 1
Link to comment
Share on other sites

The application is BE only , no frontend .
I added more or less just a land list whith many columns that i added. I never expected such an impact.

Debug mode has almost no impact 30 ms.  Tracy is installed , but i could need some hints how to use it to track those bottlenecks.

The only hook added just composes a field from several others.  No repeaters added . 

Concerning to ProDevTools 
The hook whith the biggest runtime was Tracy whith about 200 ms 🙂
Turning it of made the page significantly faster. 

Still page isn't really responsive but most time intensitive hook now takes  13 ms

Rendering of /site/templates/admin.php  still takes about 220 ms.
After Installing and adding modules it was about 60-80 ms which i feel is still very much as Applications like Dadabik come whith a rendering time of about 10 ms on this server.   

Ok turning tracy back on for further testing.

Concerning to tracy
Selector Queries (11) none of them took more than 1 ms
The PDO queries do not show how long they took... Is there a way to show this ?

Php Memmory limit is 128M
Peak of allocated memory 23.01 MB

PHP is 7.1.2
PW is 3.0.120 

 

In general it would be interesting what the factors are that slow down PW and why.

Modules installed ?
Number of Templates/Fields.?
Number of Pages?

And even on really high powered server i never had a rendering time  below 50 ms, feels kind of strange to me.   

Link to comment
Share on other sites

1 hour ago, dragan said:

The number of fields, templates […] is almost irrelevant when it comes to performance.

That's not really true. The number of fields and templates does matter as those will be autoloaded for every request hitting processwire. It's not like 10 fields make a difference, but adding an additional 100 fields or 100 templates can indeed cause a slow down. It's different with pages though. Those are really only loaded on demand.

  • Like 2
Link to comment
Share on other sites

11 minutes ago, NorbertH said:

The hook whith the biggest runtime was Tracy whith about 200 ms

It would be good to know what part/panel of Tracy is causing the slowdown - I am thinking it's probably the Field List & Values section of the RequestInfo panel - I find on complex sites I tend to disable this and it can help significantly. 

Link to comment
Share on other sites

1 hour ago, dragan said:

I would try to copy the whole site somewhere else and compare speed.

Did you try this? Either another remote server, or locally.

 

37 minutes ago, NorbertH said:

Php Memmory limit is 128M

Can you temporarily double that limit and see if that changes anything? (Drupal recommends this as the bare minimum, and Typo3's min. is 256M - I'm not comparing apples to oranges, but 128M in 2019 ain't really that much)

Link to comment
Share on other sites

Double memmory has no effect (still only about 20Mb used ). Processor says 47% , so thats quite a lot.  

Copying to another a more poverfull server helps alot , moved a nother application whith same problem recently. This is a very small box so the problem comes up very early i guess. But as soon as the application starts growing drastically(200+ fields) its always the same slowdown. Although bigger server can handle 200-300 fields.    

This field/template loading would be a good thing for optimisation. I can not imagine why its necessary to load all Fields/Templates  every time.
More and more people are building applications using PW and you cannot always avoid to add fields or templates.
Maybe its possible to cache those field loads somehow or just load the fields needed ?

But ok, so if you plan on a big applications right now its necessary to save fields and templates wherever you can ?

What we have ...

Reuse fields as often as possible.
Eg. instead using one description field per dataset, generate 1 descriptions field and reuse it as often as possible.
But what do i do f if i have a webshop where each client has a invoice address and a delivery address.  So the client needs a double set for adress data?  Somwhere i read that you should combine multiple fields into one, is there an easy example for doing this somewhere ? Another idea would be making an address template and  page fields to link the reused address fields groups to the client , but there is no clean solution for editing those addresses in the client page and this ad at least a template and 2 fields.
What else can we do , and how to save templates ?      

 

Link to comment
Share on other sites

First off, just increase raw power. Try bumping up your VPS with more power. Easy solution and this stuff is so cheap nowadays.

Upgrade the software if possible

I've had instances with 300+ fields on a template with more than 500k pages. Queries started to slow down around 200 fields. I bumped the Mariadb version to the latest, switched to INNOdb, PHP 7.2.x and it was running pretty smoother on a digitalocean medium sized VPS ($40/mo). We also had tons of runtime fields and we cached those so a page save would do the calculations instead of calculating runtime. When we eliminated those the application became really snappy with 200-300 concurrent key users   doing all kinds of imports and running reports while the 1000 regular concurrent users didn't notice lots of slowdown. Most of the slowdown nowadays come from the interface itself. We used a lot of jQuery kinda tooling inside ProcessWire admin and since the users don't have the fanciest computers, some browsers are not really capable of rendering the back-end. I guess we needed to switch to a SPA, but I'm not working on the project anymore.

58 minutes ago, NorbertH said:

Reuse fields as often as possible.

Jup, this is the key. Especially while desiging applications this is really important. 

58 minutes ago, NorbertH said:

But what do i do f if i have a webshop where each client has a invoice address and a delivery address.

Try to split up as many as you can. I don't store address information on the order, but on the client, and possible even on an address template which you can re-use. You can also use Page fields to combine all those.  To comply to GDPR you can also store the blob of address info in one or two fields on the order itself. So if a client requests to remove his data you can easily remove the accoun and associated addresses. And you still have the address data.

Another solution is to use the FieldsetPage to create on instance of all address/personal information. In the background PW creates another page containing the info. You add this field to the page once for invoice and once for the shipping info.

  • Like 4
Link to comment
Share on other sites

FieldsetPage is a good idea have to experiment with it.
As fas as i understand you can use the default saveReady hook for doing stuff with it .

Any ideas if you have multiple shipping adresses? Possibly a pagetable would do the job...

Whish we could put customized listers in those page  tabs.   

 

Link to comment
Share on other sites

Hey , just found that the Tracy Panel selector shows rendering time and memory consumtion of all the panels. 

Maybe you like to see how this impacts on a slow system on my bigger server this is almost irrelevant. 

Results for pagetree:

Tracy.png.ed84a410cdf999a18daae18b736042ec.png

I activated all for an overview .

But you where right the Request Info was consuming most of my time , as it was active by default. 

 

Link to comment
Share on other sites

12 hours ago, NorbertH said:

Any ideas if you have multiple shipping adresses?

Create two templates:  "addresses" and "address". Put in a checkbox called "primary" and attach other fields (i.e. FieldtypeStreetAddress). Create a pagefield on the user template called "addresses". Now you can have as many addresses as you want, but when the user loads it won't instantly loads all the address field. Just the page field with the ID's. If you need to render it on the front-end with the primary first, you can easily sort or assign another class to highlight the primary address. Since the Page field is really powerful and we use a relational database, try to use it as much as you can. Be careful with too many joins, but for this use case it would suffice.

  • Like 1
Link to comment
Share on other sites

@NorbertH - thanks for the screenshot - nothing out of the ordinary there. There is no need to deactivate the RequestInfo panel, just uncheck the "Field List & Values" option from its settings:

image.png.36da42fe0f90930b5bbbb8e734396022.png

That will reduce the size and time of that panel considerably.

The API Explorer and Captain Hook panels are naturally going to be large in size, but the time should be significantly lower the next time you load it because the content gets cached - for me it's 79ms.

Hope that helps a little.

  • Like 1
Link to comment
Share on other sites

  • 3 months later...

Just, asking myself, why PW needs to load all fields and Templates at Startup. Schould be possible to store installed fileds/tamplates  in a DB and load this at startup , of possibly use some kind of autoloader? 

Is there some documentation about the Bootstrap process?
Possibly someone could point out where to find the relevant parts in the initialisation?

Link to comment
Share on other sites

https://github.com/processwire/processwire/blob/649d2569abc10bac43e98ca98db474dd3d6603ca/wire/core/ProcessWire.php#L182-L217

This will point you to anything happening at startup. Most of the important stuff is part of ProcessWire::load, which triggers the initialization of most common api variables.

14 hours ago, NorbertH said:

Schould be possible to store installed fileds/tamplates  in a DB and load this at startup

That's exactly what is happening. On each request ProcessWire does load all fields/templates into memory from the db.

  • Like 2
Link to comment
Share on other sites

Hmmm , ok and concerning to Fields Array.php  Line 16

Quote

Per WireArray interface, only Field instances may be added

So if we expect a fully  complete  $fields array all of those fields need to be initialized right at the beginning too ?

The query itself shouldn't be so much time consuming?

Do i see this correct having too many unused ,modules / templates does a simmilar slowdown ?

 

 

Link to comment
Share on other sites

  • 5 months later...

 

On 2/20/2019 at 8:51 PM, NorbertH said:

Reuse fields as often as possible.

@NorbertH Ryan has created a few pro fields that are meant to be much more efficient, like Pro Fields Table and Textareas. Maybe you should try them out. Chances are, you can significantly reduce the number of fields, and hence, the number of DB queries. Most people just buy it for Repeater Matrix I guess, but Pro Fields has some other goodies as well...

  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...