Jump to content
Mirza

Every time MySQL Queries are getting locked

Recommended Posts

I have built a system in processwire, which has more than 600K pages.

A team of 40 people is using the system, DB is from AWS with 16GB Ram.

But still, select queries are getting locked.

It would be great if someone suggests how to solve this problem.

Also note: We have around 48 fields in one template.

Thanks in advance.

screenshot.jpg

screenshot_1.png

Share this post


Link to post
Share on other sites

Try to avoid using persistent connections?

I see you bumped up the timeout from 10 to 50 seconds already - did you change the wire core file?

https://github.com/ryancramerdesign/ProcessWire/blob/master/wire/modules/Session/SessionHandlerDB/SessionHandlerDB.module#L69

There's an alternative session handler here: https://modules.processwire.com/modules/session-handler-redis/

But regardless, 600k pages is a lot... some tweaking for performance will probably be necessary on any setup.

Perhaps somebody else can chime in with suggestions. I never had to deal with such huge PW-installs.

Share this post


Link to post
Share on other sites

Queries locking code be about not setting a 'reasonable' limit to number of pages to return. It could also be about too many joins. Hard to say without know exactly what's going on. These threads should be useful:

 

 

Edited by kongondo

Share this post


Link to post
Share on other sites

Thank you all, for the quick Response.

Share this post


Link to post
Share on other sites

Mirza, did you ever figure out this locking problem?  I'm seeing the same behavior in our application.  Any insight you can share would be appreciated.

Share this post


Link to post
Share on other sites

I just ran into a similar problem a couple weeks ago. The fix that did it for us was increasing innodb_buffer_pool_size from 4G to 20G BUT, after that just a couple weeks ago I got errors with data corruption, can't say if it's related. 

Share this post


Link to post
Share on other sites
14 hours ago, elabx said:

from 4G to 20G BUT, after that just a couple weeks ago I got errors with data corruption, can't say if it's related. 

Hey @elabx ,  I am tempted to say that yes it's related - (edit) it's related to your issue, not the locking one - Are you on MySQL 32bit or 64bit ? 

 

Quoting the official doc :

Quote
Default Value 134217728
Minimum Value 5242880
Maximum Value (64-bit platforms) 2**64-1
Maximum Value (32-bit platforms) 2**32-1

The size in bytes of the buffer pool, the memory area where InnoDB caches table and index data. The default value is 134217728 bytes (128MB). The maximum value depends on the CPU architecture; the maximum is 4294967295 (232-1) on 32-bit systems and 18446744073709551615 (264-1) on 64-bit systems.

 

Share this post


Link to post
Share on other sites

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Brawlz
      Hi,
      I hope this is the correct section for my problem.
      All I need is a connection to an external Database and a query gettings some data. I do this in a processwire Page-Template. I am honestly not sure if it is a problem with processwire or my code:
      $host = ‚XXXXX’; $user = ‚XXXXX‘; $pass = ‚XXXXX‘; $db = ‚XXXXX‘; $port = ‚3306‘; $mydb = new Database($host, $user, $pass, $db , $port);  $result = $mydb->query("SELECT * FROM char“);  while($row = $result->fetch_assoc()) {  print_r($row);  }  
      Produces the following error:
      Error: Exception: DB connect error 2002 - Connection timed out (in /customers/9/4/e/XXXX.de/httpd.www/wire/core/Database.php line 79)
       
      I also tried connecting without the $port variable but got the same error.
    • By Mobiletrooper
      Hey Ryan, hey friends,
      we, Mobile Trooper a digital agency based in Germany, use ProcessWire for an Enterprise-grade Intranet publishing portal which is under heavy development for over 3 years now. Over the years not only the user base grew but also the platform in general. We introduced lots and lots of features thanks to ProcessWire's absurd flexibility. We came along many CMS (or CMFs for that matter) that don't even come close to ProcessWire. Closest we came across was Locomotive (Rails-based) and Pimcore (PHP based).
      So this is not your typical ProcessWire installation in terms of size.
      Currently we count:
      140 Templates (Some have 1 page, some have >6000 pages)
      313 Fields
      ~ 15k Users (For an intranet portal? That's heavy.)
      ~ 195 431 Pages (At least that's the current AUTOINCREMENT)
       
      I think we came to a point where ProcessWire isn't as scalable anymore as it used to be. Our latest research measured over 20 seconds of load time (the time PHP spent scambling the HTML together). That's unacceptable unfortunately. We've implemented common performance strategies like:
      We're running on fat machines (DB server has 32 gigs RAM, Prod Web server has 32gigs as well. Both are running on quadcores (xeons) hosted by Azure.
      We have load balancing in place, but still, a single server needs up to 20 sec to respond to a single request averaging at around about 12 sec.
      In our research we came across pages that sent over 1000 SQL queries with lots of JOINs. This is obviously needed because of PWs architecture (a field a table) but does this slow mySQL down much? For the start page we need to get somewhere around 60-80 pages, each page needs to be queried for ~12 fields to be displayed correctly, is this too much? There are many different fields involved like multiple Page-fields which hold tags, categories etc.
      We installed Profiler Pro but it does not seem to show us the real bottleneck, it just says that everything is kinda slow and sums up to the grand total we mentioned above.
      ProCache does not help us because every user is seeing something different, so we can cache some fragments but they usually measure at around 10ms. We can't spend time optimising if we can't expect an affordable benefit. Therefore we opted against ProCache and used our own module which generates these cache fragments lazily. 
      That speeds up the whole page rendering to ~7 sec, this is acceptable compared to 20sec but still ridiculously long.
      Our page consists of mainly dynamic parts changing every 2-5 minutes. It's different across multiple users based on their location, language and other preferences.
      We also have about 120 people working on the processwire backend the whole day concurrently.
       
      What do you guys think?
      Here are my questions, hopefully we can collect these in a wiki or something because I'm sure more and more people will hit that break sooner than they hoped they would:
       
      - Should we opt for optimising the database? Since >2k per request is a lot even for a mysql server, webserver cpu is basically idling at that time.
      - Do you think at this point it makes sense to use ProcessWire as a simple REST API?
      - In your experience, what fieldtypes are expensive? Page? RepeaterMatrix?
      - Ryan, what do you consider as the primary bottleneck of processwire?
      - Is the amount of fields too much? Would it be better if we would try to reuse fields as much as possible?
      - Is there an option to hook onto ProcessWires SQL builder? So we can write custom SQL for some selectors?
       
      Thanks and lots of wishes,
      Pascal from Mobile Trooper
       
       
    • By Sergio
      All of a sudden, with nothing changed on the database or server, a website was getting error when doing a search:
      Error: Exception: SQLSTATE[HY000]: General error: 23 Out of resources when opening file './your-database-name/pages_parents.MYD' (Errcode: 24 - Too many open files) (in /home/forge/example.com/public/wire/core/PageFinder.php line 413) #0 /home/forge/example.com/public/wire/core/Wire.php(386): ProcessWire\PageFinder->___find(Object(ProcessWire\Selectors), Array) #1 /home/forge/example.com/public/wire/core/WireHooks.php(723): ProcessWire\Wire->_callMethod('___find', Array) #2 /home/forge/example.com/public/wire/core/Wire.php(442): ProcessWire\WireHooks->runHooks(Object(ProcessWire\PageFinder), 'find', Array) #3 /home/forge/example.com/public/wire/core/PagesLoader.php(248): ProcessWire\Wire->__call('find', Array) #4 /home/forge/example.com/public/wire/core/Pages.php(232): ProcessWire\PagesLoader->find('title~=EAP, lim...', Array) #5 /home/forge/example.com/public/wire/core/Wire.php(383): ProcessWire\Pages->___find('title~=EAP, lim...') #6 /home/forge/example.com/public/wire This error message was shown because: you are logged in as a Superuser. Error has been logged.  
      I tried several things, listed in this thread: https://serverfault.com/questions/791729/ubuntu-16-04-server-mysql-open-file-limit-wont-go-higher-than-65536
      But for some reason, MySQL was not getting its limit increased, but in the end, the one that did the trick was this:
      This worked for me on Ubuntu Xenial 16.04:
      Create the dir /etc/systemd/system/mysql.service.d
      Put in /etc/systemd/system/mysql.service.d/override.conf:
      [Service] LimitNOFILE=1024000 Now execute
      systemctl daemon-reload systemctl restart mysql.service Yes indeed, LimitNOFILE=infinity actually seems to set it to 65536.
      You can validate the above after starting MySQL by doing:
      cat /proc/$(pgrep mysql)/limits | grep files
    • By bot19
      I've never encountered this issue before. My local installing is using AMPPS 3.7, setup about 3 months ago.
      Everything was working fine last night, and I think possibly this morning, before it complete stopped with these errors. Please see below.
      I have not changed any config setttings, access settings.
      The only 2 other things I did yesterday was log into phpmyadmin and export the DB, then I think an Xcode update was installed yesterday.
      Can anyone help?
      I know people have run into these issues when they migrate, but I haven't migrated, setup, changed anything.
      Some people have said to just disable to do something like:
      $cfg['Servers'][$i]['AllowNoPassword'] = true; In phpMyAdmin's config.inc.php file. I tried but that didn't work.
      I even tried changing:
      $config->dbHost = 'localhost'; to 127.0.0.1, in site/config.php but no good.


    • By gebeer
      Hello,
      I would like to prevent future editing of a repeater item once a checkbox in the repeater item has been checked. Either the form inputs of the repeater item should be read only or only the field values should be rendered.
      Looking at the module code I can't seem to figure out where to hook for that functionality.
      Any ideas would be much appreciated.
×
×
  • Create New...