Jump to content

How to convert my DB from MyISAM to InnoDB without destroy my whole project?


Pixrael
 Share

Recommended Posts

I'm in big trouble, I'm almost finished a big project (about whole year of work) and now when I'm preparing it for production, I find that if I create a lot of pages in bulk, the site hangs until the transactions are finished. This is a webservice that imports information in background from third parties. I had always done the tests with little information, but now in real use it's a big problem because every time the script runs the whole site halt.

The problem is when I start the project I leave the database Engine in MyISAM default setting without predicting the problem and now I need to change it to InnoDB, I already converted all the tables using the mysql ALTER TABLE command and add $config->dbEngine = 'InnoDB'; In site/config.php .. Now the site loads perfectly and the administration also, but when I try to save a new page occur an error of type:

SQLSTATE [42000]: Syntax error or access violation: 1064 syntax error, unexpected '@', expecting $end .. inside the function "execute" in WireDatabasePDO.php

What can I do to solve this? I really need to use InnoDB to be able to write to the DB without blocking the users!

Link to comment
Share on other sites

This is the query string that Tracy show:

queryString => "SELECT pages.id,pages.parent_id,pages.templates_id,MATCH(field_page_information.data) AGAINST('customer_email: user@domain.com') AS _score_field_pa ... "

This email is a parameter I asked for, to find other related pages, I just discovered that the error is not on saving is when I try to show the page on the front-end

 

 

 

 

Link to comment
Share on other sites

Yeah, I thought of that and that is why I was asking. 

I once ran into a similiar problem, unrelated to ProcessWire. But maybe it helps when you have a look into "BOOLEAN MODE" on InnoDB, when trying to select with an "@" symbol. 

Link to comment
Share on other sites

I checked other parts of the app and everything looks good, only that template file have the problems, this is the code in the file:

$data = $page->page_information; // this is a YAML field

// more code here

$warranties = $pages->find("template=page-warranty, page_information*=customer_email: {$data->customer->email}");

That working before on MyISAM version.. here I want to find product warranties related to the current email address

Link to comment
Share on other sites

Can you try using Sanitizer first? E.g.

$search = "customer_email: {$data->customer->email}";
$value = $sanitizer->selectorValue($email);
$warranties = $pages->find("template=page-warranty, page_information*={$value}");
Quote

Sanitizes a string value that needs to go in a ProcessWire selector

Always use this to sanitize any string values you are inserting in selector strings. 
This ensures that the value can't be confused for another component of the selector string. 
This method may remove characters, escape characters, or surround the string in quotes. 

 

Link to comment
Share on other sites

31 minutes ago, androbey said:

I once ran into a similiar problem, unrelated to ProcessWire. But maybe it helps when you have a look into "BOOLEAN MODE" on InnoDB, when trying to select with an "@" symbol. 

https://bugs.mysql.com/bug.php?id=74042

https://stackoverflow.com/questions/25088183/mysql-fulltext-search-with-symbol-produces-error-syntax-error-unexpected

https://stackoverflow.com/questions/8961148/mysql-match-against-when-searching-e-mail-addresses

You found a solution?

@ryan this is a comment from the second link: I fixed this same issue, (not a syntax error either - only occurs when string has '@' in it) and i fixed by changing to NATURAL LANGUAGE MODE instead of BOOLEAN MODE

Link to comment
Share on other sites

5 minutes ago, Craig said:

Can you try using Sanitizer first? E.g.


$search = "customer_email: {$data->customer->email}";
$value = $sanitizer->selectorValue($email);
$warranties = $pages->find("template=page-warranty, page_information*={$value}");

 

Tested, do not works.. I fixed your code here:

$value = $sanitizer->selectorValue($search); //instead $email

Thanks

Link to comment
Share on other sites

23 minutes ago, Pixrael said:

I fixed your code here:

Good spot, yes. I was typing quickly ?

What version is your MySQL server, and is there a fulltext index on the field_page_information.data column?

Link to comment
Share on other sites

20 minutes ago, Craig said:

What version is your MySQL server, and is there a fulltext index on the field_page_information.data column?

Laragon said: mysql-5.7.24 and Adminer said on Table: field_page_information:

PRIMARY pages_id
INDEX data(250)
FULLTEXT data
Link to comment
Share on other sites

Hi @androbey and @Craig thanks for your support!

I finally found the bottleneck and it's not in the database itself. So I'll rollback to MyISAM, because I can't limit the system to search in the DB using the customers email. Already tested write speeds on both engines and it's very similar, at least in the order of the 100s of pages.

The problem really is PHP, specifically curl, fopen, etc. when you execute a http request, these commands block php until it receives a response. Additionally, some of the web services that I use have a rate limit, which sometimes slows down the service responses. This really doesn't worry me if it happens in the background, because I don't need to have the data in real time. But when you join curl limitations and the rate limit together, the entire app stops, sometimes up to 30 sec! that is inadmissible.

I will look for a solution, and I will let you know here.

I think PHP should improve this if it wants to remain as an application development option. It must be updated to the new times, now almost everything is based on third-party services and the iterations between platforms, and everything flows through http calls.

Link to comment
Share on other sites

You might be interested in the ReachPHP project. (Not to be confused with the React from Javascript land).

It also sounds like your setup could do with some caching layers, workers, or some abstractions to manage the data you're dealing with from different services?

  • Like 1
Link to comment
Share on other sites

3 minutes ago, kongondo said:

Are you able to abstract this to run over CLI?

I have this implemented as a REST service and not as a script. I just tested the task scheduler on the server by calling the endpoint and it worked perfectly, the app did not freeze. I have read that this happens when you make the curl call (I use wirehttp) from the browser. I was testing the app using a Chrome extension very similar to Postman, perhaps it is the cause of the problem.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...