How to convert my DB from MyISAM to InnoDB without destroy my whole project?

[Pixrael]

I'm in big trouble, I'm almost finished a big project (about whole year of work) and now when I'm preparing it for production, I find that if I create a lot of pages in bulk, the site hangs until the transactions are finished. This is a webservice that imports information in background from third parties. I had always done the tests with little information, but now in real use it's a big problem because every time the script runs the whole site halt.

The problem is when I start the project I leave the database Engine in MyISAM default setting without predicting the problem and now I need to change it to InnoDB, I already converted all the tables using the mysql ALTER TABLE command and add $config->dbEngine = 'InnoDB'; In site/config.php .. Now the site loads perfectly and the administration also, but when I try to save a new page occur an error of type:

SQLSTATE [42000]: Syntax error or access violation: 1064 syntax error, unexpected '@', expecting $end .. inside the function "execute" in WireDatabasePDO.php

What can I do to solve this? I really need to use InnoDB to be able to write to the DB without blocking the users!

[Pixrael]

?

[androbey]

Hi @Pixrael, 

do you have some more information about the query which causes the error? Is it a 'SELECT' query? 

[Pixrael]

This is the query string that Tracy show:

queryString => "SELECT pages.id,pages.parent_id,pages.templates_id,MATCH(field_page_information.data) AGAINST('customer_email: user@domain.com') AS _score_field_pa ... "

This email is a parameter I asked for, to find other related pages, I just discovered that the error is not on saving is when I try to show the page on the front-end

 

 

 

 

[androbey]

Yeah, I thought of that and that is why I was asking. 

I once ran into a similiar problem, unrelated to ProcessWire. But maybe it helps when you have a look into "BOOLEAN MODE" on InnoDB, when trying to select with an "@" symbol. 

[Pixrael]

I checked other parts of the app and everything looks good, only that template file have the problems, this is the code in the file:

$data = $page->page_information; // this is a YAML field

// more code here

$warranties = $pages->find("template=page-warranty, page_information*=customer_email: {$data->customer->email}");

That working before on MyISAM version.. here I want to find product warranties related to the current email address

[Craig]

Can you try using Sanitizer first? E.g.

$search = "customer_email: {$data->customer->email}";
$value = $sanitizer->selectorValue($email);
$warranties = $pages->find("template=page-warranty, page_information*={$value}");

Quote

Sanitizes a string value that needs to go in a ProcessWire selector

Always use this to sanitize any string values you are inserting in selector strings. 
This ensures that the value can't be confused for another component of the selector string. 
This method may remove characters, escape characters, or surround the string in quotes. 

 

[Pixrael]

31 minutes ago, androbey said:

I once ran into a similiar problem, unrelated to ProcessWire. But maybe it helps when you have a look into "BOOLEAN MODE" on InnoDB, when trying to select with an "@" symbol. 

https://bugs.mysql.com/bug.php?id=74042

https://stackoverflow.com/questions/25088183/mysql-fulltext-search-with-symbol-produces-error-syntax-error-unexpected

https://stackoverflow.com/questions/8961148/mysql-match-against-when-searching-e-mail-addresses

You found a solution?

@ryan this is a comment from the second link: I fixed this same issue, (not a syntax error either - only occurs when string has '@' in it) and i fixed by changing to NATURAL LANGUAGE MODE instead of BOOLEAN MODE

[Pixrael]

5 minutes ago, Craig said:

Can you try using Sanitizer first? E.g.

$search = "customer_email: {$data->customer->email}";
$value = $sanitizer->selectorValue($email);
$warranties = $pages->find("template=page-warranty, page_information*={$value}");

 

Tested, do not works.. I fixed your code here:

$value = $sanitizer->selectorValue($search); //instead $email

Thanks

[Craig]

23 minutes ago, Pixrael said:

I fixed your code here:

Good spot, yes. I was typing quickly ?

What version is your MySQL server, and is there a fulltext index on the field_page_information.data column?

[Pixrael]

20 minutes ago, Craig said:

What version is your MySQL server, and is there a fulltext index on the field_page_information.data column?

Laragon said: mysql-5.7.24 and Adminer said on Table: field_page_information:

PRIMARY	pages_id
INDEX	data(250)
FULLTEXT	data

[Pixrael]

Hi @androbey and @Craig thanks for your support!

I finally found the bottleneck and it's not in the database itself. So I'll rollback to MyISAM, because I can't limit the system to search in the DB using the customers email. Already tested write speeds on both engines and it's very similar, at least in the order of the 100s of pages.

The problem really is PHP, specifically curl, fopen, etc. when you execute a http request, these commands block php until it receives a response. Additionally, some of the web services that I use have a rate limit, which sometimes slows down the service responses. This really doesn't worry me if it happens in the background, because I don't need to have the data in real time. But when you join curl limitations and the rate limit together, the entire app stops, sometimes up to 30 sec! that is inadmissible.

I will look for a solution, and I will let you know here.

I think PHP should improve this if it wants to remain as an application development option. It must be updated to the new times, now almost everything is based on third-party services and the iterations between platforms, and everything flows through http calls.

[Craig]

You might be interested in the ReachPHP project. (Not to be confused with the React from Javascript land).

It also sounds like your setup could do with some caching layers, workers, or some abstractions to manage the data you're dealing with from different services?

[kongondo]

21 hours ago, Pixrael said:

This is a webservice that imports information in background from third parties.

Are you able to abstract this to run over CLI?

[Pixrael]

3 minutes ago, kongondo said:

Are you able to abstract this to run over CLI?

I have this implemented as a REST service and not as a script. I just tested the task scheduler on the server by calling the endpoint and it worked perfectly, the app did not freeze. I have read that this happens when you make the curl call (I use wirehttp) from the browser. I was testing the app using a Chrome extension very similar to Postman, perhaps it is the cause of the problem.