Deleting file field's files via API not working or timing out

[Jason Huck]

16 hours ago, netcarver said:

If HAProxy's server connection times out before Apache finishes, you'll get a 50x error reported (not sure if 502 or 504) in the browser yet the Apache process will probably complete without error behind the dropped connection. Have you looked at the "timeout server" setting (and the other timeout settings) in your HAProxy config file (/etc/haproxy/haproxy.cfg)? I think you'll need to up that beyond where you have the timeout set for the PHP script execution.

Edited to add: Have you thought about moving this processing load into a background task? 

I've been exploring the HAProxy side of it as well, though nothing I have tried so far has made any difference. It seems very similar to what's described here:

http://serverfault.com/questions/664658/server-timeout-and-retry-in-haproxy-502

I've seen some evidence of a second request in my situation as well. I haven't looked into making it a background task yet.

[Jason Huck]

Here's something. I've just discovered that if I enable debug mode in PW, the script executes without error. I do get a warning on each line from the indexer module:

Warning: trim() expects parameter 1 to be string, array given in /var/www/html/HLND/site/modules/Indexer/Indexer.module on line 304

...but otherwise everything works as expected. If I turn debug mode off, I get the 502 Bad Gateway. I just toggled it on and off a few times to verify, and it's 100% reproducible.

What does enabling debug mode do that allows the script to complete?

[adrian]

Have you tried fixing the cause of that warning: https://github.com/USSliberty/Processwire-site-indexer/blob/master/Indexer.module#L186

It would be good to know if that warning is gone whether everything works as expected without debug mode on.

[Jason Huck]

2 hours ago, adrian said:

Have you tried fixing the cause of that warning: https://github.com/USSliberty/Processwire-site-indexer/blob/master/Indexer.module#L186

It would be good to know if that warning is gone whether everything works as expected without debug mode on.

Patched the Indexer module to get rid of the warning, but otherwise no change in behavior. Works fine with debug mode on, returns 502 with debug mode off.

[Jason Huck]

I had to set this project aside for a while, but recently came back to it. In the meantime, I had upgraded the site to PW 3.x and had hopes that this issue would magically disappear. It didn't, and subsequent attempts to further optimize the import routine didn't help.

At the moment, though, I can consistently eliminate the timeouts by temporarily enabling debug mode while performing the import, e.g.:

$config->debug = true;
$import = importProductData($filepath);
$config->debug = false;

I have no idea why this works, but so far, it does. I'll have to see if the timeouts return if the data set gets larger, but it's already quite a bit larger than it was back in August when I was last working on it.