Jump to content

Jason Huck

  • Content Count

  • Joined

  • Last visited

Posts posted by Jason Huck

  1. There's not much to see. If I hit a URL like


    ...which should redirect to:


    ...I instead end up on:


    The debug log starts off like this:

    404 Page Not Found
    Checked Tue, 13 Sep 2016 13:40:14 -0400
    Request: http://xxx.xxxxxxx.xxx/http404/
    ProcessWire Version: 2.6.1
    Scanning for jumplinks...
    [Checking jumplink #1]
    - Original Source Path:       about/community
    - Compiled Source Path:       about/community
    No match there...

    ...then iterates through all of the jumplinks, including the one it *should* match on:

    [Checking jumplink #3]
    - Original Source Path:       about/locations
    - Compiled Source Path:       about/locations
    No match there...

    ...and ends like this:

    No matches, sorry. We'll let your 404 error page take over when Debug Mode is turned off.


  2. I've just discovered that Jumplinks has stopped working on an older site. Same behavior on both dev and production -- the admin UI works fine, but none of the actual redirects happen on the front end; users are just bounced to the 404 page. When I enable (Jumplinks') debug mode, I see it skip right past the relevant entry for a given URL as if it couldn't find a match. The site is running PW 2.6.1 and Jumplinks 1.3. I tried upgrading Jumplinks to the latest version in dev, but it didn't help. Any suggestions on how to further diagnose the issue?


  3. 2 hours ago, adrian said:

    Have you tried fixing the cause of that warning: https://github.com/USSliberty/Processwire-site-indexer/blob/master/Indexer.module#L186

    It would be good to know if that warning is gone whether everything works as expected without debug mode on.

    Patched the Indexer module to get rid of the warning, but otherwise no change in behavior. Works fine with debug mode on, returns 502 with debug mode off.

  4. Here's something. I've just discovered that if I enable debug mode in PW, the script executes without error. I do get a warning on each line from the indexer module:

    Warning: trim() expects parameter 1 to be string, array given in /var/www/html/HLND/site/modules/Indexer/Indexer.module on line 304

    ...but otherwise everything works as expected. If I turn debug mode off, I get the 502 Bad Gateway. I just toggled it on and off a few times to verify, and it's 100% reproducible.

    What does enabling debug mode do that allows the script to complete?

  5. 16 hours ago, netcarver said:

    If HAProxy's server connection times out before Apache finishes, you'll get a 50x error reported (not sure if 502 or 504) in the browser yet the Apache process will probably complete without error behind the dropped connection. Have you looked at the "timeout server" setting (and the other timeout settings) in your HAProxy config file (/etc/haproxy/haproxy.cfg)? I think you'll need to up that beyond where you have the timeout set for the PHP script execution.

    Edited to add: Have you thought about moving this processing load into a background task? 

    I've been exploring the HAProxy side of it as well, though nothing I have tried so far has made any difference. It seems very similar to what's described here:


    I've seen some evidence of a second request in my situation as well. I haven't looked into making it a background task yet.

  6. So that was a bit of a red herring. For some reason, putting realpath in that position caused all of the file writes to fail, so it was just skipping the image field handling completely. What I'm seeing right now is that it can handle roughly 180 rows at a time. I can skip any number of rows at the beginning, and process any consecutive batch of rows successfully right up to around 180. It sometimes varies by a few rows. It's not an issue with any particular row either -- e.g., it fails at row 181, but if I skip a row it processes row 181 just fine and fails on row 182.


  7. Increasing the max_execution_time value has no effect. The script still returns a 502 in the same amount of time. It does run successfully if I limit the number of rows.

    I'm going to play with increasing the rows and/or skipping initial rows to try and see if it's an issue with the total amount of data, or a specific row that's causing problems.

    I'll also try just running the ->deleteAll() in isolation to see if the behavior changes.



  8. Sure, here's the whole thing. This is called from a custom admin page. These actions do occur in a loop (iterating over the lines of a CSV file). I changed ->parent to ->id in case of extra overhead (retrieving and creating a page object vs. retrieving an integer value). I do set the name, title, and status of each page. Switching to a normal image fieldtype vs. CroppableImage makes no difference. This is PW 2.8.28. Thanks for taking a look!

    	function importProductData($filepath){
    		$result = new StdClass();
    		$result->lines_total = 0;
    		$result->lines_added = 0;
    		$result->lines_failed = 0;
    		$sizes = [
    			5 => '¾”',
    			6 => '1”',
    			7 => '1¼”',
    			8 => '1½”',
    			9 => '2”',
    			11 => '3”'
    		// open the file for reading
    		if(!ini_get('auto_detect_line_endings')) ini_set('auto_detect_line_endings', true);
    		if(($f = fopen($filepath, 'r')) !== false){		
    			$header = true;
    			// parse as CSV
    			while(($line = fgetcsv($f, 0, ',', '"')) !== false){
    				// skip the header
    				if($header == true){
    					$header = false;
    				// process a line
    					$title = $line[1];
    					$brand = ''; // TODO: Get brand data.
    					$product_type = ''; // TODO: Determine product type.
    					$prod_cat = (int) str_replace('F', '', $line[4]);
    					$size = $sizes[$prod_cat];
    					$uom_raw = $line[2];
    					if($uom_raw == 'EA' || $uom_raw == 'PC'){
    						$uom = $uom_raw;
    						$uom_value = null;
    						$uom = 'FT';
    						$uom_value = (int) $uom_raw;
    					$weight = (float) $line[10];
    					$itemno_mill_standard = (int) $line[0];
    					$itemno_mill_stainless = (int) $line[11];
    					$itemno_anodized_standard = (int) $line[12];
    					$itemno_anodized_stainless = (int) $line[13];
    					$active = ($line[5] == 'A');
    					$image_data_raw = $line[17];
    					if(startsWith($image_data_raw, '0x')){
    						$has_image = true;
    						$image_data = hex2bin(ltrim($image_data_raw, '0x'));
    						$image_path = wire('config')->paths->templates.'tmp/'.$line[18];
    						file_put_contents($image_path, $image_data);
    						$has_image = false;
    					// look for a page with this primary item number
    					$product = wire('pages')->get('template=product-detail,itemno_mill_standard='.$itemno_mill_standard.',include=all,limit=1');
    					// if none is found, create a new page
    						$product = new Page();
    						$product->template = 'product-detail';
    						$product->parent = wire('pages')->get(1048);
    					// disable output formatting so that the page can be saved
    					// set fields
    					$product->name = wire('sanitizer')->pageName($title.' '.$line[3]); // slug is never seen, so append UPC code to ensure it is unique
    					$product->title = $title;
    					$product->brand = ''; // TODO: Get brand data.
    					$product->product_type = ''; // TODO: Determine product type.
    					$product->size = $size;
    					$product->uom = $uom;
    					$product->uom_value = $uom_value;
    					$product->weight = $weight;
    					$product->itemno_mill_standard = $itemno_mill_standard;
    					$product->itemno_mill_stainless = $itemno_mill_stainless;
    					$product->itemno_anodized_standard = $itemno_anodized_standard;
    					$product->itemno_anodized_stainless = $itemno_anodized_stainless;
    					// set published status
    					($active ? $product->removeStatus(Page::statusUnpublished) : $product->addStatus(Page::statusUnpublished));
    					// save the page
    					// add image, save page again, and remove temporary image
    						if($product->image) $product->image->deleteAll(); // get rid of anything that's already in there
    						$product->save('image'); // try saving after delete to avoid 502's
    						$product->save('image'); // save just the image field instead of the whole page
    				}catch(Exception $e){
    					wire('log')->save('import-errors', $itemno_mill_standard.': '.$e->getMessage());
    				// attempt to fix 502 errors introduced by ->deleteAll() above
    				// debugging
    				// if($result->lines_total > 10) break;
    			wire('log')->prune('import-errors', 30);
    			wire('pages')->error('Could not open '.$filepath.' for reading.');
    		return $result;
    	// collect output
    	$out = '<h2>Import Product Data</h2>';
    	// build the form
    	$filepath = $config->paths->templates.'tmp/';
    	$uploadform = $modules->get('InputfieldForm');
    	$uploadform->action = './';
    	$uploadform->method = 'post';
    	$field = $modules->get('InputfieldFile');
    	$field->label = 'CSV File';
    	$field->name = 'productdata';
    	$field->required = 1;
    	$field->extensions = 'csv txt tab';
    	$field->maxFiles = 1;
    	$field->overwrite = true;
    	$field->descriptionRows = 0;
    	$field->destinationPath = $filepath;
    	$field->uploadOnlyMode = true;
    	$submit = $modules->get('InputfieldSubmit');
    	// check for a submission
    		// empty the tmp directory prior to starting a new import,
    		// in case there were failures on a previous attempt that
    		// left files there
    		array_map('unlink', glob($filepath.'*.*'));
    		// validate the form
    		// process the form
    		$uploaded_files = $uploadform->get('productdata')->value;
    			$file = $uploaded_files->first();
    			$this->message('File uploaded successfully.');		
    			$filepath .= $file->name;
    			$import = importProductData($filepath);
    				$this->error('Processed '.$import->lines_total.' lines with '.$import->lines_failed.' errors. Check the import-errors log for details.');
    				$this->message('Processed '.$import->lines_total.' lines with no errors.');
    			// delete the uploaded file
    				$this->message('Removed uploaded file.');
    				$this->error('Could not delete uploaded file.');
    			$this->error('Uploaded file can\'t be found.');
    	// render the form, with or without errors
    	$out .= $uploadform->render(); 
    	// display the results
    	echo $out;

    ...and for the sake of completeness, here's the contents of functions.inc:

        // Excerpts a field to $limit length
        if(!function_exists('excerpt')) {
    		function excerpt($str, $limit = 400, $endstr = '…'){
    			$str = strip_tags($str);
    			if(strlen($str) <= $limit) return $str;
    			$out = substr($str, 0, $limit);
    			$pos = strrpos($out, " ");
    			if($pos > 0) $out = substr($out, 0, $pos);
    			return $out .= $endstr;
    	// Check if a string begins with another string.
    	function startsWith($haystack, $needle){
    		 $length = strlen($needle);
    		 return (substr($haystack, 0, $length) === $needle);


  9. In this particular case, I'm iterating over lines in a CSV file, looking up pages one at a time, and updating (or creating new ones) as needed. So I'm never creating a PageArray (AFAIK, anyway). It's possible that ->deleteAll() is no less performant than other things I'm doing, but it just happens to be the thing that pushed this script over the edge. I mention it here because someone else seemed to have the same problem. The relevant code:

    // look for a page with this primary item number
    $product = wire('pages')->get('template=product-detail,itemno_mill_standard='.$itemno_mill_standard.',include=all,limit=1');
    // if none is found, create a new page
        $product = new Page();
        $product->template = 'product-detail';
        $product->parent = wire('pages')->get(1048); // magic number: ID of the parent for all product pages
    // disable output formatting
    // set individual fields (name, title, etc.)...
    // set published status
    ($active ? $product->removeStatus(Page::statusUnpublished) : $product->addStatus(Page::statusUnpublished));
    // save the page
    // add image, save page again, and remove temporary image
        if($product->image) $product->image->deleteAll(); // get rid of anything that's already in there


  10. Thanks for the suggestion, not using Tracy on this site.

    I've run plenty of imports similar to this one before, including one recently (on a different project) that imported over 6k pages, and never got a 502 until I added ->deleteAll(). Maybe I need to consider some sort of bulk operation to remove all the images for this template in one shot, rather than one page at a time. I doubt there's anything like that in the API, though. Plus, I'd prefer to only delete the images as needed, in case some other condition in the script causes that entire record to be skipped.


  11. The custom dialogs for inserting links and images in CKEditor are missing some features available in a stock install that routinely cause me headaches. Two that come to mind right now are:

    - When inserting an image, you should be able to enter just one dimension and leave the other one blank, and you should be able to use other units besides pixels (for instance, percentages).

    - When inserting a mailto: link, you should be able to specify a subject line.

    Would love to see these make their way into the PW dialogs.

  12. I'm using PW 2.8.28 with a CroppableImage field. I have an import routine that updates existing records when possible. The script runs fine unless I try to delete existing images before adding new ones. If I include $page->image->deleteAll() right before adding a new image (and after disabling output formatting), I get the following error (from nginx):

    502 Bad Gateway

    The server returned an invalid or incomplete response.


    Nothing relevant is logged in PW, or Apache or Nginx access or error logs, AFAICT. Note that it does seem to work, despite the error, but it doesn't make a great impression on clients. Any suggestions?


  13. I have a very similar situation. I'm trying to upload a CSV file to a custom admin page, created with the Admin Custom Pages module. I've constructed a simple form using the API. When I load the page, the form appears to render correctly, but it doesn't appear to be working 100% correctly.

    • Drag and drop isn't working. The browser just offers to open the dragged file. There are no JS errors in the console and it looks as if all the required JS assets are included on the page. If I instead click the "Choose File" button and select a file, it does appear to select a file (if I hover over the "Choose File" button, it displays the selected filename).

    • Submitting the form does upload the file, though overwrite isn't working. I get the following error when trying to re-upload the same file:

    "Refused file [my filename] because it is already on the file system and owned by a different field."


  14. Aside from the name vs. property issue with the Open Graph tags, it seems as though FB can be very picky about parsing og:image. It's easy to miss, because FB will infer the value based on other information on the page, but on a recent launch, in order to get it to "pass" the Open Graph Debugger, I had to add the following additional (supposedly optional) properties:

    • og:image:url
    • og:image:secure_url
    • og:image:type
    • og:image:width
    • og:image:height

    It would be nice if MarkupSEO was aware of image fields and would allow users to select the preview image from one. Then it could generate all of these additional properties automatically.


  15. I see, thanks. I wonder if there's any way to circumvent that, especially since there's nothing preventing non-superusers from actually editing the page if they know the link (or guess the ID). I didn't realize the Trash also only appears for super users. I would think that would be useful to other types of users as well.

  16. When logged into the admin as the superuser, the 404 page appears in the page tree per usual. When logged in as a "site administrator" (a custom role), the 404 page is missing. The 404 page uses the basic-page template and hasn't been edited or altered in any way. Under "who can access this page", it shows the expected permissions (site administrators can definitely edit basic pages), and in fact, if a site administrator navigates directly to the edit screen for the page, it works fine. So it's not a permissions issue per se, it just doesn't show up in the page tree for users in that role. It is a multilingual site, and English is not the default language, in case that's relevant. Any thoughts?


    • Like 1
  • Create New...