Jason Huck

I had to set this project aside for a while, but recently came back to it. In the meantime, I had upgraded the site to PW 3.x and had hopes that this issue would magically disappear. It didn't, and subsequent attempts to further optimize the import routine didn't help.

At the moment, though, I can consistently eliminate the timeouts by temporarily enabling debug mode while performing the import, e.g.:

$config->debug = true;
$import = importProductData($filepath);
$config->debug = false;

I have no idea why this works, but so far, it does. I'll have to see if the timeouts return if the data set gets larger, but it's already quite a bit larger than it was back in August when I was last working on it.

3 hours ago, Lance O. said:

Is there a way to force "www" on the links generated in the sitemap.xml file?

Your web server should be configured to have a single canonical domain to avoid fragmentation of SEO and analytics, and your sitemap should use it. If the "www" version of the domain is canonical, requests for sitemap.xml will be redirected to that domain and the generated links should match. The module simply uses the value of $page->httpUrl. ProcessWire's .htaccess file includes directives for Apache to set a canonical domain, but they are commented out by default. Look around line 123:

  # -----------------------------------------------------------------------------------------------
  # 13. OPTIONAL: Redirect users to the 'www.' version of the site (uncomment to enable).
  # For example: http://processwire.com/ would be redirected to http://www.processwire.com/
  # -----------------------------------------------------------------------------------------------

  # RewriteCond %{HTTP_HOST} !^www\. [NC]
  # RewriteRule ^ http://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

2 minutes ago, Mike Rockett said:

Also true... Perhaps I should then default it to the ext parameter type then?

Makes sense to use the same list for both purposes, as long as it's configurable. For instance, in a current project we have to handle requests for .pdf files that are now served via a PW-powered download page.

7 minutes ago, Mike Rockett said:

Regarding the first one, perhaps those options should become unavailable the second the question mark is entered?

How would you distinguish between a literal question mark and its current use to make the previous character optional? Would it have to be escaped, or is that syntax changing in v2?

9 minutes ago, Mike Rockett said:

I did think about that second one, but didn't think about filtering extensions. Perhaps a simple check for \.([a-z0-9-]{3,5})...?

I configurable list of allowed extensions would ultimately be safer and more flexible. People create all sorts of crazy URLs, and we don't know what someone might need to allow/disallow.

17 hours ago, Mike Rockett said:

Yeah, it can be quite a tough subject - but a checkbox should cover it well-enough. That said, do we trim out any included query string so it does not get matched if the checkbox is selected?

Perhaps two options when creating a new rule:

[x] Source URL may include a query string.
    [x] Append query string to target URL.

You would only use these options if you did not need to match on the contents of the query string or use it to determine the correct target URL.

18 hours ago, Mike Rockett said:

This is a good idea, but I can imagine some would want a little control over it. Perhaps a global strict mode for trailing slashes (off by default, allowing both to match)?

Perhaps global options like so:

[x] Trailing slash is optional for source URLs which do not end with the following extensions:
    [.html, .htm, .php, .aspx, .cfml, .pdf ...]

I suppose you could include the ability to override at the rule level, but would it even be necessary?

On 11/14/2015 at 0:04 AM, Mike Rockett said:

In the discussion, I mentioned there would either be a checkbox in the entity editor that allows you to make query strings optional, or that we could use square parenthises to mark them as optional, like this: bioh[?gclid={token:any}].

An option to automatically pass along (but otherwise not match on) query strings would be ideal. I'm seeing a lot of automatically appended tracking codes breaking rules. For example:

?platform=hootsuite

For now, I assume i can do this:

path/to/page/[?platform=hootsuite]

...or even:

path/to/page/[?{token:any}]

...to zap any query string, but poor HootSuite won't get credit for the referral. Tough point to cover when training non-technical administrators to use the module, though.

A configuration option to automatically assume trailing slashes are optional would also be very handy, for the same reason. That way admins wouldn't have to remember to append ? to the end of every source URL.

Another issue is that source URLs are URL encoded when stored, even if the actual source wasn't. So for example, if the source URL contains an unencoded comma and/or ampersand:

path/to/this,that&theother/

...it gets stored like this:

path/to/this%2cthat%26theother/

...which, while corrected, is not the actual source URL, so it doesn't match.

In case anyone finds this thread because they're deploying a project started in PW2.7 or older to a ServerPilot-managed container (in which case, just updating the schema isn't sufficient, as many records may have been created with invalid values), ServerPilot recommends creating a separate .cnf file to disable sql strict mode:

https://serverpilot.io/community/articles/how-to-disable-strict-mode-in-mysql-5-7.html

Don't just copy and paste that configuration, though. The specific items that need to be removed are NO_ZERO_IN_DATE and NO_ZERO_DATE.

Creating a separate .cnf file for your customizations will avoid them being overwritten by automated ServerPilot updates.

Whoops, my bad. I had read this: http://cheatsheet.processwire.com/sanitizer/properties-and-methods/sanitizer-selectorvalue-value/

...combined with this: http://processwire.com/api/selectors/#sanitizing

...and spotted this: https://github.com/processwire/processwire/blob/master/wire/core/Sanitizer.php#L1506

...and since searches containing dashes weren't working in this project, had come to the conclusion that $sanitizer->selectorValue() considered dashes to be illegal characters and was replacing them with spaces, but that's clearly not the case.

I've since discovered that dashes are ignored characters by default in the Indexer module, which is what I'm using for search. I've taken that character out of the Ignored Characters field and reindexed all objects, but still not getting results. Most likely some old results are cached somewhere. I'll verify dashes are actually present in the indexer field, and take it from there. Thanks for the clarification!

Am I right in understanding that there's no way to escape or encode a dash ("-") for use within a selector value (that has to be sanitized)? Looking at the code for $sanitizer it looks like it's just converted to a space. That seemingly makes it impossible to search for terms like "x-ray." Any workarounds for this?

36 minutes ago, Jason Huck said:

Can anyone confirm whether this module works with PW3?

Gave it a quick try. It installed without error and created a .zip file, but it failed to copy over the contents of my /templates directory, and the web-based installer in a fresh PW3 installation fails to see the profile. I guess I'll just manually copy everything over for now.

Can anyone confirm whether this module works with PW3?

22 hours ago, BitPoet said:

No, the true culprit is here:

	protected function ___renderList() { 

		$out = "\n<ol id='{$this->id}_items' data-id='{$this->id}' data-name='{$this->name}'>" . 
			$this->renderListItem("Label", "1", "itemTemplate"); 

The pageSelected function in InputfieldPageAutocomplete.js then checks if the select page's id equals the value in the item template, and with "1" matching the id of "home", it returns without appending the page. I can't definitely say if this is intentionally excluding home, but my guess is that it was only meant to exclude items already present in the list in whatever scenario those might occur. In that case, excluding the template item in the selector for the duplicate checks would be sufficient:

	pageSelected: function($ol, page) {

		var dup = false;
		
		$ol.children('li:not(.itemTemplate)').each(function() {
			var v = parseInt($(this).children('.itemValue').text());	
			if(v == page.page_id) dup = $(this);
		});

 

I can confirm that this solves the issue for me in PW 2.7 with no apparent side effects thus far. Will report back if I discover otherwise. Thanks!

I took a really quick peek inside /wire/modules/Inputfield/InputfieldPage/InputfieldPage.module, and am wondering if this block, starting at line 175 within the isValidPage method, is where the problem lies?

		if($field->parent_id && $field->parent_id != $page->parent_id) {
			if(version_compare(PHP_VERSION, '5.3.8') >= 0) {
				if(is_subclass_of($field->inputfield, 'InputfieldPageListSelection')) {
					// parent_id represents a root parent
					$rootParent = wire('pages')->get($field->parent_id); 
					if(!$page->parents()->has($rootParent)) $valid = false;
				} else {
					// parent_id represents a direct parent
					$valid = false;
				}
				if(!$valid && $editPage) $editPage->set('_isValidPage', "Page $page does not have required parent $field->parent_id"); 
			} else {
				// PHP version prior to 5.3.8
				$reflector = new ReflectionClass($field->inputfield); 
				$valid = $reflector->implementsInterface('InputfieldPageListSelection'); 
			}
		}

Specifically, this line (180):

					if(!$page->parents()->has($rootParent)) $valid = false;

Would that prevent the home page from validating, since it doesn't have a root parent (parent_id is 0)? Seems like we need an exception for the home page here, yes?

On 9/30/2016 at 6:12 PM, Robin S said:

I can confirm the behaviour in PW3. I think it's a bug - will you raise a GitHub issue for this?

Done: https://github.com/processwire/processwire-issues/issues/19

I have a Page field with no restrictions on selection, set to use Page Autocomplete. Administrators need to be able to select the home page just like any other. When they type "Home" into the field, the home page appears as an option, but when it is selected, nothing happens.

There aren't any javascript errors, and other pages can be added as expected. I do notice in the AJAX call to get the menu results there is a query param "id>0" but "Home" does appear in the list, it just can't be selected.

Page Autocomplete is the only practical UI option due to the large (over 6k and growing) number of pages in the site.

Has anyone else seen this behavior? Is this a bug or am I missing something in the configuration? This is on PW 2.7.3.

@SamC in your example, all of your variables are things that you can access directly from $page, so you could, if you wanted, just pass in $page instead of all of those individual variables. If you have anything extra you want to add, you can just add it. Just be careful not to accidentally override the built-in properties & methods.
 

$page->any_key => 'any value';
$page->another_key => 'another value';
echo wireRenderFile($page->template->name.'.inc', array('page' => $page)); // everything in $page will now be available within this template

This isn't ProcessWire-specific, but the last time I checked, MailChimp hadn't updated their official PHP SDK for version 3.0 of their API (and all prior API versions will stop working at the end of 2016), so I have been using this handy-dandy little wrapper instead:

https://github.com/drewm/mailchimp-api

FWIW, I don't do it to enforce MVC at all, which IMO is often overkill for typical web projects. I also don't have the kind of duplication you've illustrated for each template, with e.g. both home.php AND home.inc files. I route everything through a single main.php file. That file is just a series of includes for various global functions and configuration, and at the bottom I use output buffering to include the page-specific view and wrap it with a global base template. I prefer doing it that way so that all of the global markup is in a single file rather than split across separate "head.inc" and "foot.inc" includes (which then have to live in every template file). I also like having view-specific js and css assets together in the same folder as the view. That makes more sense to me than organizing assets by file type.

I could just as easily have used wireRenderFile() instead of output buffering. I provided that as the example simply because it's available in PW and this is a PW forum. I also tend to store additional properties in $page rather than creating global variables (e.g., $page->foo = 'bar' instead of $foo = 'bar'), which makes it a lot easier to use wireRenderFile(). As long as you pass in $page, you know you've got everything you need, and you can add more properties later without worrying about updating your "view bag."

On 9/24/2016 at 11:31 AM, 3fingers said:

Could you please elaborate a bit more on how to achieve this with AIOM? @Jason Huck

It's pretty simple. AIOM accepts an array of files to include, and fails gracefully when one is missing. I keep the arrays in $page for portability. I load third party libraries and site wide stuff first, then layer on the view-specific assets if available, something like this:

<?php
    $page->styles = array(
        // third party libraries first...
        'lib/normalize-css/normalize.css',
        // ...etc...
        // site-wide/global assets next. this file imports many separate stub files...
        'styles/main.less',
        // now add the view file. you can check for its existence first,
        // but a missing file won't prevent the rest from working
        'views/'.$page->template->name.'/'.$page->template->name.'.less'
    );

    // do the same thing for scripts
    // $page->scripts = array(...
?>

...then in your template:

<link rel="stylesheet" href="<?= AIOM::CSS($page->styles); ?>">

...and:

<script src="<?= AIOM::JS($page->scripts); ?>"></script>

I set all my templates up initially as "no-file" templates, using the Create a new template without a file... option, then go to the Files tab and enter "main" as the Alternate Template Filename. This points all my "logical" templates to a single "physical" template at /site/templates/main.php. From there, you can include the actual template any number of ways, the simplest being something like this:

echo wireRenderFile('./views/'.$page->template->name.'/'.$page->template->name.'.inc');

The main reason I do it this way is that I also use AllInOneMinify for asset management, and I have it set up to automatically include an associated javascript and/or LESS file for each template if present, so for the "home" template, instead of having home.php loose in the templates directory, I have:

/site/templates/views/home/

...which contains:

home.inc
home.js
home.less

25 minutes ago, Mike Rockett said:

That seems strange - it really shouldn't take so long at all... If it is indeed Jumplinks that is causing that delay, v2 of the module will more than likely solve it.

Actually, I take that back - the handler would slow things down. That said, JL2 will speed things up.

This was never benchmarked in v1, so I believe you on this one. Perhaps some pagination is in order then... Or perhaps I should switch over to an AJAX-driven table ith filtering...?

It's the same 14-second delay in both cases (and only those cases), so I'm assuming it's related to jumplinks in some way, but admittedly I haven't tried to verify that.

Is tablesorter a standard widget in the PW admin UI? You might consider switching to DataTables ( https://www.datatables.net/ ) which would give you pagination, AJAX loading, keyword filtering, etc. in the default configuration...

On the topic of pagination for the main jumplinks page, in a current project, I've migrated over 6,000 URLs from a legacy site. The structure of the URLs changed significantly in multiple ways. I had to write a custom export script for the old system, as well as a custom import script for PW, to get everything to line up correctly. There wasn't any practical way to take advantage of wildcards or mapping collections that would have resulted in a reduction of the total number of entries — it's one-to-one.

Overall, it works well enough. There are two places where the number of entries seems to impact performance. For a legacy URL that has a jumplink, performance is fine. For anything that results in an actual 404, it can take a very long time (about 14 seconds on dev) to return the 404 page.

The other thing that takes a really long time to load of course is the main jumplinks page — so yeah, some pagination (and a keyword filter, actually) would be nice...

4 minutes ago, Mike Rockett said:

@Jason Huck - did you manage to see what the issue may be?

Yes, thanks for following up. As you pointed out, Jumplinks wasn't even being invoked. I had the basic-page template set to allow URL segments, and the custom logic for that was conflicting with the normal 404 handling (both automatic and via throwing a Wire404Exception manually). The fix was as simple as moving the 404 page to its own template.

Got this sorted by moving the 404 page to its own, separate template. In this particular project, I had set the basic-page template to allow urlsegments. The 404 page relies on those as well, to display the original URL while showing the 404 page, so my custom logic was conflicting with the default routine. I'm sure there was a good chance of creating some recursion issues with that setup, too.

4 hours ago, szabesz said:

Hi Jason, I think it must be related to PHP"s output buffering, which I do not use so sorry for not being able to help more here, but I guess you should examine the way it works on the server in question.

Note that I opted for wireRenderFile() to do the "buffering", that is to store rendered template partials in variables so that the complete page can be assembled later on.

That was a good thought, but replacing output buffering with wireRenderFile() didn't change the behavior.