Jump to content

Odd CSV Import issue that ends up with fatal php error (FieldtypePage bug?))


Go to solution Solved by sjohnson,

Recommended Posts

[Posted this in another post, but realized it's doesn't really pertain to it, so I've moved it to this new topic -- again sorry]

Hi Ryan (and everyone else!),

Just starting using PW and I'm loving it... I've got a site build that requires a bunch of CSV loading and would like some help with API Importing, as I am running into some glitches with page fields.

So far I have a bunch of CSVs that work fine with my php code, but I have one that crashes, unless I put in the pageID.

I think it may be something larger as I have tried using both BatchChildEditor & ImportPagesCSV modules with a modified CSV, and both get the same error as my code.

So, here's my code, it does the following:

  • Reads a page field to get CSV filename
  • Reads 1st row of file to figure out what template and parent to use
  • Reads 2nd row for field titles
  • Processes the remaining rows as data, it also echoes out what it is doing -- not elegant, but it's only for my use
//    If no filename exit
    if ($wire->page->getUnformatted('section_slogan') == '') throw new Wire404Exception();
//    File Pointer    
    $myfile = "http://" . $config->httpHost . $config->urls->templates . "loaders/" . $wire->page->getUnformatted('csvfilename');
    $content = "<p>Using Loader File '{$myfile}'</p><hr/>";
//    Read The Data File
    if (($handle = fopen("$myfile", "r")) !== FALSE) {
    //    Get the Setup Info
        if (($data = fgetcsv($handle, 0, ",")) !== FALSE) {
            $myTemplate = $templates->get("$data[0]");
            $myParent = $wire->pages->get("path='$data[1]'");
            if (($myTemplate->id == 'NullPage') || ($myParent->id == 'NullPage')){
                throw new Wire404Exception();                
            }
        } else {
            throw new Wire404Exception();
        }
    //    Setup the parent
        $newItem = new Page();
        $newItem->template = $myTemplate;
    //    Tell User what's going on:    
        $content .= "<p>Adding New Pages based on the '{$newItem->template->name}' template to parent '{$myParent->title}' path: '{$myParent->path}'</p><hr/>";
    //    List out the fields in the template
        $num = count($newItem->fields);
        $content .= "<p> $num fields in template</p><ol>";
        foreach($newItem->fields as $itemfd) {
            $content .= "<li>" . $itemfd->name . "</li>";
        }
        $content .= "</ol><hr/>";
    //    Read header record and match-up fields to template
        $fieldmatchup = '';
        if (($data = fgetcsv($handle, 0, ",")) !== FALSE) {
            $num = count($data);
            for ($c=0; $c < $num; $c++) {
                foreach($newItem->fields as $itemfd) {
                    if($data[$c] == $itemfd->name) {
                        $fieldmatchup[$itemfd->name] = $c;
                    }
                }
            }            
        }
    //    Show the File's Field Array
        $content .= "<p>Display the Field Array</p><ul>";
        foreach($fieldmatchup as $item_key => $item_value) {
            $content .= "<li>['" . $item_key . "'] = '" . $item_value . "'</li>";
        }
        $content .= "</ul><hr/><h4>Reading File</h4><hr/>";
    //    Now add the data
        $row = 0;
        $newItem = '';
        while (($data = fgetcsv($handle, 0, ",")) !== FALSE) {
            $num = count($data)-1;
            $row++;
            $mySel = "parent=" . $myParent->id . ", title='" . $data[$fieldmatchup['title']] . "'";
            $dupPage = $wire->pages->get($mySel);
            if($dupPage->id == 'NullPage') {
                $content .= "<p>newItem (" . $mySel . ")<br/><b>-- Creating from File</b>.</p>\n";
            //    Setup blank WireArray based on a template
                $newItem = new Page();
                $newItem->template = $myTemplate;
                $newItem->title = $data[$fieldmatchup['title']];
                $newItem->parent = $myParent;
                $newItem->save();
            }
            else {
                $content .= "<p>Duplicate Found (" . $mySel . ")<br/><b>-- Updating from File</b>.</p>\n";
                $newItem = $dupPage;
            }
        $content .= "<p> {$num} fields in record {$row}: <br /></p>\n";
            $newItem->of(false);
            foreach($newItem->fields as $itemfd) {
                $key = $itemfd->name;
                if ($key != 'title') {
                    $value = $data[$fieldmatchup[$key]];
                    $content .= "[{$key}] = [{$value}]<br />\n";
                    $newItem->set($key, $value);
                }
            }
            $newItem->save();
            $content .= "<b>-- Saved: </b> <a target='_blank' href='" . $newItem->editUrl . "'>" . $newItem->title . " [" . $newItem->id . "]</a><hr/>";
        }
        fclose($handle);
    }
    $content .= "<b>{$row} records added/updated</b>";    

And here's a sample of data that works: (associated_stat & special_category are page fields)

skill,"/systems/sol/"
title,associated_stat,multiplier,special,special_category,body
Alien Archeology,Int,4,,,"Skill description"
Alien Tech,Tech,4,,,"Skill description"
Alien Weapons,Ref,4,1,Weapons,"Skill description"

And the file that's giving me issues:(pc_role_category, associated_book, sa_skill are page fields, career_skills is a multiple page field)

cp-role,"/systems/sol/"
title,pc_role_category,associated_book,page_no,verified_via,sa_skill,career_skills,body
"Merc","Combat Related","Book Name","8","Book Review","Combat Zen","Athletics|Alien Tech|Drive|Shoot","Role Description"

It's the associated_book field that's giving me the issue, if I blank it or put in the PageID it works, but with anything else it errors out with this:

Fatal Error
Call to a member function __unset() on boolean search

Source File: ...\core\wire\modules\Fieldtype\FieldtypePage.module:439

431: if($value instanceof Page) {
432: // ok
433: } else if($value instanceof PageArray) {
434: $value = $value->first();
435: } else if(is_string($value) || is_int($value)) {
436: $value = $this->sanitizeValueString($page, $field, $value);
437: if($value instanceof PageArray) $value = $value->first();
438: if($value->_FieldtypePage_remove === $value->id) {
439: $value->__unset('_FieldtypePage_remove');
440: $value = null; // remove item
441: }
442: }

So, looking at this field, and the associated page template, everything is setup the same as all the other page fields, except that the page template in question (books) has a field that references a page field (associated_system) that I'm not even referencing, so I'm not sure if that's the culprit or not, but that is the only thing that separated this template from the others is this custom label code and that the pages are outside of the parent:

post-3446-0-12908600-1465320441_thumb.pn

And yes, I've removed the custom label and it still has the same error.

With my luck it's something simple, but I can't see it... Any help would be appreciated :)

Link to post
Share on other sites
  • 2 weeks later...
  • Solution

Hi All,

I've figured out that this error seems to only happen when you are using the "Custom PHP Selection Code" on your Page Input Fields

It seems that PW uses it, but not completely, thereby causing unset errors.

I've done a bunch of rework and now my code works without issue, so in the hopes that this can help others, here's my revised code.

FYI -- This bug still exists, I'm just forcing my code to use the same selections as in the "Custom PHP Selection Code", thereby avoiding the unset errors.  Then again, maybe this is how it's supposed to work and I should have been duplicating the selection code the whole time :)

Also note that I use a folder structure like so:

  • Root
    • System 1
      • Skills
        • Skill 1
      • Roles
        • Role 1
    • System 2
      • Skills
      • Roles

Which allows me to have the same Skills, Roles, etc. names in different systems with different attributes, and still be able to link them properly.

I do this with code like "$myParent->parent->child('name=skills')->id" which allows me to join page fields within the same System together.

//	If no filename exit
	if ($wire->page->getUnformatted('section_slogan') == '') throw new Wire404Exception();
//	File Pointer	
	$myfile  = "http://" . $config->httpHost . $config->urls->templates . "loaders/" . $wire->page->getUnformatted('section_slogan');
	$content = "<p>Using Loader File '{$myfile}'</p><hr/>";
//	Read The Data File
	if (($handle = fopen("$myfile", "r")) !== FALSE) {
	//	Get the Setup Info
		if (($data = fgetcsv($handle, 0, ",")) !== FALSE) {
			$myTemplate = $templates->get("$data[0]");
			$myParent = $wire->pages->get("path='$data[1]'");
			if (($myTemplate->id == 'NullPage') || ($myParent->id == 'NullPage')){
				throw new Wire404Exception();				
			}
		} else {
			throw new Wire404Exception();
		}
	//	Setup the parent
		$errorLog = "";
		$newItem = new Page();
		$newItem->template = $myTemplate;
	//	Tell User what's going on:	
		$content .= "<p>Adding New Pages based on the '{$newItem->template->name}' template to parent '{$myParent->title}' path: '{$myParent->path}'</p><hr/>";
	//	List out the fields in the template
		$num = count($newItem->fields);
		$content .= "<div class='row'>";
		$content .= "<div class='col-sm-6'>";
		$content .= "<p> $num fields in template</p><dl class='dl-horizontal'>";
		foreach($newItem->fields as $itemfd) {
			$content .= "<dt>[" . $itemfd->name . "]</dt><dd><b>Type:</b>  " . $itemfd->type . "</dd>";
		}
		$content .= "</dl>";
	//	Read header record and match-up fields to template
		$fieldmatchup = '';
		if (($data = fgetcsv($handle, 0, ",")) !== FALSE) {
			$num = count($data);
			for ($c=0; $c < $num; $c++) {
				foreach($newItem->fields as $itemfd) {
					if($data[$c] == $itemfd->name) {
						$fieldmatchup[$itemfd->name] = $c;
					}
				}
			}			
		}
		$content .= "</div>";
		$content .= "<div class='col-sm-6'>";
	//	Show the File's Field Array
		$content .= "<p>Display the File's Field Array</p><dl class='dl-horizontal'>";
		foreach($fieldmatchup as $item_key => $item_value) {
			$content .= "<dt>[" . $item_key . "]</dt><dd>" . $item_value . "</dd>";
		}
		$content .= "</dl>";
		$content .= "</div>";
		$content .= "</div>";
		$content .= "<hr/><h4>Reading File</h4><hr/>";
	//	Now add the data
		$row = 0;
		$newItem = '';
		$content .= "<div class='row'>";
		while (($data = fgetcsv($handle, 0, ",")) !== FALSE) {
			$content .= "<div class='col-sm-6'><hr/><dl class='dl-horizontal'>";
			$num = count($data)-1;
			$row++;
			$mySel = "parent=" . $myParent->id . ", name=" . $sanitizer->pageName($data[$fieldmatchup['title']]) . "";
			$dupPage = $wire->pages->get($mySel);
			if($dupPage->id == 'NullPage') {
				$content .= "<dt>New Page:</dt><dd>[" . $mySel . "] -- Creating<br/> </dd>";
			//	Setup blank WireArray based on a template
				$newItem = new Page();
				$newItem->template = $myTemplate;
				$newItem->title = $data[$fieldmatchup['title']];
				$newItem->parent = $myParent;
				$newItem->save();
			} 
			else {
				$content .= "<dt>Duplicate Page:</dt><dd>[" . $mySel . "] -- Updating<br/> </dd>";
				$newItem = $dupPage;
			}
			$content .= "<dt>Record [{$row}]:</dt><dd>Field List<br/> </dd>";
			$newItem->of(false);
			$errorList  = "";
			foreach($newItem->fields as $itemfd) {
				$key = $itemfd->name;
				$value = $data[$fieldmatchup[$key]];
				if ($key != 'title') {
					if ($value == ""){
						$content .= "<dt>- Field:</dt><dd>[{$key}] = [{$value}] -- Clearing</dd>";
						$newItem->set($key, $value);
					} else {
						switch ($itemfd->type) {
							case 'FieldtypePage':
								$pageFilters = "";
								switch ($key) {
									case 'pc_role_category':
										$pageFilters = ", has_parent={$myParent->parent->child('name=categories')->id}";
										break;
									case 'sa_skill':
										$pageFilters = ", has_parent={$myParent->parent->child('name=skills')->id}";
										break;
									case 'career_skills':
										$pageFilters = ", has_parent={$myParent->parent->child('name=skills')->id}";
										break;
									case 'associated_book':
										$pageFilters = ", has_parent={$myParent->parent->child('name=books')->id}";
										break;
									default:
										$pageFilters = "";
								}
								$pageList = explode('|', $value);
								unset($pageIDs);
								$pageItem   = "";
								if (count($pageList) > 1) {
									$cnt = 0;
									foreach($pageList as $arrItem) {
										$cnt++;
										$value2 = $sanitizer->pageName($arrItem);
										$pageItem = wire('pages')->get("name={$value2}{$pageFilters}");
										if ($pageItem->id != 'NullPage') {
											$content .= "<dt>- Page Ref:</dt><dd><b>{$itemfd->label}</b> [{$key}][{$cnt}]: {$arrItem} -- Added</dd>\n";
											$pageIDs[] = "{$pageItem->id}";
										} else {
											$content   .= "<dt>- Page Ref:</dt><dd><b>{$itemfd->label}</b> [{$key}][{$cnt}]: {$arrItem} -- Not Found</dd>\n";
											$errorList .= "Page Ref: <b>{$itemfd->label}</b> [{$key}]: {$arrItem} -- Not Found<br/>";;
										}
									}
								} else {
										$value2 = $sanitizer->pageName($value);
										$pageItem = wire('pages')->get("name={$value2}{$pageFilters}");
										if ($pageItem->id != 'NullPage') {
											$content .= "<dt>- Page Ref:</dt><dd><b>{$itemfd->label}</b> [{$key}]: {$value} -- Added</dd>\n";
											$pageIDs[] = "{$pageItem->id}";
										} else {
											$content   .= "<dt>- Page Ref:</dt><dd><b>{$itemfd->label}</b> [{$key}]: {$value} -- Not Found</dd>\n";
											$errorList .= "Page Ref: <b>{$itemfd->label}</b> [{$key}]: {$value} -- Not Found<br/>";
										}									
								}
								if (isset($pageIDs)) {
									$newItemVal = implode('|', $pageIDs);
									$newItem->set($key, $newItemVal);
								}
							break;
							default:
								$content .= "<dt>- Field:</dt><dd>[{$key}] = [{$value}]</dd>";
								$newItem->set($key, $value);
						}
					}
				}
			}
			if ($errorList <> "") {
				$errorLog .= $errorList;
				$errorList = "<hr/><b>Loading Errors</b><hr/><p>" . $errorList . "</p>";
				$tempBody = $newItem->getUnformatted('body');
				if ($tempBody !== Null) {
					$tempBody = $tempBody . $errorList;
					$newItem->set('body', $tempBody);
				}
			}
			$newItem->save();
			$content .= "<dt> </dt><dd> </dd>";
			$content .= "<dt>-- Saved:</dt><dd><a target='_blank' href='" . $newItem->editUrl . "'>" . $newItem->title . " [" . $newItem->id . "]</a></dd>";
			$content .= "</dl>";
			$content .= "<hr/></div>";
		}
		$content .= "</div>";
		fclose($handle);
	}
	$content .= "<hr/><b>{$row} records added/updated</b>";
	if ( $errorLog !== "" ) $content .= "<hr/><b>Error Log</b><br/>{$errorLog}";

Hope this helps someone else!

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Vineet Sawant
      Hi,
      I'm trying to import some heavy data into Processwire, but I'm not sure what would be the best way to do it.
      Usually I use CSV to Pages plugin, but this time the data is too heavy(~40k rows with 10+ columns of excel sheet), thus this plugin can't help.
      I also tried Tasker plugin but I can't seem to go through the setup itself, it requires some template setup but I'm totally clueless about how to do it, so that plugin is not of any use either.
      I wanted to know from you guys how you do it and in future what would be the best way to migrate thousands of rows of data in to PW.
       
      Thanks.
       
       
    • By hellerdruck
      Hi all
      I need to export all the texts from a website to a translation company (as json or csv or txt...). How can this be done? Of course manually, but this website is huge and it would take me years...
      Also, as a second step, importing the translation ...
      Any ideas anyone? Tutorials? Plugins?
      Thanks for your help.
    • By spercy16
      After doing a Google search for the issue I saw several previous posts mentioning this same issue but cannot figure out how to fix it and shouldn't have to spend a half hour trying to. ProcessWire frequently logs out of the admin area after less than five minutes. It shouldn't time-out ever, and if someone wanted that option for security reasons they should be able to enable it through the settings in their admin panel. In addition to it not being the default setting, users also shouldn't need to edit your config files manually to change these kinds of settings. Please fix the major issue in a future release. It's absured imho that developers don't realize the inconvenience it places on other people if they have to login every time they switch back to that page. It has also logged me out without warning without any visual que that it did so. If someone was working on paragraphs of content they could easily lose their work do to this bug as well... Please don't refer me to a forum with a dozen possible solutions to the issue and fix it yourselves. I'm sick of looking at them. It's your job to troubleshoot your software, not your users!
    • By Rodd
      Hi everyone!
      I have a website in a production environment and I want to duplicate it in a local environment. I exported the content of the website (with the 'Site Profile Exporter' module) but I cannot use it actually. I've got an issue with the database. I imported this one in MAMP then.

      I also exported the pages (with the 'ProcessPagesExportImport' module), but I cannot import it to my local website because the fields don't exist. So I created this fields, but I have this error :
      How can I use the elements that already exist and are presents in my database? How can I duplicate correctly the templates, fields and pages?
      Thanks by advance
      PS: Sorry if my english is bad
       
    • By hellerdruck
      Hi all
      I need help with something. Situation: We have let's say 2'000 Files (Excel) that should be displayed (list with links) on a page. We'd need to filter these files by given Keywords or a tree structure or both. Now, I'm looking for a solution whereas our customer can synchronise the files from his local computer with the folder on the webserver. They will update and upload files on a daily basis. Therefore, it would need to synchronise rather than load the files manually in pages or repeaters. Maybe indexing would be an idea, too.
      Are there any modules for Processwire that would help achieving this? Could anyone point me in the right direction?
      Thanks in advance.
×
×
  • Create New...