Indexer

[pepita]

Hello everybody 

I like Processwire very much but I’m a newbie so I have a lot of questions ;-) I hope you understand my “English”.  Sorry about that.

I’ve installed the indexer. And now I have some troubles with the german umlaut (ä ö ü). If I search a word with an umlaut there are no results because these letters are not in the database. Should I first replace the umlauts in the search query through whitespace or is there a better way?

My second problem: I would like that the indexer also search in the content from the page field and not only in the title.

So I have a page “club” that has a page field to the page “members”. I don’t want to add every member separate in the page field because I use the members on different pages. So if there is a change I have only make on one site the update.

Sorry, it’s for me a bit difficult to explain.

Thanks for your help.

[pwired]

Hi,

Forum:

https://processwire.com/talk/topic/5416-fixing-pw-install-in-a-latin1-database-should-have-been-utf-8/

https://processwire.com/talk/topic/2053-force-utf-8-encoding-for-pw/

https://processwire.com/talk/topic/1735-get-data-from-a-latin-1-encoded-db-and-insert-it-into-pw/

General:

http://php.net/manual/en/function.utf8-encode.php#82210

Lots of inspirational solutions here:

http://stackoverflow.com/questions/910793/detect-encoding-and-make-everything-utf-8

Or maybe this:

https://github.com/neitanod/forceutf8

[Christophe]

Hello,

I don't know your "level" of mySQL.

It seems it has something to do with the collation.

You must be using utf8_general_ci. You could need utf8_unicode_ci, utf8_bin, or a language-specific collation, depending on your project.

utf8_swedish_ci seems to be used a lot in Germany.

Some are case-insensitive and others are case-sensitive (_cs, utf8_bin...).

All have their advantages and disadvantages of course.

For example, research "mysql german umlaut unicode" in Google (among others, the first links and also the links inside their targeted pages seems interesting).

I can't find it again, but I've seen, if not wrong, that we can just "query" with utf8_bin, or another collation, for example.

By indexer you mean http://modules.processwire.com/modules/indexer/?

[Far from a specialist, just trying to help.]

[pepita]

Yes, I mean http://modules.processwire.com/modules/indexer/

Thanks for your help. But it was not directly a problem with the charset. In the file “Indexer.module” on the line 310 was a preg_replace, which allows no German umlaute. So I replaced

$text = preg_replace('/[^A-za-z0-9]/', ' ', $text); 

through

$text = preg_replace('/[^A-za-z0-9äöüÄÖÜ]/', ' ', $text); 

and now it works

I think for my second problem is also the file “Indexer.module” the solution. I tried something but it doesn’t works.

foreach ($fields as $field):
   // Check if there are Repeaters
   if( $field->type == "FieldtypeRepeater" ):
      foreach( $page->get($field->name) as $repeater ):

          // Now we cicle trough all fields of repeaters.
          foreach( $repeater->fields as $rep_field ):
              $fulltext .= $this->extractTextFromField($rep_field, $repeater);
          endforeach;

       endforeach;
				
   //Fieldtype Page
   elseif( $field->type == "FieldtypePage" ):
	foreach( $page->get($field->name) as $fieldtype_page):
            // Now we cicle trough all fields of the page.
            foreach( $fieldtype_page->fields as $page_field ):
		//Check if there are Repeaters
		if( $page_field->type == "FieldtypeRepeater" ):
		   foreach( $page->get($page_field->name) as $repeater_page ):
			// Now we cicle trough all fields of repeaters.
			foreach( $repeater_page->fields as $rep_field ):
			    $fulltext .= $this->extractTextFromField($rep_field, $repeater_page);
			endforeach;
		    endforeach;
		else:
                    $fulltext .= $this->extractTextFromField($page_field, $fieldtype_page);
	        endif;
             endforeach; 
         endforeach;
 
//End Pagefield
				
  else:
      $fulltext .= $this->extractTextFromField($field, $page);
   endif;