Hidden Pages showing in Google Searches

[bbeer]

Hi All

on quite a huge site we have several oem products (pages) that are set to hidden and are not supposed to show up in any search result. Unfortunately they show up in google searches, when searched for with the title.

how can we prevent them from showing up?

[LostKobrakai]

if($page->isHidden()){

echo '<meta name="robots" content="noindex">';

}

[bbeer]

Thanks LostKobrakai

will try this right away.

[LostKobrakai]

But this will only work if you don't deny robots access to those pages, otherwise they'll index just by links from other pages. At least that's what google says about it. 

[Sergio]

Notice that it's not guaranteed that other search engines will not index your content. A robot.txt can be ignored, it's not mandatory. The safest way to hide the pages is by unpublished them or restrict access via a form or an Apache password, for instance.

[adrian]

There is a protect hidden pages option in PageProtector.

Or you could do:

if($page->isHidden()){
    throw new Wire404Exception();
}

but I am not sure you whether you actually need to give users access to these hidden pages or not. With PageProtector logged in users will be able to see the hidden pages, so maybe that is OK for your use case?

[Peter Knight]

are set to hidden and are not supposed to show up in any search result.

AFAIK Set to hidden in ProcessWire only means that pages are hidden within PW when searched via the front end via a PW search. They've no relation to Googles index.

I've always thought that Hidden: Excluded from lists and searches should be split into three

Hidden: Hide from menus and PW selector calls (unless explicitly over-ridden)

UnSearchable: Doesn't display in search results performed via the website

Don't Index: Hide from Google bots and don't display in Google's index

I'm probably approaching this from a method I used in another CMS

[Macrura]

@Peter

AFAIK if they are hidden then they shouldn't show up in searches

to not index an internal link, add rel='nofollow' to any links to hidden pages, or add the noindex meta tag;

also make sure you don't have a sitemap listing those hidden pages

if there is no link to a page somewhere on your site then it shouldn't get indexed, unless i'm missing something

[bbeer]

@ adrian

I can't throw a 404, as this pages are used in lists. I don't know PageProtector but will have a look at.

@ Peter Knight

so far there is no UnSearchable or Don't Index right now as standard feature. However, that would be great.

@ Macrura

we have now added the noIndex meta tag to those hidden pages. I hope that will solve the problem. They don't show up in sitemaps nor in xml sitemaps.

[mr-fan]

In a former CMS there was a very practical "private" state - this means that this kind of pages/content could only be accessable if a user was logged in - but they where in the menu/lists - and you can combine both states hidden + private...

[Peter Knight]

Don't forget if you're using Pete's XML site map Module that it auto adds new pages to the site map. But ...

"It also adds a field called sitemap_ignore that you can add to your templates and exclude specific pages on a per-page basis. Again, this assumes that you wish to ignore that page's children as well."