Age verification vs. scrapers, robots, seo, lighthouse, etc etc etc.

[heldercervantes]

I was naive about this and got myself into a head scratcher on a new site that I'll showcase soon.

It's a booze site, so users have to make a pinky promise that they are of legal age to access it. And so I thought "hey, let's have a cookie store the age validation, and if that's unset I redirect them to a gate page. I even pass the original page's id in there so that they return to wherever they landed." That's what I did, on PHP, and it seemed to work really well.

It's all nice and dandy until I realise that when you attept to share the page on Facebook, the scraper is redirected too and everyone ends up sharing the gate page and not the cool cocktail recipe they wanted. Page indexing is probably compromised as well, and there's no way Lighthouse will ever tell me anything constructive about a page I want to test.

I can only think of two options this late at night. First one is to check the user agent on PHP and only redirect if it looks like an actual browser. The second is to move that logic to client side JS. Checking for user agent sounds super falible. Moving the logic to JS will probably let robots scan the page and social networks read the OG info, but I suspect Lighthouse will get caught in the redirection as well.

Any clever ideas on how to get around this?

[Robin S]

4 minutes ago, heldercervantes said:

Any clever ideas on how to get around this?

Why not use a JS modal to confirm age rather than redirecting to another page? Like this: https://www.vineonline.co.nz/

[heldercervantes]

9 hours ago, Robin S said:

Why not use a JS modal to confirm age rather than redirecting to another page? Like this: https://www.vineonline.co.nz/

I'm actually combining three things into one: Age verification, cookies consent and language selection. The site also has a lot of micro animations as elements enter the page that I'd rather not have happening behind a pesky modal. Might have to give in to the modal approach, but I'm hoping I can avoid it somehow.

[Jan Romero]

I would do the user agent thing. The age “verification” is pure CYA anyway, so the less it’s triggered, the less annoying it is for everyone involved. I doubt anyone will get in trouble over it being bypassable by spoofing the UA as opposed to disabling JS or just spoofing the cookie.

edit: I just checked out Jack Daniels and Campari. Both use a JS modal. Jack Daniels’ can’t be completed with JS disabled, Campari’s doesn’t appear at all (however, without JS the site turns into an unusable mess anyway). Both sites load everything regardless of the modal so if you were 12 and too dumb to lie, you could still click view source and read all about the booze. I’ll pour one out for all the brain cells that died for nothing thinking about these things.

[JayGee]

Could you perhaps solve this by populating your page’s open graph (OG) tags with specific URLs rather then letting them auto populate from the actual link and then using URL segments or parameters to differentiate between real/spider visitors.

I think by populating OG tags it will tell fb where to pull the thumbnail, description etc from even if the human visitor has to go through the verification gateway first.

[wbmnfktr]

On 10/28/2021 at 11:48 AM, Jan Romero said:

I would do the user agent thing.

This could be seen as cloaking by Google so be very careful doing things like that. More details here:

https://developers.google.com/search/docs/advanced/guidelines/cloaking?hl=en&visit_id=637711928893360718-3584950016&rd=1

 

Feedback on this from Google in the past:

 

Quite a nice read over on this blog about this topic:

https://www.findabledigitalmarketing.com/blog/age-verification-gates-and-seo/

[heldercervantes]

Thanks for all the inputs. At least for now I'm sticking with user agent detection and keep an eye on it. I got it through Lighthouse and the major social network's scrapers, and will test it further to make sure there are no issues.

On 10/30/2021 at 1:22 PM, wbmnfktr said:

This could be seen as cloaking by Google

Yes, I'm worried about that, but I'm trying to avoid popup hell by combining age verification, cookie preference and language selection in a single place. I suppose those can all be placed in a popup as well, so I'll keep that option in mind if I run into trouble.

I'll post this project in the showcase section as soon as my client fills in the SEO stuff.

[wbmnfktr]

Another thing I found in regards to this.
If you, your client or maybe someone who does their SEO or SEM campaigns... most of the tools won't work that great right now. 
Some (maybe even most) imitate regular clients (browser namespaces and browser strings) therefore end on the gate page and can't crawl the rest of the whole page.
So... they either have to manually change that, if possible, or they won't come that far or report wrong numbers.

Nothing to worry about right now but to keep in mind.

Update / An inofficial addition to this:
maybe look up how adult sites deal with it. I can't (or don't want to) link to those pages here. Maybe even use DuckDuckGo for research here.