Jump to content

Age verification vs. scrapers, robots, seo, lighthouse, etc etc etc.


heldercervantes
 Share

Recommended Posts

I was naive about this and got myself into a head scratcher on a new site that I'll showcase soon.

It's a booze site, so users have to make a pinky promise that they are of legal age to access it. And so I thought "hey, let's have a cookie store the age validation, and if that's unset I redirect them to a gate page. I even pass the original page's id in there so that they return to wherever they landed." That's what I did, on PHP, and it seemed to work really well.

It's all nice and dandy until I realise that when you attept to share the page on Facebook, the scraper is redirected too and everyone ends up sharing the gate page and not the cool cocktail recipe they wanted. Page indexing is probably compromised as well, and there's no way Lighthouse will ever tell me anything constructive about a page I want to test.

I can only think of two options this late at night. First one is to check the user agent on PHP and only redirect if it looks like an actual browser. The second is to move that logic to client side JS. Checking for user agent sounds super falible. Moving the logic to JS will probably let robots scan the page and social networks read the OG info, but I suspect Lighthouse will get caught in the redirection as well.

Any clever ideas on how to get around this?

Link to comment
Share on other sites

9 hours ago, Robin S said:

Why not use a JS modal to confirm age rather than redirecting to another page? Like this: https://www.vineonline.co.nz/

I'm actually combining three things into one: Age verification, cookies consent and language selection. The site also has a lot of micro animations as elements enter the page that I'd rather not have happening behind a pesky modal. Might have to give in to the modal approach, but I'm hoping I can avoid it somehow.

Link to comment
Share on other sites

I would do the user agent thing. The age “verification” is pure CYA anyway, so the less it’s triggered, the less annoying it is for everyone involved. I doubt anyone will get in trouble over it being bypassable by spoofing the UA as opposed to disabling JS or just spoofing the cookie.

edit: I just checked out Jack Daniels and Campari. Both use a JS modal. Jack Daniels’ can’t be completed with JS disabled, Campari’s doesn’t appear at all (however, without JS the site turns into an unusable mess anyway). Both sites load everything regardless of the modal so if you were 12 and too dumb to lie, you could still click view source and read all about the booze. I’ll pour one out for all the brain cells that died for nothing thinking about these things.

Link to comment
Share on other sites

Could you perhaps solve this by populating your page’s open graph (OG) tags with specific URLs rather then letting them auto populate from the actual link and then using URL segments or parameters to differentiate between real/spider visitors.

I think by populating OG tags it will tell fb where to pull the thumbnail, description etc from even if the human visitor has to go through the verification gateway first.

Link to comment
Share on other sites

On 10/28/2021 at 11:48 AM, Jan Romero said:

I would do the user agent thing.

This could be seen as cloaking by Google so be very careful doing things like that. More details here:

https://developers.google.com/search/docs/advanced/guidelines/cloaking?hl=en&visit_id=637711928893360718-3584950016&rd=1

 

Feedback on this from Google in the past:

 

Quite a nice read over on this blog about this topic:

https://www.findabledigitalmarketing.com/blog/age-verification-gates-and-seo/

  • Like 2
Link to comment
Share on other sites

Thanks for all the inputs. At least for now I'm sticking with user agent detection and keep an eye on it. I got it through Lighthouse and the major social network's scrapers, and will test it further to make sure there are no issues.

On 10/30/2021 at 1:22 PM, wbmnfktr said:

This could be seen as cloaking by Google

Yes, I'm worried about that, but I'm trying to avoid popup hell by combining age verification, cookie preference and language selection in a single place. I suppose those can all be placed in a popup as well, so I'll keep that option in mind if I run into trouble.

I'll post this project in the showcase section as soon as my client fills in the SEO stuff.

Link to comment
Share on other sites

Another thing I found in regards to this.
If you, your client or maybe someone who does their SEO or SEM campaigns... most of the tools won't work that great right now. 
Some (maybe even most) imitate regular clients (browser namespaces and browser strings) therefore end on the gate page and can't crawl the rest of the whole page.
So... they either have to manually change that, if possible, or they won't come that far or report wrong numbers.

Nothing to worry about right now but to keep in mind.

Update / An inofficial addition to this:
maybe look up how adult sites deal with it. I can't (or don't want to) link to those pages here. Maybe even use DuckDuckGo for research here.

  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...