Bypassing those annoying website registration forms

Picked this tip up off slashdot - some websites (e.g. the Washington Post) insist on making you register to read their content, even if the content is actually free - the assumption is that they only want the registration so they can harvest your email address and use it for marketing purposes, and as someone who gets huge volumes of spam already (despite 500+ a day caught by the corporate filters, 40-50 still get through) I'm very loathe to give out my email address when I don't have to. Although there are services such as BugMeNot which allow you to create a throw-away email address for registration and to look up already registered addresses, the webmasters of the sites that require registration cotton on to these pretty fast and block them.

However, these selfsame sites obviously want google to index them, so they allow the google search robot in without requiring registration. Enter the User Agent Switcher plugin for Firefox. I've used this in the past to get access sites that insist you use Internet Explorer to view them, but you can add an entry to make your browser look like it is the google bot. In the Description field put whatever you want to identify the entry, and in the User Agent field put the string googlebot/2.1 (+http://www.google.com/bot.html), you can leave the rest of the fields empty. Switch to the new User Agent string and hey presto, all that content you couldn't previously access is visible. Unlike the disposable email address arms race, I don't see how this can be circumvented without making the content invisible to Google as well.

Tags : ,
Categories : Web, Tech


Re: Bypassing those annoying website registration forms

Interesting. I've tried the 'test' website, and it thinks I'm using Netscape. Here's the message I get: You're using Netscape Navigator t on an unknown operating system!

Is that what you'd expect?

Re: Bypassing those annoying website registration forms

If you are using the website I think you are, it uses JavaScript for detecting your browser type as well as looking at the User-Agent header. http://www.fiddlertool.com/useragent.aspx reports the User-Agent string back to you, and with User Agent Switcher set up as above that does indeed show that Firefox is pretending to be googlebot.

Re: Bypassing those annoying website registration forms

My favorite trick is to search for the URL itself in Google. Google will return "If the URL is valid, try visiting that web page by clicking on the following link", and if you click on it, your referer will be google.com, and most of those sites will let you in.

Re: Bypassing those annoying website registration forms

That's a neat trick as well. Of course it would be even better if they just removed the stupid registration pages ;-)

Re: Bypassing those annoying website registration forms

I was left wondering if this would be legal, at least in the UK.

Re: Bypassing those annoying website registration forms

[Trackback] Either you have a wall and only let paid subscribers in, or you don’t.