Keyword Strategy will be Shutting Down on March 28, 2013 Read details

You Can Track Rank, Just Don’t Violate Google’s Terms of Service

by Fraser Cain on January 20, 2013

If you’ve heard the news, many SEO tools have decided to stop offering any services which allow rank tracking or competitive analysis. This is because those tools relied on scraping Google results to determine the rank in the search engine. This was a clear violation of Google’s terms of service.

Google’s Terms of Service do not allow the sending of automated queries of any sort to our system without express permission in advance from Google. Sending automated queries consumes resources and includes using any software (such as WebPosition Gold) to send automated queries to Google to determine how a website or webpage ranks in Google search results for various queries. In addition to rank checking, other types of automated access to Google without permission are also a violation of our Webmaster Guidelines and Terms of Service.

I’m happy to report that Keyword Strategy was recently renewed for our Adwords API token. Although Google’s ways are mysterious, I’d like to think part of that is because we have developed an SEO tool which completely conforms to Google’s terms of service.

We still track rank and we still can help you judge the competition; all in a Google-compliant way.

Rank Tracking

Keyword Strategy’s rank tracking service works passively, extracting your keyword ranks from the traffic that comes to your server from Google. In a certain percentage of search visitors, the rank is encoded right into the URL logged by your webserver, and doesn’t require accessing Google at all.

There are advantages and disadvantages to this method, but it’s what allows to track rank on an unlimited number of keywords for every project, in a completely compliant way with Google’s terms of service.

Here’s a detailed explainer on how Keyword Strategy tracks rank.

Competition Analysis

Again, the main way that most SEO services provide competitive analysis is by scraping Google. For example, traditional KEI (Keyword Effectiveness Index) is calculated based on the number of exact matches in Google for a specific keyword. Of course, to determine the number of matches, you need to scrape Google. And once again, this is a violation of Google’s terms of service.

Instead, Keyword Strategy is based on an open-source index of 6 billion web pages called the Common Crawl. It’s not the full internet, but it’s a large enough sample that it can give you some reasonable samples of competitive data. We downloaded the title tags from the entire index, and then search it for each keyword in your project.

I personally think that the TITLE tag is a great way to consider competitive analysis. Is there aren’t a lot of examples of people using a keyword in the TITLE of their document, it might be a good target to write on.

Again, here’s a detailed explainer on how Keyword Strategy calculates competition.

Moving Forward

Now that Google is pressuring various SEO firms to comply, I’m expecting we’ll see many companies adopting more Google-friendly methods. But with SEO in general, Google has been clear about what it thought was correct behavior right from the beginning – it was right there in the terms of service.

{ 2 comments… read them below or add one }

Chris Troutner January 21, 2013 at 4:12 am

Great post! It’s great to see you posting again to this blog. I love reading them!

fraser January 21, 2013 at 3:49 pm

It’s hard to both work on the social networks, like Google+, and then explain how I’m working on the social networks… but I’ll do my best.

As soon as I have new ideas, I’ll post them quick.