Keyword Strategy will be Shutting Down on March 28, 2013 Read details

Announcing Our New Competition System

by Fraser Cain on May 10, 2012

Sample competition column

Sample competition column

As you probably know, we removed the competition and KEI metrics from Keyword Strategy a couple of months ago. We did this primarily because of the support difficulties we were having with the browser extension that gathers the data. We knew many customers appreciated the functionality, but we didn’t realize so many considered it the most important feature of our application.

So, by popular demand, we’ve implemented a new competition metric. It solves two very important problems we had with the old system:

  1. Completely server side, no extension is necessary. Phew.
  2. Completely Google-friendly. We’re using a completely open and free data source that doesn’t violate the terms of service of any search engine.
  3. Our new competition metric is based on the TITLE tag of the entire internet. Similar to the ALLINTITLE search parameter in Google.

    This is a different metric than we had before, so you’ll need to recalibrate your expectations for what is easy competition and what is difficult.

    The data source comes from a new foundation called The Common Crawl. This is a freely accessible index of the internet (5 billion pages indexed so far). We were able to extract the title tags out of all 5 billion pages, index them, and search through them to count keyword instances. As we get more comfortable with this data source, we’ll be able to provide additional tools and metrics in the future.

    As you add new keywords, it’ll take us a few minutes (or hours) to process all of them, so be patient; however, it’s way faster than the old extension method.

    Of course, this is purely in beta right now, but please test it out and let us know what you think.


    Fraser Cain and Alexander Allahverdiev
    Keyword Strategy

{ 4 comments… read them below or add one }

Chris Troutner May 11, 2012 at 5:15 pm


I was seriously disturbed when KEI and competition were removed. I’m glad you’re doing *something* to evaluate competition. I also completely agree with the concept of using keywords in the title as they are an important factor in how Google ranks an article. Keep up the good work guys!

So what do the numbers in the Competition column mean exactly? Does it mean there are that many web pages with that keyword in the title tag?

fraser May 11, 2012 at 5:19 pm

We took the entire Common Crawl database of 5 billion pages and extracted out all the titles. So we have a database of 5 billion titles. Then we took each keyword and searched to find out the total number of times that keyword appears exactly in our database. That’s the number you’re seeing in the column.

It’s going to be a different number from Google, but what’s important is the ratio. Keywords that only occur a single time are less competitive than ones that occur 10,000 times.

Try sorting the list by competition, or look for stuff that only has a single competition. See if those are interesting keywords.

Tony72 May 30, 2012 at 5:56 pm

Thank you for adding this feature.

noahmar June 27, 2012 at 8:39 pm

Awesome especially for on page SEO people.