Determining How Fast a Website Gets Indexed

You can tell a lot about how well a new website will do in the search engines right away, by how long it takes to get indexed by Google. How long it takes, which pages get indexed first, and what terms the new site gets indexed for in Webmaster Tools will tell you if you have a real challenge ahead gaining search results or if it will be easy.

I am by no means an SEO expert but I do have a decent amount of experience with this topic.

If you read my last post you’ll know that I’ve been experimenting with building a number of niche sites (just launched number four). And I’ve been working, building websites for I guess a number of years now (getting old). I’ve never been huge into SEO but it has been a curiosity of mine. I don’t do link building campaigns, article submissions and all that stuff but I think it is important that your on-page SEO is done correctly. As in, correct use of heading tags, title tags, alt tags, having lots of internal and external links to relevant content, and generally writing excellent content that is relevant to your audience. I am also slightly addicted to Analytics and Webmaster Tools, which I try not to check numerous times a day.

So just from my time spent in the web design industry, building my own sites, and regularly using Google’s search tools I’ve seen some patterns and trends that can really tell you:

1. If you are hitting your keywords (if that was important to you)
2. How much competition the site will have
3. How much promotion the site will need if traffic is a prerogative

New Sites Get the Shaft

At some point there was something called the Google Sandbox. It doesn’t still exist but it is the term people use to describe what Google does to sites they are indexing or reindexing. Basically, for new sites you will be getting zero traffic, unless you hit on a goldmine search term. Google spends a lengthy amount of time crawling your pages and trying to find out what niche your site belongs to, how popular the site is (how many links are coming in, and social media presence), and what specific keywords your site and individual pages rank for, if any. It can be frustrating because you see Google is crawling all of your pages every day, yet they aren’t getting indexed. To get into the sandbox with a pre-existing website you have to do some serious (and probably questionable) SEO, building thousands of links etc. for Google to do that to you.

Google can take a really, really long time doing this, and during that time, if you were hoping on getting traffic from Google search you will be out of luck.

As an example, this site did not start getting any measurable amount of search traffic until about four months in. Probably one of the big reasons why most blogs fail in the first couple of months. They get zero help from Google and it really isn’t fun writing blog posts that no one reads. I still don’t get a lot of Google traffic, somewhere between 70-120 visitors per day during the week, less on weekends, but it is better than the 0-10 visits I was getting. I’ve actually gotten as much or more traffic from social media and links than from Google.

There are a number of reasons why I don’t get a lot of Google traffic, which I’ll get into below.

If you are building a new website for a client, yourself or giving niche websites a shot, you should install Analytics and Webmaster Tools from day one. Day one is the day you plan on building the website and should be the same day you buy the domain, set up the hosting, and put up a simple intro page. I highly recommend getting this head start because the older your domain is the easier it will be to rank. Even one month head start will help you in the long run.

For your intro page, you can put a simple logo with a paragraph or two and maybe an unordered list of what the site will be about so Google will have a general idea of what to index you for. If you have keyword you are going for they need to be on this page in the title, headings etc. If you plan on using social media, and building an email list those two things should be on the page as well. In case you were wondering, should be doing social media before the site is completed. It is a great way to increase interest and get Google noticing your new domain.

Niche Competition

The length of time it takes to get indexed by Google in many ways can be determined by the competition in the particular niche the site will be indexed in.

Again with this site as an example, I’m writing about web design, which there happens to be thousands of blogs already doing, not to mention thousands more web design agency and portfolio websites.

The more competition in the niche the longer it will take you to get indexed. If you are building a website about Viagra or Home Loans expect it to take a while. This is for two reasons. It will obviously take Google longer to index high competition sites because it has to compare your website to thousands of other website in the niche and see where your site should fall in that mix. The other reason is SEOers used to try to build thousands of automated websites for competitive search terms in order to gain advantage in a niche and they could make constant changes to these sites to adjust to anything Google changed in their algorithm. By giving new domains very few search results and no Page Rank Google can keep these sites from effecting rankings. Essentially, every new site is guilty until proven innocent.

Which is good for the internet as a whole but sucks for you and me, just trying to build a website about a passion of ours or a business we are making. Oddly enough, these changes created even more jobs for Social Media and SEO people. Since it becomes harder to get traffic, we must use all these other alternative methods. Instead of just writing amazing content, which Google says is all we need to do, we have to go out and build links, promote our work on social media and news and bookmarking sites.

Social media is largely a waste of time but become necessary if you need to be able to get any sort of traffic to a new site in a competitive niche, since you have no other options.

Keywords

The most useful thing you can determine from even the first couple days your site is live is whether you’ve hit your keywords or not. To see this just go to Webmaster Tools > Your Site on the Web > Search Queries to see what you are being ranked for. Below is an example from this website.

Obviously, I’ve spent zero time working on keywords for this site as you can see. I’m ranked for some pretty random stuff.

If you care about search traffic you really want to pay attention to keywords. I know it sucks, and it would be nice if you could write whatever you feel like, like I do on this blog, but you will end up getting poor search results because of it, like I do on this blog. Don’t be like me! When your site is very small and new the only thing on your Search Query list should be your primary keywords. If they aren’t and/or something else is ranked or ranked over them, you know what you need to work on. The harder your niche is to break into the fewer keywords you should be trying for at the start.

As another example, one of my new niche sites got ranked for 6 of its 7 keywords in three days. That tells me several things. One, I’ve done my on-page SEO correctly. Two, there is low competition in the niche, for these terms at least. Three, that people are actively searching for these keywords. I already knew there was potential for these terms because I did my homework (researched the best keywords and the competition before building the site) but this definitely confirmed that I did my homework correctly.

Homework is always important. If you are planning or hoping to get large amounts of search traffic you need to know what people are searching for, not just building your pages around what you think might be important. The easiest way is using the Google AdSense Keyword search tool. It doesn’t work for really low or really specific terms but if you are looking for lots of traffic you need to be working the bigger search terms anyway.

Scarcity and Importance

How rare is the content on your website. Are you one of 10,000 blogs on pet grooming. Google won’t have any reason to index you quickly. Basically, Google does not find your content important. You might take offense to that. After all, you spent a whole lot of time creating it, but it’s true. Google only indexes content quickly that is very important.

So what is important to Google? Important content is content that people are searching for but cannot find, as in, it doesn’t exist. If Martians land out in Montana or some other podunk place and you post photos of you and the Martians at a barbecue on your blog, you are the only single website with that information. If millions of people are searching for “Montana Martian Landing” your post will be found and indexed in minutes.

Breaking news shows up almost immediately on Google because Google’s users find it important, therefor Google finds it important. When lots of people are searching for something Google knows it needs to find it quick so they start indexing that relevant content.

The more unique your content, the higher the likelihood that it gets indexed faster. Of course people need to be searching for that content. A combination of both unique and important will get you indexed.

Ads and Affiliate Links

Update: I forgot to mention this, but if you plan on having ads and/or affiliate links on your website do not put them on right away. Google takes a lot longer to index and add pages to the search index that are loaded down with ads. It must set off some sort of red flag. From what other people have stated, a one month period is best to wait before adding any type of ads to your website. Wait for Google to get a feel for your content and where it should be ranked.

So What To Do

If you’ve built a website, added Analytics and Webmaster Tools, submitted a sitemap, seen that it is getting crawled by Google and is still not getting pages indexed, what do you do? Well the answer sadly is wait. There is nothing you can do to make Google work any faster.

Of course you shouldn’t just be sitting idly by. Continue to write excellent, relevant and unique content. Promote that content as much as you can. Participate in your niche via comments, guest posts, forums, and social media. Eventually you’ll break through, or you’ll give up. If you don’t give up you will break through, it is just a matter of time.

If you really want to speed up the process and help your overall search engine rankings, remember these tips:

1. Make your content unique
2. Make sure people are searching for that unique content

It really is that simple and yet can be very hard to come up with in many circumstances.

Best of luck and hope this info helps you out in both getting indexed faster and the way you approach your content creation.