There was a great SEO talk at the SMX East conference that made the technical aspects of SEO much less intimidating. The presenter, from the SEO shop Botify, boiled technical SEO down to three questions:
1. What is my crawl ratio?
2. How deeply will Google crawl my site?
3. What is my crawl window?
Let’s break down each of these:
What is my crawl ratio?
Your crawl ratio is this:
[Total number of pages on your site crawled by Google in 30 days / Total number of pages on your website]
The smaller the ratio, the worse your SEO. Why does the concept of a crawl ratio even exist? Because Google is incapable of indexing every web page on the internet — there are simply too many pages. Since 2011, the number of websites has increased by 300%, reaching ~995 million, according to Botify — so Google’s crawlers can’t get to every single one. That’s why Google coined the term infinite crawl space in 2008.
How deeply will Google crawl my site?
Google probably doesn’t crawl every page of your site. This is a potential problem because if a page isn’t crawled, it isn’t indexed by the search engine. That means no search rankings for you (and less revenue from, or eyeballs on, those pages).
In addition, Google’s crawlers might be entering your site through old, junky pages that you thought you had retired. In that case, you are wasting what is known as your crawl budget — the amount of time Google spends crawling and indexing your site. Common causes of crawl budget waste are too many page redirects, slow load time (Google’s crawler is busy — it ain’t gonna wait!) and excessive no-index or canonical tagging.
What is my crawl window?
Say you have a webpage that is updated weekly. How often is it being indexed by the search engine? That’s the concept of a crawl window. If your crawl window for a very dynamic page is too narrow, that’s a bad thing.
The bottom line when considering all these questions is that you need to make sure your site is properly indexed by Google. That’s why companies use third-party web crawling tools like Screaming Frog and Botify to replicate what Google does when it crawls your site.