How To Optimise Your Site’s Crawl Budget
Technical errors such as 404’s and redirects can be confusing for the average website owner, particularly if they do not come from a web development background. This is where the expert team at Absolute Digital Media come in with our on-page and technical services. Crawl budget is often an overlooked element when it comes to SEO, and while many people may have heard of it, few people really know what it is, how it correlates with SEO and ultimately how to optimise their site for it. Here, we’re taking a closer look.
What Is Crawl Budget?
Across the internet, there are a series of ‘spiders’, ‘crawlers’, and ‘bots’, which are computer programs which are designed to ‘visit’ and ‘crawl’ all of the web pages on the internet in order to understand what they are about and their overall purpose. There are typically three different types of spiders (search engine spiders, web services’ spiders, and hacker spiders), but we’re only going to be talking about search engine spiders in this particular blog post.
Crawl budget is the term used to describe the number of times that a search engine spider will crawl your website, over a certain period of time. So, if Google crawls your site 2000 times in one month, then your crawl budget will be 2k per month.
Google posted on their blog in 2017 that crawling isn’t a ranking factor itself, however, it can have a number of implications for SEO.
Why Is Crawl Budget Important?
The simple fact about crawl budget is that you want to optimise this as much as possible, in order to ensure that Google discovers all of your most important pages as quickly as possible, and as many of them as possible in one crawl. If you are posting new content on your site on a frequent basis, and this is deemed to be highly valuable content (all of your content should be), then you will want to ensure that your content is found quickly. If you are able to effectively manage your budget and optimise this to its full potential, then your pages are likely to be found much faster.
How Does Crawl Budget Work?
Google’s crawl operates in a series of steps in a recursive manner when crawling each site. This graph from Google below shows that the crawl will first look at a txt robot and then separate this into a set of URLs. Then these URLs will be fetched.
How To Determine Your Crawl Budget
The best way to determine your crawl budget is by logging into Google Search Console. This data is relatively general, but you are able to see your average crawl stats to see how many site’s pages are crawled every day. So, if the average Google crawls per day is 15, you can work out your monthly crawl budget by 15*30 = 450. While this is prone to fluctuation, this can give you an idea of your crawl budget.
Google will provide you with an assigned crawl budget, based on two factors:
- Popularity – the more popular pages will be crawled more regularly
- Staleness – If your webpage isn’t updated on a regular basis, then it will crawl the page less frequently. Google aims to ensure that URLs do not become stale in the index.
Matt Cutts, former Head of Web Spam at Google, said in 2010: “The number of pages that we crawl is roughly proportioned to your Page Rank.”
What Can Impact Crawl Budget?
If you’re looking to optimise your crawl budget, then you’re going to want to ensure that you know exactly what can impact it. Quite simply, a low-value-add URL will negatively impact how often your site is crawled and indexed, and having a high number of these on your site can be particularly damaging to your crawl budget. A low-value-add URL is defined as a URL which has:
- Faceted navigation and session identifiers – which can create duplicative URLs
- On-site duplicate content
- Soft error pages – for example, soft 404s
- Hacked pages
- Infinite spaces and proxies – links that provide little or no new content to be indexed
- Low quality and spam content
How Can You Optimise Your Crawl Budget?
There are a number of different ways that you can maximise your crawl budget in order to ensure that each and every part of your website is crawled quickly and effectively. Here are some top tips on how you can optimise your crawl budget:
- Avoid any elongated redirect chains. If you have a broad number of 301 and 302 redirects all of which appear in a row on your site, then you can be certain that crawlers will stop crawling.
- Ensure URL parameters are correct, as dynamic URLs generated by some CMS or website engines can breed content duplication.
- HTTP errors all need to be corrected, to ensure the budget is not wasted on any broken links or server errors.
- Keep your sitemap as up-to-date as possible, as this is an important part of website crawling. Organising this properly can help search bots to understand which content is new and find it much faster than through a series of internal links within the site.
- Keep the site’s IA and internal linking as neat as possible, in order to ensure that all content is easily navigable and discoverable by search bots.
Crawl budget may not be a ranking factor per se, however technical SEO which has a huge impact on crawl budget optimisation is a major part in ensuring that your site is ranked highly by Google and other search engines. To find out more about the technical SEO services that we offer here at Absolute Digital Media, get in touch on 0800 088 6000 today.