SEO managers understand how essential rankings are for any business with a presence online. What Google says, goes. And that can make a big impact on your site’s ability to organically reach your ideal customers.
Understanding how and why Google crawls your site is a critical step toward optimizing your site and tapping into the potential organic search holds for your business. That’s why SEO managers, and anyone working in the digital field, need a rock solid grasp on how Google crawls, evaluates, and ranks their websites. That knowledge is the key to building a solid SEO strategy. And, while there are plenty of levers to pull in the SEO world that can help you understand how Google is seeing your site in real time, perhaps the most important is a log file analysis.
Let’s first look at how Google’s bots analyze a website. Then, we’ll take a closer look at the log file analysis and how you can perform your first one today.
Google is an incredibly far-reaching amorphous entity that is constantly searching the dustiest corners of the web to document every available site. To keep their database current, and its algorithm meeting the needs of users, Google needs to consume and catalog the entire internet regularly.
In order to do this, they need impossible manpower. Enter Googlebot. Googlebot is just what it sounds like: a robot (well, a collection of robots). These bots are known as web crawlers, built and used by Google to find and evaluate content all over the world wide web.
When a Googlebot crawls a website, it takes in all of the relevant data it can find: text, pictures, graphics, metadata, header tags, etc. Then, the bots place all of that information in a catalog for your site-- a kind of file that Google references when making algorithmic decisions.
Using the information gleaned by its bots, Google evaluates the relevancy of your site and web pages. They do this with a complex and ever-changing algorithm that evaluates the usefulness of your site for various search terms. But, while the algorithm itself is complex, its purpose is not. Google wants to stay in business. And, in the simplest sense, they do that by continuing to answer the search queries of users better than any other competitors. By focusing your attention on best meeting the needs of your ideal customers on your site, you will fight side by side with Google’s algorithm rather than against it.
Google has a lot to do. It’s bots can’t spend all day on your site just because you’d like them to. They will give a limited crawl budget to your site when they locate it, and it is up to you to make the best of that time. Relevance and keyword rankings are determined by these crawls, so be sure that your SEOs know how to maximize the limited time Google allocates to your site.
This limited budget stress is where the log file comes in handy.
A log file is a file stored on your web hosting server that documents events occurring on an operating system, which in this case would be your hosted domain. There are different types of log files (error log files, access log files, etc.) but when you run a log file analysis you’ll be specifically looking at the access log file.
All sites should have access logs set up with their web hosting server by default but you will need to reach out to your hosting service provider to verify if you want to be sure.
A log file analysis is the investigation of existing log files which should provide the insights needed to:
The log file analysis has three steps: data gathering, analysis, and implementation. I’ll walk you through each element to show how each phase feeds into the next.
Before you begin the log file analysis, you need to be sure you’re looking at the correct data. Use Screaming Frog Log File Analyzer to help you locate the right information. Here’s what to look for:
Once you have gathered the pertinent website logs and data, you’ll be able to move on to the actual analysis.
To analyze all this data I use the following toolset:
In executing the log file analysis, here are a few things to look for:
In answering these questions you’ll gain valuable insights on what may be holding back your website’s performance and how you can improve it. But, the journey doesn’t stop there. Once you have these insights, you must work to implement them. You’ll want to build out a list of items that need tackling, how you plan to implement those changes, as well as a plan to improve the crawlability of your site going forward.
Some of the items we’d recommend you focus on include:
Once this list of recommended changes has been built, you’ll need to work with your web development team to prioritize your next steps. I recommend rating each item on a scale of 1-5 on these three categories:
Once the priority has been established, you’ll work with your web development team to implement these fixes in a manner that works best for their development cycles.
Sounds like a lot of work, but it’s worth it. To show you just how important this analysis can be, here’s a brief case study that demonstrates the impact a log file analysis can have on an SEO strategy.
During a recent client engagement, we were working to increase e-commerce transactions brought in from Google organic traffic.
We began the journey as we generally do, by performing a series of technical audits. As we examined Google Search Console, we noticed that there were some indexation irregularities. Specifically, there were pages missing in Google’s indexation and overall coverage of the site. This is a common symptom of a crawability issue.
So, we ran a log file analysis to identify ways we could improve how Google crawls the site. Some of these findings included:
We created an action plan based on these findings, and worked with a web development team to ensure they were addressed.
Once the log file findings were implemented, we saw the following results (comparing 30 days with the previous 30 days):
As with all SEO strategies, it’s important to make sure Google acknowledges the changes you’re making and rewards your site accordingly. Running a log file analysis is one of the best ways you can make sure this happens, regardless of the other technical SEO fixes you are implementing.
