Improving your site’s “crawlability” and long-term health

Website maintenance is the Rodney Dangerfield of digital marketing, it gets no respect! Peeling the onion a layer deeper, SEO maintenance in particular is something you’ve probably never given the time of day. In our experience, great SEO starts can really be summarized into these three categories:

  • Technical – Site health, “crawlability”, structured data, etc.
  • Authority – Backlinks, domain authority, trust, etc.
  • Content Strategy – Keyword research, on-page SEO, etc.

While authority building and content strategy are extremely important and necessary components of a successful SEO effort, it ultimately does not matter if search engines can’t properly crawl the site. Lucky for you, once you get your technical SEO fundamentals in place (Plug-ins, XML Sitemaps, Robots.txt, etc.) the game ultimately becomes more defense than it is offense. SEO maintenance helps us stay on top of critical crawl issues and opportunities to help Google and other search engines improve the efficiency and detail in which they can view the site.

Doing the Work

It’s SEO maintenance time! Break out your spreadsheets and let’s get moving. We recommend keeping track of your maintenance tasks and metrics over time to ensure you’re keeping an eye on trends/issues as they develop. The important thing? Do this every month, at least. You must stay on top of SEO maintenance if you want to see successful growth in the long term. Lastly, as each site is different in scope/size/shape please think of the following tasks as a benchmark for you to develop on. Without further ado, let’s dig in!

Monthly Tasks for Ongoing SEO Maintenance

Manually review XML Sitemap & Robots.txt files 

  • XML Sitemap file is loading
  • No redundant, incorrect, or unnecessary URLs in sitemap
  • Robots.txt file is loading
  • User agents defined and no harmful disallow statements present in robots file
  • Document and triage issues

Review Coverage & Enhancements in Google Search Console

  • Site is verified in Google Search Console & XML Sitemap has been submitted
  • No critical errors or warnings present in “Coverage” charts
  • No critical trend spikes in valid or excluded URLs
  • Review errors and valid URls in “Enhancements” tab
  • Document and triage issues

Search Console

Total URL trends

Detail Trends

Monitor Page Speed & Load Metrics of Key URLs

  • Document PageSpeed Insights scores f0r Desktop/Mobile. We like to look at not just the aggregate ratings, but also UX load time metrics like First Contentful Paint (FCP) and Largest Contentful Paint (LCP).
  • Consider using your own tools, metrics, and benchmarks for this quality check. PageSpeed Insights and Core Website Vitals from Google are a great place to start. Each website will have it’s own “standard” for speed.
  • Lastly, just go to the website on your own desktop and mobile devices for a qualitative check. Load time unreasonable? Let’s fix it.

Resolve Broken Links & 404s

  • Use a tool like to crawl the site for broken internal/external links.
  • Monitor 404 landing page errors with Google Analytics or a plug-in such as Redirection.

Perform UX & Responsiveness Checks

  • Perform a quick click-through audit on desktop, tablet, and mobile.
  • Identify any critical UX or responsiveness issues.
  • If desired, use an accessibility or performance crawler like Google’s Web Dev tools for a more detailed and technical review.

Set Up, Schedule, & Review Your Favorite Crawl Tool

If you’re an SEO-savvy reader and you’ve made it this far, you’ve probably wondered why we haven’t brought up 3rd party crawl tools like Moz, SEMRush, and Ahrefs yet. The truth is, these paid tools are incredibly helpful for active monitoring and labeling of crawl issues but are not 100% necessary and are not 100% accurate.

To the untrained eye, most site-crawl reports will make your site looks like dooms day is here and everything is wrong. Crawl tools are basically nervous Nellies, always crying wolf and occasionally have something good to say. To that end, we still use them for key indicators and information alongside our tried and trusted manual checks above.

Crawl tools also catch more detailed and complex issues like duplicate content, title/meta errors, security warnings, orphaned content, canonical inconsistencies, etc. Ultimately, the right “SEO doctor” is needed to dissect the hundreds of issues in crawl reports and identify what is a real issue vs. nice-to-have vs. not applicable.

Honorable Mentions

SEO Maintenance never really stops – with ongoing changes and frequent algorithm updates, there are infinite tasks that you can add to your own personal SEO maintenance checklist. Here are some contenders that we didn’t cover above.

Backlink Toxicity/Health: Monitor and disavow spammy backlinks. Our favorite tool for this is SEMRush.

Industry SERP Changes: Search your primary terms and read documentation on the latest SERP changes to your industry.

Position Tracking: Use a keyword tracking tool to keep an eye on ranking fluctuations.

Traffic Changes: Look at Google Analytics to monitor significant organic landing page increases/decreases for month over month.

Brand Mentions: You can use Google Alerts or a paid monitoring tool to track mentions of your brand or site across the web.

Backlink Tracking: It’s important to keep an eye on new backlinks, referring domains, and domain authority metrics.

Competitor Tracking: Looking at new competitor positions or backlinks can be a good way to stir up some strategy for the month.