Contact Us

    If you have a project and would like to get a quote please submit the form bellow or call now: (647) 271-7463

    Back

    Google’s Mixed Messages: Does Googlebot Really “Follow” Links?

    Last updated on

    In a recent episode of Google’s “Search Off The Record” podcast, Analyst Gary Illyes provided clarity on how Googlebot handles links during the crawling process.

    Illyes’ explanation challenges the common belief that Googlebot navigates websites by following links in real-time.

    He revealed that, instead of following links sequentially, Googlebot collects them for processing at a later stage.

    This misconception appears to have originated from Google’s own documentation.

    Contradictory Information

    “It’s my pet peeve,” Illyes remarked during the podcast, referring to Google’s support pages.

    He elaborated:

    “On our site, we keep saying Googlebot is following links, but no, it’s not following links. It’s collecting links and then revisits those links later.”

    What The Documents Say

    Google’s official documentation on crawlers states:

    “Crawler (sometimes also called a ‘robot’ or ‘spider’) is a generic term for any program that is used to automatically discover and scan websites by following links from one web page to another.”

    The document suggests that Googlebot navigates the web by actively following links in real-time.

    This discrepancy between Google’s public messaging and the actual behavior of their crawler raises questions about other potential misunderstandings within the SEO community.

    Implications For SEO

    This revelation could have significant implications for our understanding of Google’s crawling process:

    • Crawl Budget: If Googlebot gathers links before revisiting them, our perception of crawl budgets may change. The initial “collection” phase might be less resource-intensive than previously assumed.
    • Site Architecture: Although a well-structured site remains crucial, the notion that Googlebot must navigate through multiple clicks to reach deeper pages might be outdated. This could reshape our strategies for internal linking and site depth.
    • Crawl Frequency: This insight might help explain why certain pages are crawled more frequently than others, regardless of their position within the site’s hierarchy.

    Looking Ahead

    Many SEO strategies are based on the assumption that Googlebot navigates websites by following internal links in a manner similar to a site visitor.

    However, if Illyes’ description is accurate, it indicates that Googlebot’s behavior is more intricate than previously believed.

    While this doesn’t undermine current SEO best practices, it underscores the importance for SEO professionals to remain updated on the subtleties of how Google operates.

    Original news from SearchEngineJournal