The option to manually adjust your crawl rate in Google Search Console is no longer available.
Google has officially removed the crawl rate setting tool from Google Search Console, as anticipated. In November, Google announced plans to phase out the crawl rate limiter legacy tool by January 8, 2024. That date has passed, and the tool has been discontinued.
Google explained that the tool’s removal was due to advancements in its crawling algorithms and the availability of other tools for publishers. As a result, the feature was deemed unnecessary.
Now, attempts to access the crawl rate limiter for any property in Google Search Console will lead to a notification about the tool’s discontinuation.
The crawl rate limiter was a feature in the older version of Google Search Console. It allowed website owners to request Google to reduce the frequency of site crawling. Google generally advised against using this feature unless there were clear server load issues directly attributable to the Googlebot.
This is what the tool looked like:
Google has removed the crawl rate tool from Google Search Console, a decision driven by advancements in their crawling technology and other tools provided to publishers. Gary Illyes of Google explained that the improvements in their crawling logic have rendered the tool less useful. He detailed how Googlebot now adjusts its crawling frequency in response to how a site’s server reacts to its HTTP requests. For instance, if a server consistently returns HTTP 500 status codes, Googlebot will immediately slow down its crawling. Similarly, crawling is automatically reduced if server response times significantly increase.
Illyes noted that the crawl rate limiter tool had a slower impact, often taking over a day to apply new limits. He also mentioned that the tool was infrequently used, with a tendency among those who did use it to set the crawl speed to the minimum.
With the deprecation of the crawl limiter tool, Google has adjusted the minimum crawling speed to a lower rate, akin to the old crawl rate limits. This means Google will continue to respect past settings for sites with low search interest, avoiding unnecessary bandwidth usage.
For site owners facing crawling issues, Google recommends consulting a specific help document and using a report form to inform them of any problems.
The removal of this tool is significant for those who previously relied on it. It’s advisable to monitor how this change affects your site’s crawl rate in the upcoming days and weeks.
Original news from SearchEngineLand