What is smart crawling?

Smart crawling allows the crawler to run more frequently, leveraging the XML sitemap(s) of your site. It uses the lastmod timestamps in the sitemap to detect if a page was updated since the last crawl. This allows the crawler to only re-crawl recently modified pages, saving time and resources when re-crawling the site.

Smart crawling has the following requirements in order to run correctly:

  • The website must have one or more XML sitemaps
  • The sitemap(s) must have lastmod timestamps showing when each page or file was last modified.
  • The sitemap(s) must include all pages or files that should be crawled and indexed.

Smart crawling can only be enabled by Cludo staff. Therefore, if you wish to enable smart crawling, please contact support.