What is smart crawling?

Smart crawling allows the crawler to run more frequently, leveraging the XML sitemap(s) of your site. It uses the lastmod timestamps in the sitemap to detect if a page was updated since the last crawl. This allows the crawler to only re-crawl recently modified pages, saving time and resources when re-crawling the site. By default, the standard frequency of smart crawling is set to every 6 hours.

Smart crawling has the following requirements in order to run correctly:

• The website must have one or more XML sitemaps
• The sitemap(s) must have lastmod timestamps in W3C datetime format showing when each page or file was last modified. For smart crawling to be successful, both the date and time must be present.
• The sitemap(s) must include all pages or files that should be crawled and indexed.

Smart crawling can only be enabled by Cludo staff. Therefore, if you wish to enable smart crawling or request more frequent crawling intervals, please contact support.