What is smart crawling?

Smart crawling allows the crawler to run more frequently, leveraging the XML sitemap(s) of your site. It uses the lastmod timestamps in the sitemap to detect if a page was updated since the last crawl. This allows the crawler to only re-crawl recently modified pages, saving time and resources when re-crawling the site. By default, the standard frequency of smart crawling is set to every 6 hours.

Smart crawling has the following requirements in order to run correctly:

• The website must have one or more XML sitemaps
• The sitemap(s) must have lastmod timestamps in W3C datetime format showing when each page or file was last modified. For smart crawling to be successful, both the date and time must be present.
• The sitemap(s) must include all pages or files that should be crawled and indexed.

Can I set up smart crawling?

Currently, it’s not possible to enable smart crawling directly in MyCludo.

To enable smart crawling, submit a support request, letting us know which crawler you would like to enable smart crawling for. A support agent will confirm that your site is eligible for smart crawling and inform you when it is done. There will be no downtime during this change.