What is async crawling?

Async crawling is meant for websites where content is loaded asynchronously (AJAX-generated content).

AJAX-generated content allows the web page and web browser to process data without having to reload the page. For example, if you hit a “Submit” button on the page, AJAX processes the information and updates the content without the browser reloading. This can often occur when pagination occurs and is loaded using AJAX.

With a standard crawler, only the static HTML content of the pages will be crawled, resulting in the async content not being picked up.

In cases where AJAX technology is being used, Cludo can use an async crawler to index the content. Async crawling is always configured in collaboration with Cludo, as the crawler needs to be configured to support the specific configuration on the website.

How to set up async crawling

The async crawling feature cannot currently be set up using MyCludo and requires interference from a trained Cludo employee, as this is always configured manually depending on the site structure.

In order to have async crawling configured, please reach out to support.

Best practice for async crawling

If your website has content that is rendered asynchronously, you may want to look into async crawling. However, before deciding if there is a need for this, it is recommended to consider the following:

• Which part of the page is loaded asynchronously?
  • Are these parts of the page important content for the search engine?
• Is it possible to add all relevant information as meta tags in the HTML, making it possible for the crawler to fetch the needed information this way?

As async crawling requires more resources, it is recommended to first assess if there are easy-to-implement workarounds that will allow the crawler to index the page normally.