What is async crawling?

Async crawling is meant for websites where content is loaded asynchronously (AJAX-generated content).

AJAX-generated content allows the web page and web browser to process data without having to reload the page. For example, if you hit a “Submit” button on the page, AJAX processes the information and updates the content without the browser reloading. This can often occur when pagination occurs and is loaded using AJAX.

With a standard crawler, only the static HTML content of the pages will be crawled, resulting in the async content not being picked up.

In cases where AJAX technology is being used, Cludo can use an async crawler to index the content at an additional price.

Async crawling is always configured in collaboration with Cludo, as the crawler needs to be configured to support the specific configuration on the website. If you think you need async crawling on your site, don’t hesitate to contact support.