Tag: Crawling

Why is this page not indexed?

Once a crawler has crawled the defined domain(s), you may experience a specific page not being added to the search index. This will typically be due to one of the following reasons: Feel free to contact support if you have further questions on why a page was not indexed as . . . Read more

What is a Crawler?

A crawler is usually the first step to creating a functional site search. The crawler is in charge of checking all available pages on the given domain(s) and indexing any pages and files according to the configuration. Once indexed, the pages and files can be searched for, leveraging an engine. . . . Read more

How to configure a crawl delay

A crawl delay will limit the frequency at which the crawler will request pages on a website. This results in an overall slower crawl but can prevent overloading the website with too many requests at once. This is rarely needed, but it can be useful for “sensitive” servers that don’t . . . Read more

How to set up async crawling

The async crawling feature cannot currently be set up using MyCludo and requires interference from a trained Cludo employee, as this is always configured manually depending on the site structure. In order to have async crawling configured, please reach out to support.

How to set up a crawler

A crawler searches a URL for pages and adds them to a search index to be used by an engine. Setting up a Crawler Boundaries Structure

How to review the crawler log

The crawler log offers insights into the activity of the crawler. It allows for further investigation into which pages were indexed and which were not and why.

How to exclude pages

It is possible to exclude pages in different ways, each with their own set of pros and cons. It is recommended to read and understand all the ways of excluding pages before setting up the exclusion logic to ensure the best setup for your needs. How to exclude pages via . . . Read more

Best practice for async crawling

If your website has content that is rendered asynchronously, you may want to look into async crawling. However, before deciding if there is a need for this, it is recommended to consider the following: As async crawling requires more resources, it is recommended to first assess if there are easy-to-implement . . . Read more

Best practice for crawlers

Setting up a crawler is a required step in configuring a functional search engine. You should consider both the configuration of the crawler as well as how many crawlers you should create. Language Crawlers are language-specific, so sites with support for multiple languages should configure one crawler per language. Remember . . . Read more