Once a crawler has crawled the defined domain(s), you may experience a specific page not being added to the search index. This will typically be due to one of the following reasons: Feel free to contact support if you have further questions on why a page was not indexed as . . . Read more
A crawl delay will limit the frequency at which the crawler will request pages on a website. This results in an overall slower crawl but can prevent overloading the website with too many requests at once. This is rarely needed, but it can be useful for “sensitive” servers that don’t . . . Read more
The crawler log offers insights into the activity of the crawler. It allows for further investigation into which pages were indexed and which were not and why.
It is possible to exclude pages in different ways, each with their own set of pros and cons. It is recommended to read and understand all the ways of excluding pages before setting up the exclusion logic to ensure the best setup for your needs. How to exclude pages via . . . Read more