How many requests does the crawler make?

Our crawler will always attempt to make as many requests as possible, often requesting multiple pages per second, but the actual frequency of requests depends on the server response from the website.

Some websites might also have a crawl delay set in their robots.txt, which can impact how many requests the crawler is allowed to make. For more information on this, visit our article on how to set up a crawl delay.