Crawling content on intranets
In order to index a page, the crawler must have access to it. In certain cases, the content to crawl may be located behind a login and therefore is not reachable by the crawler by default. Ways to allow crawling behind login IP whitelist Whitelisting the crawler can be done . . . Read more