Ways to index content

Indexing content is an important part of having an efficient search functionality. There are three distinct ways to index content that can be utilized.

Crawling

Crawling is the most common method of indexing content and is automatically performed when setting up and activating a crawler. Crawling is the activity of systematically browsing websites to collect content for indexing purposes. You can read more about how Cludo’s crawlers work in this article.

Pushing pages

Utilizing the Update Content tool enables the pushing of recently published or updated pages into the index. This tool prompts the crawler to visit the pushed page and immediately index it. This way of indexing content is valuable when including recently published content as a part of the search results. You can read more about pushing pages in this article.

Indexing through the API

Utilizing the API for indexing is a direct and efficient way to manage and index content.

URL pushing

It is possible to push pages to the index through the API. By utilizing the pushurls endpoint, the crawler will look to index the page immediately, just like the Update Content tool. This feature is especially useful when triggering as soon as a page is published or updated. You can read about how to use the pushurls endpoint in this article.

Data pushing

If you haven’t configured any crawlers for certain pages, you can directly push search results into the index by using the push endpoint. You can read about how to use the push endpoint in this article.