How to set up date crawling

You can configure your crawler to index dates, which can be utilized for various purposes such as date filtering or Date Freshness Boosting.

To set this up:

1. Go to Configuration › Crawlers and open the crawler you want to configure.
2. Go to the Content tab.
3. Under Page Fields, find the Date field. This field is included by default, so you don’t need to add it as a custom field; unless you’re editing an older crawler, in which case you might need to add it as a custom field.
4. Click the edit icon next to the Date field to open Edit Source Field.
5. Under Primary Source, choose how the date should be found:
   • Automatic (default) — the crawler scans the page for common date signals, in this priority order, and uses the first valid date it finds:
     1. Metadata tags such as article:published_time, og:published_time, og:updated_time, dcterms.created, dc.date, or <meta itemprop="datePublished"> / dateModified
     2. <time datetime="..."> elements or <span itemprop="datePublished"> / dateModified
     3. Elements whose class, ID, or nearby text suggest a date (e.g. containing “date”, “published”, “timestamp”)
     4. datePublished / dateModified values inside JSON-LD structured data
   • XPath — point to a specific element on the page if Automatic doesn’t pick up the right value
   • Other sources (Meta Tag, OpenGraph, Structured Data, CludoTag) if the date lives in one of those specific locations
6. If you pick a source other than Automatic, set the Data Type to Date so the value is parsed and formatted correctly.
7. Click Save, then Apply to save the crawler configuration.

When selecting the Date data type, the indexed value will appear in the Date format. You can verify the correct date format by using the Test crawler feature. If you notice two different fields, one with the original name and the other with ‘_date’ attached, it indicates successful formatting.

For assistance with setting up date crawling, feel free to contact support.