How to set up date crawling

You can configure your crawler to index dates, which can be utilized for various purposes such as date filtering or Date Freshness Boosting.

To set this up:

1. Go to Configuration › Crawlers and open the crawler you want to configure.
2. Go to the Content tab.
3. Under Page Fields, find the Date field. This field is included by default, so you don’t need to add it as a custom field; unless you’re editing an older crawler, in which case you might need to add it as a custom field.
4. Click the edit icon next to the Date field to open Edit Source Field.
5. Under Primary Source, choose how the date should be found:
   • Automatic (default) — the crawler scans the page for common date signals, in this priority order, and uses the first valid date it finds:
     1. Metadata tags such as article:published_time, og:published_time, og:updated_time, dcterms.created, dc.date, or <meta itemprop="datePublished"> / dateModified
     2. <time datetime="..."> elements or <span itemprop="datePublished"> / dateModified
     3. Elements whose class, ID, or nearby text suggest a date (e.g. containing “date”, “published”, “timestamp”)
     4. datePublished / dateModified values inside JSON-LD structured data
   • XPath — point to a specific element on the page if Automatic doesn’t pick up the right value
   • Other sources (Meta Tag, OpenGraph, Structured Data, CludoTag) if the date lives in one of those specific locations
6. If you pick a source other than Automatic, set the Data Type to Date so the value is parsed and formatted correctly.
7. Click Save, then Apply to save the crawler configuration.

Setting the date format

When you select the Date data type, the crawler will automatically determine what format the date is on based on the crawler’s configured language. If that doesn’t match what’s on the site, you can tell the crawler how the date is written on your page using the Date format field. The format describes the pattern, not the value: a page showing 04/17/2023 needs the format MM/dd/yyyy.

yyyy is the year, MM the month, dd the day, HH the hour and mm the minutes. Slashes, dashes and spaces are matched literally, so they need to appear exactly as they do on the page.

Case matters. Month is uppercase MM, minutes are lowercase mm, and there is no DD or YYYY. A format using those will not work.

Checking that it worked

Run the Test crawler and look at the extracted fields. If the format is correct you will see two: your field name, and a second with _date added. The _date version is the one used for sorting and filtering.

If the _date version is missing, the crawler could not read the date. The field is dropped rather than indexed as empty, so a search that sorts on it falls back to relevance ordering instead of showing an error.

For assistance with setting up date crawling, feel free to contact support.