Overview

Computational modeling of semantic change aims to study how the meanings of words evolve over time. Existing benchmarks have mostly focused on comparing two time periods, but semantic change is often gradual and unfolds across decades. This task extends the evaluation setting from two time periods to ten, enabling the modeling of individual word senses and their dynamics over time.

The task is based on a novel, multilingual benchmark covering Swedish, English, Italian, Spanish, Dutch, and Russian, with manually annotated data across ten time periods and approximately 120,000 human annotations.

Task Description

Subtask 1: Diachronic Word Sense Induction

The objective of Subtask 1 is to discover how the meanings of a word evolve over time without access to predefined sense inventories.

Participants receive dated usages of a target word and must automatically identify the underlying senses. The predictions are used for two complementary tracks through which Subtask 1 is evaluated.

Goal

Assign each usage of a target word to the senses it expresses.

Participants are free to induce any number of senses. The sense identifiers themselves are arbitrary; only the grouping of usages is evaluated.

Input

For every usage, participants receive:

• the sentence,
• the target word,
• the publication year,
• a textual definition of the target sense.

Example

A system could produce:

Here, cluster 1 represents the animal sense, while cluster 2 represents the computer device.

Multiple senses can be assigned to the same sentence. However, if your system produce only one label per sentence, read how to format your submission in the Data and Submission Format section.

Subtask 1a: Diachronic Sense Assignment

Evaluation

Performance is measured using BCubed F1, a clustering metric that evaluates how well predicted sense assignments match the gold standard.

Scores are computed independently for each target word and then macro-averaged across all target words for each language.

Subtask 1b: Sense Dynamics

Goal

Estimate how the distribution of senses changes over time.

Instead of evaluating individual usages, this track focuses on the temporal evolution of each induced sense.

Evaluation

A semantic change score is computed from the annotated sense distributions using Jensen-Shannon Divergence (JSD) between consecutive time periods.

Participant systems are evaluated by comparing their predicted rankings with the gold rankings using Spearman's rank correlation coefficient.

Higher correlation indicates better modeling of semantic change.

Subtask 2: Hypothesis-Driven Change Detection

Unlike Subtask 1, participants are given a target sense definition.

The objective is to identify which usages correspond to that specific meaning.

This setting reflects many real-world humanities applications, where researchers are interested in tracing the evolution of one particular concept rather than discovering all possible senses.

Input

For every usage, participants receive:

• the sentence,
• the target word,
• the publication year,
• a textual definition of the target sense.

Output

For each usage, systems must predict whether the usage expresses the provided sense.

Evaluation

Performance is measured using F1-score, computed independently for each target word and then averaged across all words.

Evaluation Summary

Data

The benchmark contains dated usages from diachronic corpora spanning approximately 1880–2023. Each language is divided into ten time periods of roughly equal length.

• Swedish: 40 target words, plus 10 development words
• English: 30 target words
• Italian, Spanish, Dutch, and Russian: 15 target words each

For each target word, usages are sampled from each time period. Each usage is manually annotated by at least three annotators.

How to participate

The task, together with data and the evaluation, is hosted on Codabench and more information is available on: SemEval-2027 Task 3 Codabench

Important Dates

• Training data ready: 1 September 2026
• Evaluation start: 10 January 2027
• Evaluation end by: latest date 31 January 2027
• Paper submission due: February 2027
• Notification to authors: March 2027
• Camera ready due: April 2027
• SemEval workshop: Summer 2027 (co-located with a major NLP conference)

Organisers:

References:

• Nina Tahmasebi, Adam Jatowt, Lars Borin. Survey of Computational Approaches to Lexical Semantic Change Detection. Nina Tahmasebi, Lars Borin, Adam Jatowt, Yang Xu, Simon Hengchen (eds). Computational Approaches to Semantic Change. Berlin: Language Science Press.
• Francesco Periti and Nina TahmasebiA Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change. (2023) In Proc. of NAACL2024
• Stefano Montanelli and Francesco Periti, Lexical Semantic Change through Large Language Models: a Survey. ACM Computing Surveys (2024)
• Pierluigi Cassotti, Lucia Siciliani, Marco DeGemmis, Giovanni Semeraro, Pierpaolo Basile, XL-LEXEME: WiC Pretrained Model for Cross-Lingual LEXical sEMantic changE. (2023) In Proc. of ACL2023
• Simon Hengchen, Nina Tahmasebi, Dominik Schlechtweg, Haim Dubossarsky. Challenges for Computational Lexical Semantic Change. Nina Tahmasebi, Lars Borin, Adam Jatowt, Yang Xu, Simon Hengchen (eds). Computational Approaches to Semantic Change. Berlin: Language Science Press.