SENTiVENT Metric

Hugging Face Evaluate metric for document-level SENTiVENT annotations: economic events, event arguments, implicit sentiment, ABSA-style targets, event-sentiment links, raw event polarity, discontinuous spans, and an optional factuality/context score profile.

Dataset Metric files GitHub

Usage

import evaluate
from datasets import load_dataset

test = load_dataset("GillesJacobs/sentivent", "sentivent_unified_document", split="test")

# Runnable empty baseline; replace this list with model predictions.
predictions = [
    {
        "id": row["id"],
        "tokens": row["tokens"],
        "annotations": [],
        "relations": {"event_sentiment": [], "event_coreference": []},
    }
    for row in test
]

metric = evaluate.load("GillesJacobs/sentivent_metric")
result = metric.compute(predictions=predictions, references=list(test))
summary = {
    "sentivent_score": result["sentivent_score"],
    "sentivent_macro_score": result["sentivent_macro_score"],
    "sentivent_relaxed_score": result["sentivent_relaxed_score"],
    "sentivent_relaxed_macro_score": result["sentivent_relaxed_macro_score"],
}
print(summary)

# {
#   "sentivent_score": 0.0,
#   "sentivent_macro_score": 0.0,
#   "sentivent_relaxed_score": 0.0,
#   "sentivent_relaxed_macro_score": 0.0
# }

What It Scores

Event extraction Trigger identification, main type, subtype, raw polarity, and trigger-attached argument roles.

Sentiment and ABSA Sentiment expressions, polarity, targets, and target-polarity tuples.

Event-sentiment links Links between event triggers and sentiment expressions, including polarity.

Discontinuous spans Native span-set matching without filling gaps into continuous spans.

Classification reports Per-label, micro, macro, and weighted rows under strict top-score components.

Main Outputs

sentivent_score: fixed support-weighted mean of five strict component F1 values, or twelve when factuality is enabled.
sentivent_macro_score: unweighted mean of the same strict components.
sentivent_relaxed_score and sentivent_relaxed_macro_score: complete parallel span-overlap scores, or null when disabled.
strict.* and enabled relaxed.*: per-subtask precision, recall, and F1; the relaxed profile is omitted when disabled.
strict.factuality and enabled relaxed.factuality: structural-alignment macro-F1 components with explicit aligned accuracy_support.
weights.support_counts: fixed train+validation coefficients used by the selected profile.

The support-weighted score is a mean of component F1 values, not pooled micro-F1. The scorer uses integer publication supports. Components with no slice gold and no predictions are omitted and the fixed supports are renormalized; prediction-only components remain active at F1 0.0. Per-slice counts never replace the coefficients, and arithmetic is not rounded between components.

Score version 0.4.0 fixes the extraction supports at event_trigger=5,256, event_argument=11,492, sentiment_expression=3,341, sentiment_target=3,933, and event_sentiment_link=77, totaling 24,099. With include_factuality=True, the extraction_factuality profile totals 55,146. It adds event_scoped_polarity=5,256, event_realis=5,256, event_modality=5,256, event_negated=5,256, sentiment_scoped_polarity=3,341, sentiment_uncertain=3,341, and sentiment_negated=3,341.

Event factuality alignment requires a non-null event type. Aligned accuracy divides correct labels by accuracy_support, the aligned records with a non-null gold label. Untyped sentiment-target carrier nodes are retained in the dataset but diagnosed and excluded from typed event-factuality scoring, activity, and supports. Component reports expose those exclusions as gold_invalid_structure and predicted_invalid_structure. Structurally complete unmatched non-null labels remain false negatives or false positives for macro-F1. Canonical source self-scores are 100.

Metric Module

This is a static Space used as the storage location for the Evaluate metric module. The Python implementation is in sentivent_metric.py; the Space does not need a running app for evaluate.load("GillesJacobs/sentivent_metric").