Score

score

Per-record score columns and a corpus data card.

Score A (always, free) flattens result.signals into Arrow-friendly sct_* columns. Score B (off by default, :class:ScoreConfig) adds a pluggable quality score computed on the ORIGINAL text and a PII-normalised MinHash signature computed on the anonymised text. All values are value-free: counts, shares, type names, codes, scores and hashes, never entity text.

QualityScorer

Bases: Protocol

Scores a batch of ORIGINAL (not anonymised) texts, one float each.

ScoreConfig dataclass

ScoreConfig(
    quality: bool = False,
    quality_model: str | None = None,
    quality_revision: str | None = None,
    minhash: bool = False,
    minhash_num_perm: int = 128,
    minhash_seed: int = 1,
    minhash_shingle: int = 5,
    minhash_min_tokens: int = 5,
)

Score B switches; every one is off by default.

ONNXQualityScorer

ONNXQualityScorer(
    model_dir: str | Path,
    *,
    revision: str | None = None,
    batch_size: int = 32,
    max_length: int = 512,
    providers: list | None = None,
)

Quality head exported by scripts/export_quality_head.py.

model_dir holds model.onnx (token ids -> score in [0, 1]), tokenizer.json and sct_quality.json (sources and revisions). Only ONNX and JSON are loaded, never pickles. revision pins the head revision recorded at export time.

minhash_tokens

minhash_tokens(text: str) -> list[str]

Casefolded tokens; char tokens when most letters are in a no-space script.

minhash_signature

minhash_signature(
    text: str, config: ScoreConfig
) -> list[int] | None

MinHash of config.minhash_shingle-token shingles of anonymised text.

None when fewer than minhash_min_tokens non-placeholder tokens.

minhash_metadata

minhash_metadata(config: ScoreConfig) -> dict[str, Any]

Everything needed to compare signatures across runs.

score_record

score_record(
    result: ProcessResult,
    *,
    original: str | None = None,
    quality: QualityScorer | float | None = None,
    minhash: ScoreConfig | None = None,
) -> dict[str, Any]

Flat sct_* columns for one ProcessResult.

Score A always. Both Score B columns are added when quality (a precomputed float, or a scorer run on original) or minhash (a :class:ScoreConfig, run on result.lm_text) is given.

score_records

score_records(
    results: Sequence,
    originals: Sequence[str],
    config: ScoreConfig = ScoreConfig(),
    *,
    scorer: QualityScorer | None = None,
) -> list[dict[str, Any]]

Score a batch; the quality scorer runs once over all originals.

Passing scorer turns quality on, as ScoreConfig(quality=True) does.

build_data_card

build_data_card(
    rows: Iterable[dict[str, Any]],
    *,
    models: Iterable[dict[str, Any]] = (),
) -> dict[str, Any]

Corpus summary from score rows plus the NER model provenance (K5).

render_data_card

render_data_card(card: dict[str, Any]) -> str

Markdown view of :func:build_data_card.