Changelog¶

0.10.0 - 2026-11-11¶

Users can opt into one privacy policy across format matches and model findings. Caller-declared fields add explicit protection without guessing field meanings. Encrypted local references keep reversible maps separate from cleaned output. Document extraction and corpus selection add private, resumable data workflows. Credential-format scanning and language abstention are optional. Detector routing remains experimental; existing full execution stays the default. Trained Torch and ONNX quality scores agree in the tested profiles. Breaking changes: none to default behavior; unified callback ordering differs when explicitly enabled.

Added¶

Unified detection, policy and rendering with value-free findings and explicit Unicode source mapping quality.
Application-owned detector registries, caller context, field schemas and versioned framework summaries.
Encrypted SQLite token references with expiry, rotation and logical purge operations.
Report-first corpus selection, explicit deduplication bases, native DataTrove recipes, caller reference checks and checkpoint replay.
Optional offline Docling PDF extraction with local verified artifacts and structural projection of cleaned text.
Optional datafog-core credential-format scanning for API keys, Bearer tokens, JWTs and private keys.
Opt-in language abstention and experimental calibrated detector routing with explicit fallback behavior.
Cross-feature tests for encrypted references, field protection, extraction, corpus selection, replay and public error traces.

Compatibility and limits¶

Legacy processing, language selection, replacement modes and three-value result unpacking stay unchanged by default.
Unified callbacks run before all detection. Legacy callbacks still run between regex replacement and NER.
Reversible framework/document output remains unsupported; direct text processing can use encrypted references.
PDF source files remain unredacted. Corpus staging and signatures need caller-managed private storage and retention.
Logical token deletion cannot erase journals, backups or storage media. Keys and authorization remain application-owned.
The trained Torch quality scorer passed functional controls, but gibberish scored highest. No quality-accuracy claim is established.
GPU inference, quality score parity and dataset resume passed on one prepared RTX 4090 setup. See the GPU qualification evidence.

0.9.0 - 2026-11-09¶

This cleanup preserves text output while simplifying configuration. Streaming and whole-document cleaning share the same masking rules. Batched documents keep matching fake values across cleaning stages. Existing output formats and result tuples stay the same. Benchmark reruns refresh cached predictions when shared adapter code changes. Configure cleaners through TextCleanerConfig and set worker limits with max_workers. Breaking changes: deprecated config globals, model tuples, worker arguments and cleaner helper aliases are removed.

Changed¶

Configure cleaners with TextCleanerConfig, including ner_models for language-keyed model choices. Changing sct.config globals no longer configures a cleaner.
Use max_workers on process_batch and aprocess_batch; the deprecated batch_size keyword is removed. Model and dataset batch sizes keep their existing meaning.
The release benchmark uses --max-workers, report setting max_workers, and -W in throughput filenames. Archived reports keep their recorded formats.
Removed deprecated cleaner properties NormaliseText, ProcessStopwords, ProcessDateTime, ProcessContacts and ProcessSpecialSymbols. Use utility classes from their own modules when needed.
Removed the empty OTTER_NER_MODELS and MMBERT_NER_MODELS placeholders. Default model routing and pinned revisions are unchanged.
NER window planning lives with chunk splitting. Benchmark process execution is shared by quality and speed workflows.

0.8.4 - 2026-11-09¶

Streaming retries keep text after a processing failure. Both redactors stay open when finish() fails during processing. The windowed redactor retains the failed window for another attempt. Previously processed output remains available after the error. Breaking changes: none.

Fixed¶

Windowed streaming consumes a prefix only after processing succeeds, so a failed window can be retried without losing text.
Both streaming redactors close only after finish() succeeds. Retry finish(), or use feed("") after a failed windowed feed().
Streaming finalization still rejects reentrant feed() and finish() calls, so callbacks cannot add text to an input snapshot already being processed.

0.8.3 - 2026-11-06¶

Leak fixes, cleaner internals, faster import, clearer errors.

Leak fixes: process_json now masks strings in tuples and non-dict mappings, and reversible mode masks digits glued to a placeholder.
Ensembles use GLiNER: with default labels, the ONNX + GLiNER vote used to ignore GLiNER's spans.
Honest signals: sensitive now means PII was found, and values your policy keeps no longer count.
Faster start: import sct is about 1.1 seconds faster and loads NER libraries only when needed.
Clearer errors: sct-clean, sct-report and MCP report bad input and failed model loads as usage errors, not tracebacks.
Type checkers work: the package ships py.typed, and CI checks it with mypy.

Heads-up: reversible output changes, and streaming and MCP now refuse reversible cleaners. sensitive is false for a date or number alone. process_batch refuses a plain string, and batch_size is now max_workers there. Legacy config.* globals that were ignored now take effect. Importers of private names must update, see below.

Behaviour changes¶

In reversible mode, a digit run glued to a placeholder is now masked and restores exactly: a@b.com-107683 gives <EMAIL_0>-<NUMBER_0>, and John Smith-107683 gives <PERSON_0>-<NUMBER_0>. A quote, dash, * or % touching a placeholder is now kept, so it comes back on restore.
The streaming redactors and the MCP server refuse a reversible cleaner with a clear error, because they have no way to return its token map.
signals['sensitive'] and is_sensitive_result now mean pii_count > 0, so a replaced date, year, number, currency or HTML tag alone is no longer sensitive.
entity_counts, pii_count and the metrics callback no longer count entities a PrivacyPolicy kept, allowed or ignored. metadata['findings'] still lists them.
OCR redaction no longer covers regions that hold only entities a policy exempted. It still covers everything replaced in the text, including dates and years.
Framework adapters (DataTrove, LangChain, Haystack) write metadata['sct'] merged over all listed columns, so PII in a later column marks the document as sensitive. Score columns stay on the first column.
process_batch and aprocess_batch take max_workers for the worker thread limit. batch_size is a deprecated keyword alias, and passing both raises TypeError.
Every TextCleanerConfig field now has a legacy module global with its upper-case name, for example config.NER_BACKEND. These were silently ignored before and now take effect. The exceptions are the computed supported_languages and the deprecated ner_models_list, which NER_MODELS_LIST still feeds. New global config.NER_MODELS takes a language-keyed dict and wins over NER_MODELS_LIST, and config.NER_ENSEMBLE_DEFAULT_KEYS now takes effect when changed.
process_batch raises ValueError for a plain string instead of treating each character as a text. Use process() for one text.
WindowedStreamingRedactor raises BufferError for a broken or cut escape followed by a very long token. It used to stream this input with output that differed from whole-text cleaning.
ONNX model loading raises ModelLoadError when a network or auth error hits model.onnx_data or tokenizer_config.json, instead of treating the file as missing. GLiNER and Presidio construction errors are also wrapped as ModelLoadError.
score_records now runs a supplied scorer when ScoreConfig.quality is False, and adds the sct_quality and sct_minhash columns. Before, the scorer was silently ignored.
sct-clean refuses reversible mode before cleaning on any non-POSIX platform, including Windows on Python 3.13+, where os.fchmod exists but cannot restrict the token map.
Importing SCT no longer changes the process-wide ONNX Runtime log level or adds a warnings filter for lingua. Other ONNX Runtime output in the host application is no longer silenced.
The torch extra now requires transformers>=5.0; 4.x could not be installed alongside the huggingface-hub>=1.5.0 requirement.

Added¶

The presidio extra now also installs GLiNER. New quantize extra installs onnx for load_onnx_ner_model(quantize=True).
process_dataframe and process_rag_chunks results gain sensitive and signals; ProcessedRAGChunk also gains metadata_findings.
process_dataframe accepts a single column name as a string.
ModelLoadError is exported from sct.
MCP detail='full' adds the warning findings_not_collected, and its outputSchema describes the signals keys.

Fixed¶

With ner_device=None, GLiNER and Presidio backends no longer crash at load when onnxruntime-gpu is installed but no GPU is usable, for example with CUDA_VISIBLE_DEVICES empty. GLiNER running in torch uses CPU, SCT logs a warning, cuda_fallback is set and provenance shows cpu. GLiNER in ONNX mode gets no such warning, and ONNX models still try CUDA first. An explicit ner_device='cuda' still fails to load.
With ner_device=None, GLiNER and Presidio backends use the GPU when torch can, even with the CPU onnxruntime package. They used to stay on CPU without saying so. GLiNER in ONNX mode is unchanged.
With ner_device=None, a cuda_device_id that torch cannot see falls back to CPU with a warning instead of failing at load.
docs/PERFORMANCE.md no longer claims a CUDA version for the onnxruntime-gpu 1.29 benchmark run, because the build was not recorded. It adds notes on older NVIDIA drivers and on GPU memory use on shared GPUs.
Synthetic mode gives each type a fake of its own kind, including Presidio's types: a company stays a company, a date stays a date, and a number stays a number. A type with no fake generator, such as your own label, is written as <TYPE> instead of a person name. A fake no longer contains the original value as a whole word.
presidio_gliner no longer masks inside SCT's own tags, so <EMAIL> stays <EMAIL> and is not counted as a finding. Text outside a tag and tag-shaped text from your source are still masked.
Without Faker, synthetic mode raises one clear ImportError naming squeakycleantext[synthetic].
process_json and RAG chunk metadata mask strings inside tuples and non-dict mappings, returned as lists and dicts, and inside tuple dict keys, following json_key_cleaning. Sets and bytes are still not masked.
The ensemble backends no longer ignore GLiNER's spans with default labels. person, organization and location reach the vote as PER, ORG and LOC. A tag you request as PERSON, ORGANIZATION or LOCATION stays under that name, and your own gliner_label_map still wins.
ner_propagate_masked and ner_merge_location_gap also work on the PERSON, ORGANIZATION and LOCATION spans of the plain gliner backend, with the same output tokens.
presidio_gliner honours replacement_mode='reversible' and 'synthetic'. Masked spans are unchanged.
process_json masks a custom_deny_terms value in a JSON key when check_to_ascii_unicode is on (a key José stayed readable).
Empty input includes metadata['policy'] and metadata['findings'] under the same conditions as other documents.
WindowedStreamingRedactor no longer raises a false BufferError with a one-character custom_patterns rule, keeps processed windows across a caught BufferError, and keeps a date whole when a cut escape ends the buffer.
TextCleaner no longer keeps document text in per-thread state after processing, including after a step raises.
AnonymizationMap.to_dict and from_dict keep reserved tokens, so placeholders allocated after a round trip do not collide.
sct-clean writes the token map before printing masked text.
MCP no longer names the root folder when SCT_MCP_ROOT cannot be opened, and gives the unique-keys remedy only for duplicate-key errors.
A plain string for quasi_identifiers in assess_reidentification_risk means one field, not separate characters.
A float, zero or negative model_max_length in tokenizer_config.json is read safely, with a fallback to 512.

Deprecated¶

batch_size on process_batch and aprocess_batch; use max_workers.
TextCleaner.NormaliseText, ProcessStopwords, ProcessDateTime, ProcessContacts and ProcessSpecialSymbols are now private attributes. The old names still work and raise a DeprecationWarning.

Removed or changed for deep importers (undocumented names)¶

GeneralNER now takes a TextCleanerConfig plus its collaborators. process_batch is now ner_process_batch, ner_data is entities_with_position_key, ner_ensemble is vote_entities, and positional_tags is required on ner_process*.
Removed: AnonymizeResult, eight unused ProcessContacts methods, ProcessStopwords.STOP_WORDS_*, the no_emoji argument of to_ascii_unicode, and DETECTOR, LANGUAGES, LANGUAGE_NAME in sct.utils.resources.
Removed the stub setup.py and the root requirements.txt; use pyproject.toml and its extras.

0.8.2 - 2026-11-03¶

Smarter numbers, your own rules, safer streaming.

Numbers stay, IDs go: prices, percents, times and short numbers are kept; ID-shaped numbers (Order 4711, 123-45-6789) are still masked. On the benchmark DEV split precision rises from 0.68 to 0.81 with recall at 0.94.
Amounts with a currency code stay (1250 EUR), and codes or dates are no longer taken for phones.
Your own term lists: words to always mask, words to never mask, and your own regex, each with a time limit.
Apple GPU: ner_device="mps" runs NER about 1.6x faster on Apple silicon.
Safer streaming: the windowed redactor no longer shows a phone that whole-text cleaning masks.

Heads-up: output text changes for numbers and some phones. replace_numbers_mode='all' and phone_reject_embedded=False restore 0.8.1.

Number and phone masking¶

New replace_numbers_mode (default 'identifiers', legacy global REPLACE_NUMBERS_MODE): <NUMBER> only for whole digit runs of an identifier shape. Kept as written: amounts and decimals (1,250.00, 12.5), numbers beside %, ‰ or a currency sign, times and ranges (14:30, 1000-3000), and digits that are part of a longer token (glued to a Latin/Greek/Cyrillic letter, digit or _, or joined by - / + * . _ | , or a dash to one). : = # do not join, so ID:98765, id=98765 and #12345 are still masked. Digits of every script count (Arabic-Indic, Devanagari, full-width); letters of unspaced scripts do not glue (金额50000元). 'all' = 0.8.1. check_replace_numbers stays the on/off switch.
A whole digit-only token of three or more hyphen-joined groups with at least replace_numbers_min_digits digits in all is one <NUMBER> (SSN: 123-45-6789, 1234-56-78-90), so grouped IDs do not leak. Date-shaped tokens (2023-11-05, 05-11-23) and two-part ranges (100-200) are kept.
New replace_numbers_min_digits (default 4, legacy global REPLACE_NUMBERS_MIN_DIGITS). Chosen on DEV only (9 datasets, 4,298 docs, macro): 4 digits gave F2 0.881 / recall 0.893 / precision 0.856 with years and dates off and the phone rule below, the best F2 of 4 to 7 (7 digits: F2 0.869, precision 0.865). The 0.8.1 default scored F2 0.850 / recall 0.944 / precision 0.680 on the same docs. Not yet confirmed on the test split.
Code fragments: INV-2023-107683 no longer gets a partial INV-<YEAR><NUMBER>. The number step leaves the code whole; the year step still takes 2023 when years are on (INV-<YEAR>-107683; the training preset has years off and keeps the code as written).
New phone_reject_embedded (default True, legacy global PHONE_REJECT_EMBEDDED): a digit-only phone match glued to a Latin/Greek/Cyrillic letter or _, directly or across a joiner (ID9876543210, ref-0612345678), or that is exactly a yyyymmdd date or a yyyymmddhhmm[ss] timestamp (20231105123456) is left as it is. A phone that only starts like a date (2023010123) is still a phone. Digits across a joiner do not count, so 06-12345678 keeps its 0.8.1 06-<PHONE>. Formatted and international phones, Tel:0612345678 and phones glued to unspaced scripts (电话9876543210) are unchanged; Tel0612345678 (glued to a Latin word) is no longer masked. DEV: precision +0.02, recall -0.01. False = 0.8.1.
Both rules reach the plain, reversible and synthetic modes, process_batch and the streaming redactors. The browser manifest exports the new number pattern (with fold_decimal_digits); phone_reject_embedded is server-side and listed under the manifest's limitations. The training preset and the adapters keep numbers off as before. ProcessContacts.replace_numbers / replace_phone_numbers keep the 0.8.1 patterns.
Not measured: the joiner set differs from the DEV simulation (: = # removed, , and dashes added, grouped numbers such as 1 234 567 kept); a rerun of the benchmark will show the exact numbers.
Three ID shapes the plain digit-run rule rejected are masked again in 'identifiers' mode, each a new field with a legacy global (default True): numbers_letter_prefix_ids (AB1234567, X12345678Z: 1-3 uppercase letters, 6+ digits, 0-2 uppercase letters; not USD1000000), numbers_two_part_ids (123456/7890, 12345-678901, 123456-789K, 12345.67890; not year pairs 2023-2024, round ranges 1000-3000, times, % or currency) and numbers_spaced_pair_ids (123 45, a spaced 5-digit postal code, and spaced IDs with 3-6 groups and 7+ digits such as 123 45 6789, also when spaces and hyphens mix (123 45-6789, 1234 56-78-90), always masked whole; thousands-grouped amounts like 1 234 567 stay). Format rules only. DEV (9 datasets, language macro, NER threshold 0.8): F2 0.888 -> 0.901, recall 0.919 -> 0.937, precision 0.814 -> 0.811, over-mask 0.029 -> 0.033. Pooled gold ID/number recall 0.633 -> 0.791 (0.8.1: 0.809) and phone recall 0.965 -> 0.990 (0.8.1: 0.997); most lost phones were 12345-678901-shaped and come back as <NUMBER>. 1234/5678 is now one <NUMBER>; a range like 12345-67890 cannot be told from an ID and is masked. The streaming redactors keep 123 45 and any space/hyphen-joined digit run in one window.
With replace_numbers_mode='identifiers' the year step leaves a year-like group of a multi-group ID run (123-45 2020, 123-45-2020) to the numbers step, which masks the whole run; before, the year was masked alone and 123-45 stayed visible. A lone year (born in 2020) and a run of years (1999 2000 2001) are still <YEAR>. The browser export keeps the old order (listed in its limitations).
The streaming split guard for digit-group runs no longer restarts inside a long digit run: 32,000 digits with no separator took ~5 s per buffer, now about 1 ms.
New numbers_keep_currency_code_amounts (default True, legacy global NUMBERS_KEEP_CURRENCY_CODE_AMOUNTS): in 'identifiers' mode a whole digit run right before or after an uppercase ISO 4217 code, with at most one space (1250 EUR, EUR 1250, USD 1,250), is an amount and stays. Only that amount shape: an SSN-like, slash-, dot- or space-joined ID beside a code (SSN 123-45-6789 USD, 123456/7890 EUR, EUR 123 45 6789, 6789 45 123 EUR) is masked whole, never in part (an earlier build of this release kept the first two and masked only one group of the last two). It reuses the code set that keeps USD1000000 out of letter-prefix IDs; eur and EURO do not count. The IBAN and card steps still mask EUR NL91ABNA0417164300 and USD 4111 1111 1111 1111. Browser export and the streaming guard follow it. False masks these numbers as before.
A spaced/hyphen-joined run of any length with 7+ digits stays one <NUMBER> (national ID 12 34 56 78 90 12 34; an earlier build of this release capped runs at 6 groups and left a 7-group ID visible). Thousands grouping keeps an amount only up to 6 groups (1 234 567 890 123 456). The streaming guard reads runs after the date, IBAN/card and phone steps; a run masked whole may split where both halves keep 7+ groups, so a 10,000-number table streams instead of raising BufferError, and comes out as several <NUMBER> tokens where the whole document gives one. A run of up to 13 groups never splits; a run the rule does not mask whole (beside a currency sign) is never split, so a buffer too small for it raises BufferError. Years no longer end a run in the guard: an earlier build split EUR 12 2023 2023 45 123 123 after 12 and masked differently from the whole document.
A date written with spaces (05 11 2023) is handled as before this release: the date step has no spaced shape, so with years on its year is <YEAR>; with years off the whole run is one <NUMBER> (an earlier build of this release masked only 2023).

User term lists¶

New custom_deny_terms ({LABEL: (term, ...)}), custom_allow_terms ((term, ...)), custom_patterns ({LABEL: regex}) and custom_terms_case_sensitive (default False), each with a legacy global (CUSTOM_DENY_TERMS, CUSTOM_ALLOW_TERMS, CUSTOM_PATTERNS, CUSTOM_TERMS_CASE_SENSITIVE). All empty by default, so output does not change.
Terms match as whole words, Unicode-aware: no letter, mark or digit of a spaced script may touch them; in CJK and Thai the term edge is the word edge. Deny terms and patterns run before the built-in regex steps; allow terms are applied last to regex and NER detections (also to your own deny terms and patterns). A regex value that also appears outside an allowed term is masked everywhere.
Every replacement mode (<LABEL>, reversible <LABEL_0> restored by deanonymize, synthetic Faker value for labels Faker knows, else <LABEL>), process_batch, both streaming redactors (the windowed split guard will not cut a term; it refuses allow terms, see Fixes), process_json, regex_counts and signals (user labels count as PII; values never appear).
Clear ValueErrors at config time: label not upper case, a string instead of a tuple of terms, empty terms, one term under two labels, an invalid regex, a regex that can match the empty string.
New custom_patterns_timeout (default 2.0 seconds per pattern per document, None = no limit; legacy global CUSTOM_PATTERNS_TIMEOUT) and custom_patterns_on_timeout ('raise' default | 'warn'; legacy global CUSTOM_PATTERNS_ON_TIMEOUT), using the regex module's timeout. 'raise' raises sct.CustomPatternTimeout (a TimeoutError naming the label, never the text); 'warn' logs it, adds the custom_pattern_timeout warning code and processes that document without the pattern. Applies in process, process_batch, both streaming redactors (also the windowed split guard) and process_json.
Terms are normalised like the text before matching (check_fix_bad_unicode, check_to_ascii_unicode, check_remove_emoji), so an NFD José or full-width ＡＣＭＥ term still matches. A Unicode case variant (İ for i) gets its term's label. An allowed term inside a longer detection (an email bob@Acme.com) no longer hides that detection. When a pattern match holds an earlier deny term (Ann in .+ secret), the reversible map keeps the source text, so deanonymize restores Ann secret. The allow check runs once per value with a binary search, so long documents stay fast. A custom_deny_terms label with no terms raises ValueError.
The windowed streaming redactor does not split a deny term or pattern match that the buffer end cuts off (a phrase fed one character at a time), nor a pattern match from the context it reads (a lookbehind such as (?<=account )\d{6}). A pattern that can grow across a whole line (.+ secret) may then need BufferedStreamingRedactor.
Server-side only: the browser manifest lists them under its limitations. Benchmarks never use them.

Apple GPU¶

ner_device accepts 'mps'. ONNX models and GLiClass stay on CPU; auto still never picks the Apple GPU.
If torch has no MPS backend, NER runs on CPU with a logged warning and the new mps_fallback code in metadata['warnings']; provenance shows cpu.
ONNX on CoreML was measured and left out: not reliably faster, 35 minutes and 82 GB for the first compile.

Fixes¶

WindowedStreamingRedactor: the split guard also runs custom_deny_terms, custom_patterns and fuzzy dates in the cleaner's step order, and never cuts a digit run near the overlap, so a phone they make visible (117 235 991 12345 with a 12345 pattern, 117 235 991 15 Janury 1990) no longer leaks. One feed() now splits until the buffer fits max_buffer_chars (before, one large feed kept everything after its first split; it now raises BufferError when no safe split exists). With whitespace normalised, a window that holds only whitespace adds no space (hello + 100 spaces + world streamed as hello world). With custom_allow_terms set it raises ValueError: an allowed value is kept only if every occurrence in the document is allowed, which an emitted window cannot know. No option: pure leak and bound fixes; allow terms are new in this release.
WindowedStreamingRedactor only splits where every masking step up to phones (HTML, URL, email, IP, date, IBAN, card, ID, phone) finds the same matches in each half as in the whole buffer, read in the cleaner's step order (each step on the text the steps before it leave). Before, a split right after a phone whose rule reads past its match (a trunk-0 or + number counts its digits ahead: 05 234 567 1, +1 2 3 4 5 6 7 8 x) emitted it whole, and the guard read phones and digit runs on the raw text, so a phone the cleaner finds only once an IP or card next to it is masked (ok 490 872 276 1.2.3.4, 4111111111111111 117 235 991) or a number run that ends at a masked IP (01 023 59 9 90.217.217.177) could split and leak, even with an overlap longer than every value. Such a split now moves earlier, or the stream raises BufferError. A 40,000-case streaming fuzz finds no leak with an overlap longer than every value (24 before); a value longer than overlap_chars still unfinished at the buffer end can still split, as documented. Streaming is about 20% slower. No option: a pure leak fix.

0.8.1 - 2026-11-02¶

Fewer leaks, same speed. 0.8.1 closes the gaps found by our new open-dataset benchmark.

Fewer names and IDs slip through: leaked PII drops on all 12 test datasets, for example 18% to 8% on REDACT and 42% to 12% on long court documents.
Bank and card numbers get their own labels (<IBAN>, <CARD>) instead of being treated as phone numbers.
Less collateral damage: JSON and longer text in brackets are no longer deleted during cleanup.
Cleaner output: URLs no longer swallow the full stop, and Name <email> is read as an email.
No slowdown.

Heads-up: some defaults changed, so output text can differ from 0.8.0. Every change has a setting to switch it back.

Benchmarks¶

Scorer: the partial-leak check now splits values into runs of letters, digits and combining marks ([\w\p{M}]+). Python \w stops at vowel signs, so Devanagari, Tamil and Kannada words broke into 1-2 character pieces under the 4-character floor and a partial leak in those scripts was never counted. Leak rates for Brahmic-script data go up (benchmark section 14 measured multiconer 19.0 -> 25.8, naamapadam 16.2 -> 22.2); this is a scoring fix, SCT output is unchanged. Thai (no spaces) is still undercounted.
Scorer: new type-agnostic masked_recall (with gold_masked) next to partial recall, overall, per type and per language, for every PII dataset; a gold span counts when any prediction overlaps it whatever its type. The summary gains a "masked R" column.

Performance¶

The email step now runs a cheap pre-check (constants.EMAIL_GATE: @ or a bracketed at) and skips the email regex when the text has neither. Every email match needs that separator, so output is byte-identical; no config field. On email-free text the 0.8.0 regex was 5-18x slower than 0.7.1 (English 17x) and took about 25% of NER-off run time (benchmark section 12); the pre-check costs about 0.01 ms per 1,400-character document against 1.7 ms for the regex. Covers TextCleaner (all replacement modes) and ProcessContacts.replace_emails / replace_emails_with_fn. Not re-measured on the benchmark mix yet; the other NER-off slowdowns (dates on by default, language ID, styled-character folding) are unchanged.

Behaviour changes¶

url_trailing_punct_outside (default True, legacy module global URL_TRAILING_PUNCT_OUTSIDE): . , ; : ! ? ' at the end of a URL path or query stay outside the URL unless more URL text follows. 0.8.0 turned Visit https://example.com/path. Next. into Visit <URL> Next., losing the sentence's full stop; the 0.8.0 fix for a fully qualified host's final dot covered only a bare host, not a path or query. In the 0.8.0 benchmark dumps, nemotron had 40 of 94 gold URLs run past the end (37 by ., 3 by ,) and bayanat 3 of 31 (benchmark section 11); nothing leaked, but strict URL F1 suffered. Format regex only. /a.html, ?a=1,2&b=3, /v1.2/x, ;jsessionid= and example.com./?q= are unchanged. Not re-measured yet. False restores the 0.8.0 URL rule.
ner_snap_word_edges (default True, legacy module global NER_SNAP_WORD_EDGES): a NER span whose start or end falls inside a word now moves out to the word edges before the ensemble vote, for every backend. 0.8.0 widened only WordPiece (##) models, so the SentencePiece XLM-R fallback and GLiNER left pieces visible: gretel Via della Salute became <LOCATION>e, and 7-13% of placeholders were glued to a letter or mark (multiconer 7.4%, naamapadam 13.2%, gretel 8.1%; benchmark section 9). A word is a run of Unicode letters and marks (wide CJK characters never join); the ner_subword_widen cap and same-script rule apply, so an unspaced Thai clause or a Latin name glued to kana is left as tagged (ner_subword_widen=0 also turns snapping off). Many Bengali glues are case suffixes on an already masked stem, so the leak gain is small; the main gain expected is strict span F1. Unicode-property rule only. Not re-measured yet. False restores 0.8.0 spans.
html_tag_strict (default True, legacy module global HTML_TAG_STRICT): the HTML rule matches only tag shapes (, </a>,  , <!DOCTYPE ...>, <?xml ...?>, comments, multi-line tags) and entities. 0.8.0 used <.*?>, which ran before the email rule, so John Smith <john.smith@example.com> became John Smith <HTML> (typed as markup, not counted as PII: REDACT showed 12.8% of EMAIL gold as HTML), <https://...> became <HTML>, and If a  lost the words between the brackets. Now those read as <<EMAIL>>, <<URL>> and plain text. The rule applies to the plain and reversible paths, the streaming split guard and the browser export. False restores <.*?>. Not re-measured yet.
ner_ensemble_vote (default 'max', legacy module global NER_ENSEMBLE_VOTE): the NER ensemble keeps a span when its best model score reaches ner_confidence_threshold. 0.8.0 averaged the scores of every model on the span, so when WordPiece widening moved a weak second vote onto the same span, agreement pulled a strong hit under the cut: সুইডেন in a Bengali sentence got XLM-R 1.0 and wikineural 0.40, mean 0.70 < 0.85, and stayed visible. The new vote can only add masks. Benchmark section 1 (cut 0.85 to 0.7, a related lever) shows leak -5 to -23 pts, recall +1 to +7, precision -0 to -3 on tab, multiconer, iahlt, thainer, naamapadam, thai_pii, nemotron, stockmark and bayanat; the vote change itself, and openpii, gretel and REDACT, are not measured yet. The cut stays at 0.85. 'mean' restores 0.8.0.
ner_propagate_masked (default True, legacy module global NER_PROPAGATE_MASKED) and ner_propagate_min_score (default None = the effective NER cut, legacy NER_PROPAGATE_MIN_SCORE): after NER, every other case-sensitive whole-word copy of a masked PER/ORG/LOC value in the same document gets the same type, so Dear Mr. Okonkwo ... Signed Okonkwo no longer masks one copy and leaves the others visible. The values come from the model's own spans; values under 2 characters or without a letter are not spread, and copies inside a placeholder or another span are left alone. Reversible maps stay exact. A replay on the 0.8.0 benchmark output fills missed gold spans (TAB 357/1216, REDACT 919/3760, gretel 524/5309, nemotron 49/226, iahlt 11/129) at a precision cost: about 50% of the new spans in TAB and gretel and 38% in REDACT fall outside gold, partly real PII the gold misses and partly wrong tags now repeated; raise ner_propagate_min_score to trade leak for precision. Not measured end to end yet. Known limit: only exact copies of the whole masked value spread, so a single word of a masked multi-word value stays visible unless the model tags it: a surname after a full name (Lars Petersen ... Petersen), or Okonkwo after the model merged Okonkwo\nOkonkwo into one span.
ner_merge_location_gap (default 0 = off, legacy module global NER_MERGE_LOCATION_GAP): when set, two neighbouring LOCATION spans become one when the text between them holds at most that many characters outside placeholders, no line break and no sentence end (.!?;: before whitespace, 。！？). The NER models tag street, district, city and country as separate entities and the NUMBER rule takes house numbers and postcodes, so 1234 Elm Street, Apt 5B, Springfield, IL 62704, USA gave five spans with Apt 5B and IL visible; with 12 it gives <NUMBER> <LOCATION>. Placeholders inside the gap are taken into the merged span; reversible maps restore exactly. Format rule only, no word lists. A replay on the 0.8.0 benchmark output (benchmark section 8) at gap 20: LOCATION leak thai_pii 55% to 42%, gretel -4.5 pts, REDACT -3.1, openpii -2.2, tab flat; at 12 the gains are about a third of that. The cost is non-gold text masked inside merges (about 4 characters per merge on openpii) and short lists of places (Paris, London and Berlin) becoming one <LOCATION>, so it stays off until measured end to end. Known limit: Thai prefixes glued to a tagged name (ถนน, เขต) stay visible at the edge of a span (word-boundary snapping, a separate finding), and text the models never tag (road names after an abbreviation, landmark text before the address) is outside any merge.
check_replace_iban_cards (default True, legacy CHECK_REPLACE_IBAN_CARDS), with replace_with_iban="<IBAN>" and replace_with_cards="<CARD>" (legacy REPLACE_WITH_IBAN, REPLACE_WITH_CARDS): before the phone step, an IBAN (country code, check digits, 11-30 letters/digits, unspaced or in groups of 4) whose ISO 7064 mod-97 checksum holds becomes one <IBAN>, and a 13-19 digit number starting 2-6 (spaces/hyphens allowed) whose Luhn checksum holds becomes one <CARD>. 0.8.0 gave NL91ABNA<PHONE> (country, check digits and bank code visible) and <PHONE> <PHONE> for a card. Values with a bad checksum go to the phone/number steps as before. Benchmark section 3: bayanat IBAN 97/122 and gretel 51/77 pass the checksum; few synthetic values do (openpii cards 191/2,096). Not measured end to end yet. False = 0.8.0.
check_replace_identifiers (default False, legacy CHECK_REPLACE_IDENTIFIERS), with replace_with_identifiers="<ID>" (legacy REPLACE_WITH_IDENTIFIERS): a token of 6+ uppercase letters and digits (any script) with 2+ digits and a letter, not glued to another letter or digit (RBCON68T08, XK4729QP31), becomes <ID>. Format rule only. Off by default and in the training preset: the shape also takes product and reference codes (COVID19); benchmark section 3 simulates openpii leak 13.1 to 7.6, but 83% of gretel hits and 27% of REDACT hits are unlabelled reference codes.
New types IBAN, CARD, ID in reversible maps, synthetic fakes, regex_counts and the signals PII types. The browser export does not include these rules (listed in its limitations). False restores 0.8.0.
bracket_content_max_chars (default 32, legacy BRACKET_CONTENT_MAX_CHARS), bracket_content_max_words (default 3, legacy BRACKET_CONTENT_MAX_WORDS) and keep_json_braces (default True, legacy KEEP_JSON_BRACES): bracket/brace removal no longer deletes JSON documents and bracketed prose. 0.8.0 removed every [...] and {...} up to the first closing bracket, so Order: {"name": ..., "items": [{...}]} done became Order: done and nested JSON left "items": "payment": (benchmark section 4: REDACT lost 14.8% of gold spans, 108/1,000 docs lost more than half their text; gretel 7.0%). Now a {...} that parses as a JSON object (json.raw_decode, also inside invalid outer JSON) is kept whole, and only innermost spans of at most 32 characters and 3 whitespace-separated words are removed ([img.png], [12], [citation needed], {{template}}); a [...] never spans a brace. Format rules only. Limits come from the cached benchmark text: with both limits at least 87% of innermost [...] spans in fineweb-2 are still removed. Not measured end to end yet. Unspaced scripts (CJK, Thai) are held by the character limit only. Both limits 0 and keep_json_braces=False restore 0.8.0.
Dates: new date_builtin_month_languages (default None = all, legacy DATE_BUILTIN_MONTH_LANGUAGES) picks which built-in month languages apply. Thai month names are not built in (no word lists); pass them with custom_month_names ({"th": ("ก.พ.", ...)}) and 24 ก.พ. 2562 (Buddhist-era year) becomes one <DATE>.
Dates: new date_month_name_slash_year (default True, legacy DATE_MONTH_NAME_SLASH_YEAR): a month name, / and a 2- or 4-digit year is one <DATE> (October/55, Oct/2019); 0.8.0 gave October/<NUMBER>. False restores 0.8.0.
Fix: month names are regex-escaped in build_date_regex, so a dotted abbreviation such as ก.พ. (or one passed via custom_month_names) no longer matches any character in place of its dots.

Fixes¶

JSON preservation (keep_json_braces): the object scan is linear on malformed input. '{"a":' * 20000 took seconds and then raised RecursionError out of process(); a '{' with no '}' after it is now skipped, a failed parse resumes after the point it failed, and nesting past the recursion limit ends the scan.
JSON preservation: replacement tokens (<NUMBER>, <PERSON>) count as JSON values, so {"n":42,"name":"Alice"} stays as {"n":<NUMBER>,"name":"Alice"} instead of being removed as brace content.
NER propagation: the whole-word check treats combining marks as part of a word, so a name राज no longer masks the start of राजा (<PERSON>ा).
html_tag_strict: an HTML comment ends at -->, so  and comments holding tags are one <HTML>; an unclosed <!-- still ends at the next >.
Browser export: the date rule follows custom_month_names, date_builtin_month_languages and date_month_name_slash_year (shared helper resources.configured_month_names), so it matches Python.
date_builtin_month_languages=() with fuzzy dates on no longer brings back every built-in month name.

0.8.0 - 2026-11-01¶

Faster, more languages, ready for data pipelines.

Up to 4x faster batch cleaning on multi-core machines, plus GPU support.
75 languages detected instead of 4.
More accurate: fewer false masks and more PII found on our benchmarks.
Plugs into your stack: Hugging Face datasets, DataTrove, LangChain and Haystack adapters, a training preset and an MCP server.
Data reports: sct-report writes quality scores and data cards without exposing any text.

Heads-up: this release has breaking changes. Read "Behaviour changes" in the changelog before you upgrade.

Highlights¶

Speed: on the release benchmark (evaluation/results/comparison-0.7.1-0.8.0.md) process_batch on 16 Linux cores is 4.2x faster on short documents (3.69 to 15.55 docs/s) and 3.2x faster on long documents (0.65 to 2.08 docs/s); on 64 cores 29.2 docs/s (short). Single-document process is unchanged within noise (0.95x to 1.10x). On a high-end GPU, single documents are about 17x and batches about 4.5x faster than 64 CPU cores. CPU and GPU numbers, hardware advice and CUDA setup: docs/PERFORMANCE.md.
Correctness, release benchmark (30 documents, 84 entities, 9 languages): exact-span TP/FP/FN 83/1/1 (P = R = F1 = 0.99; 0.7.1: 82/2/2) and 0 of 67 PII values leaked. evaluation/results/benchmark-0.8.0.md.
New multilingual PII eval (220 synthetic records, 11 languages): the default ONNX ensemble scores 0.93 / 0.95 recall / precision overall. GLiNER multi v2.1 (0.45 / 0.68) and GLiNER multi PII v1 (0.57 / 0.91) score lower, so the default model is not switched. docs/MULTILINGUAL_EVAL.md.
Linux worker processes (fork, forkserver) are tested in CI; a crash when a forked child freed an inherited ONNX session is fixed.

Added¶

Docs: a shorter README with a new pipeline diagram (resources/sct_flow.excalidraw, .svg, .png). The reference moved to docs/: INSTALL.md, CONFIG.md, API.md, NER_BACKENDS.md, REPLACEMENT_MODES.md, PRIVACY.md, ADAPTERS.md, EVALUATION.md, LANGUAGES.md; release notes of 0.4.5 to 0.6.0 moved here. README links are full GitHub URLs, so they also work on the PyPI page.
ProcessResult.signals: a value-free public view (language, confidence, PII counts, share and types, regex and entity counts, warnings, model provenance). Never entity text. Details under "Behaviour changes".
Presets: sct.preset_config("training", **overrides), TextCleanerConfig.from_preset(), sct.PRESETS. training removes only PII and repairs broken unicode, for training and RAG text. Details under "New options".
Framework adapters (docs/INTEGRATIONS.md), each behind its extra: sct.integrations.hf.clean_dataset / SCTMap ([datasets], Dataset and IterableDataset, score=True adds score columns), sct.integrations.datatrove.SCTCleaner ([datatrove], counts pii, pii_<TYPE> and sensitive stats), sct.integrations.langchain.SCTDocumentTransformer ([langchain]) and sct.integrations.haystack.SCTDocumentCleaner ([haystack], to_dict / from_dict round-trip through Pipeline.dumps). Shared options: columns (default the main text field), output="overwrite" (default) or "new_column" (<col>_clean, raw kept), config (default the training preset; reversible mode is rejected). Document adapters write metadata["sct"] = signals of the first column. sct.integrations._runtime (CleanerHandle, clean_columns, clean_records) keeps one cleaner per process, rebuilt after a fork and pickled as config only. docs/RECIPES.md covers Ray, Spark, Polars, DuckDB, dlt and Daft (Polars and DuckDB run by hand).
Score columns and data cards (docs/SCORE.md): sct.score.score_record / score_records give flat sct_* columns. Score A (language, language score, PII count, share and types, warnings) is always on. Score B is off by default (ScoreConfig): a quality score on the original text (ONNXQualityScorer, any QualityScorer callable; scripts/export_quality_head.py exports FineWeb2-HQ-Plus to one ONNX graph) and a PII-normalised MinHash of the cleaned text ([score] extra, character shingles for Han, Kana, Thai, Lao, Khmer and Myanmar text). build_data_card / render_data_card.
sct-report command: JSONL or Parquet in; value-free report JSON, optional Markdown data card (--card), per-row score JSONL (--scores), MinHash metadata, and k-anonymity on the columns named in --quasi-identifiers only.
MCP server (docs/MCP.md): warnings-first output, outputSchema, tool annotations, ToolErrors with a remedy, optional file input (clean_text(path=...) under SCT_MCP_ROOT), SCT_MCP_PRESET. Details under "Behaviour changes".
Language: detection over all 75 Lingua languages with a confidence score; any Lingua language can be pinned; a language's own ner_models entry runs only that model.
GPU choice: ner_device, cuda_device_id, onnx_cuda_provider_options; every result records which models, revisions and execution providers ran (metadata['provenance']).
Multilingual PII eval set: evaluation/multilingual_pii.jsonl (en, nl, de, es, zh, ja, th, ar, hi, bn, ta; native-script names, a code-switch slice), built deterministically by scripts/build_multilingual_eval.py and scored by scripts/run_multilingual_eval.py.
Fuzz harness for styled, full-width, native-digit and invisible-character PII: scripts/fuzz_styled_pii.py renders emails, phones, IPs, URLs, dates and numbers with Unicode variants (font, wide, small, circled forms, native digits, ZWSP/ZWNJ/ZWJ, IDNA dots, typeset spaces and dashes, glued CJK/Thai prose), runs every replacement mode and the stream, and reports leaks, restore errors, stream mismatches and stream errors (a styled value that leaves no split where its plain spelling does, counted only when every token fits the window less its overlap and the longest stretch with no split, a space inside a value included, is no longer than in the plain spelling), restore errors against the cleaner's own normalised text (so an ftfy mojibake repair is not a restore error), and a slow check (fold time on 2000 vs 8000 characters of styled contact punctuation) with a shrunk minimal cause; tests/test_fuzz_styled_pii.py replays the seeds of every cluster found. Not generated, as out of scope: an IDNA dot before a native digit (a native-digit IPv4 host) and two addresses glued through one run of prose (no boundary to find).
New extras: datasets, datatrove, langchain, haystack, score.
CI: a required linux-workers job (Python 3.13 fork, 3.14 forkserver) runs the worker-process tests and the adapter tests with their own coverage gate; lint covers scripts/.

Behaviour changes¶

Dates: check_replace_dates (legacy global CHECK_REPLACE_DATES) now defaults to True, so a full date becomes one <DATE> before the year and number steps: 2023-10-01 was <YEAR><NUMBER><NUMBER>, now <DATE>, in placeholder, synthetic and reversible mode, counted once as DATE in regex_counts. The numeric pattern now covers ISO 8601 and day/month/year with one consistent -, / or . separator in either order (2024/1/15, 2024.01.15, 15.01.2024, 01/15/2024), with an optional time (T or space, seconds, fraction, Z or offset). Month and day must be in range (year-last takes day/month or month/day, so 31/31/2024 and 14.25.2024 are not dates), year-first with dots needs two-digit month and day, and a digit joined by . or , on either side blocks a match, so versions (3.12.0, 10.1.12.2024, CalVer 2024.1.5), IPs, decimals and phone numbers are not dates. check_replace_dates=False restores the 0.7.1 output. The training preset keeps dates as written.
ASCII folding: check_to_ascii_unicode (legacy global CHECK_TO_ASCII_UNICODE) now defaults to False, so the library, sct-clean and sct-mcp keep every script: 日本語 stays 日本語 (was Ri Ben Yu), München stays München. On the release benchmark the default NER ensemble found 0 of 8 Hindi, Bengali and Tamil names with folding on (they came out as transliterated text) and over-tagged folded Japanese (69 tags for 3 names); with folding off it matches the gold count in every language. Folding also broke IDN URLs (https://例え.jp/パス became https://Li e.jp/pasu, not replaced). check_to_ascii_unicode=True, sct-clean --to-ascii or SCT_MCP_TO_ASCII=1 restore 0.7 output. IPv4 addresses written in other digit scripts (١٩٢.١٦٨.١.١, ८.८.८.८) are now replaced without folding (they leaked as <NUMBER>.١.١).
Python 3.11 is no longer supported; 0.8.0 needs Python 3.12 or newer. pip on Python 3.11 keeps resolving to 0.7.1.
lingua-language-detector minimum raised from 2.1.0 to 2.2. Lingua 2.2 can detect the language of some texts differently from 2.1.1, which changes the language-specific NER model, stopwords and date handling chosen for those texts.
ONNX NER models no longer share one lock per model: threads in process_batch now run ONNX inference at the same time (InferenceSession.run is thread-safe). Output is unchanged. Torch models and GLiNER still run one call at a time. With the ONNX Runtime default threads, a session uses every physical core of the machine and ignores taskset/CPU affinity; set onnx_intra_op_threads to cap CPU use. On the benchmark, fewer threads per session was slower, not faster, because the batched NER pass runs a few large calls rather than one per worker.
Two language keys that use the same model repo with different revisions now load each revision. Before, the second key silently reused the first key's revision. The INT8 quantized cache directory is now keyed by the resolved Hub commit (<model>@<digest>, a lowercase hash of the commit), so a quantized file is only reused for the exact weights it came from, also when a branch such as main moves. The 0.7.1 directory (org__model) is not reused, so each quantized model is quantized again once.
ner_batch_size (default 8, new legacy module global NER_BATCH_SIZE) now batches model calls: NER chunks of one document, and of all documents in one process_batch call, go through each ONNX, torch or GLiNER model ner_batch_size chunks per forward pass (rows right-padded). process_batch runs the steps before NER per document in threads, then one batched NER pass over all documents (ONNX batches run on the same number of threads), then the steps after NER. Output text, languages and finding spans are unchanged; padded batches can move scores in the last digits (max 5e-6 on the release and long-document sets with the default models). ner_batch_size=1 keeps the 0.7.1 call pattern and scores exactly. GLiNER2 still runs one chunk per call.
Throughput with the default models on a 16-core CPU (scripts/benchmark_release.py --batch 64, docs/s, ner_batch_size 1 vs 8): short documents 14.8 vs 14.9; long documents (about 8 joined documents each) 2.66 vs 2.38 with cpu-count workers, 1.50 vs 1.64 with one worker. With many workers on CPU, threads already fill the cores, so batching mainly adds padding and memory (peak RSS on the long set about 6 GB vs 12 GB). The release benchmark on another machine (16 cores) measured the opposite: ner_batch_size 8 vs 1 was 15.55 vs 10.31 docs/s (short) and 2.08 vs 1.82 (long) with 16 workers. Which setting is faster depends on the CPU; measure before changing it. ner_batch_size=1 always uses less memory.
With ner_batch_size > 1 and NER on, metrics_callback duration_ms includes the time a document waits for the shared NER pass of its batch.
NER now collects the spans of all chunks of a document in document coordinates, votes once and replaces once per document, instead of voting and replacing chunk by chunk. Chunks and model input are unchanged. Differences, only for documents longer than one chunk:
replacement_mode='reversible': placeholder numbers (<PERSON_0>...) are assigned right to left over the whole document, as for short documents, so indices can differ. Deanonymisation still round-trips.
replacement_mode='synthetic': Faker values are drawn in whole-document order, so the fake values can differ.
Placeholder mode: two entities of the same type separated only by spaces at a chunk boundary now merge into one placeholder (<PERSON> instead of <PERSON> <PERSON>), as they already did inside a chunk. If any chunk has a label outside PER/LOC/ORG/MISC, the whole document uses the plain right-to-left replacement that short documents with such labels use.
Privacy-policy output and findings are unchanged.
ProcessResult.metadata is now always a dict (it was None when no extended feature was on). It always holds two value-free count maps: regex_counts (every regex replacement per type: URL, HTML, EMAIL, IP, DATE, YEAR, PHONE, NUMBER, CURRENCY, each {"count", "chars"} with chars = length of the replaced source text) and entity_counts (NER findings per entity type, same shape). Counts cover placeholder, synthetic and reversible mode and fuzzy dates; marker text already in the source and IPs kept by replace_ips_public_only are not counted. With replace_with_currency_symbols=None each symbol is one count. Entity text stays out of metadata unless include_entities=True or a privacy policy is set.
New ProcessResult.signals (sct/signals.py: public_view, merge_views, PUBLIC_VIEW_KEYS, WARNING_CODES): a value-free view with the same keys for every input, including empty text and process_json (language, confidence and source, chars, PII count, chars, share and types, regex and entity counts, sensitive, warnings, model provenance). Metadata gains source_chars; empty or whitespace input now gets full metadata with the empty_input warning instead of None. StructuredResult.signals merges every string leaf; process_rag_chunks puts metadata["sct"]. Score columns, sct-report, the framework adapters and the MCP server read it.
repr(ProcessResult) shows only lm_text, language and signals; repr of DetectedEntity, MapEntry and AnonymizationMap no longer shows entity text or originals.
sct.adapters.is_sensitive_result(result) now takes only the result and reads these counts (any regex replacement or NER finding). The old (cleaner, source, result) signature and the marker-count guessing are gone; it was wrong for empty or custom markers (a phone replaced with "" was not sensitive). process_json, process_rag_chunks and OCR region selection use the new counts.
Language detection now covers all 75 Lingua languages (high accuracy, about 1.4 ms per short document and 246 MB peak for the detector). Text in Hindi, Thai, Arabic, Chinese, Japanese, Bengali, Tamil and other languages is labelled with its own language instead of English/Dutch/German/Spanish or None. EN/NL/DE/ES labels are unchanged on a 41-sentence regression set, and release-benchmark outputs and scores are identical (its zh/ja/hi/bn/ta documents were None and ran the same MULTILINGUAL + ENGLISH models). Texts in other languages (French, Italian, Portuguese, Polish, ...) that were mislabelled as one of the four now get their own label, so they run the MULTILINGUAL + ENGLISH NER fallback and English stopwords instead of the mislabelled language's. Very short or ambiguous texts can get a near language (for example "xyz" is now POLISH with low confidence). detect_all_languages=False restores the 0.7 detector.
language= accepts any Lingua language as a pin or in a tuple (language='hi' raised ValueError before unless the language was also in extra_languages/ner_models/...). supported_languages now only means "has stopwords, dates or an NER key".
A language with its own entry in ner_models / torch_ner_models (for example {"HINDI": "your/model"}) and no ner_ensemble route now runs only that model; before, it ran MULTILINGUAL + ENGLISH. The built-in FRENCH/PORTUGUESE/ITALIAN keys, which point at the default multilingual model, keep the fallback.
ProcessResult.metadata also always holds language_confidence (Lingua top-1 confidence; 1.0 when pinned; None when unknown), language_source (pinned, detected, fallback when a language tuple fell back to its first entry, none) and warnings (sorted codes language_unknown, language_fallback, language_low_confidence, ner_fallback_models).
ner_confidence_threshold and gliner_threshold now default to None ("not set") instead of 0.85 / 0.4; legacy NER_CONFIDENCE_THRESHOLD is None too. Before, PII mode replaced any value equal to the default, so an explicit ner_confidence_threshold=0.85 or gliner_threshold=0.4 was silently turned into 0.3. Now an explicit value is always used. New read-only properties give the values that run: effective_gliner_threshold (explicit, else 0.3 in PII mode, else 0.4) and effective_ner_confidence_threshold (explicit, else the GLiNER threshold in PII mode, else 0.85). Code that read cfg.ner_confidence_threshold as a number must use the property. Explicit values outside 0..1 raise ValueError. ner_backend and gliner_labels keep the old "equals the default" rule in PII mode (an explicit gliner_labels=('person', 'organization', 'location') still becomes the PII labels).
metadata['detector'] (only set with include_entities=True or a privacy policy) is gone. Every result now has metadata['provenance'] = {"backend", "models"}: one {"key", "id", "revision", "provider"} entry per NER model that runs for the document's language, with the resolved Hub commit as revision for ONNX models and the execution provider (or torch device) that actually ran it. {"backend": None, "models": []} when NER is off.
metadata['warnings'] gets cuda_fallback when CUDA was requested but a model runs on CPU.
ner_device=None (auto) keeps GLiClass on CPU as before; only ner_device='cuda' moves it to the GPU.
The URL regex no longer takes ", < or > into the path (they are never raw in a URL). <a href="https://x.org/a">docs</a> now becomes <a href="<URL>">docs</a> instead of <a href="<URL> ... eating the rest of the tag; this only shows when HTML is kept (check_replace_html=False).
URL regex: user info (user:pw@) stops at / ? # " < > @, so https://x.org/?to=alice@example.com&cc=bob%40example.net is one <URL> (the percent-encoded email leaked before) and <a href="https://x.org">alice@example.com</a> keeps its markup. A bare host with a query or fragment (x.org/?q=1, host#frag) is replaced whole; a bare trailing ? stays punctuation.
Phone regex: grouped national numbers with a trunk 0 (06 12 34 56 78, 010-6552 1234, 020 7946 0958, 9 or more digits, optional extension) and 3-3-3 groups (699 123 456) are now replaced whole; before they were missed or cut. 00 stays the international prefix. Number tables with such groups can now become <PHONE>.
Email regex: the local part accepts RFC punctuation after its first character (! # $ % & ' * + -) and single dots, and cannot end with a dot, so o'brien@x.org and a!b@x.org are replaced whole. = / | { } ~ ^ ? and backtick stay out, so key=alice@x.org keeps key=.
Marker-shaped text already in the input (<LONDON>, also after unicode repair: ＜LONDON＞, <LONDON>) is no longer protected from NER, and reversible mode never assigns a placeholder spelled like one already in the input. Only SCT's own placeholders are protected.
metadata['entity_counts'] counts overlapping findings once (merged under the label with the highest score, the label that is replaced).
WindowedStreamingRedactor never splits an email, phone, URL or IP between windows (matched on an NFKC view, so fullwidth digits count), and holds back a phone-like digit run at the end of a feed until the next feed completes it. If no safe split point is left, it raises BufferError.
process_dataframe reads cells by position, so frames with duplicate row or column labels are cleaned (before, those cells were skipped).
Reversible mode on empty input returns an empty AnonymizationMap (or stores one in the token store); sct-clean --token-map writes it.
sct.risk.assess_reidentification_risk groups on (type, value), so True and 1, or 1 and 1.0, are different quasi-identifier values. sct-report JSON-encodes values the same way ([1] and "[1]" differ).
MCP clean_text / clean_json return {warnings, summary, text|value, signals} in that order; detail="full" adds value-free findings (entity_type, score, start, end, action, reason, plus path for JSON). stat_text, language, policy and the nested findings[].entity / findings[].finding shape are gone. Both tools declare outputSchema and read-only, non-destructive, idempotent, closed-world annotations. Errors are fastmcp.exceptions.ToolError with a remedy (a PII-mode mismatch raised ValueError). create_mcp_server(cleaner=None, *, root=None, max_bytes=None, preset=None); concurrent first calls build one cleaner.
GLiNER on CUDA: after an ONNX-to-torch fallback, and for GLiNER2, the model now moves to the chosen GPU; presidio_gliner passes the torch device (map_location) and reports it in provenance.

New options¶

Presets: sct.preset_config(name="training", **overrides), TextCleanerConfig.from_preset(...) and sct.PRESETS. "training" only removes PII and repairs broken unicode, for training and RAG text: URL, email, phone, IP replacement, NER (default ONNX ensemble), ftfy repair and language detection on; ASCII transliteration, numbers, years, dates, fuzzy dates, isolated letters and symbols, HTML, backslash-escape decoding, bracket and brace removal, currency, whitespace normalisation, casefolding, stopwords, punctuation and stat_text off, so line breaks, indentation and code are kept. "sct" is the library default config. Any field overrides; an unknown name raises ValueError. GLiNER PII stays opt-in (ner_mode="pii"). Known limit: a URL with a path swallows a trailing sentence period (see https://x.org/a. becomes see <URL>).
check_decode_unicode_escapes (default True, legacy module global CHECK_DECODE_UNICODE_ESCAPES): part of check_fix_bad_unicode; decodes literal backslash escapes (\u00e9, \n) before ftfy. False keeps them as written, for code and JSON.
detect_all_languages (default True, legacy module global DETECT_ALL_LANGUAGES): auto-detect over all 75 Lingua languages; False = 0.7 detector over supported_languages. To restrict detection, pass a language tuple.
language_margin_warning (default 0.15, legacy module global LANGUAGE_MARGIN_WARNING, 0..1) and language_confidence_warning (default 0.0 = off, legacy module global LANGUAGE_CONFIDENCE_WARNING, 0..1): metadata['warnings'] gets language_low_confidence when the detected language leads the runner-up by less than the margin, or its confidence is below the confidence threshold. Never changes routing. Measured on the multilingual eval set and release benchmark (250 texts, plus 10/20/40-character prefixes, 994 detections, 83 wrong): the earlier default (confidence below 0.5) fired on 272 and caught 74 wrong; the margin default fires on 178 and catches 73. On full texts it fires on 11 of 250 instead of 20 and catches all 4 wrong detections instead of 3 (it also flags a Hindi text detected as Marathi at confidence 0.57). Most of the old false alarms were correct English with names. language_confidence_warning=0.5, language_margin_warning=0.0 restores the earlier warning.
fix_character_width (default False, legacy module global FIX_CHARACTER_WIDTH): part of check_fix_bad_unicode. ftfy's width fix turned the Chinese comma ， into , while 。 stayed, mixing punctuation in CJK text, and widened half-width katakana. The default now keeps CJK punctuation and katakana width and converts only full-width ASCII letters and digits (plus ＠．－＿＋ and the ideographic space inside them), so full-width phones and emails are still detected and replaced. fix_character_width=True restores the 0.7 behaviour.
ner_subword_widen (default 1.0, legacy module global NER_SUBWORD_WIDEN, None or >= 0): WordPiece models widen an entity to its whole word only when the added text is at most this many times the tagged span. Thai and Japanese kana have no spaces, so a whole clause is one WordPiece word: นครราชสีมา became ย้ายไปนครราชสีมาเมื่อปีที่แล้ว... as one LOCATION. None = always widen (the earlier 0.8.0 fix), 0 = never (0.7). Multilingual eval (sct-default): Thai recall 0.65 to 0.85.
ner_subword_widen_same_script (default True, legacy module global NER_SUBWORD_WIDEN_SAME_SCRIPT): WordPiece widening never adds a piece whose letters are in another Unicode script than the entity's edge letter. mBERT reads はDavid Millerで and ถึงคุณSarah Lee as one word, so the Latin name took the kana or Thai around it. Multilingual eval (sct-default): 573/56/87 to 578/51/82 (recall 0.868 to 0.876, precision 0.911 to 0.919), code-switch slice 0.78/0.82 to 0.83/0.88, ja 0.63/0.68 to 0.70/0.75, th 0.85/0.89 to 0.87/0.91, PERSON 0.95/0.94 to 0.97/0.96; the 5 changed documents only lose kana/Thai from a PERSON span. Release benchmark outputs identical, leaks 0/67. False restores widening across scripts.
legacy_type_names (default False, legacy module global LEGACY_TYPE_NAMES): one name per PII type on every surface. Before, phone and IP were <PHONE> / <IP> as tags but PHONE_NUMBER / IP_ADDRESS in reversible placeholders (<PHONE_NUMBER_0>), regex_counts, signals['pii_types'], score columns, DataTrove pii_<TYPE> stats, MCP summaries and PII-mode GLiNER tags (phone_number gave <PHONE_NUMBER>). Now all of them say PHONE / IP, like EMAIL, URL, DATE, YEAR, NUMBER and the NER types PERSON, LOCATION, ORGANISATION, MISC, which already had one name. The short names were kept because they are what the default tag text already showed, so default placeholder output does not change. PII_LABEL_MAP maps phone_number -> PHONE and ip_address -> IP; PII mode now expands positional_tags from the configured gliner_label_map (a custom target such as TEL was filtered out before). Privacy-policy rules key on the new names. The eval datasets (evaluation/*.jsonl) label phones PHONE; the release benchmark still maps the old names. legacy_type_names=True restores the 0.7 names; tag text stays set by replace_with_*.
fold_styled_characters (default True, legacy module global FOLD_STYLED_CHARACTERS): part of check_fix_bad_unicode. With the new default check_to_ascii_unicode=False, styled PII leaked: john@𝐞𝐱𝐚𝐦𝐩𝐥𝐞.com, ⓙⓞⓗⓝ@example.com, https://𝐞𝐱𝐚𝐦𝐩𝐥𝐞.com/a stayed in the output and ｈｔｔｐｓ：／／ｅｘａｍｐｌｅ．ｃｏｍ／ａ was only half replaced. Letters and digits whose Unicode decomposition is  or <wide> now fold to their NFKC ASCII form, <circle> ones too when they touch ASCII letters/digits or URL punctuation, and full-width URL/email punctuation folds between ASCII letters/digits; ＠ also folds between letters of any script, ． beside an ASCII letter and a ：／／ scheme after ASCII letters, so 山田＠example.com, taro＠例え.jp and https：／／例え.jp/パス are replaced; inside an address-shaped run (word characters around an @) full-width ．－＿＋ fold beside any script, so john@例え．テスト and josé＿smith@example.com are replaced whole; after a scheme:// or www. full-width ：／？＝＆＃％～ fold beside any script, and ． too in the host, so https://例え.テスト？email=..., https://例え．テスト/?token=... and www.例え.テスト？token=... are replaced with their query; a single circled letter before ＠ folds (ⓙ＠example.com) while 東京－大阪 in prose keeps its form (also with fold_styled_characters=False). CJK punctuation in prose, a lone ①, super/subscripts and squared units keep their form. Release benchmark and multilingual eval unchanged (eval texts normalise byte-identically). False restores the earlier 0.8.0 behaviour.
ner_device (default None = auto, CUDA when available; 'cpu', 'cuda'), cuda_device_id (default 0) and onnx_cuda_provider_options (default None, extra CUDAExecutionProvider options such as {"gpu_mem_limit": 8 << 30}, passed to ONNX Runtime unchecked). Legacy module globals NER_DEVICE, CUDA_DEVICE_ID, ONNX_CUDA_PROVIDER_OPTIONS. Before, every model ran on GPU 0. GLiNER in ONNX mode (gliner_onnx=True) keeps its own session setup.
json_key_cleaning (default 'pii', legacy module global JSON_KEY_CLEANING): how process_json (and sct-clean --json-input, the MCP clean_json tool, RAG chunk metadata) cleans dict keys. 'pii' runs only the regex PII steps on keys (emails, phones, URLs...; no NER, because models tag lone words such as message or user out of context, which turned common keys into <MISC> and two such keys into a duplicate-key error; the steps the training preset turns off stay off), so with the library default {"k": 1} keeps its key instead of becoming {"": 1}, {"b": 1, "c": 2} no longer raises "duplicate key", and keys are not ASCII-folded. 'pii_ner' adds NER on keys; 'full' cleans keys with the whole config (0.7 behaviour); sct-clean --json-input reports a duplicate key as a usage error instead of a traceback; 'off' keeps keys as written (keys can then carry PII). Values are unchanged.
synthetic_seed (default 42, legacy module global SYNTHETIC_SEED) and synthetic_per_document (default True, legacy module global SYNTHETIC_PER_DOCUMENT): synthetic mode seeds each document's fakes from the base seed, a hash of the document text and the fake's position, so documents get different fake names and emails (before, every worker thread restarted one seeded stream, so a process_batch gave every document the same first fake per type, and sequential output depended on call order), the same document always gets the same fakes in process, process_batch (any workers, batched NER or not), a new cleaner or an adapter worker process, and repeated values inside one document keep one fake. False keeps the 0.7 single stream per thread.
synthetic_locale (default None, legacy module global SYNTHETIC_LOCALE): synthetic fakes follow the detected document language. Before, every fake was US English (Faker()), so Dutch text got John Smith and New York. The Lingua language maps to a Faker locale through Python's locale.normalize territory table, then Faker's bare or first locale of that language (DUTCH -> nl_NL, GERMAN -> de_DE, CHINESE -> zh_CN, JAPANESE -> ja_JP, ARABIC -> ar_AA, HINDI -> hi_IN); 53 of Lingua's 75 languages (English included) get a native locale, the rest (and no detected language) use en_US. Faker locales without a provider fall back to its en_US one. Fakes stay deterministic per document across threads and processes, also for locale providers that draw from Python's global random (th_TH, uk_UA ID numbers, uk_UA plates) or order cities by hash seed (it_IT). A string forces one locale; 'en_US' restores the 0.7 fakes.
ner_sees_tags (default True, legacy module global NER_SEES_TAGS): NER reads the same text in every replacement_mode. Before, synthetic mode put the fake email, phone and URL in before NER and reversible mode put <EMAIL_0>, so NER read a different sentence than in placeholder mode and found different entities (with the small test model, Jan de Vries werkt bij Philips in Eindhoven, mail jan@vries.nl. got an ORGANISATION only in synthetic mode; reversible offsets after a regex value were shifted). Now the regex steps put the placeholder-mode tag in the text NER reads, and each regex value goes back in after NER, in order (same fakes as before, per-document mapping and regex_counts unchanged; a tag already in the source stays). Multilingual eval (sct-default, reversible mode): 578/51/82 to 584/50/76 (recall 0.876 to 0.885, precision 0.919 to 0.921), LOCATION 0.97/0.90 to 0.99/0.93, ORGANISATION 0.66/0.89 to 0.69/0.87, bn recall 0.78 to 0.83, ja 0.70/0.75 to 0.73/0.77, ta 0.87/0.88 to 0.88/0.90, ar precision 0.80 to 0.79; release benchmark outputs identical, leaks 0/67. A regex value inside an NER span goes back into that span's reversible map entry (University of ann@example.com as one ORGANISATION restores the email, also when the span cuts the tag, as in of <EMAIL) and the other occurrences of its tag still get their values. False restores the 0.7 order.
synthetic_low_confidence_locale (default 'en_US', legacy module global SYNTHETIC_LOW_CONFIDENCE_LOCALE): with synthetic_locale=None, a document whose language detection gets the language_low_confidence warning uses this Faker locale instead of the detected language's. Before, Kontakt: Anna Schmidt, anna.schmidt@firma.de, Telefon 030 1234567. was detected as Danish (0.17) and got Danish fakes. Deterministic across processes as before. None follows the detected language anyway.
synthetic_collapse_period (default True, legacy module global SYNTHETIC_COLLAPSE_PERIOD): a synthetic fake ending in . drops that period when the text already has a . right after the entity (de_DE names with a suffix such as B.Eng. gave B.Eng.. before a full stop). False keeps both periods.
synthetic_reference_date (default '2026-01-01', legacy module global SYNTHETIC_REFERENCE_DATE): synthetic DATE and DATE_OF_BIRTH fakes fall in ranges that end on this ISO date instead of today, so the per-document seed gives the same dates on any day; Faker also reads it as today while generating, so native SOCIAL_SECURITY_NUMBER/TAX_ID fakes (zh_CN, et_EE, fi_FI, lv_LV, no_NO, sv_SE, pl_PL, sr_BA) encode the same birthday on any day. None uses the clock.
sct-clean --preset {sct,training} (default sct) and SCT_MCP_PRESET / create_mcp_server(preset=...) (default sct) pick the config preset; training keeps numbers, case and whitespace as written. sct-clean --to-ascii / SCT_MCP_TO_ASCII=1 turn on ASCII folding.
onnx_intra_op_threads and onnx_inter_op_threads (default None, the ONNX Runtime default, same as 0.7.1). Legacy module globals ONNX_INTRA_OP_THREADS / ONNX_INTER_OP_THREADS. Not applied to GLiNER's own ONNX session (gliner_onnx=True).
SCT_MCP_ROOT (default unset = off) and SCT_MCP_MAX_BYTES (default 1 MiB): let MCP clean_text(path=...) read regular UTF-8 files under the root. Relative paths only; .., hidden parts and symlinks are refused (opened folder by folder with O_NOFOLLOW); off on Windows.
ScoreConfig (sct.score, not TextCleanerConfig): quality=False, quality_model=None, quality_revision=None, minhash=False, minhash_num_perm=128, minhash_seed=1, minhash_shingle=5, minhash_min_tokens=5. (data_card is gone: score_records never read it; use build_data_card / sct-report --card.)
Adapter options columns, output ("overwrite"), config (None = "training"); a single string for columns is one column.
ner_chunk_overlap (default 0, legacy module global NER_CHUNK_OVERLAP): each NER window adds this many tokens of the neighbouring text on both sides of its chunk (GLiNER-only: 2 characters per token), and a span is kept from the window whose chunk holds its start; a neighbouring window's span is kept too when no span of the owning window that passes the confidence threshold (and, with a privacy policy, is replaced or redacted by it) covers it. A span seen by several windows is one vote per model, with its best score and that score's label, so repeated context (even with a different label) never counts as extra models in ensemble voting. A document that fits one window is never split, so overlap only affects documents that are chunked. This finds entities that a chunk boundary would split (sentence, whitespace or hard cut). The default 0 keeps the 0.7.1 chunks; overlap changes the context the model sees in long documents, so detections can change either way, and inference grows by about 2 * overlap / window. An entity longer than the overlap that starts at the very end of a chunk can still be cut. Must satisfy 2 * overlap < window // 2; this is checked again for each document, so a model loaded later with a smaller limit raises ValueError instead of running with a negative chunk budget.

Fixes¶

check_remove_stext_custom_stop_words now works: False keeps the built-in stopword lists in stat_text and ignores custom_stopwords. Before, no step read it, so custom sets always replaced the built-in lists.
HTML entities decode everywhere, each on its own (0.7 used ftfy's "auto", which stopped decoding from the first line holding a < on). john@example.com left the address in the output, and WindowedStreamingRedactor never found a split in markup with entities: the right half (no <) decoded what the whole text kept, so each space was rejected after normalising the full prefix and suffix again ("intro " + "word & " * 800, 9062 characters, took 7.4 s and 3205 normalise calls, emitted "intro " and the next feed raised BufferError; now 0.03 s, 5 calls). Escaped < and > (<, <, >) stay escaped while normalising and decode after it, only outside tags and hidden from the tag step only (contacts, URLs and the reversible map see the plain bracket: john<at>example.com gives <EMAIL>, and a URL holding < restores with <), so escaped code stays visible text in every mode and the stream: Use for bold gives <HTML>Use for bold<HTML> (decoding every entity first had made it <HTML>Use <HTML> for bold<HTML>; 0.7 gave <HTML>Use <HTML>b<HTML> for bold<HTML>). No new option: check_fix_bad_unicode=False keeps every entity as written. WindowedStreamingRedactor also no longer splits inside an HTML tag that holds a space (<a title="x y">) when check_replace_html is on.
NER no longer tags text inside a regex placeholder: with the default models, call +1-202-555-0147 became call <<MISC>> (the model tagged PHONE in <PHONE>; 0.7.1 did the same). Spans inside an upper-case <TAG> / <TAG_n> marker are dropped; custom replace_with_* text is not protected.
International phone numbers no longer swallow a closing parenthesis of the sentence: (tel +31 6 1234 5678) now gives (tel <PHONE>), not (tel <PHONE>. Area-code parentheses must be balanced (+44 (0)20 ... unchanged).
ner_backend='presidio_gliner': a detected language that the Presidio analyzer has no NLP engine for is analyzed as English instead of failing (the default AnalyzerEngine only has English).
A warning is logged when CUDA was requested but ONNX Runtime fell back to CPU (missing CUDA/cuDNN libraries or a driver that does not match the onnxruntime-gpu build). Before, this happened silently. New docs/PERFORMANCE.md gives CPU and GPU throughput and CUDA setup notes.
NER windows now fit every tokenizer of the models a document's language runs, not only the one with the smallest limit. Models loaded for other languages do not change a document's windows. With the default models, the multilingual model (WordPiece) can need far more tokens than XLM-RoBERTa for the same text and silently truncated the rest of a window, so it never saw entities past that point. Such texts now get more, smaller windows; on the release set 2 of 30 documents (and 2 of 4 joined long documents) are affected. Default-config output of the release set is unchanged, but these documents run more model calls (single-run Mac benchmark of the release set: 6.9 to 5.4 docs/s).
Overlapping detections (for example John Paul Smith and Paul from two ensemble models) are merged into one span before replacement in reversible and synthetic mode and when a custom label is present. Before, right-to-left replacement left part of the outer entity visible (<PERSON_1> Smith). Placeholder mode with standard labels already merged them through Presidio. In the release benchmark, reversible output of zh-glued-001 changes from the garbled <LOCATION_1>OCATION_0> to <LOCATION_0>.
ONNX models exported with a fixed batch dimension of 1 run one row per call instead of failing under ner_batch_size > 1.
replacement_mode='synthetic': the regex steps and NER now share one consistency cache per document under ner_batch_size > 1, so the same value replaced by both gets the same fake value (as with ner_batch_size=1).
The INT8 quantized cache directory name is collision-free: commits are hashed and repo ids that contain __ are %-quoted.
ONNX models with external weights (the default XLM-RoBERTa-large models) load from a huggingface-hub cache that stores each blob in its own directory (hub/blobs/<xx>/<sha256>, seen with huggingface-hub 1.32.0). onnxruntime refused them with "External data path escapes model directory"; 0.7.1 fails the same way. In that layout SCT now hard-links model.onnx and model.onnx_data into ~/.cache/sct_linked/<model>@<commit hash> (under cache_dir when set; one directory per snapshot, so a branch that moves back to an already-cached commit loads that commit's weights) and loads them from there; if hard links are not possible (another filesystem) the files are copied once (about 2.2 GB per model) and a warning is logged; later loads, and other processes starting at the same time, reuse the copy while its size matches and its source blob is not newer. Other cache layouts load in place as before.
Chunk planning ignores fixed padding set in a model's tokenizer.json. Such a tokenizer counted every text as its padded length, so NER windows were split down to single characters and entities were missed (0.7.1 had this for the primary tokenizer; 0.8.0 had spread it to every ensemble tokenizer). None of the default models set padding.
Custom replace_with_numbers / replace_with_currency_symbols markers are inserted literally; before, a backslash in them was read as a regex group reference.
Linux worker processes: freeing a cleaner (its ONNX session) inside a forked child crashed (exit 139) or hung in garbage collection, for example in Hugging Face num_proc or DataTrove workers. Live NER objects are now pinned in the child at fork (os.register_at_fork) and never destroyed there. forkserver was not affected.
Framework adapters build one cleaner per handle also when several threads ask at once; the lock is replaced after a fork so a child never deadlocks.
Hugging Face clean_dataset returns only the columns SCT wrote, so remove_columns=["text"] with output="new_column" really drops the raw text, and a stale input column named like an output column is replaced. Its cache fingerprint is stable across runs (datasets hashed a frozenset in per-process order).
NER windows respect a tokenizer's saved truncation max_length when it is below model_max_length, so inference never cuts a planned window.
A language route given as a list (["ENGLISH", ...]) is no longer changed in place when GLiNER is added to it.
scripts/benchmark_release.py imports on Windows (no resource module); peak RSS is reported as null there.
Isolated-symbol removal (library default, off in training) only removes symbols with whitespace or a text edge on both sides. Before, a ) or full stop after a placeholder or bracket was dropped: (born 1980). became (born <YEAR>, bel +31 20 123 4567. lost its full stop, and a Chinese ， after a word was dropped (0.7.1 did the same). On the release benchmark 11 of 30 outputs now keep such a . or comma; scores are unchanged.
replacement_mode='synthetic' with NER: the model no longer rewrites a fake email or phone that a regex step just inserted (anna@beispiel.de became James Howard@Bernard LLC); those fakes are protected like placeholders.
WordPiece NER models (any tokenizer with a ## continuation prefix, such as protectai/bert-base-NER-onnx): entities widen to whole words, so John Doe no longer becomes <PERSON>e. SentencePiece models (the defaults) are unchanged.
Worker processes no longer call the Hub when the cache is warm. After model.onnx resolves, the other files load from that commit (served from the cache with no request) and the missing model.onnx_data probe uses the Hub's cached "no such file" answer (the default ensemble made one HEAD per load, an unpinned model eight). A CleanerHandle that is forked or pickled after loading pins unpinned models to the commits the parent loaded. A forked worker on Linux could hang forever in DNS (getaddrinfo) during that request.
language_confidence is rounded to 6 decimals. Lingua's value differed in the last digit between processes, so worker and in-process metadata did not match.
import sct no longer imports presidio_anonymizer (which pulls presidio_analyzer, GLiNER and torch when installed); it loads on first NER replacement.
MCP: an SCT_MCP_ROOT / create_mcp_server(root=...) that is not an existing directory raises ValueError at start-up instead of failing on every call.
WindowedStreamingRedactor checks for a split contact token on HTML-entity-decoded text too, so a phone written with  ,   or   between its digit groups is no longer cut between windows.
WindowedStreamingRedactor reads every decimal digit as ASCII when it looks for a split contact token, as the cleaner does, so a phone in Arabic-Indic or Devanagari digits (٠٦ ١٢ ٣٤ ٥٦ ٧٨) is no longer cut between windows and left unredacted. The browser manifest marks its phone and IP rules with fold_decimal_digits: true and states how a client runs them.
An empty Presidio type map (legacy_type_names=True with no label map) is used as given again, so PHONE_NUMBER / IP_ADDRESS privacy-policy rules match analyzer findings.
ner_backend='presidio_gliner': findings mapped to PER / LOC / ORG (the documented label map and the PII label map) become PERSON / LOCATION / ORGANISATION, as with the other backends, so privacy-policy rules on those names match (with default_rule keep, Anna Schmidt was kept).
WindowedStreamingRedactor folds multi-digit styled numbers (⑫ -> 12) when it looks for a split contact token, so a phone written with circled 10-50 is no longer cut between windows and left unredacted.
Haystack SCTDocumentCleaner.to_dict() leaves out a derived PII label map, so a loaded config re-derives it (as pickle does) and a later dataclasses.replace(legacy_type_names=...) gets the new phone/IP names.
Pickle, deepcopy and Haystack to_dict() leave out the shared PII label map only in PII mode, where it is re-derived; a config moved to another ner_mode with dataclasses.replace keeps it, so a GLiNER name span keeps its PER tag and privacy rule after a worker round trip.
An NER span that cuts a regex placeholder (of <EMAIL) is widened to the whole placeholder, so placeholder and synthetic output no longer keep a fragment (Proctor>.); reversible mode was already repaired.
ner_chunk_overlap: window offsets come from the tokenizer of the document's language route, not always the primary model's tokenizer.
A worker forked while another thread was building the language detector gets a fresh detector lock instead of inheriting it held (a possible hang).
URLs whose host is a private or loopback IPv4 address (10.x, 127.x, 172.16-31.x, 169.254.x, 192.168.x) or a bracketed IPv6 address ([::1]:8080) become one <URL>. Before, such a URL was not matched as a URL and query values such as ?email=alice%40example.com stayed in the text.
URLs keep interior ), ] and } when more URL text follows (?f=(a)&email=..., or a dot and a word: report(1).pdf?email=...), so the rest of the query is replaced too; a bracket closing around a URL still stays outside it.
URLs keep a run of closing brackets when more URL text follows it (ASP.NET /(S(id))/a.aspx?email=..., ?q=f(a(b))&email=...), so the query after it is replaced too.
URLs keep closing brackets followed by URL sub-delimiters (; , : ! $ * @) and more URL text (?q=(report);email=alice%40example.com, /a(b);jsessionid=...), so that value is replaced too; a bracket before prose punctuation and a space, or before a possessive 's, still stays outside.
NER: when one model gives two labels for the same span (repeated window context, WordPiece widening), it is one vote with its best score and that score's label, also without ner_chunk_overlap. Before, the two labels could average a 0.99 PER below the confidence threshold.
sct.integrations.hf.clean_dataset declares the output column features, so a first batch of only None or [] no longer types a column as null and fails on the next batch.
assess_reidentification_risk type-tags values inside tuples and frozensets too, so (1,) and (True,) are different quasi-identifier values.
Merged signals of a JSON value: language_source comes from the longest leaf of the chosen language, not the longest leaf overall.
The email domain stops at general, CJK and full-width punctuation. With fix_character_width=False (the new default) seikyu@example.com（担当：鈴木花子） was one <EMAIL> up to ）, so the name inside was hidden from NER and labelled EMAIL; now the address ends at .com. Domains in any script (例え.jp, उदाहरण.भारत) still match, and the letters of the CJK punctuation block (々〆〇, by Unicode category) stay domain letters (info@佐々木.jp), as do the join controls ZWNJ and ZWJ (U+200C/D) that Persian and Devanagari domains use.
Isolated-symbol removal keeps a quote, dash, * or % that touches a letter, digit or mark of any script. Before only ASCII letters and digits counted, so 東京-大阪 became 東京大阪 and ইসলাম-এর became ইসলামএর (which also made a WordPiece model widen a name over the suffix).
Email local parts take combining marks (প্রিয়া@example.com, संपर्क@example.com were left whole or cut to संपर्<EMAIL> without ASCII folding); full-width ＠． after a mark fold too. Email and URL hosts take letters beyond the BMP (info@𠮷野.jp) and punycode TLDs (example.xn--p1ai was cut at .xn); a URL with an IPv4 host takes every valid octet (http://10.0.0.0/?..., .255) and only a whole address, so query values no longer stay visible.
Phones and IPs are matched on a view where every decimal digit is ASCII and full-width . - are ASCII, with the source offsets kept: ٠٦ ١٢ ٣٤ ٥٦ ٧٨, ٩٨٧٦٥ ٤٣٢١٠, ١٩٢．١٦٨．١．١ are replaced and 1.2.3.4١ is one <IP>.
dataclasses.replace(cfg, legacy_type_names=...) on a ner_mode="pii" config re-derives the label map and positional tags (an explicit gliner_label_map stays); the presidio_gliner backend maps analyzer types through gliner_label_map and PHONE_NUMBER/IP_ADDRESS to PHONE/IP unless legacy_type_names=True.
With ner_sees_tags, a regex value stays hidden through the cleanup steps, so a tag they remove (replace_with_email="[EMAIL]" under bracket removal) no longer loses the value in reversible mode. A forked child gets a fresh synthetic random lock. llms.txt lists the current check_replace_dates, language_confidence_warning and language_margin_warning defaults.
With ner_sees_tags in reversible mode, a display marker that is part of an NER placeholder (replace_with_email="PERSON" beside <PERSON_0>) is no longer counted inside that placeholder, so the email is restored; before, deanonymize returned the bare marker.
Fuzz round 1 (scripts/fuzz_styled_pii.py, 6000 seeds; research/0.8.0-fuzz-round1.md): with fold_styled_characters=True every compatibility form of contact punctuation is folded by one table built from Unicode data (decomposition <wide>,  or  with NFKC in @ . - _ + : / ? = & # % ~), so small forms (john﹫example﹒com, 214﹒44﹒189﹒208, http﹕//...﹖q﹦..., 018﹣2896, ﹢31 ...) and the font plus ﬩ are replaced like full-width ones; a run may mix forms (http：/／x) and folds between digits of any script (١٩٢．١٦٨); a word that ends in a combining mark counts as a word before ＠/． (https://ळौछनै．de/...); a circled number folds beside any decimal digit or contact punctuation in any form and between space-separated digit groups (06 ⑬ 01 68 36); a plus form folds before any decimal digit.
Fuzz round 2 (9000 seeds; research/0.8.0-fuzz-round2.md): with fold_styled_characters=True a lone circled number after two or more space-separated digit groups folds as the last group (093 622 17 ㉝, 06 73 86 88 ㊺), so the phone is replaced; a lone one before a number stays a list marker (① 2024年).
Fuzz round 3 (12000 seeds; research/0.8.0-fuzz-round3.md): prose of a script written without spaces (Unicode line-break class SA, ID or CJ: Thai, Lao, Khmer, Myanmar, CJK, kana) glued to a Latin/Greek/Cyrillic letter or a digit ends an email local part, an email or URL TLD and a URL path/query, and an email may start right after it. So ดูa@b.jpกขคcj9rf@xyz.nl gives ดู<EMAIL>กขค<EMAIL> (the first TLD no longer takes the next local part), https://x.com/f?sid=xyzปถว06 10 29 37 39 keeps the phone whole, and an email local part no longer takes a phone group and Thai prose before it. An address written in one such script stays whole (https://th.wikipedia.org/wiki/ภาษาไทย, info@佐々木.jp). Known limit: an IRI path that switches from Latin letters or digits straight into such a script (/wiki/東京2020オリンピック) is replaced only up to the switch. Format-regex rule, the same in Python and the browser export; no option. Release benchmark 82/2/2 to 83/1/1 (zh-long-001 keeps 邮箱是 outside the EMAIL); multilingual eval 584/50/76 to 608/34/52 (recall 0.885 to 0.921, precision 0.921 to 0.947; 12 ja/zh documents: EMAIL, and the names the email span hid).
Fuzz round 5 (18000 seeds; research/0.8.0-fuzz-round5.md): with fold_styled_characters=True invisible format characters (Unicode category Cf: ZWSP, word joiner, soft hyphen, bidi marks) beside an ASCII letter/digit, any decimal digit, a circled letter/digit or contact punctuation are dropped when the other side is one of those, a letter of any script, a space or the text edge, so a line-break hint copied inside a value no longer hides it (06\u00ad71427605, 192.168\u200b.1.1, jan.de\u200bvries@example.com, 06\u200b 12 34 56 78 were left whole). Between two other letters they stay (ZWSP between Thai words, ZWNJ in Persian); Latin prose loses its soft hyphens too (Donau\u00addampf -> Donaudampf). WindowedStreamingRedactor skips Cf characters on its split-guard view, so such a phone is not cut between windows.
Fuzz round 6 (21000 seeds; research/0.8.0-fuzz-round6.md): with fold_styled_characters=True the phone step (whole text, reversible, synthetic and the stream's split guard) reads every space of Unicode category Zs as a space and every dash of category Pd as -, one code point for one, so phones set off the way word processors and PDFs write them (06\u00a012\u00a034\u00a056\u00a078, +31\u20096\u20091234\u20095678, 06\u20111234\u20115678) are replaced; they were left whole with check_replace_numbers=False (the training preset). The text keeps those characters. A lone circled number between or after digit groups set off by any Zs space folds as one (06\u202f98\u202f\u3256\u202f03), and an invisible format character before a dash of any form drops (131\u2060\u20107237). WindowedStreamingRedactor emits the whitespace it splits at as " " when check_normalize_whitespace is on, as the whole text gets it (a split at \u202f gave a different output). False keeps the earlier view (ASCII, full-width and ideographic forms only).
Email and URL host labels are matched in one way only (X+(-+X+)*), so a dotted host with no TLD fails in linear time: a@ or http:// before five 20-digit labels took 5 to 10 seconds, six took over 100 (found when the email hypothesis test hung in the gate). Same matches; no option.
Email local parts take ZWNJ/ZWJ as domain labels do (प्\u200dॄघप@example.org was not matched at all).
An address-shaped run (both the fold_styled_characters fold and the full-width-only fold) continues across invisible format characters (category Cf) inside a word and keeps them, so a ZWNJ/ZWJ in a label no longer stops a full-width dot after it from folding (john@نامه\u200cای．ایران, john@क्\u200dष．भारत were left whole). The email TLD takes ZWNJ/ZWJ between its letters as other labels do (ab@नाम.भा\u200dरत was cut before \u200dरत). Release benchmark unchanged; multilingual eval texts normalise byte-identically. No new option.
Numeric dates are matched on the decimal-digit view of the text (as phones and IPs are), so dates in native digits are replaced again (١٥/٠١/١٩٩٠, १५/०१/१९९०, ۱۵/۰۱/۱۹۹۰; the 0.8.0 day/month ranges were ASCII only); the browser date rule gets fold_decimal_digits. A numeric date glued to letters of an unspaced script is a date, as numbers are (วันเกิด15/01/1990ครับ); a Latin/Greek/Cyrillic letter beside it still blocks a match. With fold_styled_characters=True dates read any space as a space and any dash as -, as phones do (1987\u201105\u201112). A circled number beside any dash folds as beside - (2026\u2013⑩\u201302).
With fold_styled_characters=True an IDNA label separator (。｡, as Python's idna codec reads them) in a mail domain or URL host is a dot when an ASCII letter or digit follows (alice@example。com, https://www。example。com/?token=... were left whole); before other text 。 ends a CJK sentence and stays. Squared Latin letters (🄹🄾🄷🄽, Unicode <square> forms of one letter) fold like circled ones; squared units (㎏) stay. Every format character but ZWNJ/ZWJ inside a mail domain is dropped whatever script is beside it (info@例\u200bえ.jp was left whole).
Custom regex markers that are not <TAG>-shaped (replace_with_email="EMAIL") are protected from NER like <EMAIL>, unless the source already holds them; NER split EMAIL into E and IL, so reversible restore wrote the address twice and placeholder mode left MA behind.
Marker text the source already holds (The EMAIL field, <LONDON>) loses NER protection only where it occurs in the source; the regex step's own markers of the same text keep it (a literal EMAIL let NER split every emitted EMAIL, so reversible restore wrote an address twice).
A URL whose host starts with IPv4-shaped labels and goes on with a Unicode label or a native digit is one host name (http://10.0.0.1.école.fr/a?token=..., https://1.2.3.1௨3.example.de/?q=... matched only the IPv4, so the query was left); the IPv4 alternative now rejects every continuation the host grammar accepts. No option.
WindowedStreamingRedactor never splits a window inside a date when check_replace_dates=True (the cleaner's own date regex, custom formats included); 15 January 1990 across a split was left as text (or <NUMBER> January <YEAR>). A date longer than overlap_chars that is still incomplete in the buffer can still split, as any token longer than the overlap. No option.
A URL host ends at CJK and full-width prose punctuation (，、「), as a mail domain does; https://example.com，電話：06 12 34 56 78 took ，電話：06 as host and port, so the training preset left 12 34 56 78. With fold_styled_characters=True the URL fold stops there too (prose keeps its ：), a host ends where a glued next scheme starts (a.jp芢。https://... keeps its 。), and a circled run after a digit and an IDNA dot folds as a host label. No new option.
With fold_styled_characters=True every format character but ZWNJ/ZWJ inside a URL host is dropped, as in a mail domain (https://例\u200bえ.jp/?token=... and \u2060 were left whole, query included).
With fold_styled_characters=True phones read full-width and small parentheses as ( ) (+49（30）123456, +49﹙30﹚123456 were left whole); the source keeps them. A format character beside a parenthesis drops inside a contact run, and a circled number in parentheses glued to a digit folds (+80（⑫）808...). An international number ends before a glued bracket unless digits follow it (+49 30 21513405（自宅）), so a CJK prose bracket no longer stops the match. In a URL a full-width ＠ folds after any user-info character and prose punctuation right before it stays in the URL (https://alice:pass!＠example.com/?token=... and pass！＠ were left whole, password and query included). No new option.
A dash or dot look-alike inside a URL host or mail domain no longer hides the whole value: with fold_styled_characters=True every dash (Unicode category Pd: ‐ ‑ ‒ – —) is a label hyphen and every character whose NFKC form is . (․ ﹒) a label dot there, also after www (https://my‐site.example.com/?token=... and www․example․com were left whole, query included). Whatever the fold, a http://, https:// or ftp:// run whose host fails the host grammar is still a URL up to a space or prose punctuation (check_replace_urls). No new option.
Long tokens no longer take quadratic time: the URL fold scanners match a scheme once per [A-Za-z0-9+.-] run, the address-fold scanners start once per run, the URL user info has one way to split, and a mail local part is matched lazily up to 64 characters (RFC 5321) and a longer one whole from the start of its run, once per run (see the next entries). "http://" + "a1" * 8000 took 1.0 s in process() (4x per doubling), a 16000-dash rule line 3 s; both are now linear. An IDNA dot right before a glued scheme with a leading digit (。1https://) now stays, as before other glued schemes. The styled-punctuation joiner finds the first styled form with a lazy lookahead and takes the run possessively, so a letter followed by a long run of styled punctuation with no letter or digit after it is linear ("a" + "－" * 32000 + " " took about 6 s, now 0.02 s; same matches). No new option.
An IPv4 after a dot that ends a word is found (服务器.8.8.8.8, and x@ad.io銄黟。8.8.8.8 where fold_styled_characters=True reads the 。 as a host dot of the run that started at the address); a dot before it blocks a match only after a digit (1.2.3.4.5 stays). No new option.
A mail domain or URL host ends at punctuation of any script (Unicode category P), as at ,: the Arabic comma, the Devanagari danda and » after an address were taken into its TLD (ahmed.ali@example.com، أو lost the comma). Multilingual eval 608/34/52 to 612/29/48 (recall 0.927, precision 0.955). No new option.
The middle dot between two ls, the one punctuation IDNA allows in a label in a Latin context (RFC 5892 A.3), stays inside a mail address or URL host (info@col·legi.cat was kept, https://www.col·legi.cat/?token=... left its query). Elsewhere · still ends a label. No new option.
That middle dot also carries the styled-address fold: info@col·legi。cat (an ideographic full stop) and josé＿col·legi@example.cat (a full-width low line) were kept whole or in part, because the fold run stopped at the dot before the styled separator was normalised. An invisible line-break hint beside the dot (col\u00ad·legi) drops like any other in the run. Linear scan kept. No new option.
A mail local part longer than 64 characters is masked whole ("a" * 65 + "@example.com" was left as text after the 64-character bound above, as were 200 and 5000 characters; 5000 now takes 0.07 s). The longer form starts only at the start of a run of local-part characters or at a script change, so each run is scanned once and the regex stays linear; there it also takes leading punctuation (' + 65 letters + @) into the mask. No new option.
The other context rules of RFC 5892 for a label: a katakana middle dot ・ beside Hiragana, Katakana or Han (A.7, https://例・え.jp/?token=... left its query, a@例・え.jp was kept) and a Hebrew geresh or gershayim after a Hebrew character (A.4, A.5, a@צה״ל.co.il) stay inside a mail domain or URL host, in the regexes and in the fold's URL and address runs. Elsewhere ・ still ends a host and CJK prose keeps it. A label whose Han or kana sits away from the dot (ab・cd例) still ends at it (the rule is read on the dot's neighbours). No new option.
A run of katakana middle dots between Han or kana in a mail domain or URL host with no TLD (info@例・例・…・え, or a Japanese @ before a ・ list) backtracked exponentially (about 6 s at 22 dots, doubling per dot) and could stall a worker. The dot now has one parse, so the scan is linear. No new option.
A long run of styled punctuation after a letter or digit with no letter or digit after it ("a" + "－" * 32000 + " ") took quadratic time in the styled-character fold (6 s at 32000, 4x per doubling). The run is now taken whole once a styled form is seen in it, so the scan is linear (0.02 s). Same output; no new option.
With fold_styled_characters=True circled and squared letters and digits inside a mail address or URL run fold by NFKC whatever sits beside them (ab@éⓣ\u2024net, a@Ŗ🅂Ń.io, https://Ŗ🅂Ń.io/?token=... were kept: their neighbours were not ASCII). Outside an address run the neighbour rule stays, so list markers in prose (① 見て, Ⓐ plan) keep their form. No new option (part of fold_styled_characters).
An address fold run takes every local-part character the email grammar takes, so a styled ＠ or ＿ beside ASCII punctuation folds and the whole address is replaced (anna.smith!＠example.com was kept, josé＿smith.work@example.com left josé＿). With fold_styled_characters=True every dash (Pd) and NFKC dot in the local part is a hyphen or dot as in the domain (john.smith\u2011billing@example.com left john.smith‑), and styled forms of ! # $ % & ' * fold there too. Unchanged multilingual eval and release benchmark. No new option.
A process forked while another thread generates a synthetic SSN/TAX_ID with reference_date restores the real clock in the child; the child kept that thread's held date for good (zh_CN TAX_ID with the wrong birth year). No option.
A URL scheme glued to CJK/Thai letters is matched (เว็บhttps://...); only an ASCII letter, digit, _, / or . before it blocks a match.
A URL path or query ends at CJK and full-width prose punctuation (，。！（・; full-width URL syntax ／？＃％＆＠：．－＿ excepted), so CJK prose glued after a URL is no longer swallowed with a phone after it (...?q=zz（備考）！06 08 66 90 80 left 08 66 90 80).
Trunk-0 phones count 9+ digits from the first two, so 018 566 00 17 (3-3-2-2) is one <PHONE>; before, a 3-digit area code needed 11 digits.
WindowedStreamingRedactor keeps its overlap on the text its split guard reads, and reads the unfinished last token of the buffer too unless it changes the view before it (a cut escape). Before, the guard left that token out but measured the overlap on the whole buffer, so after a long run of CJK text with no spaces it could split at the last space it read, with no text to its right: in 、﬩④𝟡 ㉚ ⓪⑦⓪㊵⑲③（... the lone ㉚ lost the digit group that folds it and the stream emitted +49 ㉚ <PHONE> (fuzz seed 15673, window 80/24). Such a phone now stays in one window, or the stream raises BufferError when the window holds no other space. No option.
WindowedStreamingRedactor folds punctuation on its split-guard view only where the cleaner does (contact punctuation forms), so a phone glued before （ is no longer cut between windows.
WindowedStreamingRedactor composes a combining mark with the letter before it on its split-guard view, as the cleaner's NFC step does, so a date with a decomposed accent (15 fe\u0301vrier 1990) is no longer cut between windows and left unredacted. No option.
WindowedStreamingRedactor decodes backslash escapes (\uXXXX, \UXXXXXXXX, \xNN, \N{...}, octal and one-letter escapes) on its split-guard view with the cleaner's own codec when check_decode_unicode_escapes=True (the default), so a JSON-escaped phone or date (06 12\u0033\u0034 56 78, 06 12\u00a034 56 78, 15 f\u00e9vrier 1990) is no longer cut between windows and left unredacted. With check_fuzzy_replace_dates=True every fuzzy date candidate (15 Janury 1990, Janury 15, 1990) is kept in one window too. No new option; both follow the existing ones.
WindowedStreamingRedactor decodes HTML entities on its split-guard view with ftfy's own decoder to the same fixpoint as the cleaner (06 12&#32;34 56 78), keeps entities that decode to several characters (15 January 1990) and repairs mojibake word by word (15 fÃ©vrier 1990), so those phones and dates are no longer cut between windows and left unredacted. No new option.
WindowedStreamingRedactor drops terminal colour codes (\x1b[32m) and the control characters ftfy removes (NUL, U+FFFC ...) on its split-guard view, as the cleaner's ftfy step does, so a date or phone in a coloured log line (\x1b[32m15 février 1990\x1b[0m) is no longer cut between windows and left unredacted. No option.
WindowedStreamingRedactor's split guard no longer keeps its own copy of the cleaner's pre-regex steps (the entity, escape, ftfy, NFC, fold and digit rules in the four entries above). It calls the new TextCleaner.normalise(text), the same function process runs before its regexes, and matches each regex on the view its step uses (phone and date separator view, IP digit view, numbers when check_replace_numbers=True). A whitespace split is taken only where the normalised buffer is the two halves' own normalised text with at most one whitespace character between them (\r\n counts as one) and no match covers it, so the guard and the cleaner cannot differ again. Steps the copy never had now count: emoji removal (06 12🙂 34 56 78), ASCII folding (06 12 〇〇 56 78), grouped numbers (5 379) and CRLF line breaks were cut between windows; text with only \r\n between rows now splits. The unfinished last token of the buffer is left out of the guard's view, so a feed that ends inside an escape (\U000000) no longer stops the escape decoding of the whole view. Each feed normalises the buffer about twice more (a cut that lands in a match jumps back to the last candidate whose own view ends before the match, found by a bisect on view offsets, so it stays linear; a character NFKC makes longer, ⑫ -> 12, no longer makes that jump skip a safe split). Speed on a 210 KB document (1 KB feeds, NER off, best of 7): 0.97 s -> 1.18 s mixed prose, 1.00 s -> 1.13 s a phone table (whole-document process 0.45 s). No option.
Seeded synthetic fakes from Faker providers that draw from the module-level random (th_TH and uk_UA SSN and TAX_ID, uk_UA plates) now draw from a private seeded Random swapped into that provider module, not from the reseeded global random. Another thread using random at the same time no longer changes the fake, and its own draws are no longer thrown away. No option (synthetic_seed as before).
A fully qualified URL host with its final dot (https://portal.example.com./?email=..., www.example.com./a?u=, http://localhost./) stays one URL up to its port, path, query or fragment; the match stopped at .com and left the query values visible in every mode. A sentence-ending period after a URL stays outside it. Part of check_replace_urls.
Synthetic DATE_OF_BIRTH fakes with synthetic_reference_date are drawn in UTC, so the same seed gives the same birth date under any worker timezone (TZ=America/Los_Angeles moved some dates by a day).
AGENTS.md (contributors): no word-list fixes for model mistakes; every behaviour is a safe default plus a config option.
Dates (fuzz date shapes; every mode and the stream's split guard, which reads the cleaner's own date view): a month-name date glued to Thai or CJK text is found (生日是15 January 1990谢谢, เกิด15 January 1990ครับ): it ends where no digit, _ or Latin/Greek/Cyrillic letter touches it, the rule numeric dates already used (it was \b, which never ends between a digit and a Thai or Han letter); the fuzzy date candidates use the same edges. CJK and Korean unit dates (1990年1月15日, full-width digits too, 1990년 1월 15일), Korean dots each followed by a space (1990. 1. 15.) and Han-numeral dates (一九九〇年一月十五日, digits picked by their Unicode numeric value, no word list) are dates. With fold_styled_characters=True a month-name date's day may be a circled number (⑮ January 1990, January ⑮, 1990), the date view reads every character whose NFKC form is . as a dot (1990﹒ 1﹒ 15﹒), a circled letter folds beside any Latin letter (26 ⓓéⓒⓔⓜⓑⓡⓔ 1990 was 26 ⓓécembre 1990), and a format character (Cf) between a letter of any script and a space or the text edge drops (1990년 6월\u2060 1일; Thai or Persian prose loses such a character before a space too). Buddhist-era years (15/01/2533) were already dates (any 4-digit year). The fuzz harness generates these shapes (styled and glued month-name dates, unit, dotted, unpadded and Han-numeral dates). No new option: check_replace_dates and fold_styled_characters switch them. Multilingual eval unchanged (precision 0.955, recall 0.927), release benchmark 83/1/1, leaks 0/67.

Known limits¶

Dates: only full dates. Year-month (1990年1月) and month-day (1月15日) stay, as January 1990 does; Japanese era years (令和6年1月15日) stay (no installed date library reads them, and era names would be a word list); 1990.1.15. stays (CalVer 2024.1.5. has the same shape; 1990. 1. 15. is found), and a dot-only date with a one-digit month or day (生日1990.1.15谢谢) becomes <YEAR><NUMBER>.15 (1990.01.15 is found); Thai month names are not in the built-in month data (add them with custom_month_names); a circled number glued to a unit character (1月⑮日) stays a list marker.
In ensembles with GLiNER, windows are sized for the token-classification models, not for GLiNER's own word limit.
Only process_batch is faster. process of one document is not batched across documents.
GLiNER2 runs one chunk per call; torch models and GLiNER keep one lock per model, so they do not run in parallel across threads.
ner_chunk_overlap (off by default) can still cut an entity longer than the overlap that starts at the very end of a chunk.
06-12345678 becomes 06-<PHONE>.
Two addresses glued with no separator in a script written without spaces can be read as one address or split at the wrong place.
ORGANISATION is the weakest NER label in the multilingual eval (0.75 recall), mostly in non-Latin scripts; Bengali and Tamil name spans can be a few characters off. WordPiece widening makes the mBERT and XLM-RoBERTa spans agree more often, and the ensemble averages their scores, so a weak mBERT score can drop an agreed span below the threshold: some ar/bn/ta organisation names and 1 Thai name masked by the pre-review 0.8.0 code are not masked now (docs/MULTILINGUAL_EVAL.md).
A URL with a path swallows a trailing sentence period (see https://x.org/a. becomes see <URL>).
Adversarial strings such as +000000000(000)0 can leave (000)0 after phone replacement (Python re takes the first alternative that matches, not the longest).
Score B's quality head was tested with a small ONNX graph, not the exported FineWeb2-HQ-Plus model; its speed is not measured. GLiNER PII mode (ner_mode="pii") was not run end to end for this release.
An unpinned NER model makes one Hub request per worker when the parent never loaded it before forking; pin ner_model_revisions or set HF_HUB_OFFLINE=1.
With an empty regex marker (replace_with_email="") and ner_sees_tags, NER in synthetic/reversible mode reads the indexed placeholder or fake, not the empty text placeholder mode leaves, so its findings can differ there.
WindowedStreamingRedactor can still cut an HTML tag that holds a space when the tag reaches past overlap_chars from the cutoff (its > is not in the buffer yet).
A URL host with an ideographic full stop before a one-character styled label (https://ab｡⑧․cd.ex.com/p?sid=secret1) ends at that stop; the rest of the host, the path and the query stay (Linux fuzz run I, seed 19487; contrived).
An uppercase L·L next to a styled separator (INFO@COL·LEGI。CAT) does not carry the styled-address fold the lowercase l·l gets.
CI does not install the mcp extra; FastMCP-only tests skip there.

0.7.1 - 2026-10-30¶

This release fixes PII that 0.7.0 left in the output. On the release benchmark (evaluation/results/comparison-0.7.0-0.7.1.md) exact-span recall goes from 0.62 to 0.98 and leaked PII values from 6 of 67 to 0.

Behaviour changes¶

More PII is found and replaced: glued and international phone numbers, IP addresses, and GLiNER findings between 0.3 and 0.85 in PII mode. Output text changes for inputs that contain them.
IP addresses are now replaced with <IP> by default (<IP_ADDRESS_n> in reversible mode), including private ones.
New config flags: check_replace_ips (default True), replace_ips_public_only (default False) and replace_with_ips (default "<IP>"). They are the last TextCleanerConfig fields, so positional arguments keep their 0.7.0 meaning.
The browser manifest rejects replacement tokens that contain digits, or that a later enabled rule would rewrite (for example an HTML token holding an email); the browser has no protection for emitted tokens.
process_json finding paths for dict keys use ["@key", <index>] instead of the key text.

Fixes¶

Phone and year rules now find digits glued to letters in any script (for example 电话9876543210, वर्ष2023में).
The number rule now finds numbers glued to letters in scripts written without spaces (金额5000元). Digits inside Latin, Greek and Cyrillic words stay as they are (GPT-4o, Qwen3-8B, H2O). Set replace_numbers_in_words=True to replace those too (GPT4 becomes GPT<NUMBER>).
IPv4 and IPv6 addresses (public and private) are replaced with <IP> before the date, phone and year steps. Set replace_ips_public_only=True to keep private, loopback and reserved addresses, or check_replace_ips=False to turn the step off.
Phones now run before years, so +31 6 1234 2019 and 2001:4860:4860::8888 are no longer split by the year rule. The browser manifest uses the same order.
IPv6 is matched in every valid compressed form, including IPv4 tails (::abcd:192.0.2.1, 1:2:3:4:5::192.0.2.1), and invalid forms are not matched, also in the browser rule.
Replacement tokens and synthetic values (<EMAIL1>, a synthetic IP or phone) and IPs kept by replace_ips_public_only are no longer rewritten by the later phone, year and number steps.
Phone detection covers more international formats (+34 612 345 678, +91 98765 43210) and Indian mobiles (98765 43210, 098765-43210).
Oversized NER chunks are split again, also long Chinese, Japanese and Hindi text without spaces: to the tokenizer's token budget for ONNX/torch models, and to a cap of 2 * max_tokens characters in GLiNER-only mode (a character estimate, not a token count; token-budget chunking is planned for 0.8.0). Text is split after 。！？ and । ॥, and reversible map offsets for NER entities are now document-relative. Long whitespace runs at a chunk cut no longer break chunk alignment. Chunk planning ignores truncation set in a model's tokenizer.json, and a slice that grows when re-tokenized is split again, so text past the model limit is no longer left out of model input.
Languages the stop-words package does not support (for example THAI) no longer crash TextCleaner creation.
PII mode: the final ner_confidence_threshold now follows gliner_threshold (0.3 by default) instead of a hidden 0.85, so GLiNER findings between 0.3 and 0.85 are no longer dropped. An explicit ner_confidence_threshold other than 0.85 still wins (0.85 equals the default and cannot be told apart), and 0.0 is no longer turned into 0.85.
process_json and MCP clean_json no longer return original dict keys in finding paths: a key's own findings use ["@key", <index>].
is_sensitive_result compares placeholder counts, so <EMAIL> alice@example.com (HTML step off) is reported as sensitive. Marker-like tags removed by the HTML step (<EMAIL> Alice with replace_with_html="") no longer cancel a new placeholder.
TextCleanerConfig can be pickled and deep-copied, and the copy keeps the exact state. dataclasses.replace(cfg) no longer grows the default 5 NER models to 8 and 4 supported languages to 7. A partial ner_models mapping gets the same defaults whatever its mapping type.

0.7.0 - 2026-09-19¶

SqueakyCleanText now supports privacy workflows beyond single-text cleanup. Applications can apply policies, inspect decisions, and use consistent adapters.

New features¶

Added versioned privacy policies, structured findings, detector provenance, and metrics callbacks.
Added external token storage for reversible anonymization maps.
Added JSON, DataFrame, RAG, command-line, and MCP adapters.
Added exact-span PII evaluation with language, domain, label, and custom slice reports.
Added pinned model candidates, benchmark tooling, and strict model adoption gates.
Added buffered streaming, OCR image redaction, browser manifests, and corpus risk indicators.

Improvements¶

Pinned default ONNX, PyTorch, GLiNER, and GLiClass model revisions.
Preserved source offsets across normalized NER chunks.
Made process_batch(batch_size=...) control document concurrency.
Added Python 3.14 CI and an 85 percent coverage gate.
Added optional backend CI, standard lockfiles, and PyPI Trusted Publishing.
Updated reversible mode to restore emails, phone numbers, URLs, dates, numbers, HTML, and currencies.

Fixes¶

HTML replacement now honors its configured replacement token.
Presidio now receives the detected document language.
Default configuration no longer emits a deprecation warning.
MCP findings no longer return original entity values.
Overlapping policy findings now redact their complete combined span.

Migration notes¶

gliclass_onnx=True now raises an error because upstream GLiClass has no native ONNX loader.
Structured adapters reject reversible mode when they cannot return each token map safely.
Reversible fuzzy date replacement remains unsupported and now fails during configuration.

0.6.1¶

Improved GLiNER ONNX fallback behavior and test infrastructure.

0.6.0¶

PII detection mode (ner_mode='pii'): auto-configures GLiNER with 60+ PII entity labels (personal, financial, healthcare, identity, digital)
Synthetic replacement (replacement_mode='synthetic'): Faker-generated realistic values instead of <TAG> placeholders, with per-document consistency
Reversible anonymization (replacement_mode='reversible'): indexed placeholders (<PERSON_0>) with AnonymizationMap for round-trip deanonymization
Document classification (check_classify_document=True): zero-shot GLiClass pre-classification before text processing
ProcessResult: process() returns ProcessResult (backward-compatible 3-tuple) with .metadata for anonymization maps and classification results
GLiNER ONNX mode (gliner_onnx=True): load GLiNER with pre-built ONNX weights from HuggingFace Hub (auto-set for PII + ONNX backend)
Bi-encoder support: auto-detects ModernBERT and other bi-encoder GLiNER models, caches label embeddings, dynamic context windows (2048-8192 tokens)
Entity description labels: ZERONER-style natural-language descriptions for improved zero-shot accuracy
Presidio GLiNER backend (beta): opt-in ner_backend='presidio_gliner' for Presidio's context-aware recognition pipeline
ModernBERT ONNX export: updated export script with ModernBERT support (English, 8192 token context)
Dynamic chunk sizing: GLiNER chunk size adapts to model's actual context window instead of hardcoded 384

0.5.x¶

aprocess_batch(): async batch processing for FastAPI / aiohttp integrations
warmup(languages): pre-load NER models at startup to eliminate first-request latency
custom_pipeline_steps: attach arbitrary (text: str) -> str callables after the built-in pipeline
French, Portuguese, and Italian NER support via a shared multilingual ONNX session
Improved NER sentence boundary detection with abbreviation guard

0.4.5¶

Frozen TextCleanerConfig dataclass: immutable, thread-safe, per-instance configuration
ONNX-first NER inference: torch-free base install (~400 MB models vs ~7 GB)
Thread-parallel batch processing via ThreadPoolExecutor
Five NER backends: onnx, torch, gliner, ensemble_onnx, ensemble_torch
GLiNER zero-shot NER for custom entity types (PRODUCT, EVENT, SKILL, etc.)
Ensemble voting across backends for improved recall
Lazy per-language model loading
Multilingual date detection and fuzzy date matching
Configurable emoji removal, bracket/brace content removal, and smart case folding
stop-words replaces NLTK (50 KB bundled vs 30 MB download)
PyTorch and Transformers moved to optional extras
Migrated to pyproject.toml (PEP 517), Python 3.11-3.13, ruff linter