TextCleaner

TextCleaner

TextCleaner(
    cfg: TextCleanerConfig | None = None,
    *,
    privacy_policy: PrivacyPolicy | None = None,
    token_store: TokenStore | None = None,
    include_entities: bool = False,
    metrics_callback: Callable[[ProcessingMetrics], None]
    | None = None,
    detector_registry: DetectorRegistry | None = None,
    unified_policy: UnifiedPrivacyPolicy | None = None,
)

Runs one configured cleaning pipeline; process and process_batch return ProcessResults.

ProcessResult.metadata always has source_chars, regex_counts, entity_counts, language_confidence, language_source, warnings and provenance. These options add keys:

• replacement_mode='reversible': anon_map, or anon_map_reference when a token_store is given.
• check_classify_document: classes.
• privacy_policy: policy and findings.
• include_entities: findings.

Initialize the text cleaning pipeline.

Parameters:

Name	Type	Description	Default
cfg	TextCleanerConfig | None	Immutable configuration. If None, uses TextCleanerConfig defaults.	None
privacy_policy	PrivacyPolicy | None	Chooses each NER entity's action in place of the replacement; adds the policy and findings metadata keys. Requires replacement_mode='placeholder' (ValueError otherwise).	None
token_store	TokenStore | None	Reversible mode saves each map here, and metadata has anon_map_reference in place of anon_map.	None
include_entities	bool	Add sensitive entity text to legacy findings or unified findings_v2_values. Unified values reference the canonical stage and include values from redacted spans.	False
unified_policy	UnifiedPrivacyPolicy | None	Apply one policy to every selected detector in unified mode.	None
metrics_callback	Callable[[ProcessingMetrics], None] | None	Called with value-free ProcessingMetrics after each non-blank document; empty or whitespace input runs nothing and does not call it.	None

process

process(
    text: str, *, context: DetectionContext | None = None
) -> ProcessResult

Process a single text. Maintains backward compatibility.

Returns:

Type	Description
ProcessResult	ProcessResult (unpacks as 3-tuple: lm_text, stat_text, language).

process_batch

process_batch(
    texts: list[str],
    max_workers: int | None = None,
    *,
    contexts: Sequence[DetectionContext] | None = None,
) -> list[ProcessResult]

Process multiple texts with bounded document concurrency.

Parameters:

Name	Type	Description	Default
max_workers	int | None	Maximum number of worker threads (default os.cpu_count()). Documents per NER model call is cfg.ner_batch_size.	None
contexts	Sequence[DetectionContext] | None	One immutable caller context per input, in unified mode only.	None

Returns:

Type	Description
list[ProcessResult]	List of ProcessResult.

aprocess_batch async

aprocess_batch(
    texts: list[str],
    max_workers: int | None = None,
    *,
    contexts: Sequence[DetectionContext] | None = None,
) -> list[ProcessResult]

Async version of process_batch for use with asyncio-based frameworks (FastAPI, aiohttp).

Runs process_batch in a thread-pool executor so it doesn't block the event loop. max_workers is the maximum number of worker threads, as in process_batch.

warmup

warmup(languages: list[str] | None = None) -> None

Pre-load legacy NER models each language routes to, to avoid first-request latency.

A language that fails to load is logged as a warning and skipped; it loads (or fails) on first use. Unified processing raises ValueError because its providers belong to separate workers.

Parameters:

Name	Type	Description	Default
languages	list[str] | None	Language names to pre-load (e.g. ['ENGLISH', 'DUTCH']). If None, pre-loads models for all supported languages.	None

normalise

normalise(text: str) -> str

The steps before the first regex step (unicode fix, ASCII, emoji), as process runs them.

The streaming split guard reads this same view, so the two cannot differ.