Skip to content

Limits

Limits and design principles

  • Safe defaults, every choice configurable. Each behaviour is a config field; nothing is forced.
  • No guarantee that all PII is found. Models miss entities, mostly organisations and non-Latin scripts. Review output before you publish it.
  • Declared fields need explicit rules. Use a caller-owned schema to protect known PII fields; unselected fields remain unchanged.
  • Reversible maps are personal data. Anyone with the map can restore the values. Store maps apart from the cleaned data, with access control.
  • Synthetic data is not anonymisation. Fake values keep the data useful; they do not make it anonymous in a legal sense.
  • Offsets name their text stage. Legacy findings use post-regex positions. Unified findings use canonical positions and explicit source mapping quality.

Known limits

Details in CHANGELOG.md.

  • Dates: only full dates become one <DATE>. 1990年1月, 1月15日, Japanese era years and 1990.1.15. do not; the year and number steps still change them (1990年1月 becomes <YEAR>年1月, or <YEAR>年<NUMBER>月 with replace_numbers_mode='all'). Thai month names need custom_month_names.
  • In ensembles with GLiNER, windows are sized for the token-classification models, not for GLiNER's own word limit. Torch models and GLiNER keep one lock per model, so they do not run in parallel across threads.
  • Only process_batch is faster; process of one document is not batched across documents.
  • ner_chunk_overlap (off by default) can still cut an entity longer than the overlap that starts at the very end of a chunk.
  • Number masking (replace_numbers_mode='identifiers', the default) is a format rule: it cannot tell a bare 4+ digit amount with no sign, separator or currency code (price 1250) from an ID, and a value joined into a code (INV-2023-107683) stays visible unless you mask it yourself. phone_reject_embedded leaves a phone glued to a Latin word (Tel0612345678) unmasked.
  • 06-12345678 becomes 06-<PHONE>. Adversarial strings such as +000000000(000)0 can leave (000)0 after phone replacement.
  • ORGANISATION is the weakest NER label (0.75 recall in the multilingual eval); Bengali and Tamil name spans can be a few characters off.
  • A URL path that switches from Latin letters straight into a script written without spaces is replaced only up to the switch (https://x.org/a東京secret becomes <URL>東京secret).
  • The trained quality scorer passed Torch and ONNX parity controls. Quality-filter accuracy remains unverified.
  • An unpinned NER model makes one Hub request per worker when the parent never loaded it before forking; pin ner_model_revisions or set HF_HUB_OFFLINE=1.
  • WindowedStreamingRedactor raises BufferError when the buffer passes max_buffer_chars (default 4096) with no safe whitespace split; raise the limit or use BufferedStreamingRedactor. A date or IBAN longer than overlap_chars (default 256) that is still unfinished at the buffer end can be split and leak; it refuses custom_allow_terms.
  • Retrying a failed streaming call can repeat external actions performed by custom steps or callbacks. SCT retains text for retry, but cannot undo those actions. Make such actions idempotent if you retry them.